#include <sys/ptrace.h>
#include <signal.h>
main(int ac, char **av) {
int pid; if (ac>1) pid=atoi(av[1]);
ptrace(PTRACE_ATTACH,pid,0,0);
ptrace(PTRACE_DETACH,pid,0,0);
kill(pid, SIGCONT); }

P.S. I also quit World of Warcraft. ;-)

Though for me, it was really special – Alvydas Mituzas, my Dad, got a Knight’s Cross, for his lifetime merits and service for our country, spanning forty years of creativity and dedication. Anyone knowing my Dad really know what the award is for, but for me it is also a symbol of support for virtues that I’ve seen in my whole life – hard work and imagination combined for public service and public good, no matter how difficult it is to start, or finish. Stories of his past are fascinating, and even I can learn more and more of them over the years. We’re proud of him, and we’re lucky to be his family.

It takes time to understand one is ‘creative commoner’. I do have a t-shirt with such caption, but it is much more comfortable once you start feeling real power of use and reuse of information. Few anecdotes…

Dear Mr. Mituzas,

Thank you for making your photographs available under a
Creative Commons license. I am writing to inform you that
the American Society of Civil Engineers has featured a
silhouette of “Up we go” on the cover of its new book,
“Constructability Concepts and Practice.”

https://www.asce.org/bookstore/book.cfm?book=7742

Per the terms of the license, the following credit appears
on page ii of the book: “Front cover photograph by Domas
Mituzas used under a Creative Commons license.
http://creativecommons.org/licenses/by/2.0/.”

I will be happy to send a copy of the book to you if you
will provide me your mailing address.

I got this email back in summer, 2007. Did I just steal a job from professional photographer? Or would they just leave blank book cover? Will this lead to a better bridge in future? Did I join a civil cause? All I know now, is that I’m book cover photographer, albeit quite cheap one. Also, by using CC license I simply used lingua-franca of world I’m in – and now my content can evolve into shapes that I couldn’t expect, and that would be limited by non-portable licenses.

Other anecdote is way more internal. I have cheap point-and-shoot camera (same one to shoot book cover pictures :) that I use during my travels. It fits well into my jeans pocket, it doesn’t provide me any self esteem in professional photography. Still, I get to places, I take pictures, I place them on my flickr photostream, and I license them under creative commons. And fascinating things happen – my pictures appear on top of Wikipedia articles (like this one), without any intervention of mine. People just use it, I can sit back, relax, and see how the contribution widens.

Of course, there other different stories. My colleague (and manager) runs a wiki about his own town, Bielepedia, and he wants to exchange information with Wikipedia. Now he can’t, as well as quite a lot of other free content community projects. Though of course, some may believe license difference doesn’t mean much, in this case it means that we’re building borders we don’t need nor we have intent to maintain.

I live and breathe Wikipedia technology, but I do not feel competent enough to go and push content itself around, and it just shows up there itself (oh, of course, there’s army of committed volunteers who help with that). So, I benefit the project just by being creative commoner, and I may benefit lots of other projects. We at Wikipedia technical team are very open in what we do, and try to spread our know-how in many directions. Documents I wrote about how we do things ended up downloaded hundred thousand times, and I really hope that some of that know-how will end up used and reused.

I guess I’m taking this to extremes – I ended up talking to people in government of Lithuania, journalists and non-profit activists. Imagine a government, that would commit to open licensing for produced content. Well, no need to imagine – US federal institutions release information to public domain, but in Europe it is way more restricted. Still, what one has to realize – at government level it is not only a right to be given, it also has to be a right that has to be protected. Nowadays that means going to copyright powerhouses that serve large record labels and movie studios, and will charge for services, that government has to provide for free (and does in other areas, like looking for your stolen car).

We have lots and lots of talks about knowledge-societies at government levels, but we never get to the point, that every individual is part of that, and first of all we have to teach those rights, and guard them. But of course, to prove, that our rights have to be guarded, we have to show how great our work is – and how powerful can our sharing be. To achieve that we have to build bridges between license islands, talk same languages, and of course, create.

I’m a creative commoner. So should be you.

P.S. So should be Wikimedia Foundation. I’m extremely excited about the work being done to make it reality (thanks Erik, Mike, Mako, everyone!), and you know my personal position on the matter by now :)
P.P.S. I probably have to do some more investment in free culture. My friends say Nikon D90 is way to go ;-)

• Table A has two million rows
• Table B has two thousand rows
• Find all rows in A, which have any B row as substring in some field.

So, my colleague Scott was there, and gave the answer which satisfied the guy:

SELECT * FROM a JOIN b ON a.field LIKE CONCAT('%',b.field,'%');

I started telling Scott, that this will result in too many nested loops, and that better combined pattern matching should be used. Well, my actual words were something like “use RLIKE”. So, folks replaced LIKE in above query with RLIKE, didn’t see too much of improvement and made some fun of me.

So, I thought I should provide some decent response to mocking :-) I downloaded ‘most common American last names of 19th century’ from some website out there, took ~2000 of them, also built a list of all Wikipedia article titles, that have space in them (just to reduce dataset a bit, and make more humans show up there).

My initial poking showed around double speed increase when using combined pattern of RLIKE, and using PCRE UDFs provided double speed over RLIKE. I have no idea what I did wrong back then (or doing wrong now), but simple LIKE with nested row lookup is faster on my current test. Still, there’s something else I wanted to show :)

GNU grep has ‘-F’ functionality, which Interprets PATTERN as a list of fixed strings, separated by newlines, any of which is to be matched. Actually, it is quite well optimized, and uses nice algorithm, located in file kwset.c. This is what some comment in that file tells:

The algorithm implemented by these routines bears a startling resemblence to one discovered by Beate Commentz-Walter, although it is not identical.

See “A String Matching Algorithm Fast on the Average,” Technical Report, IBM-Germany, Scientific Center Heidelberg, Tiergartenstrasse 15, D-6900 Heidelberg, Germany. See also Aho, A.V., and M. Corasick, “Efficient String Matching: An Aid to Bibliographic Search,” CACM June 1975, Vol. 18, No. 6, which describes the failure function used below.

So, let’s try standing on shoulders of giants. Actually, I’m not even that smart to find this, it was actually Tim who researched and implemented this as PHP extension to make some of Wikipedia’s code faster.

So I shamelessly took few files from grep, and wrote some crappy MySQL UDF glue (it is just for demonstration, would need proper engineering to make it usable for general purposes).

So, what kind of performance would a custom-tailored algorithm for the task give…

Simple LIKE matching:

select straight_join * from names join lasts on
binary name like concat("%",last,"%") limit 1000;
1000 rows in set (3.80 sec)

Combined RLIKE matching (I don’t know why it is slower – it was much better on some previous test):

SELECT CONCAT("(",GROUP_CONCAT(last SEPARATOR "|"),")")
from lasts into @re;

SELECT * FROM names WHERE BINARY name RLIKE @re;
1000 rows in set (25.99 sec)

Combined PECL UDF matching:

SELECT CONCAT("/(",GROUP_CONCAT(last SEPARATOR "|"),")/")
from lasts into @re;

select * from names where preg_rlike(@re,name) limit 1000;
1000 rows in set (8.10 sec)

Algorithm theft:

SELECT fss_prepare(last) FROM lasts;
SELECT fss_search(name) FROM names LIMIT 1000;
1000 rows in set (0.02 sec)

Damn it, too fast, this must be some mistake, let’s try more rows:

10000 rows in set (0.07 sec)
100000 rows in set (0.62 sec)
551971 rows in set (3.50 sec)

So, by using customized algorithm we got 600x performance. What does Scott say about this?

domas: my answer took about 10 minutes.
       yours has taken what, week and a half?

.. Bastard… Oh well, priorities priorities :)

By default memcached would have slabs based on assumption, that smallest object size will have 48 bytes of data (thats without item header), and will increase the slab sizes in +25% steps:

slab class   1: chunk size    104 perslab 10082
slab class   2: chunk size    136 perslab  7710
slab class   3: chunk size    176 perslab  5957
slab class   4: chunk size    224 perslab  4681
...

So, in this case, it allocates at least 104 bytes per object, and next steps are way behind. Fortunately, there’re some quick steps to have better efficiency:

Configuration!

There’re two parameters for this:

• -n – minimum space allocated for key+value+flags, defaults at 48
• -f – chunk growth factor, default 1.25

With these settings memcached will define just 38 slabs, ranging from 104 to 458992 byte sized chunks. Even in case of mixed workloads one can use -n 5 -f 1.05 – this will define ~170 slabs, where low values will increase in 8-byte chunks, and smallest slabs would look like this:

slab class   1: chunk size     64 perslab 16384
slab class   2: chunk size     72 perslab 14563
slab class   3: chunk size     80 perslab 13107
...

It would get way higher memory efficiency for larger objects too (6k steps at 100k object sizes, rather than 30k steps with default configuration). Of course, more slabs means that there’re more eviction queues, and in case distribution of object sizes changes, it would have more memory fragmentation, though thats nothing a restart can’t resolve ;-)

Internal storage

Every data item has a header, which includes pointers to other item structures, and additional metadata. One of obvious things that will be fixed in later releases is the CAS (compare-and-swap) metadata (8 bytes per object), which is stored for every object – though very rarely used by users (one needs to use special breed commands). In future versions this might get resolved, and a very dirty hack would be changing cas_id to be uint8_t in memcached.h (heeeee!).

There’re also multiple pointers inside the header – and on 64 bit systems they take 64 bits – though in theory chunks inside memory pages (slabs can have multiple memory pages assigned) can be addressed with 16-bit pointers. Of course, the easy workaround here is simply compiling memcached as 32-bit binary – though then it won’t be able to address more than ~3GB per-instance (and running multiple memcached instances is straightforward).

There’s another internal CPU-vs-memory optimization, where objects internally end up aligned at 8-byte boundaries. Hacking CHUNK_ALIGN_BYTES at slabs.c (I set it to 2) allows us to have chunk sizes increased in much smaller steps.

Data!

Have small keys. Have small data. Compression at application will reduce network i/o, less roundtrips and system calls, and better memory efficiency in the end. Pack integers inside keys into base250 or so (skip whitespace), store binary data.

It is usually way less cycles to have more efficient storage, compared to additional cycles when a cache miss happens :)

Testing & summary

I tried to simulate the most edge case out of all edge cases – integer key and one byte data objects being inserted into 64M memcached instance.

• I could fit in 645k objects inside regular memcached, 932k after factor changes.
• Simple 32-bit build fit in 763k objects, 1164k after factor changes, 1180k after alignment change
• After removing CAS support, it fit 1378k, 1461k after reducing key with base250

So, I could have such slab size distribution (it facilitates objects up to 45k in size):

slab class   1: chunk size     33 perslab 31775
slab class   2: chunk size     34 perslab 30840
slab class   3: chunk size     35 perslab 29959
slab class   4: chunk size     36 perslab 29127
...

MySQL!

64M-sized MEMORY table will be able to store 2087k (INT,TINYINT) entries. When people aim for no-eviction storage, MySQL can be way more efficient.

Interesting though, the PK will take as much space as data itself (what simply asks PK to be held together with data, like InnoDB does (it will actually fit 2500k entries in 64M). With custom MySQL engines this shouldn’t be too difficult for those who really hit the edge cases, right? :)

Oh well, last I’ve heard, memcached is going to have storage engine support too, I wonder how fun will be hacking those (someone ages ago plugged BDB into memcached and called it Tugela.. ;)

Thats how it shows me my mail:

P.S. Can anyone make an island out of Portugal?

• LTV – 21:00 – The Company
• LNK – 21:00 – Bad Company
• TV1 – 21:00 – In Good Company

We are sorry…
We could not process your order because of geographical restrictions on the product which you were attempting to purchase. Please refer to the terms of use for this product to determine the geographical restrictions.

We apologize for any inconvenience this may have caused you.

Go to hell, Amazon.

The problem is very simple – valid PNG files can be uploaded to various sites, and then shown to users. The problem is that IE does autodetection, and if it suspects that the file may be HTML, it executes it as HTML, with all Javascript inside. The images can be properly normal images, that show your kitten or wife or whatever. Still, IE will execute any exploit code that is included in them. Exploit code can actually load the actual image, so nobody will even realize they’re looking at image and not at an attack that hijacks their sessions, steals cookies and does all other sorts of evil things.

So, whenever anyone says IE is secure, just tell them to look at this problem.

The best part is that smartest kids immediately found ways to cash out the ‘GSM LTLs’ – by using ‘call to pay’ service providers, and getting 50% cash efficiency.

One GSM provider (Omnitel, TeliaSonera company) reacted by establishing daily SMS limit that works (only 6LTL worth of SMSes per day), whereas other provider (Tele2) established limit that doesn’t (phones would get disconnected next day only).

And of course, this has brought down GSM providers, or at least their SMS networks – at Christmas. Way to go, marketing people. Way to go.

For all the international people: 1 EUR= 3.4528LTL

Edit: 0.02LTL for SMS

Here some Greenpeace activist tried to convert me (don’t know into what). I knew their position, so went into discussion about nuclear energy. It was fascinating to observe how facts-free religious position was presented, with Chernobyl (the crazy experiment gone wild – they forgot that) being the only example of what is wrong with energy, as well as the waste (see, you have to secure it, unlike CO2, which can be set free).

Of course, every field has its own problems, but thats where technology and engineering helps. The Greenpeace guy was all for wind or hydro power, but these ones are not intervening only for those who never leave the cities – the impact on landscape is obvious (and ecologists can measure impact on biosphere). The feeling of Greenpeace being sponsored by Big Oil doesn’t abandon too much – wind or water will never replace real needs of energy, and nuclear power is too big of a threat for energy monopolies.

Now, what made me rant about this – the guy asked me where I was from and after hearing I’m from Lithuania (Litauen!!) he explained to me that it was different mentality issue. See, us from Eastern Europe are simply Wrong. Even if I just paid a visit to vicinity of IAEA headquarters.

I believe that there’re far more problem than massive consumption. It is also massive growth of civilization overall, and we’ll have to achieve technological breakthroughs, or we’ll somehow will have not only to turn off our lights, but turn off some world population together with their needs. Who are they going to be?

 (F) [3] mysql. Neteisinga užklausa SELECT * FROM krepselis WHERE uzsakymas_id=220479 ORDER BY laikas. Klaida: Can’t open file: ‘krepselis.MYI’ (errno: 145). File:core.php Line:1824 

‘krepselis’ being ‘shopping cart’ and ‘uzsakymas’ being ‘order’, I realized, that error 145 will keep me hungry. Ah, and error 145 is ‘Table was marked as crashed and should be repaired’. So please, run REPAIR TABLE, I am hungry!!!

Lithuanian airlines (flylal) decided instead of using proper plane to use some chartered one. That plane used to be chartered to ferry livestock.. ergh.. tourists to warm countries, but this time it was the one carrying me. The only real trouble about this was simply that they… had 20% less legroom. Maybe it is slight inconvenience for regular traveller, but it has somewhat more serious implications for anyone who is 2m tall. What saved mine (and possibly their) nerves was that I had no neighbors around. I just stretched my legs on next two seats and crashed sleeping (though woke up with unhappy backbone ;-).

Today I had some “happy hours” too. This morning we were going to Bologne, and to save our time we decided to use self-serve ticketing machines. Everything went smooth, we selected our train, class, no-discount tickets (one has to get a special card if wants to use junior rates), did feed all 52EUR where it simply hanged. The screen was showing that 50EUR were given, 2EUR coin was not accounted, coin gates were closed, and… ‘Cancel’ did not work. Some random railroad worker could not help us at all, as we had our interface in ‘English’, but finally, after some running and shouting we got few technicians coming, writing down papers, asking us to sign and finally giving money back. Of course, as just 10 minutes were left until departure of our train we had to whine about it, so we had our tickets without any queues from a human afterwards. Yay for ticket machines!

This was the bestest top-quality train (Eurostar!), where one has to extend the table, in order to place his legs somewhere, so for backwards journey I decided to take first class – it should deal with it. Indeed it did, the table had much better design, there seemed to be more space, and the chairs were ergonomic. Now the problem with ergonomic chairs is that they’re very comfortable for people they are designed to be comfortable. Anyone taller is surely excluded, so the huge head support system (it has to have a fancy name) was somewhere behind and below my shoulders. That is the constant problem with any ergonomic chairs, and I’m still in progress of finding a new chair for my home-office, but at least I can choose to sit on a chair that is not that irritating. Maybe I should’ve travelled second class, as all normal people. :)

Of course, the guy at “international tickets” counter didn’t actually speak English (I sure know Italian is most important language, at least in operas). Then… it appears I didn’t read the email properly.

The email their automated system sent me did have two sections. One of them was Order details, which contained ‘Booking code (DOSSIER)’. Another was ‘Ticket details’, which contained a reservation number. Of course, I had wrong code. I know it was frustrating for him too, as the intranet application responded to any clicks in about a minute, so he had a really important reason not to try to help me. I already mentioned he did not speak English.

So, after one hour and two metro trips I presented booking/order/dossier code instead of reservation number, and got my tickets. Oh, and by the way, international ticket office starts working at 7:30, train leaves at 7:10. I am all happy I had this story this afternoon, not the morning of departure.

Update: Well, now I feel stupid and outdated. I could have opened that email in my mobile phone. :-)

What was fun (and made me write this spoiling post), is that it is easy to spot MySQL usage patterns in science fiction:

• Metaverse – the virtual reality internet, has its counterparts already in our current non-fictional world. “Second Life” is a nice [MySQL powered] platform for virtual life and creativity, and it is now in state of early Metaverse evolution.
• Central Intelligence Corporation (created after the merger of Library of Congress and CIA, as they were duplicating each other anyway) operates huge resources of user-provided content, like Flickr, YouTube or Wikipedia [MySQL powered again]. Though it had somewhat better interfaces and APIs, I guess that is what Web 5.0 is about ;-)

I really liked the idea how asynchronous replication and lack of locking effects virtual reality worlds – even these effects were taken into account by the author. Add neurolinguistics, civilization history, brilliant satire, and you really get the best geek fiction. Thumbs up for the book. The Diamond Age seems to be the next read.

Main conference will happen in Harvard (uh oh!), and the developer meetings at MIT (uh oh uh oh!). Few years ago, when I was finishing school, I had a dream to study at MIT. I did do SAT stuff, wrote essays, got recommendations, and then… didn’t use all that, because I pitied 50$ (or few hundred $, if I’d apply to more universities there). That probably was part of screwed up independence feeling I have developed in those years, I didn’t ask anyone, I didn’t tell anyone. So, one of my biggest dreams was destroyed not because I failed to reach it. It was blown up simply because I failed to try.

Now I’ll meet MIT people, I’ll walk in MIT campus and I’ll remember how one can fail if he doesn’t try. At least lately it is somewhat different. I try, and sometimes achieve. If anyone reads this, please do try to achieve your dreams. Otherwise you’ll end up writing such blog posts. Dreaming sometimes just isn’t enough.

There are gates to the yard, which can be opened remotely, and there’s intercom. And few meters inside there’s a white button to open the gates, if you have to get outside. What is really interesting – most people do not notice or understand the purpose of the button, though it is the single visible button, and it really contrasts with environment.

Anyway, military academy training proved to be successful, instead of calling me or climbing over the fence (that would be too childish), more sophisticated approach was used:

Priceless.

Now as collection is finalized, I’m forced to make some decision with future of this army (>200 units ;-) Hell!

Flying from LA to Vegas was another “random search” – a guy with gloves was carefully looking for WMDs.

Of course, why I’m writing this post – in Vegas I got the SSSS tag on my ticket, which meant changing lanes twice and getting into the fancy one – the one with dust particle blowers/analyzers, then each electronics equipment (and I did carry _some_) had to be checked for traces of _something_.

I still have two flights to make – will see how much of random searches I’ll get. Hail Vilnius airport, travel there isn’t that bad. But recently they started to ask to take shoes off too. I wonder how much time will it take for them to fire dust particles as well.

We were still walking late at night:

Besides riding double-deckers, we did take a tube, where steep and long spiral staircases are common, if you want to get out. And right, you don’t have to forget you’re in London – taking a minor tube journey may surprise you with heavy shower, though before that you could not see clouds :) Here, we went to British Museum, it’s all wet:

Inside you may spend hours, and variety of cultures represented may force you into deep discussions about all worlds being together (and, heh, Muslim cartoons).. Here, your stairs to the wisdom:

So, next early morning, full of impressions I was riding a cab home…

I always try to get emergency exit seats (not because I’m afraid, though), so every time I have to hear what should I do in case of disaster. This time it was funny:

• Steward: Please follow instructions how to open emergency door in case of disaster
• Me: Oh sure, if windows doesn’t fall off first
• Steward: ?????
• I gently touch the decoration frame around the window and it falls off. Window stays intact.
• Steward: ?!!!?!!!!?
• Me: :-D