We had a thousand or so threads waiting on LOCK_open, and the only thread holding the mutex was this darling:

Thread 902 (Thread 1637624128 (LWP 22113)):
#3  mutex_spin_wait (mutex=0x2aaaac3232b8,
        file_name=0x8b3bf7 "trx0trx.c", line=220) at sync0sync.c:565
#4  trx_allocate_for_mysql
#5  ha_innobase::allocate_trx
#6  check_trx_exists
#7  ha_innobase::info
#8  ha_innobase::open
#9  handler::ha_open
#10 openfrm
#11 open_unireg_entry
#12 open_table
#13 open_tables
#14 open_and_lock_tables
#15 mysql_insert

So, kernel mutex, which is quite contended, will escalate to LOCK_open on table open, which will block all queries from happening on the server. Fix? Nearly same code as previous “Opening tables” fix – don’t call ha_innobase::info() as part of open(), or move the transaction establishment code outside of info().

buf_read_ahead_random(...){ ...
       if (srv_startup_is_before_trx_rollback_phase) {
                /* No read-ahead to avoid thread deadlocks */
                return(0);
        }

This variable is tested at two functions – buf_read_ahead_linear() and buf_read_ahead_random() and nowhere else. So yeah, “server startup is before transaction rollback phase” is another way of saying “don’t do read ahead, please please”.

gdb -ex "set  srv_startup_is_before_trx_rollback_phase=1" \
    --batch -p $(pidof mysqld)

And many servers bottlenecked on this became much much much faster (and 2000 concurrent threads running dropped to 10). Of course, this is most visible in high-latency-high-throughput I/O situations, but we’re hitting this contention spot on local disk setups too.

Don’t forget to have the fix if gdb decides to be nasty and locks up your server :)

#include <sys/ptrace.h>
#include <signal.h>
main(int ac, char **av) {
int pid; if (ac>1) pid=atoi(av[1]);
ptrace(PTRACE_ATTACH,pid,0,0);
ptrace(PTRACE_DETACH,pid,0,0);
kill(pid, SIGCONT); }

It can simply take hours on such systems just to open tables – and the major portion of time spent is randomly diving into InnoDB tables to populate index statistics. It obviously sounds like low hanging fruit – as statistics aren’t needed while you are opening a table, they’re needed just for querying the table.

So, I threw in few thousand tables to my machine, and tried opening them with ten connections. Standard InnoDB code was opening 13.5 tables a second. After spending few minutes and moving (this is pure prototype, not suitable for production) statistic collection post ha_innodb::open(), I noticed performance increase.

Tables were opened at 105-a-second speed. A bit better, ~8x better.

Merry Christmas, MySQL!

Transaction 1 and 2 case is way too easy, try adding few hundred transactions that contend over same set of locks. To do that, InnoDB deadlock monitor will recursively brute-force lock graph, until it hits a 200-transaction-long chain (it will say it is a deadlock), or until it runs out of paths to check. Still, with the power of modern hardware that will still be milliseconds.

Unfortunately, InnoDB will also hold kernel_mutex at that time, so lots and lots of InnoDB operations will not happen at that time. To be exact, InnoDB will rarely do anything else, while deadlock check is happening.

To illustrate that, I have a very simple testcase (that in certain conditions stalls the server for half an hour, even if it is not being ran):

UPDATE t1 SET b=b+1 WHERE a=1;

With few threads it executes nearly 20000 times a second on my desktop machine. With ten threads it executes 14000/s. With 50 threads it is only 3000/s. With 100 threads it falls down to 639 operations a second. At 140 threads it is already just 266.

I built InnoDB without deadlock detection (tiny tiny patch), and tried same test. Similar performance with 10 threads, still doing 10000 operations a second at 100 threads:

Though I illustrated edge case here, its purity actually didn’t show how bad this can go – this situation can happen not only because of high contention on single row, but simply because someone holds up the row lock for a bit too long (there’s always that sleep between UPDATE and COMMIT, too). It can take a single transaction to cause a lock convoy, and once transactions queue up, and update rate falls down below 100/s, all MySQL will be doing is checking for deadlocks, even if they never happen.

On many systems deadlock detection is causing way more issues, than lack of it would. Most deadlocks happen on transactions that are somewhere in the middle of their lock wait anyway :)

There’s some discussion about it at MySQL Bug#49047

I feel ashamed that I ever wanted you to support 4.0->5.1 replication, and apologize for that. I really understand that it was really egoistic of me even to consider you should be involved in this.

I even understand that 5.0 is running out of active support (I’m not questioning that you’ll stop supporting 4.1 entirely too), and you’ll stop doing pretty much anything to 5.0, except “critical security fixes” (w00t, I managed to get one into 4.1, 8 year old MITM flaw :).

I really understand that supporting more than one release is very very very difficult, and people should do only adjacent version upgrade.

I’m not asking you much, but, maybe you could then support 5.1 to 5.1 replication? I don’t want much, just:

• Gracefully recover after slave crashes.
• Don’t have single serial reading of pages for replication stream as a bottleneck – either read-ahead properly (you can do that with RBR!!!), or apply events in parallel (you can do that with RBR too!)
• Allow to edit replication filters without restarting servers.
• Allow to enable and disable binary logging of events received from master, as well as enabling and disabling binary logging without restarting the instance.

I hope it isn’t too much too ask! It is just supported replication between two same version instances.

Thanks!

See, developers are perfectionists, and their perfectionism also includes the crazy idea that all memory has to be deallocated at server shutdown, as otherwise Valgrind and other tools will complain that someone leaked memory. Developers will write expensive code in shutdown routines that will traverse every memory structure and deallocate/free() it.

Now, guess what would happen if they wouldn’t write all this expensive memory deallocation code.

Still guessing?

OS would do it for them, much much much faster, without blocking the shutdown for minutes or using excessive amounts of CPU. \o/

InnoDB: Doing recovery: scanned up to log sequence number 354164119040
InnoDB: Doing recovery: scanned up to log sequence number 354169361920

On a machine with bigger logs it will start spending nearly 100% CPU somewhere in recv_scan_log_recs. Guess what it does…. -fno-inline builds to the rescue:

#0  mem_block_get_len at ./include/mem0mem.ic:86
#1  mem_heap_get_size at ./include/mem0mem.ic:591
#2  recv_scan_log_recs at log/log0recv.c:2727

And:

samples  %        symbol name
8467     72.9222  mem_heap_get_size
291       2.5062  recv_add_to_hash_table
95        0.8182  mem_block_get_len

To speak in layman’s terms, InnoDB does SUM(LENGTH(allocation)) on its relatively wide memory (tens, hundreds of thousands of entries) arena, FOR EVERY LOG SEGMENT, to make sure it didn’t run out of available 32GBs. Hehehe, lame.

As for now, I’ll just killed the check and have my recovery much much faster – finished in 3 minutes, what it wasn’t able to do in 30 before.

P.S. This is different from what I wrote before (and magic Yasufumi’s patch)
P.P.S. Now I got to learn to reuse LOG_DUMMY table during the recovery process, as it is next low hanging fruit there…

I will do this to rest of your code, if you continue breaking replication for me.

– Domas

This particular benchmark has two favorite parts, that go with each other together really well:

I didnt change absolutely any parameters for the servers, eg didn’t change the innodb_buffer_pool_size or key_buffer_size.

And..

If you need speed just to fetch a data for a given combination or key, Redis is a solution that you need to look at. MySQL can no way compare to Redis and Memcache. …

Seriously, how does one repay for all the damage of such idiotic benchmarks?

P.S. I’ve ranted at benchmarks before, and will continue doing so.

P.S. I also quit World of Warcraft. ;-)

I don’t like objections, therefore I object to them. Of course, I understand, that some people want to play major parts in dramas and soap operas, others are spectating and enjoying it, but most people I know and respect simply want to be efficient and get their work done.

These people, many of whom were my colleagues back at Sun/MySQL, sometimes can be labeled as ones who don’t care about greater picture, or the common good. Unfortunately, the topic of common interest depends from religion to religion, from society to society, from person to person.

My approach to common good is doing what I want to do for the community, and not what community wants me to do. If that relation is not compatible for any of the sides, there is no relation at all. Now, what happens now, is that there is some user community (and I include all the people who are protecting their free-software-usage interests) which has demands for vendors.

When PG zealots keep telling that “PG is open, MySQL is closed-development by a company”, you will often see MySQL camp not listening to that at all. But there is some true in that, every MySQL user was using a product that was developed inside a citadel (even some builds were called ‘Citadel’), and company had a say over lots of issues.

Start with Windows licensing at the beginning, GPLv2 versus GPLv3 or any other issue, that was affecting our user community at large, it has always been top-down decision, nobody has ever asked me as a user what kind of license I would want – and all those decisions were made by MySQL, the semi-independent company.

I never actually craved for too much control of the project – I’m always free to fork (and I have forked at least 3 releases of MySQL, and now have to deal with yet another fork at work ;-). I have went through various stages of relations with MySQL – and at every of them I had to understand that open-source software carries its own set of risks, but also has its own set of mitigations. I’m free to support my organization needs at various outcomes, I don’t have any data lock-in, I don’t have any vendor lock-in, the only impact is a risk that I should have calculated years ago, not today.

So even if the argument starts with “MySQL is being acquired by competitor, and it is risky situation”, there are few real messages out there: “we need to protect people who don’t care about risks” – sounds very much like bailout money for all the people with crazy spending, as well as “if your entry campaign is about doing lots of common good, you may not be allowed to profit in the future”.

This doesn’t make open-source a good investment (would Sun pay 1B$ if it forecasted today’s EC position?), and it not being a good investment means there won’t be opportunities for people to balance common good and business opportunities in future.

If a community was supposed to have a voice in this situation, it should’ve had a fair share of board members in project management. If a community didn’t get its fair share of management at the organization, it should’ve found another organization, or accepted the risks.

In my opinion, if you are right now in the camp of supporting objections, it is not because you’re seeing a lot now, it is mostly because you didn’t see anything before.

My favorite features are python scripting and non-stop debugging. I was toying around with python scripting for a while, and was planning to make backtraces make sense. Having hands that open means that one can see PHP backtraces, when gdb’ing apache, see table names and states when MySQL thread access handler interfaces, or remote IPs and users, when it is writing to network. Process inspection can simply rock, if right tools are created using these new capabilities, and I’m way too excited when I think about those. “Always have debugging symbols” gets way more meaning now.

Another issue I’ve been trying to resolve lately is avoiding long locking periods for running processes (directly attaching to process can freeze its work for a second or so, which isn’t that tolerable in production environments). GDB is getting closer to the async debugging capabilities – where one can run a debugger without actually stopping anything.

So, congratulations GDB team, now it is job for us to find all the uses of the tool. It has been invaluable so far, but this is much much more.

• “Sending data” – reading data from tables (or looking it up)
• “Copying to tmp table” – reading data from tables (or looking it up)
• “Copying to tmp table on disk” – query needs a rewrite
• “statistics” – looking up data from tables
• “Sorting result” – reading data from tables (or looking it up)

Locking is fun:

• “update” – There is big chance that it is waiting for a lock, can be updating as well, on busy systems
• “Locked” – Even bigger chance that it is MyISAM table waiting to be converted to InnoDB

Table opening is even funnier:

• “Opening tables” – some other idiot thread is holding the LOCK_open or similar mutex
• “Closing tables” – some other idiot thread is holding the LOCK_open or similar mutex
• “Opening table” – the idiot thread (or not that smart DBA who thinks that table_cache should be set to current value)
• “Flushing tables” – someone decided that flushing tables is good idea, but forgot to kill long running queries
• “Waiting for tables” – same as “Flushing tables”

If you have replication:

• “Connecting to server” – could not connect to server, waiting to retry
• “Reconnecting after a failed master event read” – master and slave have same @server_id
• “Registering slave on master” – master and slave have same @server_id
• “Waiting to reconnect after a failed master event read” – master and slave have same @server_id

Few more tricky ones:

• “Sleep” – last millisecond it was probably not sleeping, due to how process scheduling and kernel level buffering works probably it has finished right before kernel switched to the ‘SHOW PROCESSLIST’ thread.
• “Writing to net” – usually, not writing to net (client has blocked) – actual writing to net is usually nearly non-blocking operation, as long as client doesn’t sleep.
• “Killed” – actually, someone tried to kill it, but it doesn’t care, as it doesn’t check for thd->killed flag

And in the user column:

• “unauthenticated user” – are you sure your DNS is working right?
• “system user” – it isn’t user, these are (hopefully plural) replication threads, that don’t have any “user” credentials
• “root” – either that is you running the PROCESSLIST, another session you forgot you are running or you forgot to create separate user for your application

I probably didn’t write quite a few important ones, but some of them are self-evident (such as “init” and “end”), and others probably will never show up :)


/*
* Author: Domas Mituzas
* Released to public domain
*
* apxs -c -i mod_dumpcore.c
* and...
* LoadModule dumpcore_module .../path/to/mod_dumpcore.c
* CoreDumpDirectory /tmp/cores/
* and...
* sysctl -w kernel.core_pattern=/tmp/cores/core.%p.%t
*/

#include "httpd.h"
#include "http_config.h"
#include <sys/prctl.h>

static int dumpcore_handler(request_rec *r)
{ prctl(PR_SET_DUMPABLE,1,0,0,0); return DECLINED; }

module AP_MODULE_DECLARE_DATA dumpcore_module = {
STANDARD20_MODULE_STUFF, NULL, NULL, NULL, NULL, NULL,
dumpcore_register_hooks };


P.S. I was quite astonished to find out that nobody ever needed this, I remember quite a few discussions after which we fixed this in MySQL two years ago.

Update: there’s also ‘echo 1 > /proc/sys/fs/suid_dumpable’ or ’sysctl -w fs.suid_dumpable=1′ – now I recall whole story, RHEL3 and RHEL4 didn’t have this, so we had to do prctl() hack, whereas later Linux kernel versions allowed this workaround.

I don’t know how effective your scare-mongering cash-extortion tactics are, but they don’t really help neither your users, nor vendors, nor anyone else.

It all starts when major vulnerability databases start authoritatively spouting out crap like this:

A vulnerability has been reported in MySQL, which can be exploited to compromise a vulnerable system.
The vulnerability is caused due to an unspecified error and can be exploited to cause a buffer overflow. (Secunia)

Or crap like this:

MySQL is prone to a buffer-overflow vulnerability because if fails to perform adequate boundary checks on user-supplied data.
An attacker can leverage this issue to execute arbitrary code within the context of the vulnerable application. Failed exploit attempts will result in a denial-of-service condition. (Securityfocus)

Of course, there must be some reason to publish such claims, right? Of course, if you brought in cash for all these security vendors, they’d definitely tell you that in future they will give you all available updates, but for now:

Due to the very limited available information, it is not possible to suggest an effective workaround.

Why would one post such crap? Because people want to sell things, like vulnerability databases full of rubbish:

A working commercial exploit is available through Intevydis. This exploit is not otherwise publicly available or known to be circulating in the wild (Securityfocus)

This is how the Security Vendor that unleashed this all classifies the vulnerability:

• Name: MySQL 5.x exploit
• Status: 0day
• Details: Remote buffer overflow exploit. Tested on Debian Linux 5.0 with mysql-server 5.0.51a-24+lenny1
• Listener: LINUXMOSDEF
• Platform: Linux x86

If you’d look it up, Debian just had security advisory DSA-1877-1 (released at September 2nd, one day before Secunia and Securityfocus went live with their stuff), which tells us about denial of service and execution of arbitrary code possibility. The +lenny2 package fixes it, the +lenny1 package is vulnerable. At this point, I don’t really have anything against Debian security people – Linux distribution security contacts are always eager to communicate, share, discuss, and, well, admit failures – and as they are not in money extortion business, it is very easy to forgive them. Still, the advisory mentions “potential execution of arbitrary code via format string specifiers”. See, original exploit at milw0rm did mention “format string vulnerability”, which is source of  [easily understandable] confusion here.

Various C libraries allow passing %n format specifier to printf() calls, which is:

The number of characters written so far is stored into the integer indicated by the int * (or variant) pointer argument.  No argument is converted.

This means that if you allow someone to pass a format string, he can overwrite memory of your application with specially crafted data (though, it isn’t that trivial to exploit it). MySQL though, as it has to be very portable, has its own version of printf, to avoid any OS-specific behaviors, and that implementation does not have %n, which means by passing arbitrary format string you cannot execute arbitrary code. Phew.

So, what is the bug? Let me present you Bug#45790:

• If server has General Query Log enabled (thats very very low percentage of systems out there) and….
• User has right to create databases (which isn’t a right given away to every user out there….) then…
• He can shut down (well, crash) the server!

Summary of this whole security catastrophe in practice would be:

System administrator can shut down MySQL server

My heart pounds and I hurry to upgrade every machine to 5.0.84 (sarcasm aside, do it anyway, it has great fixes ;-).

P.S. I may be wrong, and there is an exploit which will convert your database clusters into botnet zombies, capable of much more than regular botnets – imagine all the multicore servers attached to SANs completely dominating the world. Scary. Thats why we try to work on all security threats – even if it takes quite a few hours too long for what it deserves.

Nowadays big media events can cause some troubles, but generally they don’t bring huge traffic spikes anymore. Say, Michael Jackson’s English Wikipedia article had peak hour of one million page views (2009-06-25 23:00-24:00) – and that was merely 10% increase on one of our projects (English Wikipedia got 10.4m pageviews that hour). Our problems back then were caused by complexity of page content – and costs got inflated because of lack of rendering farm concurrency control.

Other interesting sources of attention are custom Google logos leading to search results leading to Wikipedia (of course!). Last ones, for Perseids or Hans Christian Ørsted sent over 1.5m daily visitors each – but thats mere 20 article views a second or so.

What makes those spikes boring nowadays is simply the length of long-tail. Our projects serve over five million different articles over the course of an hour (and 20m article views) – around 3.5m articles are opened just once. If our job would be serving just hot news, our cluster setup and software infrastructure would be very very very different – and now we have to accommodate millions of articles, that aren’t just stored in archives, but also are constantly read, even if once an hour (and daily hot set is much larger too).

All this viewership data is available in raw form, as well as nice visualizations at trendingtopics, wikirank and stats.grok.se. It is amazing to hear about all the research that is built on this kind of data, and I guess it needs some improved interfaces and APIs already for all the future uses ;-)

For example, moving a slave to another master down a replication topology could be expressed this way (this is a working code, actually):

slave = mysql(options.slave)
oldmaster = mysql(slave.get_master())
newmaster = mysql(options.newmaster)

oldmaster.lock()
oldpos = oldmaster.pos()
newmaster.wait(oldpos)
newmaster.lock()
oldmaster.unlock()
slave.wait(oldpos)
slave.change_master(newmaster)
newmaster.unlock()

I’m sure transaction group/global IDs would simplify the process a lot, but still, having building blocks one can write pretty much self-documenting narrow code, shuffle actions done without having to rethink whole programming logic too much. Implementation of methods like .sync(), .clone(), .promote() ends up environment-specific, but may save quite some time afterwards too.

As much as I’d like everyone around to get their data management actions written down into scripts, I’d like every DBA action I do to be written down in such code too :-) I’d love to have code, which detects resource shortages, orders servers, deploys software and re-shards data automatically… well, you know what I mean :)

mysql> show variables like 'log_slave_updates';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| log_slave_updates | OFF   |
+-------------------+-------+
1 row in set (0.00 sec)

mysql> set global log_slave_updates=1;
ERROR 1238 (HY000): Variable 'log_slave_updates' is a read only variable

Of course, there are few options, roll in-house fork (heheeeee!), restart your server, and keep warming up your tens of gigabytes of cache arenas, or wait for MySQL to ship a feature change in next major release. Then there are evil tactics:

mysql> system gdb -p $(pidof mysqld)
                       -ex "set opt_log_slave_updates=1" -batch
mysql> show variables like 'log_slave_updates';
+-------------------+-------+
| Variable_name     | Value |
+-------------------+-------+
| log_slave_updates | ON    |
+-------------------+-------+
1 row in set (0.00 sec)

I don’t guarantee safety of this when slave is running, but… stopping and starting slave threads is somewhat cheaper, than stopping and starting big database instance, right?

What else can we do?

mysql> show slave status \G
...
     Replicate_Do_DB: test
...
mysql> system gdb -p $(pidof mysqld)
          -ex 'call rpl_filter->add_do_db(strdup("hehehe"))' -batch
mysql> show slave status \G
...
      Replicate_Do_DB: test,hehehe
...

It is actually possible to add all sorts of filters this way, rpl_filter.h can be good reference :) So now that you want to throw out some data from your slaves, restart isn’t needed. Unfortunately, deleting entries isn’t possible via rpl_filter methods, but you can always edit base_ilists, can’t you?

P.S. having this functionality inside server would definitely be best.

Say, even though you can select configuration variables using @@syntax, you can’t do same for much more interesting to DBAs status variables in any more interesting logic.

Apparently instead of doing

SHOW STATUS LIKE "questions"

one has to do this now (I’m being dramatic here, above hasn’t been removed yet, but hasn’t been expanded for better usage either):

SELECT VARIABLE_NAME, VARIABLE_VALUE
FROM INFORMATION_SCHEMA.GLOBAL_STATUS
WHERE VARIABLE_NAME="QUESTIONS"

Do note, those SQL standard followers will get caps-lock button swapped with space bar soon.

Of course, we, DBAs, know that one can simplify stuff by creating stored routines:

CREATE FUNCTION `gstatus`(v varchar(64)) returns varchar(1024)
return
( SELECT variable_value
  FROM information_schema.global_status
  where variable_name=v LIMIT 1
)

So we can do such simple things as:

mysql> select m.gstatus("questions");
+------------------------+
| m.gstatus("questions") |
+------------------------+
| 140                    |
+------------------------+
1 row in set (0.00 sec)

Of course, this leads to solution of one of most common DBA problems, how to get decent status variable values per time:

CREATE PROCEDURE m.report(in timer float)
begin

DROP TEMPORARY TABLE IF EXISTS status_old;
CREATE TEMPORARY TABLE status_old
SELECT * FROM INFORMATION_SCHEMA.GLOBAL_STATUS;

SELECT SLEEP(timer) into @x;
SELECT
    s.variable_name status,
    (s.variable_value-o.variable_value)/timer value
FROM INFORMATION_SCHEMA.GLOBAL_STATUS s
    JOIN status_old o USING (variable_name)
WHERE s.variable_value>0;

DROP TEMPORARY TABLE status_old;

end

So, the “show me changes-per-second for values in last 0.5s” would look like this:

ysql> call m.report(0.5) //
+-----------------------------------+---------+
| status                            | value   |
+-----------------------------------+---------+
| ABORTED_CLIENTS                   |       0 |
| ABORTED_CONNECTS                  |       0 |
| BYTES_RECEIVED                    |  532662 |
| BYTES_SENT                        | 1140894 |
...
| QUERIES                           |    2884 |
| QUESTIONS                         |    2878 |
| SELECT_FULL_JOIN                  |       2 |
| SELECT_RANGE                      |     196 |
| SELECT_SCAN                       |     146 |
...
| THREADS_CACHED                    |      12 |
| THREADS_CONNECTED                 |     -28 |
| THREADS_CREATED                   |       4 |
| THREADS_RUNNING                   |      -2 |
| UPTIME                            |       2 |
| UPTIME_SINCE_FLUSH_STATUS         |       2 |
+-----------------------------------+---------+
125 rows in set (0.53 sec)

Query OK, 0 rows affected, 1 warning (0.54 sec)

So, by spending five minutes on writing very simple INFORMATION_SCHEMA procedure we can resolve one of usual nightmares in MySQL DBA environments.

I can get back now to the initial idea of this post – if one DBA can write such small neat thing in few minutes, would you imagine how useful can a collaboratively built repository of DBA-assisting stored procedures in functions, and how we can spit at all the SQL standard verbosity, and make our systems easy to manage? :) I think we shouldn’t allow not to share such utilities, as widespread use and “expect it already there” would make overall work much much easier. Let’s use and reuse (and someone should set up framework for building such thing ;-))

You can find most of my views on various issues in our question pages (I was somewhat boiling when answering the What will you do about the WMF mishandling it’s funding? one – it probably takes great effort to phrase such a bad question, and so easy to answer it :), as well as Wikipedia Signpost ‘interview’.

I was appointed to the Board back in January 2008, after holding various other volunteer (at some point in time – ‘officer’) positions within the organization since 2004 – and brought in the core technology and operational efficiency skill set there. The appointment was supposed to be somewhat temporary, but board restructure appeared to be much longer process than we expected – both the chapters part, and nomination committee work. As a community member, after the restructure I was in ‘community-elected’ seat, though I never participated in any election – so that wasn’t too fair to the actual community, need to fix that :)

So, even though I wasn’t too visible to actual community (people would notice me mostly when things go wrong, and I’m not in best mood then, usually :-), I feel that the values I’ve worked on, evangelized and supported for all these years – efficiency and general availability of our projects – can win mindshare not only of our read-only users I work mostly for, but also eligible voters.

And I do think, that internal technology expertise has to be represented on board, as things we’ve been doing, and methods we’ve been using, are very much unique in the technology world. Oh, and somewhere I mentioned, our technology spending is close to 50%, that has to be represented too :-)

The whole plan is:

1. Compile all code with -fprofile-generate in {C|CXX|LD}FLAGS
2. Run the binary
3. Run your application/benchmark against that binary
4. Recompile all code with -fprofile-use (above steps will place lots of .gcda files in source tree)
5. PROFIT!!! (note the omission of “???” step)

How much profit? I measured ~7% of sysbench performance increase (and probably would see much higher value in CPU-tight benchmarks). YMMV. Can such PGO be useful for every user out there? Maybe – but the best results are achieved once looking at actual use patterns – though of course, lots of them are similar everywhere around.

Also, I am showing the actual profiling process with too much of pink. Apparently gcc/gcov profiles tend to get corrupted in multithreaded applications, so I did multiple profile/build passes, until I managed to assemble final binary. :-)

Now I have to figure out how to use -combine flag in gcc, and treat whole MySQL codebase as one huge .c file (apparently compilers can make much much better decisions then).

But there’s one thing that definitely got worse – gcc, the compiler, has a change in default compilation options – it omits frame pointers from binaries in x86_64 architecture. People have advocated against that back in 1997 because of very simple reasons, that are still very much existing today too – frame pointers are needed for efficient stack trace calculations, and stack traces are very very useful, sometimes.

So, this change means that oprofile is not able to give hierarchical profiles, it also means that MySQL crash information will not include most useful bit – the failing stack trace. This decision has been just because of performance reasons – frame pointer takes whole register (though on x86_32 that meant 1 out of 8, on x86_64 it is 1 out of 16), which could be used to optimize the application.

I tested two MySQL builds, one built with ‘-O3 -g -fno-omit-frame-pointer’ and other with -fomit-frame-pointer instead – and performance difference was negligible. It was around 1% in sysbench tests, and slightly over 3% at tight-loop select benchmark(100000000,(select asin(5+5)+sin(5+5))); on a 2-cpu Opteron box.

The summary suggestion for this flag would be very simple. If you don’t care about fixing or making your product faster, you can probably use 1% speed-up. But if getting actual real-time performance data can lead to much better performance fixes, and fast introspection means qualified engineers can diagnose problems much faster, 1% or even 3% is not that much. So, add ‘-fno-omit-frame-pointer’ to CFLAGS and CXXFLAGS, and enjoy things getting back to normal :)

By the way, while I was at it, I did some empiric tests of other options, but one that irks me most is not using ‘-g’ on production binaries. See, debugging information, symbol tables, etc – they all cause around 0% performance difference. The only difference is that e.g. mysqld binary will weight 30M, instead of 6M (though that fat will not be loaded into RAM, and will only cost diskspace).

Why does debugging information matter? It doesn’t, if you don’t attempt to be power-user. It does, if you enjoy crazy debugger tricks (like one here or here). Oh, and of course, bonus GDB trick, how to run KILL without connecting to server:

(gdb) thread apply all bt
...
(gdb) thread 2
[Switching to thread 2 (Thread 0x44a76950 (LWP 23955))]#0  ...
(gdb) bt
#0  0x00007f7821e68e1d in pthread_cond_timedwait...
#1  0x000000000052dc0e in Item_func_sleep::val_int (this=0x12317f0)...
#2  0x0000000000501484 in Item::send (this=0x12317f0, ...
...
#15 0x00000000005caf29 in do_command (thd=0x11da290) ...
...
(gdb) frame 15
#15 0x00000000005caf29 in do_command (thd=0x11da290) at sql_parse.cc:854
854	  return_value= dispatch_command(command, thd, packet+1, ...
(gdb) set thd->killed = THD::KILL_QUERY
(gdb) continue

And the client gets ‘ERROR 1317 (70100): Query execution was interrupted’ :-)

Given a dictionary, find words that have most anagrams. With few-hundred-thousand word dictionary, on 8-way machine this should complete in less than 12 hours.

The supercomputing lab inspired solution was something like this (I may be wrong, years pass though):

• Put all the words into an array
• Start generating letter sequences: aaaa, aaab, aaac, …
• For every generated word, traverse the array, find how many matches it hits
• Get sequences with most matches

See, the problem was not how to make this algorithm efficient, but how to parallelize it, or maybe rewrite in C (thats what they do in supercomputing labs). It took few minutes to write following code:

wordlist=open("/usr/share/dict/words")
anagrams={}
for line in wordlist:
  word=line.strip()
  sorted=list(word)
  sorted.sort()
  sorted="".join(sorted)
  if sorted not in anagrams:
    anagrams[sorted]={'count':1,'words':[word,]}
  else:
    anagrams[sorted]['count']+=1
    anagrams[sorted]['words'].append(word)

(I just canonized words into base-anagram-form – sorted letters, and ran collision test)

It took maybe half a second to run it on my 1.2ghz ibook back then. The end of the story back then was that we get one critic less.

Now, why I remember this story today – Yasufumi Kinoshita just did exactly the same to us, MySQL users. Simply, brute force is not the answer, if you can spare the minute and rethink stuff. Thanks, for reminding this.

http://mituzas.lt/tech/query-cache-tuner/

Though for me, it was really special – Alvydas Mituzas, my Dad, got a Knight’s Cross, for his lifetime merits and service for our country, spanning forty years of creativity and dedication. Anyone knowing my Dad really know what the award is for, but for me it is also a symbol of support for virtues that I’ve seen in my whole life – hard work and imagination combined for public service and public good, no matter how difficult it is to start, or finish. Stories of his past are fascinating, and even I can learn more and more of them over the years. We’re proud of him, and we’re lucky to be his family.

When carrying out more database benchmarking, but this time with PostgreSQL, XFS and Btrfs were too slow to even complete this test, even when it had been running for more than an hour for a single run. Between EXT3, EXT4, and NILFS2, the fastest file-system was EXT3 and then its successor, EXT4, was slightly behind that. Far behind the position of EXT4 were NILFS2 and then Btrfs and XFS.

There were few other benchmarks, e.g. SQLite showed ‘bad performance’ on XFS and Btrfs.

*clear throat*

Dear benchmarkers, don’t compare apples and oranges. If you see differences between benchmarks, do some very very tiny research, and use some intellect, that you, as primates, do have. If database tests are slowest on filesystems created by Oracle (who know some stuff about systems in general) or SGI (who, despite giving away their campus to Google, still have lots of expertise in the field), that can indicate, that your tests are probably flawed somewhere, at least for that test domain.

Now, probably you’ve heard about such thing as ‘data consistency’. That is something what database stack tries to ensure, sometimes at higher costs, like not trusting volatile caches, enforcing certain write orders, depending on acknowledgements by underlying hardware.

So, in this case it wasn’t “benchmarking file systems”, it was simply, benchmarking “consistency” against “no consistency”. But don’t worry, most benchmarks have such flaws – getting numbers but not understanding them makes results much more interesting, right?

Oh, and… thanks for few more misguided people.

There are few use cases for it. One is for all these people who benchmark stuff and want selective OS cache purges, another is for those who run high performance databases. Remember the O_DIRECT serialization everywhere? Well, XFS does direct I/O in parallel, unless there are cached pages (and they can happen because of any random outside-of-database activity, like ‘file’ command). Once you ‘uncache’ the files, XFS will be very much parallel again \o/ \o/