In my quest of making MySQL usable, I managed to hit contention that wasn’t spotted by performance masters before. Meet most useless mutex ever (this is actual contention event, not just a hold):
Count nsec Lock
1451 511364 mysqld`ut_list_mutex
nsec ---- Distribution --- count Stack
2048 |@@ | 132 libc.so.1`mutex_lock_impl
4096 |@@@ | 186 libc.so.1`mutex_lock
8192 | | 3 ut_malloc_low
16384 |@@ | 137 mem_heap_create_block
32768 |@ | 105 row_sel_store_mysql_rec
65536 |@ | 89 row_search_for_mysql
131072 |@ | 99 ha_innobase::general_fetch
262144 |@@ | 131 ha_innobase::rnd_next
524288 |@@@ | 195 rr_sequential
1048576 |@@@ | 223
2097152 |@@ | 137
4194304 | | 14
----------------------------------------------------------------
ut_list_mutex guards a memory structure which has all memory blocks allocated by InnoDB (via ut_malloc/ut_free) in it.
It has two uses:
- Printing “Total memory allocated” in SHOW INNODB STATUS (though this can still be implemented lock-free)
- Deallocating all memory on shutdown (though, all modern operating systems do that anyway, so this is purely just to shut up valgrind)
If you have any BLOB/TEXT data in your tables, you’re definitely hitting this contention spot (it is #1 contention in such cases).
Fix? Kill the eyecandy, replace ut_malloc and ut_free with direct calls to malloc() and free(), oh and of course, use scalable allocators like tcmalloc or Hoard.
Tags: innodb, mutex, performance
Domas,
Looking at the code for 5.0.67 and based on the call stack that you provide, the call chain is mem_heap_create_block -> mem_area_alloc -> ut_malloc -> ut_malloc_low. There are a couple of things to note:
* mem_area_alloc/mem_area_free is protected by a separate mutex that can be hot
* mem_area_alloc only calls ut_malloc when the mem_pool_t runs out of memory
* most of us don’t get to use dtrace. Were you using it?
* I have not had see the ut_malloc mutex get hot. What workload did you use?
* the Google patch changes mem_area_malloc/mem_area_free to call malloc/free directly. Even without tcmalloc, most system libraries are probably better than what is provided by mem_area_alloc/mem_area_free/mem_pool_t for multithreaded usage
damnit, to figure out where the problem was happening I rebuilt 5.1.30 with -fno-inline and now I’m hitting mem_pool mutexes, rather than ut_malloc one. Debugging inlined functions has always been pain :) Maybe I had some specific situation/edge case, where it was being hit, but I reproduced it on both 4.0 and 5.1 (I’m not sure I even have anything else on this Solaris box).
Will try to understand what is happening in code path more.
Yes, I was using dtrace plockstat thing here.
The workload was BLOB reading en-masse. :)
We avoid blobs as much as possibe. I think that Innodb has special code paths for large allocations which blobs might use.