Can't initialize function 'INNODB'; Plugin initialization function failed

So running mysql-5.1.24rc with INNODB-Plugin-1a I ran into "Can't initialize function 'INNODB'; Plugin initialization function failed", and noticed a more serious issue with the whole plugin architecture.


Before without plugin if the engine died so did mysqld. Now, mysqld can stay up and accept connections. But what if you have 1000 mysql instances, and all of your Nagios checks where to test if mysqld was up and accepting connections? This entire test is now invalid. I now need a Nagios check to test if the plugin-engine is up, by doing a write to the engine.

Now to the cause of the issue:

Here is my condition. 5.1.24rc with INNODB-Plugin was killed by OOM killer (mysqld memory leak?). When it came back, my expectation was for innodb to go through a recovery process, it didn't. Strange. Then I got this message from replication.

Slave: Unknown table engine 'InnoDB' Error_code: 1286

So I did a show plugin command and innodb showed that it was installed. Since I'm out of ideas I thought that I would unload the various INNODB plugins.

Tada found the problem, I could not uninstall them something is wrong with the plugin. So, to make everything smooth for a reinstall of the plugin.

1. TRUNCATE mysql.plugins // I don't know if this is needed
2. restart mysql
3. INSTALL PLUGIN INNODB SONAME 'ha_innodb.so';

Now I have an error
InnoDB: mmap(26600275968 bytes) failed; errno 12
InnoDB: Fatal error: cannot allocate the memory for the buffer pool

Weird

I look at whats allocated. mysqld is consuming 11GB when nothing is loaded, and the various key buffers are sized very low. So, just to test a theory I lower the innodb bufferpool from 26GB to 19GB. Since 19+11 = 30GB and I have 32GB total.

1. start mysql
2. INSTALL PLUGIN INNODB SONAME 'ha_innodb.so';

Now it hangs. But, now this looks a familiar failure, I think its now recovering, so I look in the mysql error log and indeed it is recovering!