Stump the Murph: ulimit, pam and linux

There is a game that a small group of friends and I have been playing since my Friendster years. It's called Stump the Murph. Basically if there is some weird problem in Linux mainly but it's in a variety of subjects-that we can't figure out we pass it to one of our friends Kevin Murphy. In 8 years I believe I stumped him once but I can't remember what it is so it doesn't count.

Here is the problem

SQLSTATE[HY000] [1135] Can't create a new thread (errno 11); if you are not out of available memory, you can consult the manual for a possible OS-dependent bug

"Obviously" this means you need to raise the ulimit for the process running mysql. I say "obviously" because this error means different things. In most cases it means that the server ran out of memory. perror 11 says OS error code  11:  Resource temporarily unavailable, yet when there is enough memory there may be a pam_limit issue. In my case there is.


So I did the following

in /etc/security/limits.conf I added this

mysql   soft    nofile  10240
mysql   hard    nofile  1537454
mysql   soft    nproc   32768
mysql   hard    nproc   65535

yet when I test the changes su - mysql
I get
 

su: pam_limits(su-l:session): Could not set limit for 'nofile': Operation not permitted

So my next course of action is to check

/etc/pam.d/system-auth

wait a second it has

session required pam_limits.so

and

/etc/pam.d/su calls

session         include         system-auth

thus I don't need to add session required pam_limits.so

Now the game of Stump the Murph begins:

In about 1/2 hour Murph figured out the solution! He deduced that since 

cat /proc/sys/fs/file-max
1537454

you can't set the hard limit of nofile to 1537454 because in theory you could starve the kernel from file descriptors thus from murph's suggestion I did

mysql      soft    nofile  10240

mysql      hard    nofile  768727
mysql      soft    nproc   32768
mysql      hard    nproc   65535

#MAKE SURE root has the same or greater settings!!!

root      soft    nofile  10240
root      hard    nofile  768727
root      soft    nproc   32768
root      hard    nproc   65535

Thanks Murph!