MySQL server connections capped at 1013

MySQL

For the last few days I've been trying to figure out why my MySQL read slave servers are capped at 1012+1 (+1 for the superuser) connections when I have the max_connections set to 5000. It seems that no matter what I change, the "Historic max_used_connections" never increases beyond 1013.

The mysql read slaves support a high traffic web site. I have a cluster of 3 read only slaves. Each read slave seems to be capped at 1013 max connections. I can take two out of the cluster so all connections go to one and still only 1013. No error messages show up in the mysql log file.

A mysqladmin status on one of the read slaves shows this:
Uptime: 1661 Threads: 128 Questions: 17634224 Slow queries: 0 Opens: 2441 Flush tables: 1 Open tables: 938 Queries per second avg: 10616.630

Same server, mtop shows this:
load average: 0.05, 0.15, 0.19 mysqld 5.1.66-log up 0 day(s), 0:28 hrs
106 threads: 4 running, 906 cached. Queries/slow: 1.6K/0 Cache Hit: 99.96%
Opened tables: 0 RRN: 278 TLW: 1.0K SFJ: 0 SMP: 0 QPS: 1

Each server has 12 dual-core Intel CPU's (24 cores) with 140 GB of RAM running RHEL 6.3, Kernel 2.6. (x86_64)

Any idea what would be limiting/capping the connections at 1012+1 ?

EDIT
Also wanted to add two things:

1) I am running the "free/community" edition of MySQL that you get from a Red Hat 6.3 YUM install. Could there be some limitation in that version?

2) I've changed the /etc/security/limits.conf file on the MySQL read slaves, increasing values for users mysql and root (did not help my situation):

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1161534
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 10240
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 32768
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Best Answer

We had the same issue. Put this at the top of the mysql start script.

ulimit -u 102400 -n 102400

Has to be at mysql start time.