See /etc/launchd.conf

For some reason, I had this in /etc/launchd.conf:

limit maxfiles 2048

This apparently overrode my settings in /etc/sysctl.conf, either by being considered more important or by being executed later in the bootup process.

After I removed this and rebooted, my settings from /etc/sysctl.conf had taken effect.

Though not directly answering your question you will find two solutions at superuser:

1. To adjust open files limits on a system-wide basis in Mac OS X Yosemite, you must create two configuration files. The first is a property list (aka plist) file in '/Library/LaunchDaemons/limit.maxfiles.plist' that contains the following XML configuration:

   <?xml version="1.0" encoding="UTF-8"?>
   <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
     <plist version="1.0">
       <dict>
         <key>Label</key>
           <string>limit.maxfiles</string>
         <key>ProgramArguments</key>
           <array>
             <string>launchctl</string>
             <string>limit</string>
             <string>maxfiles</string>
             <string>65536</string>
             <string>65536</string>
           </array>
         <key>RunAtLoad</key>
           <true/>
         <key>ServiceIPC</key>
           <false/>
       </dict>
     </plist>
   

   This will set the open files limit to 65536. The second plist configuration file should be stored in /Library/LaunchDaemons/limit.maxproc.plist with the following contents:

   <?xml version="1.0" encoding="UTF-8"?>
   <!DOCTYPE plist PUBLIC "-//Apple/DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
     <plist version="1.0">
       <dict>
         <key>Label</key>
           <string>limit.maxproc</string>
         <key>ProgramArguments</key>
           <array>
             <string>launchctl</string>
             <string>limit</string>
             <string>maxproc</string>
             <string>2048</string>
             <string>2048</string>
           </array>
         <key>RunAtLoad</key>
           <true />
         <key>ServiceIPC</key>
           <false />
       </dict>
     </plist>
   

   Both plist files must be owned by 'root:wheel' and have permissions '-rw-r--r--'. This permissions should be in place by default, but you can ensure that they are in place by running sudo chmod 644 <filename>. While the steps explained above will cause system-wide open file limits to be correctly set upon restart, you can apply them manually by running launchctl limit.

   In addition to setting these limits at the system level, we recommend setting the at the session level as well by appending the following lines to your 'bashrc', 'bashprofile', or analogous file:

   ulimit -n 65536
   ulimit -u 2048
   

   Like the plist files, your bashrc or similar file should have -rw-r--r-- permissions. At this point, you can restart your computer and enter ulimit -n into your terminal. If your system is configured correctly, you should see that maxfiles has been set to 65536.

   Adjust the maxfiles & maxproc limit as you need it

   Sourced from: http://docs.basho.com/riak/latest/ops/tuning/open-files-limit/

2. Modifying the /etc/launchd.conf per a lot of google queries and SO suggestions didn't seem to work for me in Yosemite (10.10). What did end up working, after numerous change/reboot/test combinations, was modifying (or creating if it doesn't exist) the /etc/sysctl.conf file.

   This is what I had to put in to make it work

   kern.maxfiles=65536
   kern.maxfilesperproc=65536
   

   I'm not sure if 'kern.maxfiles' needs to be in there, but when I had it in there by itself I still had the same issue, when I added the 'kern.maxfilesperproc' everything started working.

If one or both of those solutions work for you, please credit the original posters.