I'm not sure what documentation you've read so apologies if I'm repeating anything here.
To distribute reads to secondary nodes, most drivers allow you to set a readPreference value for the current session. Clients set read preference on a per-connection basis. With slaveOk, the driver should will always send queries to the secondaries, if they're available.
Distributing reads to secondaries requires the use of
ReplicaSetConnection with ReadPreference.SECONDARY.
See “rs.slaveOk()” for more information and this link.
In the mongo shell, to enable secondary reads, issue the following command :
rs.slaveOk()
The PHP documentation for it is here but I'm guessing that may be the documentation you're referring to.
As a FYI, here's an old discussion about it on the MongoDB Google Group.
If you're still having issues, I'd recommend using the MongoDB Google Group and providing some further information such as the version of MongoDB you're using, the version of the PHP driver, your log files, rs.conf() and rs.status().
As a FYI, you have to be careful with read scaling as sending too many reads to the secondaries can often result in the secondaries lagging the primary and becoming stale, thus requiring a full resync.
I think you can get what you want if you first rs.add() the new secondary, and when it's done you rs.remove() the arbiter.
There will be a brief period of thime with 4 nodes, howerever.
Best Answer
OK we found why there is a discrepancy.
We were able to list the collections on the secondary node with
From there, I realized the extra collection on the primary was
system.profile
(I'd forgotten I'd turned on the profiler) and it's 517 documents accounted for the different object counts from db.stats()