Does exists some console utility for keyboard layout switching in Mac OS X

Put the .keylayout in /Library/Keyboard Layouts/ instead of ~/Library/Keyboard Layouts/.

I believe this is all possible. In my answer on TeX.SE, I give the MathUnicode.keylayout file that I made. Essentially, my advice is: go ahead and play around with this file.

A couple of thoughts:

• The XML specification for .keylayout files is available from Apple.

• On line 22 you can see that I refer to the anyOption modifier. This refers to the left or right option key. But you can also write option or rightOption if you wish to distinguish between the two.

• I'm not sure whether caps can be treated as a "hold down" key, as you want it, rather than a "press on / press off" key.