Writing to NSUserDictionaryReplacementItems with defaults

1. Is there a way to see what key combination is already taken, and for what purpose?

You can see if there is a conflict with your chosen shortcut. From "If a keyboard shortcut doesn’t work":

1. Choose Apple menu > System Preferences, and then click Keyboard.
2. Click Keyboard Shortcuts. A yellow warning triangle appears next to conflicting keyboard shortcuts.

2. Is there a key combination convention or rule for the different modifier keys? What key would be appropriate for what task? (e.g. when would I choose Command or Shift or Option or Ctrl or one of the function keys)

For dictionary lookups, Apple has already defined Ctrl+Cmd+D as "Display the definition of the selected word in the Dictionary app."

The OS X Human Interface Guidelines on Keyboard Shortcuts states the following about when to use these modifier keys and for what purposes. The recommended priority is to use Cmd frequently, use Option sparingly, avoid using Ctrl and use Shift for complementary functions of a common command (of course, you may observe that in certain instances Apple does not follow these guidelines).

Quote from the Keyboard Shortcuts page:

As much as possible, use the Command key as the main modifier key in a keyboard shortcut. For example, Command-P uses Command to modify the P key. For a command that complements another more common command, you can add Shift to the shortcut. For example, the shortcut for the complementary Page Setup command adds Shift to the shortcut for Print to give Shift-Command-P.

Use the Option key sparingly. If there’s a third, less common command that’s related to a pair of commands that use Command and Shift-Command, you can use Option-Command for the third command’s keyboard equivalent. Use combinations like these very rarely. You can also use Option for a keyboard shortcut that’s a convenience or power-user feature. For example, the Finder uses Option-Command-W for Close All Windows and Option-Command-M for Minimize All Windows.

As much as possible, avoid using the Control key. Because the Control key is already used by some of the universal access features—as well as in Cocoa text fields where Emacs-style key bindings are often used—it should be used as a modifier key only when necessary.

List multiple modifier keys in the correct order. If you use more than one modifier key in a shortcut, always list them in this order: Control, Option, Shift, Command.

As far as I've been able to determine there's still no proper way to do this. However, you can make it work by killing cfprefsd after using the above command. So I just use something like this in the Terminal app:

defaults write ~/Library/Group\ Containers/TEAMID.com.company.AppGroup/Library/Preferences/TEAMID.com.company.AppGroup PreferenceName -bool YES; killall cfprefsd

Not great, but better than nothing.

Update for macOS Sierra (10.12): Unfortunately using the file path doesn't seem to work anymore. Strangely it works on files that are elsewhere—like if I copy an App Group plist to the desktop—but I can't get it to work right on an App Group plist in its normal location.

It's worth noting that man defaults has included this note for a while:

The defaults command will be changed in an upcoming major release to only operate on preferences domains. General plist manipulation utilities will be folded into a different command-line program.

Like I said though, this isn't an issue with arbitrary plists, it seems specific to plists in ~/Library/Group Containers/.