MacOS – MPlayerX frame skip keyboard shortcut

Go to System Preferences -> Keyboard -> Keyboard Shortcuts. Then, on the left side, select Exposé & Spaces. On the right side you will see the current keyboard shortcuts for Exposé and Spaces. If you double click on the shortcut command (literally double click ^1 for instance) you will be able to change this to whatever you want. Just do the key-combo you want to use.

So, once you double-click on the shortcut, it'll show it as editable. Then, press the keys you want the short cut to be, all at once, for instance press Apple and J at the same time, and it'll change the shortcut.

The function keys work because they are system-wide "media player control keys". Keystrokes always go to the system, which usually passes them exclusively to the active application, but some have special handling. There's a chance that you could reconfigure VLC so that the system-wide media key that means "rewind" does skip back instead. Check that out.

Otherwise you're going to need some kind of additional keyboard shortcut app like QuicKeys. (See also these suggestions). Don't give up yet! You almost certainly can cobble together a solution.