Use xkb to make shift insert parenthesis

As always, figured this one out moments after posting the question. The problem was simply that I was using Hyper_L which by default is assigned to Mod4. By changing Caps Lock to Hyper_R instead, it worked. It still had to be bound to Mod3 though as a real modifier is still needed.

Here's the updated config which now produces the desired behaviour:

partial
xkb_types "hyper" {
    virtual_modifiers Alt,Hyper;

    type "HYPER" {
        modifiers= Shift+Control+Alt+Hyper;

        map[Shift]= Level2;
        preserve[Shift]= Shift;

        map[Control]= Level2;
        preserve[Control]= Control;

        map[Shift+Control]= Level2;
        preserve[Shift+Control]= Shift+Control;

        map[Alt]= Level2;
        preserve[Alt]= Alt;

        map[Shift+Alt]= Level2;
        preserve[Shift+Alt]= Shift+Alt;

        map[Control+Alt]= Level3;
        preserve[Control+Alt]= Control+Alt;

        map[Shift+Control+Alt]= Level2;
        preserve[Shift+Control+Alt]= Shift+Alt;

        map[Hyper]= Level2;

        level_name[Level1]= "Extra";
        level_name[Level2]= "Normal";
        level_name[Level3]= "Ctrl+Alt";
    };
};


partial function_keys
xkb_symbols "hyper_f21" {

    replace key <FK01> {
        type= "HYPER",
        symbols[Group1]= [           F21,             F1, XF86Switch_VT_1 ]
    };
    replace key <FK02> {
        type= "HYPER",
        symbols[Group1]= [           F22,             F2, XF86Switch_VT_2 ]
    };
    ...
};

partial modifier_keys
xkb_symbols "caps_hyper" {
    replace key <CAPS> {
        [ Hyper_R ]
    };
    # Remove Hyper (Hyper_L/Hyper_R) from Mod4, was added by "pc" layout
    modifier_map none { <HYPR> };
    # Now make physical caps lock (mapped to Hyper_R above) control Mod3.  Mod3 is
    # associated with the virtual modifier "Hyper" because the existing XKB config
    # links Hyper_L and Hyper_R with this virtual modifier.  Therefore Mod3 becomes
    # the virtual modifier "Hyper" because they both share the same keysym Hyper_R.
    modifier_map Mod3 { <CAPS> };
};

Again, this does:

• F1 -> F21
• Caps + F1 -> F1
• Shift + F1 -> Shift + F1 (unchanged)
• Alt + F1 -> Alt + F1 (unchanged)
• etc. - all others unchanged

I found a way to turn Caps Lock into Shift Lock, thanks to a Super User answer:

xmodmap -e "keycode 66 = Shift_Lock"

The following turns my left Ctrl key into shift lock:

xmodmap -e "keycode 37 = Shift_Lock"

To make the left Shift key lock, use

xmodmap -e "keycode 50 = Shift_Lock"

Right Shift key, use

xmodmap -e "keycode 62 = Shift_Lock"

When Shift Lock is active, your keyboard's Caps Lock and Scroll Lock LEDs should both be lit.