If I were in a similar situation, I would do the following:

• Change the default keyboard shortcut for Emoji & Symbols from ⌃⌘Space to ⌃⌥⌘Space so that the default ⌃⌘Space can be used as a shortcut with one of the scripts, the primary one.

• Create two AppleScript scripts, one for showing Emoji & Symbols, (the primary one for me), and the other for showing Character Viewer. (This could be done in one script however, for coding simplicity two will be used.)

• Use FastScripts to assign a keyboard shortcut to each. The first for My Emoji & Symbols to ⌃⌘Space and for My Character Viewer, use ⌃⇧⌘Space as shown in the image further below.
  
  ^{Note that I have no affiliation with Red Sweater Software, the developer of FastScripts, and am just a satisfied user.}

  • Note that in place of using FastScripts, an Automator Service can be used. However, I prefer the former because it has worked better for me then the latter.

The following was tested and worked under macOS High Sierra...

First, change the global keyboard shortcut for the Emoji & Symbols menu item:

• System Preferences > Keyboard > Shortcuts > App Shortcuts

  • Click the [+] to add a new shortcut with the following settings:

  • Applications: All Applications

  • Menu Title: Emoji & Symbols
  • Keyboard Shortcut: ⌃⌥⌘Space
            _{See image further below.}

Next, create two AppleScript scripts (.scpt) files in ~/Library/Scripts/, one named My Emoji & Symbols and the other named My Character Viewer, adding the following to each:

For My Emoji & Symbols:

do shell script "defaults write com.apple.CharacterPaletteIM CVStartAsLargeWindow -bool false"
delay 0.5
tell application "System Events" to keystroke space using {control down, option down, command down}

For My Character Viewer:

do shell script "defaults write com.apple.CharacterPaletteIM CVStartAsLargeWindow -bool true"
delay 0.5
tell application "System Events" to keystroke space using {control down, option down, command down}

• Note the only difference between the two is the value of -bool, being true or false for the CVStartAsLargeWindow key. Also, the value of the delay command is to allow time for the defaults command to make the write before System Events triggers the new global keyboard shortcut for the Emoji & Symbols menu item. The value of the delay command may need be adjusted, so do so as needed.

Then in FastScripts > Preferences... > Script Shortcuts, assign ⌃⌘Space (or ⌃⌘␣) to My Emoji & Symbols, and assign ⌃⇧⌘Space (or ⌃⇧⌘␣) to My Character Viewer.   _{See image below.}

• Note that FastScripts will need to be added to: System Preferences > Security & Privacy > Privacy > Accessibility   _{See image below.}

If using Automator Services, use a Run Apple Script action with the example AppleScript code, show above, replacing the default code in each and assign keyboard shortcuts via: System Preferences > Keyboard > Shortcuts > Services

• _{Note that the mentioned naming conventions and or paths, et cetera, have changed for some items in macOS Mojave and will need to be adjusted accordingly. I am not yet running macOS Mojave however, have seen posts on some of the differences. You'll need to adjust as necessary if running macOS Mojave.}

Key Symbol Legend:

• Shift: ⇧
• Control: ⌃
• Option: ⌥
• Command: ⌘
• Space: Space

System Preferences > Keyboard > Shortcuts > App Shortcuts

FastScripts > Preferences... > Script Shortcuts
     _{• Note that the ␣ character, shown in the shortcut below, is for the space character.}

System Preferences > Security & Privacy > Privacy > Accessibility