Bash – Double variable substitution in bash

The \ must be used to prevent the expansion of $$ (current process id). For triple substitution, you need double eval, so also more escapes to avoid the unwanted expansions in each eval:

#! /bin/bash
l0=value
l1=l0
l2=l1
l3=l2
l4=l3

echo $l0
eval echo \$$l1
eval eval echo \\$\$$l2
eval eval eval echo \\\\$\\$\$$l3
eval eval eval eval echo \\\\\\\\$\\\\$\\$\$$l4

When you run a command through ssh, it is run by calling your $SHELL with the -c flag:

-c    If the -c option is present, then commands are read from 
      the first non-option argument command_string.  If there  are
      arguments  after the command_string, the first argument is 
      assigned to $0 and any remaining arguments are assigned to
      the positional parameters.  

So, ssh remote_host "bash -c foo" will actually run:

/bin/your_shell -c 'bash -c foo'

Now, because the command you are running (export foo=bar) contains spaces and is not properly quoted to form a whole, the export is taken as the command to be run and the rest are saved in the positional parameters array. This means that export is run and foo=bar is passed to it as $0. The final result is the same as running

/bin/your_shell -c 'bash -c export'

The correct command would be:

ssh remote_host "bash -c 'export foo=bar'"