How to diff files ignoring comments (lines starting with #)

First, you need to protect the pattern from expansion by the shell. The easiest way to do that is to put single quotes around it. Single quotes prevent expansion of anything between them (including backslashes); the only thing you can't do then is have single quotes in the pattern.

grep -- 'foo*' *.txt

(also note the -- end-of-option-marker to stop some grep implementations including GNU grep from treating a file called -foo-.txt for instance (that would be expanded by the shell from *.txt) to be taken as an option (even though it follows a non-option argument here)).

If you do need a single quote, you can write it as '\'' (end string literal, literal quote, open string literal).

grep -- 'foo*'\''bar' *.txt

Second, grep supports at least¹ two syntaxes for patterns. The old, default syntax (basic regular expressions) doesn't support the alternation (|) operator, though some versions have it as an extension, but written with a backslash.

grep -- 'foo\|bar' *.txt

The portable way is to use the newer syntax, extended regular expressions. You need to pass the -E option to grep to select it (formerly that was done with the egrep separate command²)

grep -E -- 'foo|bar' *.txt

Another possibility when you're just looking for any of several patterns (as opposed to building a complex pattern using disjunction) is to pass multiple patterns to grep. You can do this by preceding each pattern with the -e option.

grep -e foo -e bar -- *.txt

Or put patterns on several lines:

grep -- 'foo
bar' *.txt

Or store those patterns in a file, one per line and run

grep -f that-file -- *.txt

Note that if *.txt expands to a single file, grep won't prefix matching lines with its name like it does when there are more than one file. To work around that, with some grep implementations like GNU grep, you can use the -H option, or with any implementation, you can pass /dev/null as an extra argument.

^{¹ some grep implementations support even more like perl-compatible ones with -P, or augmented ones with -X, -K for ksh wildcards...}

^{² while egrep has been deprecated by POSIX and is sometimes no longer found on some systems, on some other systems like Solaris when the POSIX or GNU utilities have not been installed, then egrep is your only option as its /bin/grep supports none of -e, -f, -E, \| or multi-line patterns}

As Gilles says, you'll need to use a tool that knows about the format of your config files. For your particular example, you can use a short Python script that uses the built-in ConfigParser module.

• First, let's say your original configuration file on the server is original.cfg:

  [config]
  ip=127.0.0.1
  port=22
  

• Now put your changes in a new file called update.cfg:

  [config]
  user=root
  

  This file should have new or changed entries listed under the section heading where you would like them to go.

• Then run a script like this one:

  #!/usr/bin/env python
  
  import ConfigParser
  
  config = ConfigParser.ConfigParser()
  
  # Read the original config file
  with open('original.cfg', 'r') as f:
      config.readfp(f)
  
  # Also read in all the changes we'd like to make
  with open('update.cfg', 'r') as f:
      config.readfp(f)
  
  # Write the full new config file out
  with open('output.cfg', 'w') as f:
      config.write(f)
  

There are, of course, plenty of variations on how to do this. For example, the config changes could be coded directly into the Python script rather than read from a separate update.cfg file. This should give you a good base to get started, though.