Sed to remove all whitespace from a line

sedwhitespace

I want to use sed to remove all space characters from a text file. At present I am using this command:

sed 's/ //' test.txt > test2.txt

This works in the sense that it removes the first space character of each line but leaves the rest.

Is there a way to tell sed to repeat for a space characters on a line before moving on to the next or is it a case of scripting multiple run thoughs on the same file?

At present the file is small for testing but soon this will be ramped up on to much larger files so minimum rinse and repeat would be great. Any thoughts?

At present I get this using the command above:

bruteforce  
allthe kings men  
moneymoney money  
ifi was a rich girl

When what I would like is this:

bruteforce  
allthekingsmen  
moneymoneymoney  
ifiwasarichgirl

Best Answer

You should use the g (global) flag, which makes sed act on all matches of the pattern on each line, rather than the first match on each line:

sed 's/ //g' test.txt > test2.txt

Related Solutions

How to match whitespace in sed

The character class \s will match the whitespace characters <tab> and <space>.

For example:

$ sed -e "s/\s\{3,\}/  /g" inputFile

will substitute every sequence of at least 3 whitespaces with two spaces.

REMARK: For POSIX compliance, use the character class [[:space:]] instead of \s, since the latter is a GNU sed extension. See the POSIX specifications for sed and BREs

Fill the last line up with nothing but emptiness to a specific position

Try this:

sed -i -e 's/.\{80\}/&\r\n/g' \
  -e 's/$/################################################################################/' \
  -e 's/\(\r\n.\{80\}\)#*$/\1/' "C:\Folder\test.txt"

(I have broken the command into multiple lines just for readability in presentation; you can do it all on one line.) The s/$/###…###/ appends 80 # characters, and then the s/$\r\n.\{80\}$#*$/\1/ looks for this sequence:

\r\n
80 of any character
any number of #s
up to the end of the buffer

and replaces it with the group formed by the first two bullets. In other words, it keeps the n characters that are the last line of the file and the 80-n # characters that follow it, and discards the n # characters that follow those.

Once you've verified that this works, just change all occurrences of # in the above command to spaces.

OP's Final Working Solution

^{NOTE: The SET s10= line has 10 empty spaces in it to the right of the equal sign.}

Batch Script

@ECHO ON

SET File=C:\Folder\test.txt
::: The below s10 variable equals ten white\empty spaces - so it has 10 blank spaces in it
SET s10=          
SET s40=%s10%%s10%%s10%%s10%
SET s80=%s40%%s40%

sed -i -e "s/.\{80\}/&\r\n/g" -e "s/$/%s80%/" -e "s/\(\r\n.\{80\}\)\s*$/\1/" "%File%"
GOTO :EOF

We can make this a little easier to read and less tedious to type.

Pure sed

If your text is all alphanumeric, and there is some character (such as @) that will never ever appear in your data, then you can do something like this:
```
  -e 's/$/@@@@@@@@@@/' \        # Append 10 @ characters to the line.
  -e 's/@/        /g' \         # Change each @ to eight spaces.
```
If you want to make your script unambiguously readable, at the cost of making it somewhat more verbose, take it one step further:
```
  -e 's/$/@@@@@@@@@@/' \        # Append 10 @ characters to the line.
  -e 's/@/@@@@@@@@/g' \         # Change each @ to eight @s (there are now 80 @s).
  -e 's/@/ /g'        \         # Change each of the 80 @s to a space.
```
If there's no single character that never appears in your data, but there is some (short) character sequence, like (), that never appears, you can use the same principle:
```
  -e 's/$/()()()()()()()()()()/' \  # Append 10 () pairs to the line.
  -e 's/()/        /g' \            # Change each () to eight spaces.
```
Be careful when dealing with characters that are special in regular expressions; e.g., ., *, [, ^, &, etc. (And don't include the #-comments in your script in those places; to break a simple command across multiple lines, the \ must be the last character on the line (before the newline) and not in a comment.)

Using the shell

Same principle:

#!/bin/sh
      ︙
s="          "                  # Set $s to 10 spaces.
s80="$s$s$s$s$s$s$s$s"          # Set $s80 to 8 copies of $s; i.e., 80 spaces.
      ︙
sed -i -e 's/.\{80\}/&\r\n/g' \
  -e 's/$/'"$s80"'/' \
  -e 's/\(\r\n.\{80\}\)\s*$/\1/' "C:\Folder\test.txt"