Bash – Reading and Processing Strings Character by Character

bashterminaltext processing

I want to incrementally read a line of input from the terminal,
and allow the user some basic line editing functionality; INS, DEL, RIGHT, LEFT HOME, END, BACKSPACE

Each time the string is modified, I want to process it, to do an incremental regex search of a text file.

These edit-keys, and others, generate multiple input characters which
make it rather difficult to interpret the input, eg C-Left generates 6 characters.

Is there a simple way to achieve this char-by-char editable input?
I'm especialy interested in knowing how to do this in bash, because the rest of the processing will be bash..
Other suggestions are welcome too..

Here is how I started out, but it gets a bit out of hand with such a potential variety of control codes ..

#!/bin/bash
IFS=$'\n' 
while true ;do
  read -n 1 c
  ((${#c}==0)) && break # Exit the loop. Input length is 0  
                        # ie. The user has pressed Enter
  echo "xx=$(echo -n "$c"|xxd -p)="
  # 1b 5b 32 7e  "INS"
  # 1b 5b 33 7e  "DEL"
  # 1b 5b 43     "RIGHT"
  # 1b 5b 44     "LEFT"
  # 1b 5b 46     "END"
  # 1b 5b 48     "HOME"
  # 7f           "BACKSPACE"
done

Best Answer

If you read a character at a time with read -n, you'll have to implement a key sequence parser. You can build a slow-and-dirty solution that works on most terminals with this: consider that a function key escape sequence begins with an escape character and continues with any number of characters amongst 0-9;[]O followed by one final character not in this set.

A better way to read input is to use a proper input library. Bash uses one for its own purposes (readline). You get a limited interface to it by declaring your own keybindings with the bind built-in; specifically bind -x to run a shell command on a key press. Because of this limited interface, implementing what you want is likely to be possible but difficult.

Zsh has its own input library, zle. Its interface is a lot richer than bash's. With zle, you can define arbitrary keymaps, and you get more access to the internals of zle from shell code. Use zle to assign shell functions to zle user-defined commands (called widgets), bindkey to create and populate your own keymap, and finally vared to read a line of input using the keymap of your choice.

Related Solutions

Bash – Undo tab expansion

You're giving the keystrokes, but not the readline bindings, this can explain differences in observed behaviour (try bind -p).

The only way I can think of to do exactly what you ask is to use a custom completion function that saves the buffer state, and a separate key binding which invokes a companion shell function to restore it.

In addition to @Mark Plotnick's excellent suggestion to use menu-complete, other readline/bash options are:

shell-backward-kill-word this removes a "shell word" which in the case of an expanded file or path will be the complete file or path name
backward-kill-word this removes a "readline word", which in the case of a file or path will be a file or directory name component, including the trailing /
undo which undoes each change atomically, but this almost certainly requires several attempts to fully remove a complete expansion, this is almost certainly the function you have bound to to ctrl-_ (bash default)
set-mark and kill-region are lesser used readline features, set-mark marks the current cursor position, and kill-region deletes from the mark to the cursor. This requires you to invoke set-mark (default meta-space) in advance so that you can later use kill-region to delete back to it (no default binding)
possible-completions (bound to meta-=) which lists the completions without modifying the command buffer

You could also write your own function to modify the input buffer, and bind it to a function (here's one I made earlier) but I don't see how that would be very much better than backward-kill-word in this case.

You can test bindings on the fly without using .inputrc using bash's bind command, e.g.:

bind "\C-o: backward-kill-word"
bind "\C-o: shell-backward-kill-word"

backward-kill-word is the best fit, default binding is metactrl-h. set-mark/kill-region would be better, but I can't think of a neat way to automate the use of this though.

Bash – Read User Input Line by Line Until Ctrl+D

To do that, you'd have to read character by character, not line by line.

Why? The shell very likely uses the standard C library function read() to read the data that the user is typing in, and that function returns the number of bytes actually read. If it returns zero, that means it has encountered EOF (see the read(2) manual; man 2 read). Note that EOF isn't a character but a condition, i.e. the condition "there is nothing more to be read", end-of-file.

Ctrl+D sends an end-of-transmission character (EOT, ASCII character code 4, $'\04' in bash) to the terminal driver. This has the effect of sending whatever there is to send to the waiting read() call of the shell.

When you press Ctrl+D halfway through entering the text on a line, whatever you have typed so far is sent to the shell¹. This means that if you enter Ctrl+D twice after having typed something on a line, the first one will send some data, and the second one will send nothing, and the read() call will return zero and the shell interpret that as EOF. Likewise, if you press Enter followed by Ctrl+D, the shell gets EOF at once as there wasn't any data to send.

So how to avoid having to type Ctrl+D twice?

As I said, read single characters. When you use the read shell built-in command, it probably has an input buffer and asks read() to read a maximum of that many characters from the input stream (maybe 16 kb or so). This means that the shell will get a bunch of 16 kb chunks of input, followed by a chunk that may be less than 16 kb, followed by zero bytes (EOF). Once encountering the end of input (or a newline, or a specified delimiter), control is returned to the script.

If you use read -n 1 to read a single character, the shell will use a buffer of a single byte in its call to read(), i.e. it will sit in a tight loop reading character by character, returning control to the shell script after each one.

The only issue with read -n is that it sets the terminal to "raw mode", which means that characters are sent as they are without any interpretation. For example, if you press Ctrl+D, you'll get a literal EOT character in your string. So we have to check for that. This also has the side-effect that the user will be unable to edit the line before submitting it to the script, for example by pressing Backspace, or by using Ctrl+W (to delete the previous word) or Ctrl+U (to delete to the beginning of the line).

To make a long story short: The following is the final loop that your bash script needs to do to read a line of input, while at the same time allowing the user to interrupt the input at any time by pressing Ctrl+D:

while true; do
    line=''

    while IFS= read -r -N 1 ch; do
        case "$ch" in
            $'\04') got_eot=1   ;&
            $'\n')  break       ;;
            *)      line="$line$ch" ;;
        esac
    done

    printf 'line: "%s"\n' "$line"

    if (( got_eot )); then
        break
    fi
done

Without going into too much detail about this:

IFS= clears the IFS variable. Without this, we would not be able to read spaces. I use read -N instead of read -n, otherwise we wouldn't be able to detect newlines. The -r option to read enables us to read backslashes properly.
The case statement acts on each read character ($ch). If an EOT ($'\04') is detected, it sets got_eot to 1 and then falls through to the break statement which gets it out of the inner loop. If a newline ($'\n') is detected, it just breaks out of the inner loop. Otherwise it adds the character to the end of the line variable.
After the loop, the line is printed to standard output. This would be where you call your script or function that uses "$line". If we got here by detecting an EOT, we exit the outermost loop.

¹ You may test this by running cat >file in one terminal and tail -f file in another, and then enter a partial line into the cat and press Ctrl+D to see what happens in the output of tail.

For ksh93 users: The loop above will read a carriage return character rather than a newline character in ksh93, which means that the test for $'\n' will need to change to a test for $'\r'. The shell will also display these as ^M.

To work around this:

stty_saved="$( stty -g )"
stty -echoctl

# the loop goes here, with $'\n' replaced by $'\r'

stty "$stty_saved"

You might also want to output a newline explicitly just before the break to get exactly the same behaviour as in bash.

Best Answer

Related Solutions

Bash – Undo tab expansion

Bash – Read User Input Line by Line Until Ctrl+D

Related Question