Multiline History Entry in Bash – When is Lithist Possible?

bashcommand historyshopt

lithist If enabled, and the cmdhist option is enabled, multi-line
commands are saved to the history with embedded newlines rather than
using semicolon separators where possible.

with this hence my question: where/when/how is this possible?

I tried to enable and test the feature on my GNU bash, version 4.4.12(1)-release doing this:

shopts -s cmdhist
shopts -s lithist
echo -n "this is a test for a ";\
echo "multiline command"

I then did a

history | tail

expecting some ouput akin to this :

101 shopts -s cmdlist
102 shopts -s lithist
103 echo -n "this is a test for a ";\
    echo "multiline command"
104 history

yet instead get this:

101 shopts -s cmdlist
102 shopts -s lithist
103 echo -n "this is a test for a "; echo "multiline command"
104 history

As is obvious the multiline command (the one with bash history number 103) has not been stored with "embedded newlines rathar than using semicolon separators". Why was lithist not possible here? What did I do wrong?

Best Answer

A \<new line> is not the correct way to get a <new line> in the history.

Memory

Lets deal only with history lines as they are kept in shell memory (not disk).

Lets type a couple of commands as you did:

$ echo -n "this is a test for a ";\
> echo "two line command"

What was stored in memory as the line just written?

$ history 2
514  echo -n "this is a test for a ";echo "two line command"
515  history 2

As you can see, the "line continuation", a backslash followed by a newline, was removed.
As it should (from man bash):

If a \ pair appears, and the backslash is not itself quoted, the \ is treated as a line continuation (that is, it is removed from the input stream and effectively ignored).

We may get a newline if we quote it:

$ echo " A test of 
> a new line"
A test of 
a new line

And, at this point, the history will reflect that:

$ history 2
  518  echo "A test of
a new line"
  519  history 2

A true multi-line command:

One possible example of a multi-line command is:

$ for a in one two
> do echo "test $a"
> done
test one
test two

Which will be collapsed into one history line if cmdhist is set:

$ shopt -p cmdhist lithist
shopt -s cmdhist                                                                                                       
shopt -u lithist

$ history 3
   24  for a in one two; do echo "test $a"; done                                                                       
   25  shopt -p cmdhist lithist                                                                                        
   26  history 3

The numbers for each command changed because at some point I cleared the history (in memory) with a history -c.

If you unset the cmdhist, you will get this:

$ shopt -u cmdhist
$ for a in one two
> do echo "test $a"
> done
test one
test two

$ history 5
    5  shopt -u cmdhist
    6  for a in one two
    7  do echo "test $a"
    8  done
    9  history 5

Each line (not a full command) will be at a separate line in the history.

Even if the lithist is set:

$ shopt -s lithist
$ for a in one two
> do echo "test $a"
> done
test one
test two
$ history 5
   12  shopt -s lithist
   13  for a in one two
   14  do echo "test $a"
   15  done
   16  history 5

But if both are set:

$ shopt -s cmdhist lithist
$ for a in one two
> do echo "test $a"
> done

$ history 5
   23  history 15
   24  shopt -p cmdhist lithist
   25  shopt -s cmdhist lithist
   26  for a in one two
do echo "test $a"
done
   27  history 5

The for command was stored as a multiline command with "newlines" instead of semicolons (;). Compare with above where lithist wasn't set.

Disk

All the above was explained using the list of commands kept in the memory of the shell. No commands were written to the disk. The (default) file ~/.bash_history was not changed.

That file will be changed when the running shell exits. At that point in time the history will overwrite the file (if histappend isn't set), or will be appended otherwise.

If you want the commands to be committed to disk you need to have this set:

export PROMPT_COMMAND='history -a'

That will make each command line to be appended to file on each new command line.

Now, lets get down to business with cmdhist and lithist. It is not so simple as it may seem. But don't worry, all will be clear in a moment.

Let's say that you take the time to type all the commands below (there is no shortcut, no alias, no function, you need the actual commands, sorry).
To first clear all history in memory (history -c) and in disk (make a backup) (history -w) and then to try three times:

With the default values of cmdhist (set) and lithist (unset).
With both set
With both un-set
Setting lithist with an unset cmdhist makes no sense (you can test it).

List of commands to execute:

$ history -c ; history -w           # Clear all history ( Please backup).

$ shopt -s cmdhist; shopt -u lithist
$ for a in one two
> do echo "test $a"
> done

$ shopt -s cmdhist; shopt -s lithist
$ for a in one two
> do echo "test $a"
> done

$ shopt -u cmdhist; shopt -u lithist
$ for a in one two
> do echo "test $a"
> done

You will end with this (in memory):

$ history
    1  shopt -s cmdhist; shopt -u lithist
    2  for a in one two; do echo "test $a"; done
    3  shopt -s cmdhist; shopt -s lithist
    4  for a in one two
do echo "test $a"
done
    5  shopt -u cmdhist; shopt -u lithist
    6  for a in one two
    7  do echo "test $a"
    8  done
    9  history

The three multiline commands end as follows:

one in line numbered 2 (one single line, one command).
one in a multiline numbered 4 (one command in several lines)
one in several lines numbered from 6 to 8

Ok, but what happen in file? say it alredy ....

finally, in file:

Simple, write to file and cat it to see this:

$ history -w ; cat "$HISTFILE"
shopt -s cmdhist; shopt -u lithist
for a in one two; do echo "test $a"; done
shopt -s cmdhist; shopt -s lithist
for a in one two
do echo "test $a"
done
shopt -u cmdhist; shopt -u lithist
for a in one two
do echo "test $a"
done
history
history -w ; cat "$HISTFILE"

No line numbers, only commands, there is no way to tell where a multiline starts and where it ends. There is no way to tell even if there is a multiline.

In fact, that is exactly what happens, if the commands are written to file as above, when the file is read back, any information about multilines gets lost.

There is only one delimiter (the newline), each lines is read back as one command.

Is there a solution to this, yes, to use an additional delimter.
The HISTTIMEFORMAT kind of does that.

HISTTIMEFORMAT

When this variable is set to some value, the time at which each command was executed gets stored in file as the seconds since epoch (yes, always seconds) after a comment (#) character.

If we set the variable and re-write the ~/.bash_history file, we get this:

$ HISTTIMEFORMAT='%F'
$ history -w ; cat "$HISTFILE"
#1490321397
shopt -s cmdhist; shopt -u lithist
#1490321397
for a in one two; do echo "test $a"; done
#1490321406
shopt -s cmdhist; shopt -s lithist
#1490321406
for a in one two
do echo "test $a"
done
#1490321418
shopt -u cmdhist; shopt -u lithist
#1490321418
for a in one two
#1490321418
do echo "test $a"
#1490321420
done
#1490321429
history
#1490321439
history -w ; cat "$HISTFILE"
#1490321530
HISTTIMEFORMAT='%FT%T '
#1490321571
history -w ; cat "$HISTFILE"

Now you can tell where and which line is a multiline.

The format '%FT%T ' shows the time but only when using the history command:

$ history
    1  2017-03-23T22:09:57 shopt -s cmdhist; shopt -u lithist
    2  2017-03-23T22:09:57 for a in one two; do echo "test $a"; done
    3  2017-03-23T22:10:06 shopt -s cmdhist; shopt -s lithist
    4  2017-03-23T22:10:06 for a in one two
do echo "test $a"
done
    5  2017-03-23T22:10:18 shopt -u cmdhist; shopt -u lithist
    6  2017-03-23T22:10:18 for a in one two
    7  2017-03-23T22:10:18 do echo "test $a"
    8  2017-03-23T22:10:20 done
    9  2017-03-23T22:10:29 history
   10  2017-03-23T22:10:39 history -w ; cat "$HISTFILE"
   11  2017-03-23T22:12:10 HISTTIMEFORMAT='%F'
   12  2017-03-23T22:12:51 history -w ; cat "$HISTFILE"
   13  2017-03-23T22:15:30 history
   14  2017-03-23T22:16:29 HISTTIMEFORMAT='%FT%T'
   15  2017-03-23T22:16:31 history
   16  2017-03-23T22:16:35 HISTTIMEFORMAT='%FT%T '
   17  2017-03-23T22:16:37 history

Related Solutions

Bash – How to stop bash editing the history when I reuse and modify an entry

Turns out revert-all-at-newline is the answer. I needed to include set revert-all-at-newline on in my ~/.inputrc file, since using the set command at the bash prompt had no effect. (Then, of course, I had to start a new shell.)

Also, I found that ~/.inputrc is loaded instead of /etc/inputrc if present, which means that any defaults defined in the latter are no longer active when you create ~/.inputrc. To fix this, start ~/.inputrc with $include /etc/inputrc.

Thanks to @StéphaneChazelas for pointing me in the right direction.

Bash – does history not overwrite but append when histappend is set to off in Bash

@David Dai is correct about this bash-help thread.

Here is the relevant bash code in C.

if (history_lines_this_session <= where_history () || force_append_history)
  {
    result = append_history (history_lines_this_session, hf);
    history_lines_in_file += history_lines_this_session;
  }
else
  {
    result = write_history (hf);
    history_lines_in_file = history_lines_this_session;
  }

shopt -s histappend sets force_append_history to non-zero (true).

history_lines_this_session is the number of commands you typed this interactive bash session. When you first start bash, it is 0. It increases by 1 each time you Enter a command that is saved to history.

where_history() is an index into the in-memory history list, starting at index 0. It points to the next index to write history to. For example, when you have 0 history, it will be index 0. When you have 6 lines of history, it will be index 6.

When you first start bash and you have history in your HISTFILE, where_history() increase but will be limited to <= HISTSIZE. So where_history() has a maximum value of HISTSIZE.

After reading the HISTFILE, where_history() will increase but history_lines_this_session will still be 0. When you Enter HISTSIZE + 1 number of commands, history_lines_this_session == HISTSIZE + 1 and where_history() == HISTSIZE (because it is capped at the maximum). At this point, (history_lines_this_session <= where_history()) == false so history will be overwritten if histappend is off.

You can test this out in your bash yourself. If histappend is off, once you run HISTSIZE + 1 number of commands, history_lines_this_session will exceed where_history() by 1 and HISTFILE will be overwritten. You have to make sure other commands aren't messing with your history for the test to work though (happens on macOS). Use the following commands to turn off commands that may mess with history:

trap - EXIT
PROMPT_COMMAND=
SHELL_SESSION_HISTORY=0 # for macOS only