Bash – How to See the Exact Command Line Being Executed

bashdebuggingprocess

I have a long running bash instance (inside a screen session) that is executing a complex set of commands inside a loop (with each loop doing pipes, redirects, etc).

The long command line was written inside the terminal – it's not inside any script. Now, I know the bash process ID, and I have root access – how can I see the exact command line being executed inside that bash?

Example

bash$ echo $$
1234
bash$ while true ; do \
    someThing | somethingElse 2>/foo/bar | \
    yetAnother ; sleep 600 ; done

And in another shell instance, I want to see the command line executed inside PID 1234:

bash$ echo $$
5678
bash$ su -
sh# cd /proc/1234
sh# # Do something here that will display the string  \
   'while true ; do someThing | somethingElse 2>/foo/bar | \
    yetAnother ; sleep 600 ; done'

Is this possible?

EDIT #1

Adding counter-examples for some answers I've got.

About using the cmdline under /proc/PID: that doesn't work, at least not in my scenario. Here's a simple example:
```
$ echo $$
8909

$ while true ; do echo 1 ; echo 2>/dev/null ; sleep 30 ; done
```
In another shell:
```
$ cat /proc/8909/cmdline
bash
```
Using ps -p PID --noheaders -o cmd is just as useless:
```
$ ps -p 8909 --no-headers -o cmd
bash
```

ps -eaf is also not helpful:

$ ps -eaf | grep 8909
ttsiod    8909  8905  0 10:09 pts/0    00:00:00 bash
ttsiod   30697  8909  0 10:22 pts/0    00:00:00 sleep 30
ttsiod   31292 13928  0 10:23 pts/12   00:00:00 grep --color=auto 8909

That is, there's no output of the ORIGINAL command line, which is what I'm looking for – i.e the while true ; do echo 1 ; echo 2>/dev/null ; sleep 30 ; done.

Best Answer

I knew I was grasping at straws, but UNIX never fails!

Here's how I managed it:

bash$ gdb --pid 8909
...
Loaded symbols for /lib/i386-linux-gnu/i686/cmov/libnss_files.so.2
0xb76e7424 in __kernel_vsyscall ()

Then at the (gdb) prompt I ran the command, call write_history("/tmp/foo") which will write this history to the file /tmp/foo.

(gdb) call write_history("/tmp/foo")
$1 = 0

I then detach from the process.

(gdb) detach
Detaching from program: /bin/bash, process 8909

And quit gdb.

(gdb) q

And sure enough...

bash$ tail -1 /tmp/foo
while true ; do echo 1 ; echo 2>/dev/null ; sleep 30 ; done

For easy future re-use, I wrote a bash script, automating the process.

Related Solutions

Bash – Prevent a command from being executed after another command

You're asking for a lot with a general answer, but it should be possible to do it relatively easily to some extent.

However, bash has at least two general mechanisms that can help solve that. You can set the variable PROMPT_COMMAND and this will be executed after anything you run in your prompt:

$ PROMPT_COMMAND='echo BOFH says hi!'
BOFH says hi!
$ man man
BOFH says hi!
$

We can use this to save the last entered command and then reread it, so we can store it in memory for further use (history expansion does not work in this case):

$ PROMPT_COMMAND='history -a; tail -n1 ~/.bash_history'
PROMPT_COMMAND='history -a; tail -n1 ~/.bash_history'
$ wc < test.sh | grep .
 5 11 63
wc < test.sh | grep .

Now comes the logic part, which is completely up to you. Let's use just very simplistic logic to prevent the example in the question. The thing to remember is, that bash functions still work in there, so you don't have to cram everything on a single line, in a single variable.

$ # define the logic check
$ checkSanity() { grep -q "tar  *[tf][[:alpha:]]*[tf] " <<< "$@" && echo "be careful, don't delete this"; }
checkSanity() { grep -q "tar  *[tf][[:alpha:]]*[tf] " <<< "$@" && echo "be careful, don't delete this"; }
$ # feed it the last command
$ PROMPT_COMMAND='history -a; last=$(tail -n1 ~/.bash_history); checkSanity "$last"'
$ # simple test
$ tar tf test.sh 2>/dev/null 
be careful, don't delete this
$ tar xf test.sh 2>/dev/null 
$

Of course, this approach is useful to prevent PEBKAC (especially if you add sleep 3 at the end), but it can't break the next command on its own.

If that's really what you want, trap the DEBUG signal (eg. trap 'echo "I am not deterministic, haha"' DEBUG), since it runs beforehand. Be careful with combining these two approaches, since the output/action will be doubled:

$ df
/.../
$ trap 'tail -n1 ~/.bash_history' DEBUG
df
df
trap 'tail -n1 ~/.bash_history' DEBUG
trap 'tail -n1 ~/.bash_history' DEBUG

To make the trap break a command, you'll have to enable extdebug (shopt -s extdebug). You also don't need to save and reread history constantly, but can inspect $BASH_COMMAND to get the command about to be run. Then you just need to make sure the logic checker return 1 when it detects something bad and 0 otherwise.

          extdebug
                  If set, behavior intended for use by debuggers is enabled:
    /.../
                  2.     If  the command run by the DEBUG trap returns a non-zero value, the next
                         command is skipped and not executed.


$ checkSanity() { if grep -q "tar  *[tf][[:alpha:]]*[tf] " <<< "$1"; then  echo "be careful, don't delete this"; return 1; fi; }
$ trap 'checkSanity "$BASH_COMMAND"' DEBUG
$ # simple test
$ tar tf test.sh 2>/dev/null 
be careful, don't delete this
$ tar xf test.sh 2>/dev/null 
$

linux cron – Email Only Occasionally Sent on Output and Errors

Upon further testing, I suspect the & is messing with your results. As you point out, &>/dev/null is bash syntax, not sh syntax. As a result, sh is creating a subshell and backgrounding it. Sure, the subshell's echo creates stderr, but my theory is that:

cron is not catching the subshell's stderr, and
the backgrounding of the subshell always completes successfully, thus bypassing your || echo ....

... causing the cron job to have no output and thus no mail. Based on my reading of the vixie-cron source, it would seem that the job's stderr and stdout would be captured by cron, but it must be getting lost by the subshell.

Test it yourself in a /bin/sh environment (assuming you do not have a file named 'bar' here):

(grep foo bar) &
echo $?

EDIT #1

Best Answer

Related Solutions

Bash – Prevent a command from being executed after another command

linux cron – Email Only Occasionally Sent on Output and Errors

Related Question