Bash Error – Cannot Execute Binary File

bashexecutableshell-script

I have a script that works well when I ssh to the server to execute it myself, but has problems when Hudson, a continuous integration server, runs it.

I am automating tests on an embedded linux system (the target). The target is connected to Server A (RHEL 5) via serial and operated over minicom. Server B (FC 12) builds the tests that actually run on the target, and can ssh to Server A. Server C (RH) hosts Hudson, with Server B as a slave.

I've written a runscript (http://linux.die.net/man/1/runscript) script to do everything needed on the actual target; it boots the image, mounts a directory from Server B and executes the tests. A bash script on Server B invokes minicom with the runscript script along with some companion actions. I have a bash script on Server B which uses

ssh -t -t ServerA bashScript.sh

to get those tests run on the target. I am on Server C, I can get those tests run by ssh'ing to Server B and executing the script that ssh's to Server A which executes minicom with runscript. Whew. To review:

Server A: Hudson uses its slave mechanism to ssh to Server B.

Server B: kickOffTests.sh has the line ssh -t -t ServerA runTests.sh

Server A: runTests.sh calls a perl script which invokes minicom -S my.script ttyE1

Target, after booting: Mounts a directory from Server B, where the tests are, and enters that directory. It invokes yet another bash script, which runs the tests, which are compiled C executables.

Now, when I execute any of these scripts myself, they do what they should. However, when Hudson tries to do the same thing, over in the minicom session it complains about a line in the "yet another bash script" that invokes the C executable, ./executable, with ./executable: cannot execute binary file

I still have a lot to learn about linux, but I surmise this problem is a result of Hudson not connecting with a console. I don't know exactly what Hudson does to control its slave. I tried using the line export TERM=console in the configuration just before running kickOffTests.sh, but the problem remains.

Can anyone explain to me what is happening and how I can fix it? I cannot remove any of the servers from this equation. It may be possible to take minicom out of the equation but that would add an unknown amount of time to this project, so I'd much prefer a solution that uses what I already have.

Best Answer

The message cannot execute binary file has nothing to do with terminals (I wonder what led you to think that — and I recommend avoiding making such assumptions in a question, as they tend to drown your actual problem in a mess of red herrings). In fact, it's bash's way of expressing ENOEXEC (more commonly expressed as exec format error.

First, make sure you didn't accidentally try to run this executable as a script. If you wrote . ./executable, this tells bash to execute ./executable in the same environment as the calling script (as opposed to a separate process). That can't be done if the file is not a script.

Otherwise, this message means that ./executable is not in a format that the kernel recognizes. I don't have any definite guess as to what is happening though. If you can run the script on that same machine by invoking it in a different way, it can't just be a corrupt file or a file for the wrong architecture (it might be that, but there's more to it). I wonder if there could be a difference in the way the target boots (perhaps a race condition).

Here's a list of additional data that may help:

Output of file …/executable on server B.
Some information about the target, such as the output of uname -a if it's unix-like.
Check that the target sees the same file contents each time: run cksum ./executable or md5sum ./executable or whatever method you have on the target just before yet-another-bash-script invokes ./executable. Check that the results are the same in the Hudson invocation, in your successful manual invocation and on server B.
Add set -x at the top of yet-another-bash-script (just below the #!/bin/bash line). This will produce a trace of everything the script does. Compare the traces and report any difference or oddity.
Describe how the target is booting when you run the scripts manually and when Hudson is involved. It could be that the target is booted differently and some loadable module that provides the support for the format of ./executable doesn't get loaded (or is not loaded yet) in the Hudson invocations. You might want to use set -x in other scripts to help you there, and inspect the boot logs from the target.

Related Solutions

Linux – Monitoring what program calls an executable file

This would be hacky, but if it's a dynamically linked executable, you could set up a global preload in /etc/ld.so.preload which would only trigger a logging hook if it detected you were in the right executable.

Something like:

#define _XOPEN_SOURCE
#include <stdio.h>
#include <string.h>
#include <unistd.h>

#define TARGET "/some_executable"

__attribute__((constructor)) 
static void 
logger(int argc, char** argv){ 
    /*catch own argv right here and parent's later from /proc */

    static char buf[sizeof(TARGET)];

    readlink("/proc/self/exe", buf, sizeof(buf)-1);

    if ( 0==strcmp(TARGET, buf)){
        /* ... */
        syslog(/*...*/);
    }
}

The obvious disadvantage of this approach is it would slightly delay the execution of each dynamically linked executable on your system, but my measurements indicate the delay is quite small (<1ms where fork+exec costs about 2ms).

As for the dropped permission problem, you could have a small setuid-root binary that will unconditionally read and echo its grandparents proc files (the status file, most likely), possibly if and only if its parent is the executable whose parents you want to log. You could then spawn that setuid executable inside your logging hook to obtain the info on the executables parent (grandparent of the setuid helper).

Bash – Does a noninteractive login shell execute `~/.profile` or a file whose name is `$BASH_ENV`

No, you’re not correct. Bash behaves as documented:

the first section applies to interactive login shells, and to non-interactive shells started with the --login flag;
the third section applies to non-interactive shells, including non-interactive login shells not started with the --login flag.

A shell can be a login shell without the --login flag. If you look at /proc/$$/cmdline from a Bash shell started by SSH on a Linux system, you’ll see it was started as -bash — the leading hyphen is the usual way of starting a login shell, and isn’t covered by the first section if it ends up being non-interactive. However if one wanted a non-interactive login shell for whatever reason, one would typically use --login to get it.

Best Answer

Related Solutions

Linux – Monitoring what program calls an executable file

Bash – Does a noninteractive login shell execute `~/.profile` or a file whose name is `$BASH_ENV`

Related Question