Troubleshooting -bash: Cannot Execute Binary File: Exec Format Error

The message cannot execute binary file has nothing to do with terminals (I wonder what led you to think that — and I recommend avoiding making such assumptions in a question, as they tend to drown your actual problem in a mess of red herrings). In fact, it's bash's way of expressing ENOEXEC (more commonly expressed as exec format error.

First, make sure you didn't accidentally try to run this executable as a script. If you wrote . ./executable, this tells bash to execute ./executable in the same environment as the calling script (as opposed to a separate process). That can't be done if the file is not a script.

Otherwise, this message means that ./executable is not in a format that the kernel recognizes. I don't have any definite guess as to what is happening though. If you can run the script on that same machine by invoking it in a different way, it can't just be a corrupt file or a file for the wrong architecture (it might be that, but there's more to it). I wonder if there could be a difference in the way the target boots (perhaps a race condition).

Here's a list of additional data that may help:

• Output of file …/executable on server B.
• Some information about the target, such as the output of uname -a if it's unix-like.
• Check that the target sees the same file contents each time: run cksum ./executable or md5sum ./executable or whatever method you have on the target just before yet-another-bash-script invokes ./executable. Check that the results are the same in the Hudson invocation, in your successful manual invocation and on server B.
• Add set -x at the top of yet-another-bash-script (just below the #!/bin/bash line). This will produce a trace of everything the script does. Compare the traces and report any difference or oddity.
• Describe how the target is booting when you run the scripts manually and when Hudson is involved. It could be that the target is booted differently and some loadable module that provides the support for the format of ./executable doesn't get loaded (or is not loaded yet) in the Hudson invocations. You might want to use set -x in other scripts to help you there, and inspect the boot logs from the target.

You can't usually run binary files in NixOS, they will either need some environment variables set or to be patched with patchElf. I assume you can install and run java using the nix package manager. You can probably also create a suitable environment to run it using myEnvFun.