I do have the exact same problem than you for years as well.
For simple non-interactive uses, I like to use the binary block editor BBE.
BBE is to binary as SED is to text, including its archaic syntax and simplicity, however, it has a lot of features missing from what I often need, so I have to combine it with other tools. So, BBE is only a partial solution.
Also note that BBE hasn't had any updates or improvements for years.
Of course one can use xxd
before and xxd -r
after editing the data with text-based tools, but that won't work when the data in question is large and random access is required, for example when processing block devices.
(Note: For Windows, there is at least the costly, proprietary WinHex scripting language, but that won't get us anywhere.)
For more complicated binary editing, I usually fall back to Python as well, even though it sometimes is too slow for large files, which is it's main drawback. I hope Pyston (Python employing LLVM to compile to optimized machine code) will someday mature enough to be usable, or even better, someone will design and implement a free compact, fast and versatile binary processing scripting language, which AFAIK doesn't exist for U*IX like systems yet.
UPDATE
I also happen to use the homebrew, open source Intel x86 assembler flat assembler, or fasm for short, that evolved into much more than just an assembler.
It has a powerful, textblock-based macro preprocessor (itself a turing complete language) with a syntax in the tradition of the borland turbo assembler macro language, but much more advanced.
Also, it has a data manipulation language, which allows to binary include arbitrary files, do all kinds of binary and arithmetic manipulation on it (integer only) at "compile time" and write the result into an output file. This data manipulation language has control strutures and is also turing complete.
It is much easier to use than writing a program that does some binary manipulation in C and probably even in python. Plus, it loads blindingly fast, as it is a small sized executable with almost no external dependecies (There are 2 versions: either it only requires libc or it can run as a static executable directly on the Linux kernel ABI).
It does have some ruff edges, like
not supporting concurrency
being writting in 32 bit x86 assembly (works on x86_64 though), you probably need qemu or a similar emulator if you want to run it on anything else than x86 or x86_64
it's powerful macro preprocessor language is turing complete, this means you better have some experience with languages like Lisp, Haskell, XSLT, or probably M4 would be the best choice.
all data that is to be written into the output file are performed in a "flat" buffer in memory, and this buffer can grow but not shrink until the output file has been written and fasm terminated. This means that one can only generate files at most as large as you have main memory left in a single run of fasm.
data can only be written into a single output file for each run of fasm
yeah, it is homebrew, a really neat and clever one though
Best Answer
-e expression
evaluate the given expression asperl
code-p
:sed
mode. The expression is evaluated for each line of input, with the content of the line stored in the$_
variable and printed after the evaluation of the expression.-l
: even more likesed
: instead of the full line, only the content of the line (that is, without the line delimiter) is in$_
(and a newline is added back on output). Soperl -lpe code
works likesed code
except that it'sperl
code as opposed tosed
code.unpack "B*"
works on the$_
variable by default and extracts its content as a bit string walking from the highest bit of the first byte to the lowest bit of the last byte.pack
does the reverse ofunpack
. Seeperldoc -f pack
for details.With spaces:
(it assumes the input is in blocks of 8 bits (0-padded)).
With
unpack "(B8)*"
, we extract 8 bits at a time, and we join the resulting strings with spaces withjoin " "
.