I have a shell script that uses a single variable as an associative array (one KEY=VALUE
per line).
Throughout the execution of the script, the variable is manipulated to add, remove or modify entries:
Append:
VARIABLE="$(printf "%s\n%s" "$VARIABLE" "KEY=VALUE")"
Modify:
VARIABLE="$(printf "%s\n" "$VARIABLE" | sed -E "s,^(KEY=).*$,\1VALUE,")"
Remove:
VARIABLE="$(printf "%s\n" "$VARIABLE" | grep -E -v "^KEY=.*$")"
When executed in a terminal (or as a service on an old machine under sysv through an init script), the script runs fine, but when run as a service under systemd, after a while, the script starts to spit out error messages in the log :
sh: printf: I/O error
After a lot of trial and error, I couldn't determine exactly what command(s) in the script produced those errors, but I noticed that they start to appear when the length of the variable reaches 8000 bytes (I guess 8192, but I couldn't pinpoint it exactly since whole lines are appended).
I'm pretty sure the variable length is the problem because I implemented a routine that trims the oldest entries of the array whenever the variable length approaches 8192 bytes, and now the script does run under systemd
for a long time without errors; but that's of course not ideal, as there is some information lost.
I searched the web for information about maximum variable length in shell scripts, but didn't find anything useful:
-
dash
manual page doesn't say anything about maximum variable length. -
GNU sed documentation says:
For those who want to write portable sed scripts, be aware that some implementations have been known to limit line lengths (for the pattern and hold spaces) to be no more than 4000 bytes. The POSIX standard specifies that conforming sed implementations shall support at least 8192 byte line lengths. GNU sed has no built-in limit on line length; as long as it can malloc() more (virtual) memory, you can feed or construct lines as long as you like.
…but this applies to line length, not to the whole text length (the individual lines don't exceed 80 characters)
Anyway, since the errors appear only when the script is run through systemd
, I tried to increase LimitMSGQUEUE
and/or LimitSTACK
in the unit file, to no avail (this was a blind guess, as I don't understand exactly the concepts of message queues or process stack, but the numbers shown by systemctl show
looked like 8 KB or so). All other limits regarding memory (LimitRSS
, LimitAS
, LimitMEMLOCK
) seem high enough (way past 8192 bytes), so I don't know what to do next.
What can I do to get this script running under systemd
without errors when the variable length exceeds 8 KB ?
Best Answer
Not so much an answer, but a diagnostic...
On my system, running
dash
v0.5.8-2.10, variable length can be quite large, at least 2^30 chars. Demonstration, by doubling the length of a variable${x}
until its length in chars${#x}
exceeds about 25% of free memory (as checked by the ad hocfurp
function):First start up
dash
:Then, (within
dash
), run this code:Output (on my system, may vary depending on free memory):
Try putting something like the above code in your script, and then run it under the same
systemd
environment and check the output.