debian – Why is ‘ls’ Wrapping Items with Spaces in Single Quotes?

coreutilsdebianlsquotingupgrade

I just noticed that on one of my machines (running Debian Sid) whenever I type ls any file name with spaces has single quotes surrounding it.

I immediately checked my aliases, only to find them intact.

wyatt@debian630:~/testdir$ ls
'test 1.txt'  test1.txt
wyatt@debian630:~/testdir$ alias
alias ls='ls --color=auto'
alias wget='wget --content-disposition'
wyatt@debian630:~/testdir$

(picture)

Another test, with files containing single quotes in their names (also answering a request by jimmij):

wyatt@debian630:~/testdir$ ls
'test 1.txt'  test1.txt  'thishasasinglequotehere'\''.txt'
wyatt@debian630:~/testdir$ touch "'test 1.txt'"
wyatt@debian630:~/testdir$ ls
''\''test 1.txt'\'''  test1.txt
'test 1.txt'          'thishasasinglequotehere'\''.txt'

(picture)

update with new coreutils-8.26 output (which is admittedly much less confusing, but still irritating to have by default). Thanks to Pádraig Brady for this printout:

$ ls
"'test 1.txt'"   test1.txt
'test 1.txt'    "thishasasinglequotehere'.txt"

$ ls -N
'test 1.txt'  test1.txt
test 1.txt    thishasasinglequotehere'.txt

Why is this happening? How do I stop it properly?

to clarify, I myself set ls to automatically color output. It never put quotes around things before.

I'm running bash and coreutils 8.25.

EDIT:
Appears the coreutils developers thought (link) it would be a good idea to make that a global default despite breaking the principle of least astonishment as well as 46+ years of UNIX tradition.

Any way to fix this without a recompile?

UPDATE – October 2017 – Debian Sid has re-enabled the shell escape quoting by default. This is just getting ridiculous. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=877582

And at the bottom of the reply chain to the previous bug report, "the change was intentional and will remain." https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=813164#226

I thought this was settled. Apparently not.

UPDATE: April 2019: Just found a spurious bug report in PHP that was caused by this change to ls. When you're confusing developers and generating false bug reports, it's time to re-think your changes.

Update: Android toybox ls is now doing something similar to this but with backslashes instead of quotes. Using the -q option makes spaces render as 'question mark characters' (I have not checked what they are, since they're obviously not spaces), so the only fix I have found so far without rooting the device in question is to add this to a script and source it when launching a shell. This function makes ls use columns if in a terminal and otherwise print one-per-line, while tricking ls into printing spaces verbatim because it's running through a pipe.

ls() {
    # only way I can stop ls from escaping with backslashes
    if [ -t 1 ]; then
        /system/bin/ls -C $@ |cat
    else
        /system/bin/ls $@ |cat
    fi
}

Best Answer

Preface: While it may be quite satisfying to upvote an answer such as this and call it a day, please be assured that the GNU coreutils maintainers do not care about SO answer votes, & that if you actually want to encourage them to change, you need to email them as this answer describes.

Update 2019:
Sometime this past year the maintainers have doubled-down and now offer to any bug-coreutils@gnu.org reports about this issue only a boilerplate response pointing to an incredibly long page on their website listing problems people have with this change that they have committed themselves to ignoring.
The unceasing pressure from bug-coreutils@gnu.org reports has clearly had an effect, forcing the generation of this immense & absurd page, and potentially reducing the number of maintainers willing to deal with the problem to only one.
When this many people consider a thing a bug, then it's a bug whether maintainers disagree or not.
Continuing to email them remains the simplest way to encourage change.

"Why is this happening?"

Several coreutils maintainers decided they knew better than decades of de facto standards.

"How do I stop it properly?"

http://www.gnu.org/software/coreutils/coreutils.html:

Bug Reports

If you think you have found a bug in Coreutils, then please send as complete a bug report as possible to <bug-coreutils@gnu.org>, and it will automatically be entered into the Coreutils bug tracker. Before reporting bugs please read the FAQ. A very useful and often referenced guide on how to write bug reports and ask good questions is the document How To Ask Questions The Smart Way . You can browse previous postings and search the bug-coreutils archive.

~~Distros that have already reverted this change:~~

~~Debian coreutils-8.25-2~~
- ~~Including consequently, presumably, Ubuntu and all of the hundreds of Debian-based and Ubuntu-based derivatives~~

Distros unaffected:

openSUSE (already used -N)

"Any way to fix this without a recompile?"

Proponents would have you...

get back to the old format by adding -N to their ls alias

…on all of your installs, everywhere, for the remainder of eternity.

Related Solutions

Shell – Wrapping Commands with Single and Double Quotes

General tip: if you have two levels of nesting, avoid using single quotes in the inner command, and use single quotes around the outer command.

Additional tip: don't use backticks - `…` - to execute code, instead use $(…) around it. Dollar-parentheses is pretty much DWIM('Do what I mean') when it comes to nested quotes; backquotes have arcane, shell-dependent rules.

watch -n 3 'for x in $(my_command | grep keyword | cut -d" " -f1); do command2 "rusage[mem=7000]" "$x"; done'

If you need single quotes inside a single-quoted command, you can use '\''. Think of these four characters as the way to quote a single quote inside single quotes, though technically speaking this is constructed as end the single-quoted string, append a literal single quote, and start a new single-quoted string (still appended to the current word).

For more complex cases, either painstakingly count the quotes, or define temporary variables.

cmd='for x in $(my_command | grep keyword | cut -d" " -f1); do command2 "rusage[mem=7000]" "$x"; done'
watch_cmd='watch -n 3 "$cmd"'

This answer isn't specific to zsh. Zsh doesn't bring anything major here. You can save a bit of quoting because there's no need for double quotes around command substitutions, and sometimes there are ways to use built-ins rather than external commands which reduce the quoting needs, but the underlying issues are the same as in other shells.

Oh, and by the way, note that watch will execute your command in sh, not in zsh. If you want to execute the command in zsh, you need to run

watch -n 3 -x zsh -c "$cmd"

on Debian/Ubuntu, and

export cmd
watch -n 3 'exec zsh -c "$cmd"'

(even more quoting!) elsewhere.

bash – Why Parameter Expansion with Spaces Works Inside Double Brackets but Not Single Brackets

The single bracket [ is actually an alias for the test command, it's not syntax.

One of the downsides (of many) of the single bracket is that if one or more of the operands it is trying to evaluate return an empty string, it will complain that it was expecting two operands (binary). This is why you see people do [ x$foo = x$blah ], the x guarantees that the operand will never evaluate to an empty string.

The double bracket [[ ]], on the other hand, is syntax and is much more capable than [ ]. As you found out, it does not have the "missing operand" issue and it also allows for more C-like syntax with >, <, >=, <=, !=, ==, &&, || operators.

My recommendation is the following: If your interpreter is #!/bin/bash, then always use [[ ]]

It is important to note that [[ ]] is not supported by all POSIX shells, however many shells do support it such as zsh and ksh in addition to bash

Best Answer

Related Solutions

Shell – Wrapping Commands with Single and Double Quotes

bash – Why Parameter Expansion with Spaces Works Inside Double Brackets but Not Single Brackets

Related Question