Fedora – How to remove line wrapping from DNF & YUM commands

dnffedorarhelwrapyum

When using dnf and yum on rpm based Linux distros (RHEL/Red Hat, Fedora, CentOS, etc) the utility will automatically wrap lines to make it more friendly for the user to read. This is problematic as it makes it extremely annoying to work with the data through pipelining.

For example:

$ dnf search jenkins-ssh-credentials-plugin-javadoc
Last metadata expiration check: 6 days, 15:30:08 ago on Thu Sep  1 21:09:10 2016.
============= N/S Matched: jenkins-ssh-credentials-plugin-javadoc =============
jenkins-ssh-credentials-plugin-javadoc.noarch : Javadoc for jenkins-ssh-credentials-plugin
$ dnf search jenkins-ssh-credentials-plugin-javadoc | grep ssh
====== N/S Matched: jenkins-ssh-credentials-plugin-javadoc =======
jenkins-ssh-credentials-plugin-javadoc.noarch : Javadoc for
                                              : jenkins-ssh-credentials-plugin

You can see that once the output for DNF is put through grep it decides to wrap the data in a completely different way then when normally displayed to the user.

Multiple issues have been filed about this behavior ( #584525, #986740 ) and consistently the issues are closed as CLOSED NOTABUG because "Yum is an interactive text-based ui which is not suited, nor intended for piping.". The solution as per the Red Hat developers is to "use a different tool for the job."

It seems unreasonable to have to do this, especially when the methods supplied (install repoquery for example) don't even exist within the dnf utilities and require installing a dozen more packages just to parse the output of this data.

Ideally a user would be able to just use the data in pipelining. In lieu of that, it would be nice to have a simple one-liner which could be used to make the data usable.

Best Answer

yum and dnf write progress messages, overstriking a line by using carriage returns. Those messages are not wrapped. grep is not designed for that, assuming lines do not have embedded control characters. grep does make some assumptions about terminals, but that's a different story.

I use a sed script for filtering the overstruck lines to a "final" line (deleting all of the overstruck part). In the example script2log.sed,

# $Id: script2log.sed,v 1.3 2015/02/04 23:50:12 tom Exp $
#
# Trim ordinary ANSI sequences, then OSC sequences, then backspace
# sequences, then trailing CR's and finally overstruck sections of
# lines.
#
# There are still several interesting cases which cannot be handled
# with a script of this sort.  For example:
#       CSI K (clear line)
#       cursor movement within the line
s/␛[[][<=>?]\{0,1\}[;0-9]*[@-~]//g
s/␛[]][^␛]*␇//g
s/␛[]][^␛]*␛\\//g
:loop
s/[^␈]␈\(.\)/\1/g
t loop
s/␍␍*$//g
s/^.*␍//g
s/␛[^[]//g

that is done with these two commands, which first removes any number of trailing carriage returns on a line, and then removes any section of the line including a carriage return. What is left is only the last copy of the line to be written (the one you want):

s/␍␍*$//g
s/^.*␍//g

(and yes, those are literal carriage return characters in the script).

When I capture output from yum (or dnf), I do not try piping to grep (that is guaranteed to produce poor results). Rather, I use script to capture the output, and post-process that using sed, e.g.,

script -c "yum upgrade"
sed -f script2log.sed typescript >upgrade.log
Related Question