fedora dnf – Database Disk Image is Malformed with dnf on Fedora 29

I have cross-posted this on ask.fedora.org as I am still uncertain whether this or the other side has the larger Fedora userbase.

For a few days now I could not install updates due to this error:

# env LC_ALL=C dnf update
Last metadata expiration check: 0:18:57 ago on Wed Jan 23 16:16:14 2019.
Traceback (most recent call last):
  File "/usr/bin/dnf", line 58, in <module>
    main.user_main(sys.argv[1:], exit_code=True)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 179, in user_main
    errcode = main(args)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 64, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 99, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 123, in cli_run
    ret = resolving(cli, base)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 146, in resolving
    base.resolve(cli.demands.allow_erasing)
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 760, in resolve
    self._transaction = self._goal2transaction(goal)
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 681, in _goal2transaction
    ts.add_upgrade(pkg, upgraded, obs)
  File "/usr/lib/python3.7/site-packages/dnf/db/group.py", line 269, in add_upgrade
    ti_new = self.new(new, libdnf.transaction.TransactionItemAction_UPGRADE)
  File "/usr/lib/python3.7/site-packages/dnf/db/group.py", line 222, in new
    reason = self.get_reason(pkg)
  File "/usr/lib/python3.7/site-packages/dnf/db/group.py", line 237, in get_reason
    return self.history.swdb.resolveRPMTransactionItemReason(pkg.name, pkg.arch, -1)
  File "/usr/lib64/python3.7/site-packages/libdnf/transaction.py", line 788, in resolveRPMTransactionItemReason
    return _transaction.Swdb_resolveRPMTransactionItemReason(self, name, arch, maxTransactionId)
RuntimeError: Step: database disk image is malformed in

        SELECT
            ti.action as action,
            ti.reason as reason
        FROM
            trans_item ti
        JOIN
            trans t ON ti.trans_id = t.id
        JOIN
            rpm i USING (item_id)
        WHERE
            t.state = 1
            /* see comment in TransactionItem.hpp - TransactionItemAction */
            AND ti.action not in (3, 5, 7, 10)
            AND i.name = 'python2-rpkg'
            AND i.arch = 'noarch'
        ORDER BY
            ti.trans_id DESC
        LIMIT 1

I found that one shall do these commands in order to fix it, and I tried, but it has not resolved the problem.

• dnf clean all
• dnf makecache
• rpm --buildddb

On Ask Fedora somebody suggested that dnf is not able to recover from that and that one has to repair the SQLite database manually.

I have tried to run the command given,

sqlite3 history-<date>.sqlite ".dump" | sqlite3 history-<new date>.db && rm history-<date>.sqlite

Doing it on /var/lib/dnf/history/history-2015-10-25.sqlite had no effect. Doing it on /var/lib/dnf/history.sqlite resulted in a new error:

# dnf update
Letzte Prüfung auf abgelaufene Metadaten: vor 1:22:29 am Mi 23 Jan 2019 16:39:44 CET.
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 64, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 99, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 123, in cli_run
    ret = resolving(cli, base)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 146, in resolving
    base.resolve(cli.demands.allow_erasing)
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 760, in resolve
    self._transaction = self._goal2transaction(goal)
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 657, in _goal2transaction
    ts.add_install(pkg, obs, reason)
  File "/usr/lib/python3.7/site-packages/dnf/db/group.py", line 256, in add_install
    ti_new = self.new(new, libdnf.transaction.TransactionItemAction_INSTALL, reason)
  File "/usr/lib/python3.7/site-packages/dnf/db/group.py", line 219, in new
    rpm_item = self._pkg_to_swdb_rpm_item(pkg)
  File "/usr/lib/python3.7/site-packages/dnf/db/group.py", line 210, in _pkg_to_swdb_rpm_item
    rpm_item = self.history.swdb.createRPMItem()
  File "/usr/lib/python3.7/site-packages/dnf/db/history.py", line 291, in swdb
    self._swdb = libdnf.transaction.Swdb(self.dbpath)
  File "/usr/lib64/python3.7/site-packages/libdnf/transaction.py", line 729, in __init__
    this = _transaction.new_Swdb(*args)
RuntimeError: Exec failed: malformed database schema (1467577792)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/dnf", line 58, in <module>
    main.user_main(sys.argv[1:], exit_code=True)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 179, in user_main
    errcode = main(args)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 64, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 121, in __exit__
    self.close()
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 472, in close
    self._finalize_base()
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 455, in _finalize_base
    self.history.close()
  File "/usr/lib/python3.7/site-packages/dnf/db/history.py", line 305, in close
    self.swdb.closeTransaction()
  File "/usr/lib/python3.7/site-packages/dnf/db/history.py", line 291, in swdb
    self._swdb = libdnf.transaction.Swdb(self.dbpath)
  File "/usr/lib64/python3.7/site-packages/libdnf/transaction.py", line 729, in __init__
    this = _transaction.new_Swdb(*args)
RuntimeError: Exec failed: malformed database schema (1467577792)

Looking further into the sqlite3 .dump thing, I see this:

/var/lib/dnf# sqlite3 history.sqlite .dump 
PRAGMA foreign_keys=OFF;
BEGIN TRANSACTION;
/**** ERROR: (10) disk I/O error *****/
ROLLBACK; -- due to errors

However, when I copy this history.sqlite into /tmp and execute that .dump there, I get a sensibly looking database. Also opening it with the GUI sqliteman works fine. It seems that the database itself is okay.

Dumping the database into a new file somewhere else and copying it back then gives me this:

/v/l/dnf# dnf update
Letzte Prüfung auf abgelaufene Metadaten: vor 0:30:51 am Sa 26 Jan 2019 13:21:13 CET.
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 64, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 99, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 123, in cli_run
    ret = resolving(cli, base)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 146, in resolving
    base.resolve(cli.demands.allow_erasing)
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 760, in resolve
    self._transaction = self._goal2transaction(goal)
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 650, in _goal2transaction
    reason_obsolete = ts.get_reason(obsolete)
  File "/usr/lib/python3.7/site-packages/dnf/db/group.py", line 237, in get_reason
    return self.history.swdb.resolveRPMTransactionItemReason(pkg.name, pkg.arch, -1)
  File "/usr/lib64/python3.7/site-packages/libdnf/transaction.py", line 788, in resolveRPMTransactionItemReason
    return _transaction.Swdb_resolveRPMTransactionItemReason(self, name, arch, maxTransactionId)
RuntimeError: Step: disk I/O error in

        SELECT
            ti.action as action,
            ti.reason as reason
        FROM
            trans_item ti
        JOIN
            trans t ON ti.trans_id = t.id
        JOIN
            rpm i USING (item_id)
        WHERE
            t.state = 1
            /* see comment in TransactionItem.hpp - TransactionItemAction */
            AND ti.action not in (3, 5, 7, 10)
            AND i.name = 'libmodulemd'
            AND i.arch = 'x86_64'
        ORDER BY
            ti.trans_id DESC
        LIMIT 1


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/dnf", line 58, in <module>
    main.user_main(sys.argv[1:], exit_code=True)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 179, in user_main
    errcode = main(args)
  File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 64, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 121, in __exit__
    self.close()
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 473, in close
    self.reset(sack=True, repos=True, goal=True)
  File "/usr/lib/python3.7/site-packages/dnf/base.py", line 504, in reset
    self.history.close()
  File "/usr/lib/python3.7/site-packages/dnf/db/history.py", line 305, in close
    self.swdb.closeTransaction()
  File "/usr/lib/python3.7/site-packages/dnf/db/history.py", line 291, in swdb
    self._swdb = libdnf.transaction.Swdb(self.dbpath)
  File "/usr/lib64/python3.7/site-packages/libdnf/transaction.py", line 729, in __init__
    this = _transaction.new_Swdb(*args)
RuntimeError: Exec failed: disk I/O error
Exception ignored in: <function SwdbInterface.__del__ at 0x7f28b02e4510>
Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/dnf/db/history.py", line 262, in __del__
  File "/usr/lib/python3.7/site-packages/dnf/db/history.py", line 305, in close
  File "/usr/lib/python3.7/site-packages/dnf/db/history.py", line 291, in swdb
  File "/usr/lib64/python3.7/site-packages/libdnf/transaction.py", line 729, in __init__
RuntimeError: Exec failed: disk I/O error

I also restored the SELinux labels, but that did not change anything, it is

-rw-r--r--. 1 root root unconfined_u:object_r:rpm_var_lib_t:s0 29M 26. Jan 13:52 history.sqlite

Root can also read the file:

/v/l/dnf# head -c 1000 history.sqlite 
SQLite format 3@  ,�,.,P��⏎

How is this on Fedora 29, can I recover from this?