Ubuntu Core Snap – How to Debug Snaps

snapubuntu-core

I've successfully built a snap that I can install on my system.

However, it crashes upon start, and I'm not quite sure where to start debugging that crash.

What are the different approaches to debugging an installed app failing to start?

Best Answer

For debugging confined snaps (ie, installed without --devmode) there is a snap in the store called snappy-debug that will get better over time but is already useful. To use:

    sudo snap install snappy-debug --devmode # after snapd 2.0.7 lands, drop --devmode
    sudo snap connect snappy-debug:log-observe ubuntu-core:log-observe
    sudo /snap/bin/snappy-debug.security scanlog

This scanlog command will tail the syslog, resolving seccomp syscall numbers and make suggestions on what interfaces to use, changes to make to your snap to work within the sandbox, etc. You can filter by snap with:

    sudo /snap/bin/snappy-debug.security scanlog <snapname>

Some more tips are covered here:

https://developer.ubuntu.com/snappy/build-apps/debug

Using snap try can be helpful too.

Related Solutions

Snap – Where Can Ubuntu Snaps Write Data

When you declare apps in your snapcraft.yaml, it results in a binary wrapper being generated upon install and placed into /snap/bin/, named after your package and app name (note that if the app is a service, this wrapper is instead a systemd .service file).

That wrapper contains most of the environment under which the application will run. The two environment variables that are most relevant to this question are SNAP_DATA and SNAP_USER_DATA. You can find more information about these in the documentation, but I'll describe them here as well:

SNAP_DATA is a system-wide writable area (in /var/snap/). This might be used to host logs for services, for instance.
SNAP_USER_DATA is a user-specific writable area in the home directory of the user running the application (specifically /home/<user>/snap/). This might be used for user-specific configuration files, etc.

Both of these directories are very important to the upgrade/rollback functionality, since both of them are versioned. That is, each revision of a given snap has its own copy of these directories. Let me explain with an example.

Say you install revision 1 of the "foo" snap. That will create two directories:

/var/snap/foo/1 (SNAP_DATA)
/home/<user>/snap/foo/1 (SNAP_USER_DATA)

Now say "foo" uses both of these. Maybe it has a service that hosts a database in SNAP_DATA, and a binary that uses config files in SNAP_USER_DATA.

Now revision 2 of "foo" is released, and it's automatically updated. The first thing that happens is that /var/snap/foo/1 is copied into /var/snap/foo/2 and /home/<user>/snap/foo/1 is copied into /home/<user>/snap/foo/2. Then the new revision is fired up. It should notice that it's running on old data, and maybe it has some database migrations to run to the database in SNAP_DATA. It does that, and away it goes.

Now say those migrations fail for whatever reason, and this application needs to be rolled back. It begins using the old revision of the /snap/foo application, where SNAP_DATA was pointing to /var/snap/foo/1 and SNAP_USER_DATA was pointing to /home/<user>/snap/foo/1. This picks things up on the old version at the point before the migrations were run, since those operations were run on a copy of the data.

Long story short: don't use the home interface to store data you can be storing in SNAP_DATA or SNAP_USER_DATA, since they're an integral part of the upgrade/rollback strategy. Take advantage of them!

UPDATE for v2.0.10:

Two new data directories were also introduced:

SNAP_COMMON sits alongside SNAP_DATA, but is specifically unversioned. Every revision of the specific snap has access to this directory, so it's not copied upon upgrade/rollback etc. This might be used for particularly large, unversioned files (e.g. raw data that isn't really version-specific).
SNAP_USER_COMMON sits alongside SNAP_USER_DATA, but is again specifically unversioned. It might be used for storing non-version-specific data per user.

UPDATE for v2.15:

The files placed within /snap/bin are no longer wrappers that define the environment, but symlinks to /usr/bin/snap. So the way to determine the environment under which an application runs would be to use snap run --shell <snap>.<app>, for example:

$ sudo snap install hello-world
$ snap run --shell hello-world
To run a command as administrator (user "root"), use "sudo <command>".
See "man sudo_root" for details.

$ env | grep SNAP
SNAP_USER_COMMON=/home/kyrofa/snap/hello-world/common
SNAP_REEXEC=
SNAP_LIBRARY_PATH=/var/lib/snapd/lib/gl:
SNAP_COMMON=/var/snap/hello-world/common
SNAP_USER_DATA=/home/kyrofa/snap/hello-world/27
SNAP_DATA=/var/snap/hello-world/27
SNAP_REVISION=27
SNAP_NAME=hello-world
SNAP_ARCH=amd64
SNAP_VERSION=6.3
SNAP=/snap/hello-world/27

Ubuntu – How do system services work in snaps

From https://developer.ubuntu.com/en/snappy/build-apps/debug/

Testing a service

To test a service it must be installed first. Once it is installed, systemd's systemctl command can be used to see if the service starts and runs as expected, for example:

systemctl status snap.<name>.<appname>

Finding the logs

The journalctl command can be used to inspect the messages that the service sends to stdout/stderr, for example:

journalctl -u snap.<name>.<appname>

Services may log additional data to syslog (/var/log/syslog) or to custom log directories. Note that custom log directories must be in a path that the service can write to (usually SNAP_DATA).

Getting a core dump

To enable core dumps you have to configure a place to write them to through sysfs. For instance you can use

$ echo "/tmp/core.%e.%p" > /proc/sys/kernel/core_pattern

to ensure that your coredumps get written into the /tmp directory regardless of where CWD of the process that received a signal was.