Ubuntu – Apt-get update trouble upon missing/broken files

I'm preping a humanitarian mission in Africa where we'll leave behind two laptops in a place with low/costly connectivity to internet. I'd rather have then running Ubuntu than, say, 'another more shady option'.

Thus for updates as well setup adaptability (ie, with an application bank), I want to create a local mirror of the Xenial depots on a USB SSD hardrive for them to physically bring to a more connected/less costly outlet. I've been using apt-mirror under the instructions of this tutorial so far (beware, French ahead, but it is standart enough I suppose).

I'm running all the following on an Ubuntu 16.10 system.

My sources are as follow :

deb [arch=amd64] file:/media/andre/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/ xenial multiverse universe main 
deb [arch=amd64] file:/media/andre/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/ xenial-updates multiverse universe main
deb [arch=amd64] file:/media/andre/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/ xenial-security multiverse universe main
deb [arch=amd64] file:/media/andre/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/ xenial partner

I've had errors while executing "sudo apt-get update" upon an unfound Components-amd64.yml file, while there was a Components-amd64.yml.gz file with messed-up rights (-rw—–) belonging to root in the targeted repository.

** (appstreamcli:17139): WARNING **: No origin found for file _home_andre_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial-security_restricted_dep11_Components-amd64.yml.gz

(appstreamcli:17139): GLib-GIO-CRITICAL **: g_converter_input_stream_new: assertion 'G_IS_INPUT_STREAM (base_stream)' failed

(appstreamcli:17139): GLib-GIO-CRITICAL **: g_output_stream_splice: assertion 'G_IS_INPUT_STREAM (source)' failed

** (appstreamcli:17139): WARNING **: No origin found for file _home_andre_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial-security_multiverse_dep11_Components-amd64.yml.gz

(appstreamcli:17139): GLib-GIO-CRITICAL **: g_converter_input_stream_new: assertion 'G_IS_INPUT_STREAM (base_stream)' failed

(appstreamcli:17139): GLib-GIO-CRITICAL **: g_output_stream_splice: assertion 'G_IS_INPUT_STREAM (source)' failed

** (appstreamcli:17139): WARNING **: No origin found for file _home_andre_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial-security_universe_dep11_Components-amd64.yml.gz
AppStream cache update completed, but some metadata was ignored due to errors.
Lecture des listes de paquets... Fait
N: Can't drop privileges for downloading as file '/media/andre/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial/InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission non accordée)
E: Failed to fetch file:/media/andre/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial/universe/dep11/Components-amd64.yml  Fichier non trouvé - /media/andre/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial/universe/dep11/Components-amd64.yml (2: Aucun fichier ou dossier de ce type)
E: Failed to fetch file:/media/andre/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-updates/universe/dep11/Components-amd64.yml  Fichier non trouvé - /media/andre/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-updates/universe/dep11/Components-amd64.yml (2: Aucun fichier ou dossier de ce type)
E: Failed to fetch file:/media/andre/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-security/universe/dep11/Components-amd64.yml  Fichier non trouvé - /media/andre/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-security/universe/dep11/Components-amd64.yml (2: Aucun fichier ou dossier de ce type)
E: Le téléchargement de quelques fichiers d'index a échoué, ils ont été ignorés, ou les anciens ont été utilisés à la place.

I found some reports related to rigths and privileges. I also ran into
another one related to dep11 files and decided to act upon a comment

• https://bugs.launchpad.net/ubuntu/+source/apt-mirror/+bug/1550852

  since there was a fetch failed for some dep11 files

My conf apt-miror file thus became :

############# config ##################
#
set base_path    /media/andre/Depots/miroir 
#
set mirror_path  $base_path/mirror
set skel_path    $base_path/skel
set var_path     $base_path/var
set cleanscript $var_path/clean.sh
set defaultarch  deb-amd64
set postmirror_script $var_path/postmirror.sh
set run_postmirror 0
set nthreads     20
set _tilde 0
#
############# end config ##############
#
deb-amd64 http://archive.ubuntu.com/ubuntu xenial main restricted universe multiverse
deb-amd64 http://security.ubuntu.com/ubuntu xenial-security main restricted universe multiverse
#
## Major bug fix updates produced after the final release of the distribution.
#
deb-amd64 http://archive.ubuntu.com/ubuntu xenial-updates main restricted universe multiverse
#
## Dépôt commercial  ##
#
deb-amd64 http://archive.canonical.com/ubuntu xenial partner
#
############# clean ##############
#
clean http://fr.archive.ubuntu.com/ubuntu/ xenial
clean http://security.ubuntu.com/ubuntu xenial-security
clean http://archive.canonical.com/ubuntu xenial
clean http://extras.ubuntu.com/ubuntu xenial

But now, there was more of the same with Translation-en files!

I decided to reset the whole mess and downloading the depot anew.

When launching apt-get update, there was an unfounded "Package" file as well as some "Components-amd64.yml" files.

drwxr-xr-x 6 root root 4096 aog 19 16:31 xenial
drwxr-xr-x 6 root root 4096 aog 19 16:31 xenial-updates
__________________________________________________________________________________________
E: Failed to fetch file:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial/main/dep11/Components-amd64.yml  Fichier non trouvé - /media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial/main/dep11/Components-amd64.yml (2: Aucun fichier ou dossier de ce type)
___________________________________________________________________________________________
administrateur@femmeTantsaha1:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial/main/dep11$ ls -l
total 2116
-rw-r--r-- 1 root root  732506 apr 21 13:08 Components-amd64.yml.gz
-rw-r--r-- 1 root root 1023905 apr 21 13:09 icons-128x128.tar.gz
-rw-r--r-- 1 root root  409405 apr 21 13:09 icons-64x64.tar.gz
___________________________________________________________________________________________
E: Failed to fetch file:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-updates/main/dep11/Components-amd64.yml  Fichier non trouvé - /media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-updates/main/dep11/Components-amd64.yml (2: Aucun fichier ou dossier de ce type)
____________________________________________________________________________________________
administrateur@femmeTantsaha1:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-updates/main/dep11$ ls -l
total 748
-rw-r--r-- 1 root root 213372 aog 18 20:17 Components-amd64.yml.gz
-rw-r--r-- 1 root root 388198 aog 18 20:18 icons-128x128.tar.gz
-rw-r--r-- 1 root root 159218 aog 18 20:18 icons-64x64.tar.gz
_____________________________________________________________________________________________
E: Le téléchargement de quelques fichiers d'index a échoué, ils ont été ignorés, ou les anciens ont été utilisés à la place.

I tried to render the local depot functional by completing it with the file "/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-security/main/binary-amd64/Packages". I downloaded and decompressed "Packages.xz" at //security.ubuntu.com/ubuntu/dists … ary-amd64/ and restored the missing repository linkings.

andre@andre-dotmu:/media/andre/Depots/miroir/mirror/security.ubuntu.com/ubuntu/dists/xenial-security/main/binary-amd64$ ls -l
total 1128
-rw-r--r-- 1 root root 846812 août  24 08:42 Packages
-rw-r--r-- 1 root root 165174 août  24 07:58 Packages.gz
-rw-r--r-- 1 root root 133896 août  24 07:58 Packages.xz
-rw-r--r-- 1 root root    105 août  24 07:58 Release

For the "Components-amd64.yml" files, I decompressed the "Components-amd64.yml.gz" files found in the specified repositories.

andre@andre-dotmu:/media/andre/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial/main/dep11$ ls -l
total 4256
-rw------- 1 _apt root 2188952 avril 21 12:08 Components-amd64.yml
-rw-r--r-- 1 root root  732506 avril 21 12:08 Components-amd64.yml.gz
-rw-r--r-- 1 root root 1023905 avril 21 12:09 icons-128x128.tar.gz
-rw------- 1 _apt root  409405 avril 21 12:09 icons-64x64.tar.gz

andre@andre-dotmu:/media/andre/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-updates/main/dep11$ ls -l
total 1472
-rw------- 1 _apt root 686135 août  18 19:17 Components-amd64.yml
-rw------- 1 _apt root 239447 août  22 14:49 Components-amd64.yml.gz
-rw-r--r-- 1 root root 406642 août  22 14:49 icons-128x128.tar.gz
-rw------- 1 _apt root 165416 août  22 14:49 icons-64x64.tar.gz

Now, when launching apt-get update, other error messages appear.

    sudo apt-get update
    Réception de:1 file:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
    Réception de:1 file:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu xenial InRelease [247 kB]
//shortening//
    Réception de:12 file:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages
    Err:12 file:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu xenial-security/restricted amd64 Packages
//shortening//
    Réception de:22 file:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu xenial/main amd64 DEP-11 Metadata [2189 kB]
    Réception de:23 file:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu xenial/main DEP-11 64x64 Icons [481 kB]
    Err:22 file:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu xenial/main amd64 DEP-11 Metadata
      Impossible d'ouvrir le fichier /var/lib/apt/lists/partial
//shortening//
     Err:30 file:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu xenial-updates/main amd64 DEP-11 Metadata
      Somme de contrôle de hachage incohérente
    Réception de:31 file:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu xenial-updates/main DEP-11 64x64 Icons [205 kB]
    Ign:31 file:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu xenial-updates/main DEP-11 64x64 Icons
    Lecture des listes de paquets... Fait
    N: Can't drop privileges for downloading as file '/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial/InRelease' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission non accordée)
    W: The repository 'file:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu xenial-security Release' does not have a Release file.
    N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
    N: See apt-secure(8) manpage for repository creation and user configuration details.
    E: Failed to fetch file:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-security/restricted/binary-amd64/Packages  Fichier non trouvé - /media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-security/restricted/binary-amd64/Packages (2: Aucun fichier ou dossier de ce type)
    E: Failed to fetch store:/var/lib/apt/lists/partial/_media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial_main_dep11_Components-amd64.yml  Impossible d'ouvrir le fichier /var/lib/apt/lists/partial/_media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial_main_dep11_Components-amd64.yml - open (13: Permission non accordée)
    E: Failed to fetch file:/media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-updates/main/dep11/Components-amd64.yml  Somme de contrôle de hachage incohérente
    E: Le téléchargement de quelques fichiers d'index a échoué, ils ont été ignorés, ou les anciens ont été utilisés à la place.

For "E: Failed to fetch store:/var/lib/apt/lists/partial/_media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial_main_dep11_Components-amd64.yml", I've set the rights at 777 on "/var/lib/apt/lists/partial". Here is a list of the repository content:

administrateur@femmeTantsaha1:/var/lib/apt/lists/partial$ ls -l
total 388
-rw-r--r-- 1 root root 246846 apr 22 02:24 _media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial_InRelease
lrwxrwxrwx 1 root root    113 aog 24 09:44 _media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial_main_dep11_Components-amd64.yml -> /media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial/main/dep11/Components-amd64.yml
lrwxrwxrwx 1 root root    116 aog 24 09:44 _media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial_main_dep11_Components-amd64.yml.gz -> /media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial/main/dep11/Components-amd64.yml.gz
lrwxrwxrwx 1 root root    111 aog 24 09:44 _media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial_main_dep11_icons-64x64.tar.gz -> /media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial/main/dep11/icons-64x64.tar.gz
lrwxrwxrwx 1 root root    109 aog 24 09:44 _media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial_main_i18n_Translation-en.gz -> /media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial/main/i18n/Translation-en.gz
lrwxrwxrwx 1 root root    109 aog 24 09:44 _media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial_main_i18n_Translation-en.xz -> /media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial/main/i18n/Translation-en.xz
lrwxrwxrwx 1 root root    122 aog 24 09:44 _media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial_restricted_dep11_Components-amd64.yml.gz -> /media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial/restricted/dep11/Components-amd64.yml.gz
lrwxrwxrwx 1 root root    115 aog 24 09:44 _media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial_restricted_i18n_Translation-en.xz -> /media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial/restricted/i18n/Translation-en.xz
-rw-r--r-- 1 root root  95650 aog 24 08:58 _media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial-updates_InRelease
lrwxrwxrwx 1 root root    124 aog 24 09:44 _media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial-updates_main_dep11_Components-amd64.yml.gz -> /media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-updates/main/dep11/Components-amd64.yml.gz
lrwxrwxrwx 1 root root    119 aog 24 09:44 _media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial-updates_main_dep11_icons-64x64.tar.gz -> /media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-updates/main/dep11/icons-64x64.tar.gz
lrwxrwxrwx 1 root root    117 aog 24 09:44 _media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial-updates_main_i18n_Translation-en.gz -> /media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-updates/main/i18n/Translation-en.gz
lrwxrwxrwx 1 root root    117 aog 24 09:44 _media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial-updates_main_i18n_Translation-en.xz -> /media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-updates/main/i18n/Translation-en.xz
lrwxrwxrwx 1 root root    130 aog 24 09:44 _media_administrateur_Depots_miroir_mirror_archive.ubuntu.com_ubuntu_dists_xenial-updates_restricted_dep11_Components-amd64.yml.gz -> /media/administrateur/Depots/miroir/mirror/archive.ubuntu.com/ubuntu/dists/xenial-updates/restricted/dep11/Components-amd64.yml.gz

There seems to some symbolic links, with all but the 2 ".yml" files I added poiting to archives. And those 2 files are bringing error reports.

I'm really outta my depth, and I've been trying to make this work for a while now. Any idea?