Demonstrating the usage of 'ISO' tools with a real-life example

This is the final of three posts about my 'ISO' tools that include 'isorespin.sh', 'isocomparepkgs.sh' and 'isomimicpkgs.sh' scripts and in it I will demonstrates their usage with a worked example.

The example is about a testing PC that is currently running Ubuntu 18.04 LTS and now that Ubuntu 20.04 LTS has been released I'm considering how to upgrade it.

Normally Ubuntu does not prompt you to upgrade until the first point release becomes available (in this case 20.04.1) so in theory I could use the interim to plan however I want to upgrade now.

I'm considered two approaches to perform the upgrade. The first is the accepted way of upgrading where you simply run a command or use the Ubuntu update manager and upgrade the currently installed Ubuntu (i.e. 18.04 to 20.04). The second approach is arguably more of a migration where you backup your data then install a clean Ubuntu 20.04 and then restore your data which despite semantics is still effectively upgrading from 18.04 to 20.04.

Canonical provide good documentation on how to perform upgrades (e.g. How to upgrade from Ubuntu 18.04 LTS to 20.04 LTS today) however one point that must not be overlooked is whether your upgraded system will still have the same functionality of your existing one.

Like most users besides the applications or packages provided by the base installation I've also added packages and an upgrade will likely affect them. Unfortunately some packages are tied to a release version so maybe unavailable in a new releases or unsupported at best.

When you perform a upgrade traditionally you would remove obsolete packages in order to get a clean system and in part that is why planning for an upgrade is necessary.

The alternative approach of restoring your data after a fresh install also isn't just that simple as besides restoring your data you then need to install all the packages you added to the previous release albeit bearing in mind that not all packages may be available.

One key issue you face with the second approach is knowing what packages you have currently installed and of them which were manually installed subsequent to the initial base installation.

Looking at the second approach first, lets start by identifying the packages that will be required to be additionally installed.

On Debian and other dpkg-based distributions which includes Ubuntu you could try running the following command to gather this information:

grep " install" /var/log/dpkg.log

However this only works if the log file hasn't been rotation in which case you need:

grep " install" /var/log/dpkg.log /var/log/dpkg.log.1

But again this isn't that simple as older log files get compressed and worse still eventually deleted by log rotation (take a look at your '/etc/logrotate.d/dpkg' file) so it can't be used as a way to give you the entire history of your system.

As a simple alternative you can use my 'isocomparepkgs.sh' to compare the packages currently installed with those included in the installation ISO.

First I need to find out which ISO was used for the initial install as looking at '/etc/os-release' will only show the current version (in my case 18.04.4). Looking at '/var/log/installer/media-info' shows the release version (mine was 18.04) and release date (20180426). This indicates that the installation was done from the 'ubuntu-18.04-desktop-amd64.iso' ISO and this can be confirmed by comparing with the ISO's '.disk/info':

if [ "$(cat /var/log/installer/media-info)" == \
     "$(7z x -so ubuntu-18.04-desktop-amd64.iso .disk/info)" ]; then \
        echo is the ISO
else
        echo not the ISO
fi

However since its installation multiple 'apt upgrade' commands have upgraded my system from 18.04 to 18.04.4 so to ensure I make a true comparison I should first use 'isorespin.sh' to upgrade an 18.04 ISO the same way (as remember that point releases also introduce the HWE stack so an upgraded 18.04.4 is not exactly the same as a release 18.04.4 ISO).

Therefore I need to first simulate the elapsed upgrades with:

isorespin.sh -i ubuntu-18.04-desktop-amd64.iso -b GRUB-64 --dist-upgrade

after which I renamed the resultant ISO as 'dist-upgraded-ubuntu-18.04-desktop-amd64.iso' and then compare with my existing system:

isocomparepkgs.sh dist-upgraded-ubuntu-18.04-desktop-amd64.iso

The resultant 'isocomparepkgs.log' contains a list of packages that are only locally installed:

Even now it is not that simple as the list of packages is very long as it includes package dependencies including libraries but at least I have a list which could be used for the second approach.

This is where 'isomimicpkgs.sh' comes in useful. It will take that long list of packages and try and identify which core packages would achieve the same result if installed using 'apt' from Ubuntu sources as well as identifying those which require manual installation. So it is just a simple question of running:

isocomparepkgs.sh dist-upgraded-ubuntu-18.04-desktop-amd64.iso

and the log file shows:

A point to note here is that at some stage during using Ubuntu 18.04 I installed the 'lxde' package and later purged 'youtube-dl'. It also looks like a package dependency changed with respect to PHP as if I follow the suggested purge command exactly then I will also remove a package I want to keep ('phoronix-test-suite') again because of package dependencies (see my documentation on 'isomimicpkg.sh' for more details).

Another observation to make here is that three packages were identified as requiring manually installation.

So after verifying and running the appropriate 'apt install' and 'apt purge' commands I then renamed the resultant ISO as 'localised_18.04.iso' which is an ISO that effectively reflects my current environment (less chome, megasync and xnview).

Therefore if I was to adopt the second approach to upgrading from 18.04 to 20.04 I could use these two 'apt' commands as a basis of what to install over a clean 20.04 installation. Obviously some packages like 'linux-headers-4.15.0-99-generic' are not required whereas others like 'edid-decode' I may decide I no longer require and some packages like 'phoronix-test-suite' are just not available. But at least it is a starting point.

However the first approach was to upgrade 18.04 using the command line. Now that I have an ISO that mimics my current environment I can install this on a separate test box, run the upgrade and observe what happens without the need of lengthy data backups or restores. This way I can see what happens during the upgrade.

Or I can manually upgrade this ISO. However because of 'snaps' not working in an 'chroot' environment the upgrade is slightly more convoluted.

First I need to replace the 'bionic' repositories with the 'focal' ones:

isorespin.sh -i localised_18.04.iso -b GRUB-64 -r "--remove deb http://archive.ubuntu.com/ubuntu/ bionic main restricted" -r "--remove deb http://security.ubuntu.com/ubuntu/ bionic-security main restricted" -r "--remove deb http://archive.ubuntu.com/ubuntu/ bionic-updates main restricted" -r "deb http://archive.ubuntu.com/ubuntu/ focal main restricted" -r "deb http://security.ubuntu.com/ubuntu/ focal-security main restricted" -r "deb http://archive.ubuntu.com/ubuntu/ focal-updates main restricted"

after which I rename the resultant ISO as 'interim_first_localised_pseudo_20.04.iso'. Then in theory I could perform a 'dist-upgrade' on this ISO however this fails due to a problem with the 'texlive-binaries' package. So instead I purge that package and then perform the upgrade:

isorespin.sh -i interim_first_localised_pseudo_20.04.iso -b GRUB-64 -e texlive-binaries --dist-upgrade

creating a new ISO I call 'interim_second_localised_pseudo_20.04.iso' and renaming the log file as 'interim_second_localised_pseudo_20.04.iso.isorespin.log'. I then add back in packages removed as a result of '-e texlive-binaries' with:

isorespin.sh -i interim_second_localised_pseudo_20.04.iso -b GRUB-64 -p '"'$(sed -n '/The following packages will be REMOVED/,/The following NEW packages will be installed/p;/The following NEW packages will be installed/q' interim_second_localised_pseudo_20.04.iso.isorespin.log | sed '1d;$d;s/\*//g' | xargs | sed 's/libsynctex1 //' | sed 's/libtexlua52 //')'"' --debug

which is a command derived through trial and error as a couple of the packages must be excluded in order to successfully add back the packages. I rename the final ISO as 'localised_pseudo_20.04.iso'.

At this point I can now mimic a standard Ubuntu 20.04 ISO from this localised pseudo 20.04:

isomimicpkgs.sh localised_pseudo_20.04.iso ubuntu-20.04-desktop-amd64.iso

to give me the 'apt' commands to create a localised Ubuntu 20.04 ISO from the localised pseudo Ubuntu 20.04 ISO:

$(grep "^isorespin.sh" isomimicpkgs.log | head -1) --debug
isorespin.sh -i linuxium-ubuntu-20.04-desktop-amd64.iso -b GRUB-64 -e "libxfce4util-bin python3-pyxattr rtmpdump youtube-dl"

which I triumphantly name 'localised_20.04_from_pseudo_20.04.iso'.

Finally I can compare this 'localised_20.04_from_pseudo_20.04.iso' with the earlier 'localised_pseudo_20.04.iso' to shows me the new packages installed as part of Ubuntu 20.04 and the packages I loose as part of the upgrade:

and importantly for me this includes loosing 'phoronix-test-suite'.

This still leaves me the option of manually installing my mimic'ed Ubuntu 18.04 testing environment and performing the upgrade the official Ubuntu way.  However before doing this I want to make sure that I create the most similar environment as possible prior to the upgrade so first I capture the full list of installed packages on Ubuntu 18.04 with:

dpkg-query -Wf '${db:Status-Abbrev}${Package}\n' | grep '^ii' | awk '{print $2}' | sort > source_locally_installed_packages.18.04

Now I can install my 'localised_18.04.iso' on the separate test box. Once installed I need to examine the installation log as 'ubiquity' (Ubuntu's installer) removes some packages it believes are not required:

grep -i removed /var/log/installer/syslog

and I need to re-install them with:

sudo apt install apt-clone cifs-utils dmeventd dpkg-repack gparted grub-efi-amd64 kpartx lvm2 python-crypto python-ldb python-samba python-tdb qt5-gtk-platformtheme qttranslations5-l10n samba-common samba-common-bin

I also need to remove any packages added by 'ubiquity' so I compare the output of running a 'dpkg-query' with the previous 'source_locally_installed_packages.18.04' output and see that I need to:

sudo apt purge grub-pc-bin

As a check I can then compare the 'localised_18.04.iso' with newly locally installed packages using:

isocomparepkgs.sh localised_18.04.iso

which confirms that I have successfully replicated the environment:

Now I can perform the upgrade with

(Alt+F2) update-manager -c -d

As I perform the upgrade I am given the opportunity to monitor what will happen:

and make important decisions:

Once complete I can then mimic the resultant environment to a standard Ubuntu 20.04 ISO with:

isomimicpkgs.sh ubuntu-20.04-desktop-amd64.iso

creating a final 'localised_20.04.iso'.

I can then compare this ISO with the ones created earlier (i.e. 'localised_pseudo_20.04.iso' and 'localised_20.04_from_pseudo_20.04.iso') and see more implications and effects of upgrading for use in my final planning of which approach to use and what information I need to successfully upgrade my testing PC from Ubuntu 18.04 to 20.04.

Hopefully this worked example show the usefulness of my 'ISO' tools and how they can be applied to everyday problems rather than just being used to boot Ubuntu on a 32-bit device.

Please donate if you find the scripts useful using the following link http://goo.gl/nXWSGf as everything helps with development costs.