Saturday, 27 June 2020

Demonstrating the usage of 'ISO' tools with a real-life example

This is the final of three posts about my 'ISO' tools that include '', '' and '' scripts and in it I will demonstrates their usage with a worked example.

The example is about a testing PC that is currently running Ubuntu 18.04 LTS and now that Ubuntu 20.04 LTS has been released I'm considering how to upgrade it.

Normally Ubuntu does not prompt you to upgrade until the first point release becomes available (in this case 20.04.1) so in theory I could use the interim to plan however I want to upgrade now.

I'm considered two approaches to perform the upgrade. The first is the accepted way of upgrading where you simply run a command or use the Ubuntu update manager and upgrade the currently installed Ubuntu (i.e. 18.04 to 20.04). The second approach is arguably more of a migration where you backup your data then install a clean Ubuntu 20.04 and then restore your data which despite semantics is still effectively upgrading from 18.04 to 20.04.

Canonical provide good documentation on how to perform upgrades (e.g. How to upgrade from Ubuntu 18.04 LTS to 20.04 LTS today) however one point that must not be overlooked is whether your upgraded system will still have the same functionality of your existing one.

Like most users besides the applications or packages provided by the base installation I've also added packages and an upgrade will likely affect them. Unfortunately some packages are tied to a release version so maybe unavailable in a new releases or unsupported at best.

When you perform a upgrade traditionally you would remove obsolete packages in order to get a clean system and in part that is why planning for an upgrade is necessary.

The alternative approach of restoring your data after a fresh install also isn't just that simple as besides restoring your data you then need to install all the packages you added to the previous release albeit bearing in mind that not all packages may be available.

One key issue you face with the second approach is knowing what packages you have currently installed and of them which were manually installed subsequent to the initial base installation.

Looking at the second approach first, lets start by identifying the packages that will be required to be additionally installed.

On Debian and other dpkg-based distributions which includes Ubuntu you could try running the following command to gather this information:

grep " install" /var/log/dpkg.log

However this only works if the log file hasn't been rotation in which case you need:

grep " install" /var/log/dpkg.log /var/log/dpkg.log.1

But again this isn't that simple as older log files get compressed and worse still eventually deleted by log rotation (take a look at your '/etc/logrotate.d/dpkg' file) so it can't be used as a way to give you the entire history of your system.

As a simple alternative you can use my '' to compare the packages currently installed with those included in the installation ISO.

First I need to find out which ISO was used for the initial install as looking at '/etc/os-release' will only show the current version (in my case 18.04.4). Looking at '/var/log/installer/media-info' shows the release version (mine was 18.04) and release date (20180426). This indicates that the installation was done from the 'ubuntu-18.04-desktop-amd64.iso' ISO and this can be confirmed by comparing with the ISO's '.disk/info':

if [ "$(cat /var/log/installer/media-info)" == \
     "$(7z x -so ubuntu-18.04-desktop-amd64.iso .disk/info)" ]; then \
        echo is the ISO
        echo not the ISO

However since its installation multiple 'apt upgrade' commands have upgraded my system from 18.04 to 18.04.4 so to ensure I make a true comparison I should first use '' to upgrade an 18.04 ISO the same way (as remember that point releases also introduce the HWE stack so an upgraded 18.04.4 is not exactly the same as a release 18.04.4 ISO).

Therefore I need to first simulate the elapsed upgrades with: -i ubuntu-18.04-desktop-amd64.iso -b GRUB-64 --dist-upgrade

after which I renamed the resultant ISO as 'dist-upgraded-ubuntu-18.04-desktop-amd64.iso' and then compare with my existing system: dist-upgraded-ubuntu-18.04-desktop-amd64.iso

The resultant 'isocomparepkgs.log' contains a list of packages that are only locally installed:

Even now it is not that simple as the list of packages is very long as it includes package dependencies including libraries but at least I have a list which could be used for the second approach.

This is where '' comes in useful. It will take that long list of packages and try and identify which core packages would achieve the same result if installed using 'apt' from Ubuntu sources as well as identifying those which require manual installation. So it is just a simple question of running: dist-upgraded-ubuntu-18.04-desktop-amd64.iso

and the log file shows:

A point to note here is that at some stage during using Ubuntu 18.04 I installed the 'lxde' package and later purged 'youtube-dl'. It also looks like a package dependency changed with respect to PHP as if I follow the suggested purge command exactly then I will also remove a package I want to keep ('phoronix-test-suite') again because of package dependencies (see my documentation on '' for more details).

Another observation to make here is that three packages were identified as requiring manually installation.

So after verifying and running the appropriate 'apt install' and 'apt purge' commands I then renamed the resultant ISO as 'localised_18.04.iso' which is an ISO that effectively reflects my current environment (less chome, megasync and xnview).

Therefore if I was to adopt the second approach to upgrading from 18.04 to 20.04 I could use these two 'apt' commands as a basis of what to install over a clean 20.04 installation. Obviously some packages like 'linux-headers-4.15.0-99-generic' are not required whereas others like 'edid-decode' I may decide I no longer require and some packages like 'phoronix-test-suite' are just not available. But at least it is a starting point.

However the first approach was to upgrade 18.04 using the command line. Now that I have an ISO that mimics my current environment I can install this on a separate test box, run the upgrade and observe what happens without the need of lengthy data backups or restores. This way I can see what happens during the upgrade.

Or I can manually upgrade this ISO. However because of 'snaps' not working in an 'chroot' environment the upgrade is slightly more convoluted.

First I need to replace the 'bionic' repositories with the 'focal' ones: -i localised_18.04.iso -b GRUB-64 -r "--remove deb bionic main restricted" -r "--remove deb bionic-security main restricted" -r "--remove deb bionic-updates main restricted" -r "deb focal main restricted" -r "deb focal-security main restricted" -r "deb focal-updates main restricted"

after which I rename the resultant ISO as 'interim_first_localised_pseudo_20.04.iso'. Then in theory I could perform a 'dist-upgrade' on this ISO however this fails due to a problem with the 'texlive-binaries' package. So instead I purge that package and then perform the upgrade: -i interim_first_localised_pseudo_20.04.iso -b GRUB-64 -e texlive-binaries --dist-upgrade

creating a new ISO I call 'interim_second_localised_pseudo_20.04.iso' and renaming the log file as 'interim_second_localised_pseudo_20.04.iso.isorespin.log'. I then add back in packages removed as a result of '-e texlive-binaries' with: -i interim_second_localised_pseudo_20.04.iso -b GRUB-64 -p '"'$(sed -n '/The following packages will be REMOVED/,/The following NEW packages will be installed/p;/The following NEW packages will be installed/q' interim_second_localised_pseudo_20.04.iso.isorespin.log | sed '1d;$d;s/\*//g' | xargs | sed 's/libsynctex1 //' | sed 's/libtexlua52 //')'"' --debug

which is a command derived through trial and error as a couple of the packages must be excluded in order to successfully add back the packages. I rename the final ISO as 'localised_pseudo_20.04.iso'.

At this point I can now mimic a standard Ubuntu 20.04 ISO from this localised pseudo 20.04: localised_pseudo_20.04.iso ubuntu-20.04-desktop-amd64.iso

to give me the 'apt' commands to create a localised Ubuntu 20.04 ISO from the localised pseudo Ubuntu 20.04 ISO:

$(grep "^" isomimicpkgs.log | head -1) --debug -i linuxium-ubuntu-20.04-desktop-amd64.iso -b GRUB-64 -e "libxfce4util-bin python3-pyxattr rtmpdump youtube-dl"

which I triumphantly name 'localised_20.04_from_pseudo_20.04.iso'.

Finally I can compare this 'localised_20.04_from_pseudo_20.04.iso' with the earlier 'localised_pseudo_20.04.iso' to shows me the new packages installed as part of Ubuntu 20.04 and the packages I loose as part of the upgrade:

and importantly for me this includes loosing 'phoronix-test-suite'.

This still leaves me the option of manually installing my mimic'ed Ubuntu 18.04 testing environment and performing the upgrade the official Ubuntu way.  However before doing this I want to make sure that I create the most similar environment as possible prior to the upgrade so first I capture the full list of installed packages on Ubuntu 18.04 with:

dpkg-query -Wf '${db:Status-Abbrev}${Package}\n' | grep '^ii' | awk '{print $2}' | sort > source_locally_installed_packages.18.04

Now I can install my 'localised_18.04.iso' on the separate test box. Once installed I need to examine the installation log as 'ubiquity' (Ubuntu's installer) removes some packages it believes are not required:

grep -i removed /var/log/installer/syslog

and I need to re-install them with:

sudo apt install apt-clone cifs-utils dmeventd dpkg-repack gparted grub-efi-amd64 kpartx lvm2 python-crypto python-ldb python-samba python-tdb qt5-gtk-platformtheme qttranslations5-l10n samba-common samba-common-bin

I also need to remove any packages added by 'ubiquity' so I compare the output of running a 'dpkg-query' with the previous 'source_locally_installed_packages.18.04' output and see that I need to:

sudo apt purge grub-pc-bin

As a check I can then compare the 'localised_18.04.iso' with newly locally installed packages using: localised_18.04.iso

which confirms that I have successfully replicated the environment:

Now I can perform the upgrade with

(Alt+F2) update-manager -c -d

As I perform the upgrade I am given the opportunity to monitor what will happen:

and make important decisions:

Once complete I can then mimic the resultant environment to a standard Ubuntu 20.04 ISO with: ubuntu-20.04-desktop-amd64.iso

creating a final 'localised_20.04.iso'.

I can then compare this ISO with the ones created earlier (i.e. 'localised_pseudo_20.04.iso' and 'localised_20.04_from_pseudo_20.04.iso') and see more implications and effects of upgrading for use in my final planning of which approach to use and what information I need to successfully upgrade my testing PC from Ubuntu 18.04 to 20.04.

Hopefully this worked example show the usefulness of my 'ISO' tools and how they can be applied to everyday problems rather than just being used to boot Ubuntu on a 32-bit device.

Please donate if you find the scripts useful using the following link as everything helps with development costs.

Friday, 15 May 2020

How to create an ISO that mimics the installed packages from an ISO or those locally installed using ''

This is the second of three posts introducing a couple of new 'ISO' tools that I've developed to complement my '' script.

The second tool is '' which is a script to compare installed packages in two Ubuntu (or Ubuntu flavour), Linux Mint, neon, elementary, BackBox or Peppermint desktop ISOs or Ubuntu live server ISOs, or one such ISO with locally installed packages and identify which packages need to be added to the second ISO to mimic the first ISO or added to the first ISO to mimic those locally installed. In otherwords it will show how to update an ISO with either the packages found in another ISO or with those locally installed.

For example, say you want to create a test enviroment with the same packages as those locally installed. Normally you would install the same Ubuntu release ISO and then install the packages one by one in order to replicate the enviroment. However this is not so simple when you have a lot of packages installed.

You can start by finding the ISO that was used for the initial install by looking at the contents of the file '/var/log/installer/media-info':

linuxium@LINUXIUM:~$ cat /var/log/installer/media-info 
Ubuntu 18.04 LTS "Bionic Beaver" - Release amd64 (20180426)linuxium@LINUXIUM:~$ 

Next you can search through the 'apt' history files looking for entries showing an application installation:

See how the highlighted entry shows the package installed together with the dependent packages that were automatically installed at the same time.

However this will not show any packages that were installed using 'dpkg' or by a tool like 'gdebi'. To see all the packages we can look at '/var/log/dpkg.log' however the 'package -> dependent package' relationship is somewhat lost:

Finally these logs do not last for ever as they are rotated to save space as defined by their respective configuration files:

So one solution to the problem of creating a duplicate test environment would be to use '' to establish which packages were locally installed and not part of the original ISO and then respin the original ISO adding these packages. Except that in a more complex environment this can become a somewhat unrealistic alternative :

Which finally leads me to my other tool: ''.

        Usage: <target ISO>
        or: <source ISO> <target ISO>

The usage syntax is straightforward: you either run the script against two ISOs to see how to mimic the package of those installed in the source ISO to the target ISO or you run the script against an ISO to see how to mimic the locally installed packages to that ISO.

In the example we have been using so far you would run the command: ubuntu-18.04-desktop-amd64.iso
The resultant output looks like this:

The tool first identifies those local packages that need to be installed manually as they are not Canonical packages (e.g. 'google-chrome-stable', 'megasync' and 'xnview').

The tool has then simplified the number of packages that were needed to be installed down from 698 to 102 by attemting to reverse engineer the package dependencies and determine a minimum set of packages that obtains the same result.

However as a result it has highlighted some packages that then also need to be removed (or purged), either because they had been previously removed since original installation (for example 'youtube-dl') or due to a dependency issue (in this case it is actually 'php7.2-gd'). Therefore rather than simply removing all the identified additional packages as there could be consequencies because of package dependencies it is advisable to check each one individually first. This is best achieved by respinning the intermediate ISO using the '--interactive' option. For example:

The resultant ISO can then be compared with the locally installed packages to verify that we have successfully mimicked the local environment:

In the final post of this series I will demonstrate the usage of my 'ISO' tools in a real-life example.

The initial release of '' can be downloaded from here.

Please donate if you find the script useful using the following link as everything helps with development costs.

Saturday, 9 May 2020

Comparing packages between ISOs or between an ISO and those locally installed using ''

This is the first of three posts introducing a couple of new 'ISO' tools that I've developed to complement my '' script.

The first tool is '' which is a script to compare installed packages between two Ubuntu (or Ubuntu flavour), Linux Mint, neon, elementary, BackBox or Peppermint desktop ISOs or Ubuntu live server ISOs or between one such ISO and the locally installed packages on the system where the script is run.
Usage: <ISO> [<ISO>]
The usage syntax is straightforward: you either run the script against two ISOs to get a package comparison of those installed in each or you run the script against one ISO to get a comparison of packages in the ISO verses those locally installed.

For example, say you want to see the package difference between the original Ubuntu 18.04 ISO and the latest point release of 18.04.4. Simply enter the command: ubuntu-18.04-desktop-amd64.iso ubuntu-18.04.4-desktop-amd64.iso

The resultant output looks like this:

with the logfile 'isocomparepkgs.log' also including a side-by-side comparison:

It is important to note that the script only shows the difference between installed packages based on their name and not their version or release. Also it only shows packages and not userspace file differences. 

So the difference between a respun Ubuntu 18.04.4 ISO using the 'atom' option and the original ISO only shows the additional bluetooth package and not the wifi and ALSA UCM files which were also added as a result of respinning:

When comparing packages in an ISO with those locally installed it is important to remember than an ISO also includes several packages used only as part of the installation process:

and that the list of locally installed packages also includes all the dependent packages which can make it difficult to determine the actual packages used in the install commands:

which is a why the second post in this series may be of interest.

The initial release of '' can be downloaded from here.

Please donate if you find the script useful using the following link as everything helps with development costs.

Saturday, 25 April 2020

Canonical have announced the release of Ubuntu 20.04 LTS - 20.04 (Focal Fossa)

Canonical have announced the latest release of Ubuntu 20.04 Long-Term Support (LTS) as Ubuntu 20.04 LTS (Focal Fossa).

I’ve respun the desktop ISO using my ‘‘ script and created ISOs suitable for Intel Atom and Intel Apollo Lake devices:

Atom (-i ubuntu-20.04-desktop-amd64.iso --atom)
Apollo (-i ubuntu-20.04-desktop-amd64.iso --apollo)

I've also respun the desktop ISO with the '--server' option to create a pseudo server ISO suitable for Intel devices with a 32-bit bootloader:

Server (-i ubuntu-20.04-desktop-amd64.iso --server)

Also announced are the official 20.04 flavours of Ubuntu including Lubuntu which I've also respun to created an ISO suitable for Intel Atom devices:

Atom (-i lubuntu-20.04-desktop-amd64.iso --atom)

Downloading Note

After downloading an ISO file it is recommended to test that the file is correct and safe to use by verifying the integrity of the downloaded file. An error during the download could result in a corrupted file and trigger random issues during the usage of the ISO.

The program 'md5sum' is designed to verify data integrity using the MD5 (Message-Digest algorithm 5) 128-bit cryptographic hash. The MD5 calculation gives a checksum (called a hash value), which must equal the MD5 value of a correct ISO.

First open a terminal and go to the correct directory to check a downloaded ISO. Then run the command 'md5sum <ISO>' for example:
md5sum linuxium-atom-ubuntu-20.04-desktop-amd64.iso
'md5sum' should then print out a single line after calculating the hash:

e1a389390b80c468b00f68e999ed9fa0  ./linuxium-atom-ubuntu-20.04-desktop-amd64.iso

Compare the hash (the alphanumeric string on left) from your output with the corresponding hash below. If both hashes match exactly then the downloaded file is almost certainly intact. However if the hashes do not match then there was a problem with the download and you should download the file again.

ISO 'md5sum' hashes

e1a389390b80c468b00f68e999ed9fa0  ./linuxium-atom-ubuntu-20.04-desktop-amd64.iso
ea3e1596843c261a686091569de04895  ./linuxium-apollo-ubuntu-20.04-desktop-amd64.iso
4d871473bd0a19c21eb082dfb5617421  ./linuxium-ubuntu-20.04-server-amd64.iso
aec469c56b398e8238f4ddc9d2b825c4  ./linuxium-atom-lubuntu-20.04-desktop-amd64.iso

Please donate if you find these ISOs useful.

Tuesday, 21 April 2020

A while back I created the website which I hoped would become the 'go to' site for all matters Linix on mini PCs. Unfortunately it didn't go viral! Neither has it got much attention and the reality is that it meets the criteria of being a failed project: it has become a burden both financially and resourcefully as well as technically.

So it is time to close it down and in early May the site will quietly disappear.

Please donate if you find my remaining sites useful using the following link as everything helps with support costs.

Friday, 17 April 2020

Using '' to create a server ISO from an Ubuntu desktop ISO

To simplify the creation of a server ISO by respinning an Ubuntu desktop ISO I've added a new option '--server'. This option is compatible with existing options so you can create a server ISO that works on both 32-bit and 64-bit bootloaders found on various low cost Intel Atom mini PCs or one that works on the more recent Apollo and Gemini Lake mini PCs.

Invocation is as simple as adding '--server' to your '' command. For example to create a 'vanilla' server ISO from an Ubuntu 18.04.4 desktop ISO enter: -i ubuntu/ubuntu-18.04.4-desktop-amd64.iso --server
Here is a (speeded up) video showing this example in action:

The '--server' option is only supported with Ubuntu 18.04 and 20.04 desktop ISOs at the moment. This is because whilst removing the 'ubuntu-desktop' task essentially creates the base for a server ISO, it still leaves a small number of residual packages that need purging. This new option removes the bulk of these and those that are left are really insignificant.

The option now creates 'ubuntu' as the default user without a password, similar to how the standard desktop ISO works. It also downloads the latest version of '' to '/usr/local/bin' for convenience when installing.

The latest version of '' can be downloaded from here.

You can install the server ISO using the '' which I've also updated. Now rather than having to enter 'c' to 'change to the detected value' you simply hit enter to accept the default or enter your own value if required:

The script also now creates a swap file which is adjusted in size to be appropriate for the device's disk and memory sizes.

An updated 'installation' video can be seen here:

which again is speeded up in parts to make it more watchable.

The latest version of '' can be downloaded from here.

Please donate if you find the scripts useful using the following link as everything helps with development costs.

Tuesday, 7 April 2020

Creating and Installing a Server Respun Ubuntu Desktop ISO

Whilst Canonical release 'server' images of Ubuntu getting them to boot or install on Intel mini PCs can be difficult especially if the mini PC is limited by the BIOS enforcing a 32-bit bootloader. As a result there have been many requests for '' to respin an Ubuntu server ISO. However the structure of a server ISO is different to a desktop ISO and as a result is incompatible with the script. So I've always suggested as a workaround to first respin and install the desktop ISO and then purge the desktop packages and install the required server packages.

But what if you try and respin a desktop ISO as a server ISO?

Well this is possible however you need to add the package 'casper' in order to boot it. You also need to create a user as part of respinning otherwise you won't be able to login.

The problem is that even after successfully booting a 'pseudo' server ISO installing it is rather tedious.

Therefore I'm developing a text based script to simplify the installation and I've documented the current status below.

First you need to respin the Ubuntu desktop ISO using '' with the following options: \
-i ubuntu-18.04.4-desktop-amd64.iso \
--atom \
-e ubuntu-desktop \
-p "ubuntu-server casper gdisk network-manager linux-generic-hwe-18.04" \
-f \
-c "useradd -c ubuntu -d /home/ubuntu -m -g users -s /bin/bash ubuntu" \
-c "sed -i '/^root/ a ubuntu ALL=(ALL:ALL) ALL' /etc/sudoers" \
-c "passwd -de ubuntu"

To explain further the options included are:
'-i ubuntu-18.04.4-desktop-amd64.iso' is the desktop ISO I've currently been testing with.
'--atom' for an Intel Atom based device. Alternatively use '--apollo' for those relevant devices that don't boot the standard desktop ISO or '-b rEFInd' to include the 'rEFInd' boot manager.
'-e ubuntu-desktop' to purge (remove) the packages tagged with task 'ubuntu-desktop' which will remove the graphical interface.
'-p ubuntu-server' to install all the packages included in the metapackage 'ubuntu-server'.
'-p casper' to provice the scripts required to boot the ISO.
'-p gdisk' is required by my installation script in order to format the storage.
'-p network-manager' again required by the installation script in order to have internet access.
'-p linux-generic-hwe-18.04' as it was removed during the desktop purge.
(Note: these packages have been combined into one '-p' option)
'-f' includes my installation script (see below).
-c "useradd ..." this command creates the user 'ubuntu' for the respun ISO. Any user can be created.
-c "sed ..." allows the 'ubuntu' user 'sudo' access.
-c "passwd ..." allows a password of choice to be created on initial boot.

A further '--debug' option can be added so that the resultant logfile shows details of all the package dependencies used in the purge and install commands. Full documentation on the options can be found here.

My 'Linuxium server install' script '' can be downloaded by clicking on this link.

Using the above options creates the 'pseudo' server ISO. I've uploaded a respun ISO here which you can use for testing and I've renamed it as 'linuxium-atom-ubuntu-18.04.4-server-amd64.iso' just to avoid confusion.

Next create a 'LiveUSB' by writing the ISO to a USB using 'dd':
dd if=linuxium-atom-ubuntu-18.04.4-server-amd64.iso of=/dev/sdX bs=4M
where 'X' is the drive letter for the USB.

Then boot from the 'LiveUSB' using 'Try Ubuntu without installing' and you'll have a 'live' server ISO which you can use similar to any live ISO. Also don't try clicking the 'Install Ubuntu' option as this will fail as the required packages have already been removed by the purge command.

Prior to attempting an installation you must have a working internet connection as packages are required to be downloaded. It is not important whether it is by ethernet or wifi and the 'nmcli' command can be used to configure a connection.

Finally to install the server ISO simply run '' which is included on the ISO in 'usr/local/bin':

The script can also be run with options:
linuxium@LINUXIUM:~$ -h
Usage: /usr/local/bin/ [ OPTIONS ]
where OPTIONS include '-h' for 'help'
'-v' for 'version'
'-c' for 'check' (version)
'-y' run automatically with best guessed values (dangerous)
which are self explanatory.

The script then shows the variables it will use for the installation together with values it determines may be the most appropriate and then allows you to select what you want:

It will then ask for confirmation to start the installation (bearing in mind this will completely overwrite the existing hard drive of the device) and it requires a (case sensitive) 'Y' to continue.

The following video shows the installation on an Intel Compute Stick (STCK1A32WFC):

Importantly the installer will detect whether the bootloader is 32-bit for 64-bit and install GRUB accordingly.

The installation progress is recorded in a logfile 'linservin.log' which is also copied to the installed filesystem and is available under '/var/log/installer'.

If the 'server' ISO proves popular I'm considering adding a '--server' option to '' to facilitate the respinning of server ISOs. Otherwise feedback is appreciated to improve the current '' script.

Please donate if you find the script useful using the following link as everything helps with development costs.