Thursday, 29 June 2017

Ubuntu on the GPD Pocket using 'isorespin.sh'


Although the GPD Pocket has already started to ship the support for running Ubuntu on the device is currently still in development. This means anyone trying to run Ubuntu is faced with the usual hurdles of booting, wifi, screen rotation and all the other frequently encountered issues.

Not surprisingly you can't just download an official Ubuntu ISO and be up running problem free. However I thought it would be interesting to see if the fixes required could be incorporated into an ISO through respinning using my 'isorespin.sh' script.

I've used the solutions that chrisawcom has posted on reddit (https://www.reddit.com/r/GPDPocket/comments/6idnia/linux_on_gpd_pocket) and created a script that basically updates an official ISO with the required fixes allowing it to be respun.

The command I use to respin the ISO is:

isorespin.sh -i ubuntu-17.04-desktop-amd64.iso -u -p thermald -p "libproc-daemon-perl libproc-pid-file-perl liblog-dispatch-perl" -f linuxium-install-UCM-files.sh -f gpd_pocket -f gpd_pocket.sh -c gpd_pocket.sh -g "" -g "i915.fastboot=1 fbcon=rotate:1"

and I'll briefly cover each of the options as follows:

-i  ubuntu-17.04-desktop-amd64.iso uses an official Ubuntu desktop 64-bit ISO as the base ISO.
-u updates the kernel to the latest Ubuntu Kernel Team kernel build located at http://kernel.ubuntu.com/~kernel-ppa/mainline (current is v4.12-rc7). This is important as the proposed v4.12 kernel includes critical patches that fix many of the issues found when running Linux on an Intel Atom SOC including audio and wifi.
-p installs specific software packages required for thermal management and fan controls.
-f adds the scripts that I've written to specifically customize the ISO with audio, wifi and screen support together with the required files.
-g both removes the default kernel boot parameters and adds additional kernel boot parameters required for successfully booting the device related to screen handling.

Looking at the main configuration script (gpd_pocket.sh) in more detail this basically configures:

  • the internet access for the script
  • the Broadcom wifi driver
  • screen rotation
  • screen brighness
  • touchscreen  
  • fan

by implementing the solutions identified on reddit.

The key benefit is including all the fixes within an ISO that can then be repeatedly used for testing. Respinning the ISO produces the log file:


Script '/usr/local/bin/isorespin.sh' called with '-i ubuntu-17.04-desktop-amd64.iso -u -p thermald -p libproc-daemon-perl libproc-pid-file-perl liblog-dispatch-perl -f linuxium-install-UCM-files.sh -f gpd_pocket -f gpd_pocket.sh -c gpd_pocket.sh -g  -g i915.fastboot=1 fbcon=rotate:1' ...
Work directory 'isorespin' used ...
ISO '/home/linuxium/Documents/ubuntu-17.04-desktop-amd64.iso' respun ...
Kernel boot parameters 'i915.fastboot=1 fbcon=rotate:1' added ...
Initial kernel boot parameters 'quiet splash' deleted ...
Bootloader 'GRUB' added ...
Kernel updated with mainline kernel version '4.12.0-041200rc7-generic' ...
Package 'thermald' added ...
Package 'libproc-daemon-perl libproc-pid-file-perl liblog-dispatch-perl' added ...
File '/home/linuxium/Documents/linuxium-install-UCM-files.sh' added ...
Directory '/home/linuxium/Documents/gpd_pocket' added ...
File '/home/linuxium/Documents/gpd_pocket.sh' added ...
Command run ...
# gpd_pocket.sh
/usr/local/bin/gpd_pocket.sh: Configuring GPD Pocket ...
./linuxium-install-UCM-files.sh: Extracting UCM files ...
./linuxium-install-UCM-files.sh: Installing UCM files ...
./linuxium-install-UCM-files.sh: Reloading UCM driver ...
./linuxium-install-UCM-files.sh: Installation of UCM finished 
/usr/local/bin/gpd_pocket.sh: Installing Broadcom files ...
/usr/local/bin/gpd_pocket.sh: Reloading Broadcom driver ...
/usr/local/bin/gpd_pocket.sh: Installing rotation files ...
/usr/local/bin/gpd_pocket.sh: Installing brightness files ...
/usr/local/bin/gpd_pocket.sh: Installing touchscreen files ...
/usr/local/bin/gpd_pocket.sh: Installing fan service ...
Created symlink /etc/systemd/system/multi-user.target.wants/gpdfand.service -> /lib/systemd/system/gpdfand.service.
/usr/local/bin/gpd_pocket.sh: Starting fan service ...
Running in chroot, ignoring request.
/usr/local/bin/gpd_pocket.sh: Configuration of GPD Pocket finished.
Respun ISO created as 'linuxium-v4.12-rc7-ubuntu-17.04-desktop-amd64.iso'.

and an ISO. Anyone interested can download my respun ISO 'linuxium-gpd-v4.12-rc7-ubuntu-17.04-desktop-amd64.iso' and write to a USB using the 'dd' command:

dd if=linuxium-v4.12-rc7-ubuntu-17.04-desktop-amd64.iso of=/dev/sdX bs=4M

where '/dev/sdX' is the correct device for your USB (but first check using commands like 'df', 'blkid' or 'lsblk' to confirm its correct name).

If you want to respin the ISO yourself then download and unzip the file 'gpd_pocket.zip' which contains the required script and files together with 'linuxium-install-UCM-files.sh' for the ALSA UCM files. For the ISO you need to download an Ubuntu 64-bit desktop ISO (e.g. Ubuntu 17.04 from http://releases.ubuntu.com/17.04/ubuntu-17.04-desktop-amd64.iso or Ubuntu 16.04 from http://releases.ubuntu.com/16.04.2/ubuntu-16.04.2-desktop-amd64.iso) together with my 'isorespin.sh' script and then run the command above.

One additional option you might consider adding is '-s 200MB' to add persistence to your ISO as it means that you can retain your data, settings and any changes you make between reboots which is particularly useful when trying to get things to work.

More details about my 'isorespin.sh' script can be found here including full documentation including examples.

If you find my script useful then please donate using the following link http://goo.gl/nXWSGf as everything helps with development costs.





Interim Ubuntu 17.04 ISO with mainline v4.12-rc7 kernel, RTL8723BS and Broadcom Wifi/BT firmware, UCM audio files and full 32-bit bootloader support

UpdateThis work is superseded by my 'isorespin.sh' script which can respin an official ISO suitable for use on Intel Atom devices.

Latest 'linuxium-v4.12-rc7-ubuntu-17.04-desktop-amd64.iso' kernel respun using v7.1.2 of 'isorespin.sh' with log file of:

Script '/usr/local/bin/isorespin.sh' called with '-i ubuntu-17.04-desktop-amd64.iso -u -l rtl8723bs_4.12.0_amd64.deb -f linuxium-install-UCM-files.sh -f linuxium-install-broadcom-drivers.sh -f wrapper-linuxium-install-UCM-files.sh -f wrapper-linuxium-install-broadcom-drivers.sh -c wrapper-linuxium-install-UCM-files.sh -c wrapper-linuxium-install-broadcom-drivers.sh' ...
Work directory 'isorespin' used ...
ISO '/home/linuxium/isorespin/ubuntu-17.04-desktop-amd64.iso' respun ...
Bootloader 'GRUB' added ...
Kernel updated with mainline kernel version '4.12.0-041200rc7-generic' ...
Local package '/home/linuxium/isorespin/rtl8723bs_4.12.0_amd64.deb' added ...
File '/home/linuxium/isorespin/linuxium-install-UCM-files.sh' added ...
File '/home/linuxium/isorespin/linuxium-install-broadcom-drivers.sh' added ...
File '/home/linuxium/isorespin/wrapper-linuxium-install-UCM-files.sh' added ...
File '/home/linuxium/isorespin/wrapper-linuxium-install-broadcom-drivers.sh' added ...
Command run ...
# wrapper-linuxium-install-UCM-files.sh
./linuxium-install-UCM-files.sh: Extracting UCM files ...
./linuxium-install-UCM-files.sh: Installing UCM files ...
./linuxium-install-UCM-files.sh: Reloading UCM driver ...
./linuxium-install-UCM-files.sh: Installation of UCM finished 
# wrapper-linuxium-install-broadcom-drivers.sh
./linuxium-install-broadcom-drivers.sh: Extracting Broadcom files ...
./linuxium-install-broadcom-drivers.sh: Installing Broadcom files ...
./linuxium-install-broadcom-drivers.sh: Reloading Broadcom driver ...
./linuxium-install-broadcom-drivers.sh: Installing Broadcom bluetooth service ...
./linuxium-install-broadcom-drivers.sh: Starting Broadcom bluetooth service ...
Running in chroot, ignoring request.
./linuxium-install-broadcom-drivers.sh: Installation of Broadcom finished 
Respun ISO created as 'linuxium-v4.12-rc7-ubuntu-17.04-desktop-amd64.iso'.

I've used the Ubuntu 17.04 ISO 'ubuntu-17.04-desktop-amd64.iso' and I've updated the kernel with the 'v4.12-rc7 mainline build' and also added my wifi/bt firmware package 'rtl8723bs_4.12.0_amd64.deb', my Broadcom wifi/bt firmware script 'linuxium-install-broadcom-drivers.sh' installed by running 'wrapper-linuxium-install-broadcom-drivers.sh' and my ALSA UCM audio files script 'linuxium-install-UCM-files.sh' installed by running the 'wrapper-linuxium-install-UCM-files.sh'.

The respun ISO can be downloaded from 'linuxium-v4.12-rc7-ubuntu-17.04-desktop-amd64.iso' and written to a USB using 'dd'.

On booting if bluetooth fails to start automatically then it can be started manually with the command:

sudo systemctl start rtl8723bsbt.service

or

sudo systemctl start brcmbt.service

depending on wifi chip.

Full documentation about my 'isorespin.sh' script including examples is available from http://linuxiumcomau.blogspot.com.au/2017/06/customizing-ubuntu-isos-documentation.html.

If you find this useful then please donate using the following link http://goo.gl/nXWSGf as everything helps with development costs.

Adding and deleting kernel boot parameters with 'isorespin.sh'


Sometimes additional kernel boot parameters are required for successful booting to change specific behaviours and to enable or disable certain features (see https://wiki.ubuntu.com/Kernel/KernelBootParameters).

So I've added new functionality to my 'isorespin.sh' script to either delete the default kernel boot parameters or add additional kernel boot parameters or both.

The updated command line invocation is now:

Usage: /usr/local/bin/isorespin.sh [ -h | -v ]
       /usr/local/bin/isorespin.sh -i <ISO> [ [ -u | -k <kernel> ] | -r "<repo> ... " | -p "<pkg> ... " | -l "<pkg.deb> ... " | -f "<file> | <directory> ... " | [ -s <size>MB | GB ] | [ -b GRUB | rEFInd ] ...
       /usr/local/bin/isorespin.sh ... -w <directory> | -d "<pkg> ... " | -e "<pkg> ... " | -c "<cmd> ... " | -o "<file> | <directory> ... " | -g "" | "<kernel boot parameter> ... " ]

Kernel boot parameters are controlled by a new option:
-g or --grub will add the specified kernel boot parameters both to the GRUB/rEFInd boot menu and to the default GRUB settings or if an empty ("") argument is specified then the the ISO's initial kernel boot parameters will be removed.
The option can be entered either using the command line or by using the GUI.

The following is an example showing both the removal of the default kernel boot parameters (quiet splash) and the inclusion of two new kernel boot parameters:

isorespin.sh -i ubuntu-17.04-desktop-amd64.iso -g "" -g "i915.fastboot=1 fbcon=rotate:1"

The default kernel boot parameters are first removed by selecting the delete option: 


and the new kernel boot parameters are then added:


as text strings:




Once the script is running it will update the kernel boot parameters:


with the exact details being recorded in the log file:


Please donate if you find the script useful using the following link http://goo.gl/nXWSGf as everything helps with development costs.

Thursday, 22 June 2017

Third look at Ubuntu 17.10: spicing up Artful

UpdateThis work is superseded by my 'isorespin.sh' script which can respin an official ISO suitable for use on Intel Atom devices.


linuxium-persistence-v4.12-rc6-artful-desktop-amd64.iso

For this look at the latest daily build of Ubuntu 17.10 (Artful Aardvark) I've not only respun the ISO with the latest kernel and wifi/bt drivers together with some additional useful packages but I've also added the Cinnamon desktop.

The result is rather nice.

Canonical may be concerned about people's reaction to their decision that Ubuntu 17.10 will drop Unity and use GNOME 3.0 for the default desktop as they recently ran a poll to collect data on usefulness of some GNOME Shell extensions including 'Dash to Dock'. They have just published the results from the 18,330 responses (https://insights.ubuntu.com/2017/06/12/ubuntu-desktop-gnome-extensions-poll-results) which show an overwhelming interest for 'Dash to Dock'.

1 for not very useful and 5 for very useful

They suggest "Perhaps we work with upstream GNOME to provide a setting we could override which shows the dash outside of just the activities view".

Well given that not everyone likes GNOME 3.0 and Linux Mint is gaining popularity, when I saw the news that Cinnamon 3.4 had been released and "Here’s How To Install it on Ubuntu" I thought why not spice up Artful Aardvark?

Using my 'isorespin.sh' script I created the respun ISO with the following command:

isorespin.sh -i artful-desktop-amd64.iso -u -p "$(cat cinnamon-package-dependencies-for-artful.txt)" -p "ssh openssh-server inxi" -l google-chrome-stable_current_amd64.deb -l rtl8723bt_4.12.0_amd64.deb -f cinnamon-packages-isorespin/usr/src -f 50-ubuntu.conf -f linuxium-install-UCM-files.sh -f wrapper-linuxium-install-UCM-files.sh -f linuxium-install-broadcom-drivers.sh -f wrapper-linuxium-install-broadcom-drivers.sh -c "mv /usr/local/bin/src/*.deb /var/cache/apt/archives" -c "rm -rf /usr/local/bin/src" -c "apt-get install -y /var/cache/apt/archives/*.deb" -c "rm -f /var/cache/apt/archives/*.deb" -c "mv /usr/local/bin/50-ubuntu.conf /usr/share/lightdm/lightdm.conf.d" -c wrapper-linuxium-install-UCM-files.sh -c wrapper-linuxium-install-broadcom-drivers.sh -c "sed -i 's?/usr/share/themes/Adwaita/backgrounds/adwaita-timed.xml?/usr/share/backgrounds/warty-final-ubuntu.png?' /usr/share/glib-2.0/schemas/org.cinnamon.desktop.background.gschema.xml" -c "glib-compile-schemas /usr/share/glib-2.0/schemas" -s 200MB

The key features of the respun ISO are:
  • Based on the Ubuntu 17.10 (Artful Aardvark) Daily Build from 19th June
  • Upgraded with the latest Ubuntu mainline v4.12-rc6 kernel
  • Includes the packages ssh, openssh-server, inxi and Google Chrome
  • Includes the RTL8723BS and Broadcom wifi/bt firmware
  • Includes the ALSA UCM files
  • Configured with Cinnamon 3.4 desktop
  • Has a persistence partition of 200MB
  • Uses the rEFInd boot manager (with GRUB for 32-bit BIOS devices)

so it works on Intel Atom Bay Trail/Cherry Trail and Intel Apollo Lake devices. And now for the first time I've see wifi working on the Minix NEO Z83-4:


If you want to respin the ISO yourself you will need to resolve the issue that the Cinnamon repository is only currently available up to 'zesty'. The steps I took were as follows:
  1. Fetch the Ubuntu 17.04 ISO
  2. Respin the Ubuntu 17.04 ISO with Cinnamon
  3. Extract the file system from the ISO
  4. Capture all the packages that were installed in the ISO
  5. Identify which packages in the Cinnamon repository were actually installed in the ISO
  6. Download those installed Cinnamon packages
  7. Fetch the Artful ISO
  8. Derive the package dependencies required by the Cinnamon packages in Artful
  9. Prepare a configuration file to boot a Cinnamon session
  10. Download the Google Chrome package

Using the following commands:

wget http://releases.ubuntu.com/17.04/ubuntu-17.04-desktop-amd64.iso
isorespin.sh -i ubuntu-17.04-desktop-amd64.iso -r "ppa:embrosyn/cinnamon" -p cinnamon -p blueberry
mv linuxium-ubuntu-17.04-desktop-amd64.iso cinnamon-linuxium-ubuntu-17.04-desktop-amd64.iso
mv isorespin.log cinnamon-17.04-isorespin.log
sudo mount cinnamon-linuxium-ubuntu-17.04-desktop-amd64.iso /mnt
sudo unsquashfs /mnt/casper/filesystem.squashfs
sudo umount /mnt
mv squashfs-root cinnamon-17.04-squashfs-root
sudo chroot cinnamon-17.04-squashfs-root dpkg -l > all-packages-in-cinnamon-17.04.txt
for PACKAGE in $(grep Package  cinnamon-17.04-squashfs-root/var/lib/apt/lists/ppa.launchpad.net_embrosyn_cinnamon_ubuntu_dists_zesty_main_binary-amd64_Packages | sed 's/^Package: //'); do echo "'ii  ${PACKAGE}'"; grep "ii  ${PACKAGE}" all-packages-in-cinnamon-17.04.txt; done  | grep -v "'" | sed 's/^ii  //' | sed 's/ .*//' | sed 's/:amd64$//' > cinnamon-packages-in-cinnamon-17.04.txt
isorespin.sh -i ubuntu-17.04-desktop-amd64.iso -r "ppa:embrosyn/cinnamon" -d "$(cat cinnamon-packages-in-cinnamon-17.04.txt)" -o /usr/src
rm linuxium-ubuntu-17.04-desktop-amd64.iso
rm isorespin.log
sudo rm -rf isorespin/usr/src/linux-headers-4.10.0-19/ isorespin/usr/src/linux-headers-4.10.0-19-generic/
mv isorespin cinnamon-packages-isorespin
wget http://cdimage.ubuntu.com/daily-live/current/artful-desktop-amd64.iso -O artful-desktop-amd64.iso
isorespin.sh -i artful-desktop-amd64.iso -f cinnamon-packages-isorespin/usr/src -c "mv /usr/local/bin/src/*.deb /var/cache/apt/archives" -c "rm -rf /usr/local/bin/src" -c "apt-get install -y /var/cache/apt/archives/*.deb" -c "rm -f /var/cache/apt/archives/*.deb"
rm -f linuxium-artful-desktop-amd64.iso
mv isorespin.log cinnamon-package-dependencies-for-artful-isorespin.log
sed -z 's/or\n/or /g' cinnamon-package-dependencies-for-artful-isorespin.log | tr -s ' ' | grep 'Depends: ' | sed 's/.*Depends: //' | sed 's/ but it is not installable//g' | sed 's/(.*)//' | sed 's/.*or \([^ ]*\) or.*/\1/' | xargs > cinnamon-package-dependencies-for-artful.txt
sed 's/ubuntu/cinnamon/' cinnamon-17.04-squashfs-root/usr/share/lightdm/lightdm.conf.d/50-ubuntu.conf > 50-ubuntu.conf 
wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb 

As I had already copied the other required local package and script files into the directory I then respun the Artful ISO using the command above creating the following (abridged for readability) log file:

Script '/usr/local/bin/isorespin.sh' called with '-i artful-desktop-amd64.iso -u -p wmctrl bluez-tools gnome-icon-theme-symbolic gir1.2-keybinder-3.0 gir1.2-javascriptcoregtk-3.0 gksu python-imaging python-gi-cairo python-pexpect st
reamer gnome-panel gnome-icon-theme gnome-icon-theme-symbolic inxi gist -p ssh openssh-server inxi -l google-chrome-stable_current_amd64.deb -l rtl8723bt_4.12.0_amd64.deb -f cinnamon-packages-isorespin/usr/src -f 50-ubuntu.conf -f l
inuxium-install-UCM-files.sh -f wrapper-linuxium-install-UCM-files.sh -f linuxium-install-broadcom-drivers.sh -f wrapper-linuxium-install-broadcom-drivers.sh -c mv /usr/local/bin/src/*.deb /var/cache/apt/archives -c rm -rf /usr/loca
l/bin/src -c apt-get install -y /var/cache/apt/archives/*.deb -c rm -f /var/cache/apt/archives/*.deb -c mv /usr/local/bin/50-ubuntu.conf /usr/share/lightdm/lightdm.conf.d -c wrapper-linuxium-install-UCM-files.sh -c wrapper-linuxium-
install-broadcom-drivers.sh -c sed -i 's?/usr/share/themes/Adwaita/backgrounds/adwaita-timed.xml?/usr/share/backgrounds/warty-final-ubuntu.png?' /usr/share/glib-2.0/schemas/org.cinnamon.desktop.background.gschema.xml -c glib-compile
-schemas /usr/share/glib-2.0/schemas -s 200MB' ...
Work directory 'isorespin' used ...
ISO '/usr/src/development/linuxium/isorespin/test/v7.1.0/artful-desktop-amd64.iso' respun ...
Bootmanager 'rEFInd' added ...
Kernel updated with mainline kernel version '4.12.0-041200rc6-generic' ...
Package 'wmctrl' added ...
...
Package 'gist' added ...
Package 'ssh openssh-server inxi' added ...
Local package '/usr/src/development/linuxium/isorespin/test/v7.1.0/google-chrome-stable_current_amd64.deb' added ...
Local package '/usr/src/development/linuxium/isorespin/test/v7.1.0/rtl8723bt_4.12.0_amd64.deb' added ...
Directory '/usr/src/development/linuxium/isorespin/test/v7.1.0/cinnamon-packages-isorespin/usr/src' added ...
File '/usr/src/development/linuxium/isorespin/test/v7.1.0/50-ubuntu.conf' added ...
File '/usr/src/development/linuxium/isorespin/test/v7.1.0/linuxium-install-UCM-files.sh' added ...
File '/usr/src/development/linuxium/isorespin/test/v7.1.0/wrapper-linuxium-install-UCM-files.sh' added ...
File '/usr/src/development/linuxium/isorespin/test/v7.1.0/linuxium-install-broadcom-drivers.sh' added ...
File '/usr/src/development/linuxium/isorespin/test/v7.1.0/wrapper-linuxium-install-broadcom-drivers.sh' added ...
Command run ...
# mv /usr/local/bin/src/*.deb /var/cache/apt/archives
# rm -rf /usr/local/bin/src
# apt-get install -y /var/cache/apt/archives/*.deb
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  build-essential dpkg-dev fakeroot g++ g++-6 gcc gcc-6 gir1.2-gkbd-3.0
...
  python-pam python-pyinotify python-setproctitle python3-setproctitle
Suggested packages:
  cinnamon-screensaver-x-plugin cinnamon-screensaver-webkit-plugin
...
  python-pam-dbg python-pyinotify-doc
Recommended packages:
  cinnamon-bluetooth
The following NEW packages will be installed:
  blueberry build-essential cinnamon cinnamon-common cinnamon-control-center
...
  python-pyinotify python-setproctitle python3-setproctitle xapps-common
0 upgraded, 75 newly installed, 0 to remove and 38 not upgraded.
Need to get 26.9 MB/44.4 MB of archives.
After this operation, 180 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu artful/main amd64 python-setproctitle amd64 1.1.10-1build1 [14.5 kB]
...
Get:42 http://archive.ubuntu.com/ubuntu artful/main amd64 policykit-1-gnome amd64 0.105-6ubuntu2 [24.0 kB]
Extracting templates from packages: 100%
Fetched 26.9 MB in 23s (1127 kB/s)
Selecting previously unselected package python-setproctitle:amd64.
(Reading database ... 204306 files and directories currently installed.)
Preparing to unpack .../00-python-setproctitle_1.1.10-1build1_amd64.deb ...
Unpacking python-setproctitle:amd64 (1.1.10-1build1) ...
...
Preparing to unpack .../74-policykit-1-gnome_0.105-6ubuntu2_amd64.deb ...
Unpacking policykit-1-gnome (0.105-6ubuntu2) ...
Setting up libquadmath0:amd64 (7.1.0-6ubuntu2) ...
...
Processing triggers for dbus (1.10.10-1ubuntu2) ...
# rm -f /var/cache/apt/archives/*.deb
# mv /usr/local/bin/50-ubuntu.conf /usr/share/lightdm/lightdm.conf.d
# wrapper-linuxium-install-UCM-files.sh
./linuxium-install-UCM-files.sh: Extracting UCM files ...
./linuxium-install-UCM-files.sh: Installing UCM files ...
./linuxium-install-UCM-files.sh: Reloading UCM driver ...
./linuxium-install-UCM-files.sh: Installation of UCM finished 
# wrapper-linuxium-install-broadcom-drivers.sh
./linuxium-install-broadcom-drivers.sh: Extracting Broadcom files ...
./linuxium-install-broadcom-drivers.sh: Installing Broadcom files ...
./linuxium-install-broadcom-drivers.sh: Reloading Broadcom driver ...
./linuxium-install-broadcom-drivers.sh: Installing Broadcom bluetooth service ...
./linuxium-install-broadcom-drivers.sh: Starting Broadcom bluetooth service ...
Running in chroot, ignoring request.
./linuxium-install-broadcom-drivers.sh: Installation of Broadcom finished 
# sed -i 's?/usr/share/themes/Adwaita/backgrounds/adwaita-timed.xml?/usr/share/backgrounds/warty-final-ubuntu.png?' /usr/share/glib-2.0/schemas/org.cinnamon.desktop.background.gschema.xml
# glib-compile-schemas /usr/share/glib-2.0/schemas
Persistence partition of '200MB' added ...
Respun ISO created as 'linuxium-persistence-v4.12-rc6-artful-desktop-amd64.iso'.

As you can see I used my 'isorespin.sh' script four times in total adding repos, installing packages, adding and running scripts, adding files and directories, downloading packages, running commands, outputting directories and of course respinning the ISO.

Please donate if you find the script useful using the following link http://goo.gl/nXWSGf as everything helps with development costs.

Wednesday, 21 June 2017

Customizing Ubuntu ISOs: Documentation and examples of how to use 'isorespin.sh'


Normally Linux distribution ISOs work perfectly when written to a USB for booting as a 'Live USB' allowing both usage and installation of the distro. Unfortunately with the introduction of Intel Atom based mini PCs and tablets the issue of requiring a 32-bit bootloader to boot a 64-bit OS arose. Further complications were caused by the lack of 'mainline' support for HDMI audio and wifi/bluetooth resulting in the need for custom kernels. Although some Linux distros can work OOTB I found that for Ubuntu it wasn't that simple.

To solve the limitations of using a standard ISO I developed a script that allows an ISO to be respun and customized ("remastered") to create a new ISO. Being a script which you run locally it means you can both see and control how your ISO is respun. I wanted flexibility in the ability to:
  • Add a GRUB 32-bit bootloader to allow ISOs to boot on the many Intel Atom devices limited by their BIOS or to add the rEFInd boot manager to allow booting on Intel Apollo Lake devices.
  • Upgrade the kernel to the latest or to a specific version to benefit from recent patch functionality e.g. for audio and wifi/bluetooth.
  • Add (or remove) repositories, software packages, scripts and files to allow the installation of firmware and favourite programs.
  • Preseed or perform set-up commands and scripts and also change system defaults.
  • Add persistence so that the ISO can retain data, installed software packages and settings across reboots.
The result is my 'isorespin.sh' script which works with all official desktop 64-bit Ubuntu (http://releases.ubuntu.com) and Ubuntu flavoured ISOs (https://www.ubuntu.com/download/ubuntu-flavours), Linux Mint ISOs (https://www.linuxmint.com/download.php), KDE neon ISOs (https://neon.kde.org/download), elementary OS ISOs (https://elementary.io), Peppermint OS (https://peppermintos.com/) and BackBox Linux (https://backbox.org/). And whilst Kali is not directly supported the Kali Metapackages (https://tools.kali.org/kali-metapackages) can be added when respinning an ISO.

It can be run using either the shell CLI (command line interface) in a the terminal or using its GUI (graphical user interface). Not only will it generate a new ISO but it will also produce a log file which includes the options used when respinning the ISO and serves to document the respun ISO.

Although it is expected to run the script on a Linux machine it also works on a Linux virtual machine on Windows (see below for further details). You will need certain packages like 'squashfs-tools' and 'xorriso' installed (use 'sudo apt-get install -y squashfs-tools xorriso' to install them) plus 'zenity' if you want to use the GUI. As different Linux distros have different packages installed by default the script will first check and notify you if any other packages are required. You will also need at least 10 GB of free space but this can be on external storage (e.g. a USB) as it can be specified as a location different to where the script is run from. For some options (such as updating the kernel) you will need a working internet connection as the script needs to download software. Also the script needs to be executable which can be achieved using the 'chmod' command ('sudo chmod 755 isorespin.sh'). Finally as the script runs some commands (e.g. mount/umount) which require root access using the 'sudo' command you will initially be prompted for your password and must already have 'sudo' privileges.


Invocation

Running the script is really quite simple even though there are quite a lot of features or options making it look rather complex.


Usage: /usr/local/bin/isorespin.sh [ -h | -v | --rolling-list ]
       /usr/local/bin/isorespin.sh -i <ISO> [ [ -u | -k <kernel> ] | -r "<repo> ... " | -p "<pkg> ... " | -l "<pkg.deb> ... " | -f "<file> | <directory> ... " | [ -s <size>MB | GB ] | [ -b GRUB | rEFInd ] | ...
       /usr/local/bin/isorespin.sh ... -w <directory> | -d "<pkg> ... " | -e "<pkg> ... " | -c "<cmd> ... " | -o "<file> | <directory> ... " | -g "" | "<kernel boot parameter> ... " | ...
       /usr/local/bin/isorespin.sh ... --apollo | --atom | ...
       /usr/local/bin/isorespin.sh ... --rolling-release | --rolling-release-hwe | --rolling-release-hwe-edge | --rolling-proposed | --rolling-proposed-hwe | --rolling-proposed-hwe-edge | ...
       /usr/local/bin/isorespin.sh ... --rolling-testing | --rolling-testing-hwe | --rolling-testing-hwe-edge | --rolling-unstable | --rolling-unstable-hwe | --rolling-unstable-hwe-edge | ...
       /usr/local/bin/isorespin.sh ... --upgrade | --key  "<repo> ... " ]


Respinng an ISO is controlled by either running the script with flags and arguments (CLI) or selection options and values (GUI) and briefly these are as follows:

-h or --help will display the 'usage' message. 
-v or --version displays the version of the script. 
-i or --iso must be specified and is the name (including the full or relative path) of the ISO to be respun. 
-u or --update will update the kernel to the latest Ubuntu Kernel Team kernel builds located at http://kernel.ubuntu.com/~kernel-ppa/mainline
-k or --kernel replaces the kernel with the version specified by the argument or value which can be any of the Ubuntu Kernel Team kernel builds located at http://kernel.ubuntu.com/~kernel-ppa/mainline and is passed as the directory or folder name without the trailing '/' (e.g. '--kernel v4.11-rc7').
--rolling-list shows what new kernels are available for the kernel types of release, proposed, testing and unstable and which can be installed using the '--rolling-' option. All 'rolling' options need the 'curl' package to be installed. 
--rolling-release will update the kernel to the latest kernel for that specific release. 
--rolling-release-hwe updates the kernel to the latest HWE (Hardware Enablement) or LTS Enablement Stack kernel for an LTS release. 
--rolling-release-hwe-edge updates the kernel to the upcoming HWE Stack kernel if available. 
--rolling-proposed updates the kernel to the proposed next release kernel from the proposed repository if available. 
--rolling-proposed-hwe updates the kernel to the proposed next LTS HWE kernel from the proposed repository if available. 
--rolling-proposed-hwe-edge updates to the proposed next upcoming LTS HWE kernel from the proposed repository if available. 
--rolling-testing updates the kernel with the latest kernel from the pre-release and test kernel repository if available. 
--rolling-testing-hwe updates the kernel with the latest LTS HWE kernel from the pre-release and test kernel repository if available. 
--rolling-testing-hwe-edge updates the kernel with the latest upcoming LTS HWE kernel from the pre-release and test kernel repository if available. 
--rolling-unstable updates the kernel with the latest kernel from the unstable repository. 
--rolling-unstable-hwe updates the kernel with the latest LTS HWE kernel from the unstable repository if available. 
--rolling-unstable-hwe-edge updates the kernel with the latest upcoming LTS HWE kernel from the unstable repository if available. 
-r or --repository adds the specified repository to the respun ISO. 
-p or --package will install a single package or a set of packages (if enclosed in quotes) that are available to the ISO and would typically be installed using the 'apt-get install <package>' command. This option is very useful in preseeding your ISO with packages you always normally install immediately after booting and will help resolve the issue of the ISO running out of space as a result of multiple installations. For example I typically include "ssh openssh-server inxi" when respinning an ISO for personal use.
-l or --local allows local Debian binary packages to be installed within the respun ISO. Care must be taken in specifying the order of multiple packages to preserve any dependencies and of course all dependencies must be met for the packages to be successfully installed. 
-f or --file will copy either the specified files or directories to the respun ISO under '/usr/local/bin'. Although intended to allow local binaries or shell scripts to be saved on the respun ISO there is no limitation on the actual type of file that can be added. 
-s or --storage adds a specified amount of persistent storage to the respun ISO. A size must be specified although there is some flexibility on how much is allocated. The minimum size is 128MB which keeps the size of the ISO down but means that once a USB is created the persistence partition needs to be manually resized for normal usage. Alternatively a larger size can be specified to avoid the immediate need to resize as this can always be performed later. It is recommended to use a realistic amount somewhere between 1GB and 2GB to maintain the balance between a practical but usable ISO. When creating a respun ISO with persistence the rEFInd boot manager will automatically be selected for booting the ISO although it will also need to be selected through the '-b' option if requiring the ISO to also install rEFInd. When using the GUI the persistence partition size can be defined using the slider which limits the size between 128MB and 2048MB.
-b or --boot will add either the GRUB boot loader (which is added by default) or the rEFInd boot manager which currently is required for booting on a lot of the current Intel Apollo Lake devices. 
-w or --work-directory is used to specify where the ISO will be respun. This is primarily for overcoming the lack of space associated with mini PCs and allows the respinning to be run on temporarily connected external storage media such as a drive or USB. Once the storage has been connected and mounted it can be passed as a parameter. Likewise if a directory is then created on the mounted storage this can be passed as the argument or value for this option to ensure there is sufficient space (i.e. at least 10 GB) for the script to run.
-d or --download will download Debian binary packages from the respun ISO using the 'apt-get download' command and save them under '/usr/src'. If combined with the '-o' option the Debian binary package will be accessible after respinning the ISO. 
-e or --erase will remove Debian binary packages from the respun ISO using the 'apt-get purge' command. 
-c or --command will execute the supplied argument as a 'bash' command. Care must be taken when embedding single or double quotes within the argument as they are subject to interpretation. For more complex commands it is often both advisable and easier to write a simple shell script and use the '-c' option to execute it as part of respinning. 
-o or --output will save either the specified files or directories to an output directory called 'isorespin'. This is very useful when wanting to extract files or packages that have been created or downloaded as part of respinning the ISO. 
-g or --grub will add the specified kernel boot parameters both to the GRUB/rEFInd boot menu and to the default GRUB settings or if an empty ("") argument is specified then the the ISO's initial kernel boot parameters will be removed.  
--atom include the flags, packages, scripts and commands that I recommend when respinning an ISO for a device with an Intel Atom (Bay Trail or Cherry Trail) processor and mirrors the manual invocations of  '-l rtl8723bX_4.12.0_amd64.deb -f linuxium-install-UCM-files.sh -f wrapper-linuxium-install-UCM-files.sh -f linuxium-install-broadcom-drivers.sh -f wrapper-linuxium-install-broadcom-drivers.sh -c wrapper-linuxium-install-UCM-files.sh -c wrapper-linuxium-install-broadcom-drivers.sh'. This option requires the 'curl' package to be installed. 
--apollo include the flags, packages, scripts and commands that I recommend when respinning an ISO for a device with an Intel Apollo Lake processor and mirrors the manual invocations of '-b rEFInd'. This option requires the 'curl' package to be installed.
--key adds GPG keys to the APT keyring on the respun ISO. Only available as a CLI option.
--upgrade performs an 'apt-get upgrade' on ISO's packages. Only available as a CLI option.
The script can either be run as a local script or it can be installed by copying the script to '/usr/local/bin' where it will be accessible for all users after the script has been made executable using the 'chmod' command ('sudo chmod 755 isorespin.sh'). Depending on the options and their complexity the respinning will take quite a few minutes and the script provides updates on its progress whilst running. The respun ISO is created with a new name derived from the original ISO name and includes indicators of the options used like 'persistence' or the kernel version for example with full details being written to the log file.

Once the ISO has been respun it can be written to a USB using the standard 'dd' command.

The examples below illustrate the various options through scenarios and provide a basic tutorial to using the script. They are based on the script being installed to '/usr/local/bin':


The script's help is displayed using the command:

isorespin.sh -h


The version of the script can either be displayed by entering:

isorespin.sh -v


or it can be seen from the top of the script's 'main' menu:


or 'advanced' menu:


The script can be run using the CLI by specifying options with arguments in any order. If the script is started without any parameters it first checks to see if it can run the GUI and if so starts with the main selection menu where all required options should be selected. As seen above the GUI uses two menus with the first being a main menu of most frequently used options and an optional additional menu of the more advance options. For each option selected, a sub-menu will appear to allow the choice or value to be entered.


Respinning an ISO

The simplest usage is to respin an ISO which will automatically add the GRUB 32-bit bootloader allowing the respun ISO to boot on Intel Atom devices e.g.:

isorespin.sh -i ubuntu-17.04-desktop-amd64.iso

or by invoking the GUI:





Upgrading the kernel

To respin the ISO with the latest kernel the command would be:

 isorespin.sh -i ubuntu-17.04-desktop-amd64.iso -u

or



If a specific kernel was required for example 'v4.11.6' then the '-k' option would be used:

isorespin.sh -i ubuntu-17.04-desktop-amd64.iso -k v4.11.6

or



Adding local packages and running scripts

A more complex scenario of respinning an ISO is when you want to add packages and run scripts to fix functionality like wifi/bt and audio.

Due to current limitations of the official ISOs I've previously created scripts to add wifi/bt firmware and also add the ALSA UCM audio files:

rtl8723bs_4.12.0_amd64.deb
rtl8723bt_4.12.0_amd64.deb
linuxium-install-broadcom-drivers.sh
linuxium-install-UCM-files.sh

These can be incorporated when respinning an ISO. For the scripts you first add the file using the '-f' option and then execute the script using the '-c' option.

When I wrote the above scripts I originally intended them to be run post installation rather than as part of respinning an ISO. As a result they produce warnings and also need to be executed in the correct directory. Rather than writing a complex '-c' parameter to run them correctly it is simpler to write a further script (which I call a 'wrapper') which controls how the script is run. These 'wrappers' can then be included as part of respinning an ISO:

wrapper-linuxium-install-broadcom-drivers.sh
wrapper-linuxium-install-UCM-files.sh

For example the following command will respin the Ubuntu 17.04 ISO and upgrade the kernel to the latest version (which includes the RTL8723BS driver) installs the required RTL8723BS wifi/bt firmware and both adds and runs a script to include the UCM files for audio:

isorespin.sh -i ubuntu-17.04-desktop-amd64.iso -u -l rtl8723bs_4.12.0_amd64.deb
-f linuxium-install-UCM-files.sh -f wrapper-linuxium-install-UCM-files.sh
-c wrapper-linuxium-install-UCM-files.sh


While the script is running progress updates are displayed:


until it finally completes:


producing a log file which shows how the script was run:



A similar example but respinning the daily snapshot Ubuntu 17.10 (Artful) ISO and using the GUI starts by selecting the required options:


and then entering the values starting with the ISO. When selecting the ISO a 'file manager' styled window will be presented but only files ending with a '.iso' suffix will be displayed:


For the kernel upgrade option the choice is to upgrade to the latest mainline version:


Local packages are selected using the 'file manager' which only displays '.deb' files. To allow multiple entries you will be asked after each selection is made whether a further addition is required:



Next the two scripts are added using the 'file manager':





And finally the 'advanced' menu is displayed and after selecting the option to add a command:


the command is entered as a text string:


Now that all the selected options have been entered a confirmation screen is be displayed which shows the full command as it would look if the script had been run manually:


Having selected 'yes' the script runs and displays its progress:





until it finishes:


Similarly a log file is produced that shows how the script was run:



For a more complex example of using the script see Compiling your own kernel using 'isorespin.sh'.


Persistence

Adding persistence to an ISO is useful as it means that you can retain your data, installed software packages and settings between reboots. Persistence can be added as part of any ISO respinning simply by adding the '-s' option and defining the size of the storage to be permanently allocated in the ISO. As previously mentioned there is a balance between making it small and therefore resulting in a small ISO that requires resizing once booted (see below) verses a larger size and a corresponding larger ISO.

The following example respins the standard Ubuntu 17.04 ISO with the latest kernel and create a 2 GB persistence partition:

isorespin.sh -i ubuntu-17.04-desktop-amd64.iso -u -s 2 GB

Using the GUI it is created by:








As can be seen from looking at the size of the resulting ISO that it is large at around 3.8 GB but not too large to be totally impractical.


Kernel boot parameters

The following is an example showing both the removal of the default kernel boot parameters (quiet splash) and the inclusion of two new kernel boot parameters:

isorespin.sh -i ubuntu-17.04-desktop-amd64.iso -g "" -g "i915.fastboot=1 fbcon=rotate:1"

The default kernel boot parameters are first removed by selecting the delete option:


and the new kernel boot parameters are then added:


as text strings:



Once the script is running it will update the kernel boot parameters:


with the exact details being recorded in the log file:



Rolling kernels

The need for rolling kernels is primarily to address the shortfall that existing ISO kernels typically don't have the hardware support required for the latest devices. They ship with the 'kernel of the day' whereas the latest hardware tends to require, well, the latest kernel to fully work. Ubuntu have in part addressed this through their LTS Enablement Stacks. Another alternative to get 'newer' kernels is to enable the proposed repository however there is also a pre-release and test kernel repository together with the unstable repository where mainline kernels are migrated into Ubuntu as well as the upstream mainline kernels.


I've added an option '--rolling-list' to keeping track of what new kernels are available where with options to easily incorporate them when respinning an ISO. I've termed the kernel types as release, proposed, testing and unstable to reflect the repositories they are drawn from as described above. And if it is not abundantly obvious but any kernel other than the formally released ones are not encouraged for anyone needing a stable system or anyone who is not comfortable running into occasional or even frequent breakage.


Target processors


Respinning an ISO for a target processor includes the flags, packages, scripts and commands that I recommend for the ISO to work on devices and mirrors the manual invocations of:

For '--atom': -l rtl8723bX_4.12.0_amd64.deb -f linuxium-install-UCM-files.sh -f wrapper-linuxium-install-UCM-files.sh -f linuxium-install-broadcom-drivers.sh -f wrapper-linuxium-install-broadcom-drivers.sh -c wrapper-linuxium-install-UCM-files.sh -c wrapper-linuxium-install-broadcom-drivers.sh

For '--apollo': -b rEFInd

The files are automatically downloaded from the internet if they are not found as part of the command invocation and this should simplify respinning for those looking something quick and easy. Both these new functionality require that you have the 'curl' package installed prior to running the respin script however the script will check and warn you if it is missing.

The options have been included in the GUI:






Kali Metapackages


Kali offers Kali Metapackages which 'give you the flexibility to install specific subsets of tools based on your particular needs'. If the GPG key of 'ED444FF07D8D0BF6' is added (using the option '--key "adv --keyserver keyserver.ubuntu.com --recv-keys ED444FF07D8D0BF6") and the Kali repository is added (using the option '--repository "deb http://http.kali.org/kali kali-rolling main contrib non-free") these metapackages can be added as packages when respinning an Ubuntu ISO:


There are some restrictions/limitations. Unity isn't supported and I've found adding a GPG key to a 17.04 or 17.10 release fails. Additionally adding the 'kali-linux-full' package results in dependency issues. However it is possible to respin the recently released Ubuntu GNOME 16.04.3 and add 'kali-linux' and 'kali-linux-top10':

Script '/usr/local/bin/isorespin.sh' called with '-i ubuntu-gnome-16.04.3-desktop-amd64.iso --atom -u --key adv --keyserver keyserver.ubuntu.com --recv-keys ED444FF07D8D0BF6 --repository deb http://http.kali.org/kali kali-rolling main contrib non-free -p kali-linux -p kali-linux-top10' ...
Work directory 'isorespin' used ...
ISO '/home/linuxium/ubuntu-gnome-16.04.3-desktop-amd64.iso' respun ...
Bootloader 'GRUB' added ...
Kernel updated with mainline kernel version '4.13.0-041300rc3-generic' ...
Key 'adv --keyserver keyserver.ubuntu.com --recv-keys ED444FF07D8D0BF6' added ...
Repository 'deb http://http.kali.org/kali kali-rolling main contrib non-free' added ...
Package 'kali-linux' added ...
Package 'kali-linux-top10' added ...
Local package '/home/linuxium/isorespin/rtl8723bs_4.12.0_amd64.deb' added ...
File '/home/linuxium/isorespin/linuxium-install-UCM-files.sh' added ...
File '/home/linuxium/isorespin/wrapper-linuxium-install-UCM-files.sh' added ...
File '/home/linuxium/isorespin/linuxium-install-broadcom-drivers.sh' added ...
File '/home/linuxium/isorespin/wrapper-linuxium-install-broadcom-drivers.sh' added ...
Command run ...
# wrapper-linuxium-install-UCM-files.sh
./linuxium-install-UCM-files.sh: Extracting UCM files ...
./linuxium-install-UCM-files.sh: Installing UCM files ...
./linuxium-install-UCM-files.sh: Reloading UCM driver ...
./linuxium-install-UCM-files.sh: Installation of UCM finished 
# wrapper-linuxium-install-broadcom-drivers.sh
./linuxium-install-broadcom-drivers.sh: Extracting Broadcom files ...
./linuxium-install-broadcom-drivers.sh: Installing Broadcom files ...
./linuxium-install-broadcom-drivers.sh: Reloading Broadcom driver ...
./linuxium-install-broadcom-drivers.sh: Installing Broadcom bluetooth service ...
Created symlink /etc/systemd/system/multi-user.target.wants/brcmbt.service, pointing to /lib/systemd/system/brcmbt.service.
./linuxium-install-broadcom-drivers.sh: Starting Broadcom bluetooth service ...
Running in chroot, ignoring request.
./linuxium-install-broadcom-drivers.sh: Installation of Broadcom finished 
Respun ISO created as 'linuxium-v4.13-rc3-ubuntu-gnome-16.04.3-desktop-amd64.iso'.

Lubuntu is also supported:


and by adding 'kali-desktop-lxde' additional LXDE packages are included (note 'Other'):



Creating a LiveUSB

Once your respun ISO has been created you can write it to a USB using 'dd'. For example if your USB is '/dev/sdb' then enter:

dd if=linuxium-v4.12-rc6-ubuntu-17.04-desktop-amd64.iso of=/dev/sdb bs=4M

Make sure you select the correct device for the USB and always check first using commands like 'df', 'blkid' or 'lsblk' to confirm. ISOs can also be written to a USB using 'Rufus' on Windows if preferred.


Booting your respun ISO


Having created the USB with your respun ISO the following information may be useful when booting and running.

To boot your device select the USB's UEFI partition from the device's boot menu:


Normally you will then see a standard GRUB menu:



If the 'rEFInd' boot manger was installed (either through selection or automatically due to persistence) the following screen will be displayed for 64-bit bootloader devices:



otherwise for 32-bit bootloader devices the normal the GRUB menu screen is displayed and after selection a black screen with the following message is briefly displayed:

error: no suitable video mode found.
Booting in blind mode

before the normal Ubuntu splash screen appears and the device fully boots. This 'error' can safely be ignored.

Wifi issues

If your device uses a Broadcom wifi chip and wifi isn't working even after adding the firmware using my script then it may be missing an SDIO file. By running the command 'dmesg' immediately after booting and looking at the output if you see messages showing that the wifi firmware 'txt' file is missing it is possible to use the copy from the Windows driver that is stored in NVRAM. The file should look something like '/sys/firmware/efi/efivars/nvram-74b00bd9-805a-4d61-b51f-43268123d113' and you actually copy it using 'cat /sys/firmware/efi/efivars/nvram-74b00bd9-805a-4d61-b51f-43268123d113 > /lib/firmware/brcm/brcm/brcmfmac43241b4-sdio.txt' substituting the actual NVRAM filename on your device and the correct *sdio.txt name for your driver.

Bluetooth issues

On booting if bluetooth fails to work automatically and the firmware was installed using one of my scripts then the service can be manually started with the command:

sudo systemctl start rtl8723bsbt.service

or

sudo systemctl start brcmbt.service

depending on device's chip. If bluetooth is always required then the command could be included in '/etc/rc.local' to avoid this step. Note that this uses 'systemd' so earlier ISO releases without 'systemd' cannot use my scripts to provide bluetooth.


Resizing the persistence partition

Once booted use the 'gparted' command to resize the persistence partition. You will initially be prompted to fix the GPT to use all the available space so click 'Fix'


and then continue with the resizing:



NVRAM sync issues

Sometimes after installing your device only boots to a black screen.  Typically this happens when the NVRAM and ESP are out of sync. This can be easily fixed by performing the following:

1. Start with the device switched off.
2. Connect your LiveUSB.
3. Power on the device and press the relevant function key or key sequence to boot from the LiveUSB.
4. Open a terminal windows and enter:
type efibootmgr 
5. If the 'efibootmgr' is not installed enter:
sudo apt-get update && sudo apt-get install -y efibootmgr
6. Next remove any boot entries that may have been created through the earlier installs by entering:

      for BOOTENTRY in 0 1 2 3 4 5 6 7 8 9
      do
          sudo efibootmgr -b ${BOOTENTRY} -B
      done

7. Now create an Ubuntu boot entry by entering:
sudo efibootmgr -c -d /dev/mmcblk0 -p 1 -l \\EFI\\ubuntu\\grubx64.efi -L Ubuntu
8. Remove the LiveUSB and reboot from the newly created Ubuntu entry.


Note: If you have a dual boot installation then the ESP partition may be the second partition so modify the '-p 1' in the command above to '-p 2' or to whatever the ESP partition is. Likewise for some devices the eMMC is '/dev/mmcblk1' and not '/dev/mmcblk0' so again modify the command as appropriate. If your device uses a 32-bootloader then use 'bootia32.efi' rather than 'grubx64.efi' in the command. It also sometimes doesn't work first time and requires all the steps to be repeated which should then fix the issue.


Additional information regarding the script

Minimizing repeated downloads

The rEFInd boot manager (http://www.rodsbooks.com/refind) can be downloaded from https://sourceforge.net/projects/refind/files into the same directory as the script is run from to prevent unnecessary repeated downloads if this is a concern and will make processing quicker.

Avoiding new windows opening when respinning

To prevent the file manager opening a window each time the script mounts a temporary file system I have run the command 'gsettings set org.gnome.desktop.media-handling automount-open false' first on my system.

Lock file

The script creates a lock file '.isorespin.sh.lock' in the directory where it is run from in order to ensure exclusive access to the run-time environment. If the script or machine crashes this lock file may need to be removed in order to rerun the script and you will be prompted should this occur. However it is not recommended to run the script in parallel from multiple directories as it is both CPU and I/O intensive and severe performance degradation would occur.

Permissions

If when running the script you get the error:

bash: ./isorespin.sh: Permission denied

it means that the script needs to be made executable which can be achieved using the 'chmod' command ('sudo chmod 755 isorespin.sh').

Running the script on Windows

First download and install Oracle's VirtualBox on Windows and then create a Linux VM from a standard Ubuntu 64-bit desktop ISO and install to a VDI of fixed size (minimum 20GB). Start the new Ubuntu VM and install 'Oracle VM VirtualBox Extension Pack'. You can then download my 'isorespin.sh' script to respin the Ubuntu-based ISO of choice. By installing the extension pack you will also be able to write the respun ISO to a USB using 'dd' from within the Ubuntu VM. Unfortunately 'Bash on Ubuntu on Windows' using 'Windows Subsystem for Linux' does not support 'loop' devices which are required by my script so an Ubuntu VM is the only working alternative at this time.

Running the script on Arch Linux

Running the script on Arch Linux requires a modified PATH. The simplest way is to run the script as follows:

PATH=/usr/sbin:/sbin:/bin:$PATH ./isorespin.sh

Additionally to install the required packages it may be necessary to run:

sudo pacman -S cdrkit bc libisoburn squashfs-tools dosfstools

or similar depending on which packages are missing from you environment.


Miscellaneous issues not directly related to the script

The Linux mainline kernel

It is worth noting that certain releases of the mainline kernel include specific core functionality:

v4.9 includes the eMMC v5.0 driver required for accessing the internal storage in many devices.
v4.11 includes the HDMI audio driver.
v4.12 includes the RTL8723BS driver although you will still need the wifi/bt firmware.

Kernel freezes

Well known and well documented without a permanent fix or so it seems. Kernels after v3.16 seem to include something that caused them to randomly freeze on Intel Atom devices and then the whole system hangs. Subsequently an accepted workaround was recommended to limit the processor (CPU) to a certain power state or 'C-state' and if freezes are encountered then to pass 'intel_idle.max_cstate=1' as a boot parameter. Later kernels have included patches and the current state seems to be reliable enough to ignore the workaround. If however freezes still occur the easiest way to implement the workaround on an installed system is to edit the file '/boot/grub/grub.cfg' and modify to look like:


GRUB_CMDLINE_LINUX_DEFAULT="intel_idle.max_cstate=1"

Alternatively the following command can be entered:
sudo sed -i 's/\(GRUB_CMDLINE_LINUX=\)""/\1"intel_idle.max_cstate=1"/' /etc/default/grub
followed by:
sudo update-grub
and then a reboot as the change only needs to be made once typically following installation.

Micro SD card issues

Whilst micro SD cards have worked on Intel Atom Bay Trail devices it is only since the v4.7.2 kernel that Intel Atom Cherry Trail devices have had any form of success.

The typical error encountered is:

        mmc1: error -110 whilst initialising SD card
        mmc1: card never left busy state

and although most Class 10 and below cards from any manufacturer except Sandisk work now without problem some UHS cards still fail. I've found Samsung UHS Speed Class 3 and UHS Speed Class 1 work whereas similar Sandisk cards fail crashing the device sometimes when inserting or removing the card.

No sound from headphones or internal speakers

It may be necessary to first use the speaker controls to select the audio output before sound works. For audio on Lubuntu this will also require installing the package 'pavucontrol':

sudo apt-get install -y pavucontrol

and then using it to select the audio output e.g. HDMI. Unfortunately some devices still require additional audio drivers or specific device quirks that have not been incorporated into the mainline kernel so sound may only work over HDMI and not through internal speakers or via the headphone jack even though the UCM files have been added. This is part of ongoing Linux development and future kernel releases hopefully will include the drivers required for currently non-working devices.

Installing without an internet connection on devices with a 32-bit boot loader

Unfortunately the standard Ubuntu ISO requires an internet connection when installing on devices with a 32-bit boot loader otherwise the following error is encountered:
The 'grub-efi-ia32' package failed to install into /target/. Without the GRUB boot loader, the installed system will not boot.
My script solves this for Ubuntu ISOs as it will now add the required GRUB packages allowing the installation to successfully complete. However other Ubuntu ISOs including the Ubuntu flavours and distros based on Ubuntu (such as Linux Mint) etc. their installation without the internet on 32-bit devices is with varying success. In part it depends on the original ISO installation capability on 64-bit devices without the internet. For example as Lubuntu 17.04 cannot be installed on 64-bit devices without internet as the ISO does not contain '/pool' with required packages so likewise after respinning its installation on 32-bit devices is not possible. However Lubuntu 16.04.2 can be installed on both 64-bit and 32-bit devices after respinning (see http://cdimage.ubuntu.com/lubuntu/releases/16.04.2/release/lubuntu-16.04.2-desktop-amd64.list and http://cdimage.ubuntu.com/lubuntu/releases/17.04/release/lubuntu-17.04-desktop-amd64.list for detailed package information). Also for non-Ubuntu ISOs it depends on the availability of archived package versions as some ISOs are created with earlier GRUB package versions to those currently available. For example with Elementary OS the earlier packages are not available and it is not known where (or if) an archive copy is maintained so installing without the internet on 32-bit devices is not possible.

Suspend/Hibernate/Sleep issues

Suspend in general has been problematic on Intel Atom devices and while new patches are continually being applied to the mainline kernel it is currently not working or incorrectly working on a number of devices.

Brightness issues

Unfortunately as I don't have a tablet I cannot advise on the various screen related issues some users experience using Ubuntu. The best command appears to be 'xrandr' although its use is device specific. Many people have commented in previous posts with their solutions and I've included some of these below.

Most brightness suggestions include:

xrandr --output [device_name] --brightness floating_pvalue (0 <= x <= 1.0)

where you can determine your device by entering:

xrandr -q | grep "connected"

with the primary one being your device.

For example:

xrandr --output DSI-1 --brightness .${BRIGHTNESS}

where the variable BRIGHTNESS can take a value between 3 and 9.

Rotation issues

Some examples that users have posted include:

xrandr --output DSI-1 --rotate right
xinput set-prop <Device Touchscreen> 'Coordinate Transformation Matrix' 0 1 0 -1 0 1 0 0 1

and

xrandr -o right
xinput set-prop 'Goodix Capacitive TouchScreen' 'Coordinate Transformation Matrix' 0 1 0 -1 0 1 0 0 1

Also others have reported that screen rotation during boot works with 'fbcon=rotate:1'.

Touchscreen issues

These seem to be related to rotation issues and another example of a solution includes:

xinput set-prop "Silead GSLx680 Touchscreen" "Coordinate Transformation Matrix" 1 0 0 0 -1 1 0 0 1

which can also be set when matching InputClass section in 'xorg.conf' with:

Option "TransformationMatrix" "1 0 0 0 -1 1 0 0 1"

Installation on Intel Apollo Lake Devices

On hardware requiring the rEFInd boot manager to boot such as Intel Apollo Lake devices you will need to include the '-b refind' option when respinning the ISO to include the rEFInd boot manager when installing the respun ISO. Otherwise you will need to install the rEFInd boot manager manually (see http://www.rodsbooks.com/refind) after an installation as the ISO will only includes the installation of GRUB.


Reporting issues

Comments are welcome but when discussing an issue please include the name of your device and use 'pastebinit' or similar to post a URL to a copy of the output from 'dmesg' (which is basically a log of kernel messages).

To use the 'pastebinit' command (see https://help.ubuntu.com/community/Pastebinit) first install the command by entering 'sudo apt-get install -y pastebinit' and then to share the kernel messages log enter 'dmesg | pastebinit'. This will paste a copy of the output from 'dmesg' to Ubuntu's Pastebin and provide a URL to access it.

So when reporting an issue please include:
  • Outline of issue
  • Name of device
  • URL from pastebin (e.g. http://paste.ubuntu.com/23646692/)


Donate

Please donate if you find the script useful using the following link http://goo.gl/nXWSGf as everything helps with development costs.