Tuesday 25 April 2017

RTL8723BS wifi soon to hit mainline kernel

Bastien Nocera's RTL8723BS wifi github (hadess/rtl8723bs) has been merged into the kernel staging tree and as a result the compiling of the driver as an external module or running the driver on older kernels is no longer supported and the repository now only contains the history of the driver cleanup which is not available in the upstream-merged driver. As the final commit states:
The driver is available upstream now in the linux-next tree, in
the staging drivers section. All tests, bug reports should be
made to the linux-wireless mailing-list, using this upstream driver.
Consider the existence of this GitHub repository as historical
documentation. Patches that were shipped as part of this repository have
mostly been subsided and obsoleted by upstream kernel fixes.
Support for Bluetooth, and the firmware files are also in the process
of being made available upstream, and will be removed when that happens.
If all goes to plan it should appear in mainline kernel 4.12-rc1 whose merge window should open next week barring any last minute issues with the first builds appearing in early May.

I've consequently retired my script 'linuxium-install-rtl8723bs.sh' as we look forward to the upcoming mainline support.

Saturday 22 April 2017

Creating personalized Ubuntu, Mint and Debian ISOs for Intel Mini PCs

UpdateThis work is superseded by my 'isorespin.sh' script which can respin an official ISO suitable for use on Intel Atom devices.

Normally Linux distribution ISOs work perfectly when written to a USB for booting as a 'Live USB' allowing both usage and installation of the distro. Unfortunately with the introduction of Intel Atom based mini PCs the issue of requiring a 32-bit bootloader to boot a 64-bit OS arose. Further complications were caused by the lack of mainline support for HDMI audio and wifi resulting in the use of custom kernels. And now even with mainline support soon to eventuate there is still a lead time required by distributions before they incorporate the latest kernels.

To solve the issue of using a standard ISO but with a different kernel I developed a script that allows the ISO to be respun with the latest kernel build. But because new kernel builds are released weekly I extended the script to allow the respinning with a specified kernel.

However I always end up installing the same set of packages immediately after installation so I thought it would be useful if packages could be included during the respin. Equally adding locally compiled packages became another requirement as it allowed the respinning of ISOs to include custom kernels or third party applications like Google Chrome. 

Another limitation was that after installation I found myself wanting to run scripts to fix or add functionality like audio and wifi. So being able to incorporate these as part of a personalized ISO would simplify both the installation process and provide a more functional Live USB.

​The last issue I encountered was trying to boot an ISO on a new Apollo Lake mini PC. As the GRUB bootloader failed to boot Ubuntu I switched to using the rEFInd boot manager to allow Linux to boot itself. In doing so I solved another want I've sometimes wished for when using a Live USB namely the lack of persistence or the ability to save files such as screenshots or error messages that could still be accessed after rebooting. So I rewrote my script to include all these features. 

The invocation although command line based is really quite simple:

Usage: ./isorespin.sh [-h|-v] | [[-u | -k <kernel>] | -p "package_1 ... package_n" | -l "package_1.deb ... package_n.deb" | -a "file_1 ... file_n" | -r [-s <size><MB|GB>]] -i <ISO>

Functionality is achieved by calling the command with flags and arguments or options with parameters and these are as follows:
-u or --update will update an Ubuntu based ISO (e.g. Ubuntu or one of its flavours or Linux Mint) to the latest Ubuntu Kernel Team released mainline build. For a Debian based ISO (e.g. Debian Live or LMDE) using this option will update the current v3.16 kernel with the Canonical DKMS patches for HDMI audio and RTL8723BS wifi.
-k or --kernel replaces an Ubuntu based ISO's kernel with the one specified by the argument or parameter which can be any of the Ubuntu Kernel Team kernel builds located at http://kernel.ubuntu.com/~kernel-ppa/mainline (just use the directory or folder name without the trailing '/' as in '--kernel v4.11-rc7' noting that earlier kernels will probably not work well if at all).
-p or --package will install a single package or a set of packages (if enclosed in quotes) that are available to the ISO (which can be either Ubuntu or Debian based) and would typically would be installed using 'apt install <package>' command.
-l or --local allows local Debian binary packages to be installed within the respun ISO. Care must be taken in specifying the order of multiple packages to preserve any dependencies and of course all dependencies must be met for the packages to be successfully installed.
-a or --add allows local binaries or shell scripts to be saved under '/usr/local/bin' on the respun ISO although there is no limitation on the actual type of file itself so any file can be added if required.
-r or --refind will add the rEFInd boot manager which can be selected using the device's boot manager at boot time.
-s or --storage adds persistence to the respun ISO. Although a size must be specified there is some flexibility on how much is allocated. The minimum size is 100MB which keeps the size of the ISO down but means that once the USB is created the persistence partition should be manually resized for typical usage. Alternatively a larger size can be specified thus avoiding the immediate need to resize as this can always be performed later. It is recommended to use a realistic amount somewhere between 1GB and 2GB to maintain the balance between a practical but usable ISO. 
-i or --iso must be specified and is the location of the ISO to be respun.
Depending on the options and their complexity the respin will take a few minutes and the script provides updates its progress whilst running. The respun ISO is created with a new name derived from the original ISO name and includes indicators of the options used like 'persistence' or the kernel name or 'dkms' for example. It is recommended that at least 10GB of space is available for the script to function correctly (as the script requires a minimum of 10GB to run) and an internet connection is typically required for some options like kernel updates for example. The script runs on Debian/Ubuntu based distros and requires certain packages to be installed prior to running although it checks and notifies if anything is missing first before continuing. Also the rEFInd bootmanager (http://www.rodsbooks.com/refind) can be downloaded from https://sourceforge.net/projects/refind/files into the same directory as the script to prevent unnecessary repeated downloads if this is a concern and will make processing quicker.

Once the ISO has been respun it can be written to a USB using the standard 'dd' command. 

To perform a manual resize when using a USB with persistence first use the device's boot menu to select the USB and when the rEFInd screen appears click on 'F2' twice. A text line will appear at the top of the screen so use the arrow keys to move the cursor and delete the word 'persistence' or 'persistent' depending on whether it is an Ubuntu or Debian based ISO. This step actually disables persistence and is only required so that the partition is not auto mounted thus preventing resizing. Now press the enter key to boot. 


Once booted use the 'gparted' command to resize the persistence partition. You will be prompted to fix the GPT to use all the available space so click 'Fix' and then continue with the resizing. 

An example invocation is as follows:

./isorespin.sh -i ubuntu-17.04-desktop-amd64.iso -k v4.11-rc5 -a "../scripts/wifi/linuxium-install-rtl8723bs-binaries-for-4.11.0-rc5.sh ../scripts/linuxium-install-UCM-files.sh" -p "ssh openssh-server inxi" -l google-chrome-stable_current_amd64.deb -r -s 2GB

This will respin the latest Ubuntu 'Zesty' ISO and make the boot kernel the fifth release candidate of the latest development mainline kernel. It will add my script for installing wifi for the RTL8723BS chip on this specific kernel version under '/usr/local/bin' along with adding another script to install the UCM files required for headphone audio on certain Intel Atom mini PCs. It will also install the 'ssh', 'openssh-server' and 'inxi' packages along with 'Chrome' so they are all available for immediate use on a 'Live USB' and will be included as part of a standard installation. It will also include a persistence partition of 2GB and the resultant ISO produced as: 

linuxium@LINUXIUMONE:~$ ls -l linuxium-persistence-rEFInd-v4.11-rc5-ubuntu-17.04-desktop-amd64.iso
-rw-r--r-- 1 linuxium linuxium 4215292928 Apr 22 12:43 linuxium-persistence-rEFInd-v4.11-rc5-ubuntu-17.04-desktop-amd64.iso

The script currently supports all recognized desktop 64-bit Ubuntu and Ubuntu flavoured ISOs (https://www.ubuntu.com/download/ubuntu-flavours), Linux Mint and LDME ISOs (https://www.linuxmint.com/download.php), Debian 'Live install' ISOs (https://www.debian.org/CD/live), KDE neon ISOs (https://neon.kde.org/download), elementary OS ISOs (https://elementary.io) and Kali ISOs (https://www.kali.org/downloads). Any issues or improvement suggestions are welcome. It can be downloaded from (latest version) isorespin.sh.

Please donate if you find the script useful using the following link http://goo.gl/nXWSGf as everything helps with development costs.

Thursday 13 April 2017

Improved 'isorespin' with choice of kernels and Debian 'Live' ISO support

UpdateThis work is superseded by my 'isorespin.sh' script which can respin an official ISO suitable for use on Intel Atom devices.

Although Debian have released 'Live' ISOs for some time, I've always had problems getting them to boot on Intel Atom Bay Trail and Cherry Trail devices. But with the news of Canonical dropping Unity in favour of Gnome and with the rise in popularity of Linux Mint and their Cinnamon and Mate editions, I thought I'd revisit Debian and see whether my 'isorespin' script could be adapted to make a difference.

After modifying my script to successfully boot a Debian 'Live' ISO (see https://www.debian.org/CD/live for details and http://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid for downloadable images) I realised that as the kernel was version 3.16 the original Canonical Intel Atom DKMS packages could be applied to provide HDMI audio and RTL8723BS wifi.

Unfortunately it does mean that the same old problems are back in that HDMI audio only works with a device booting from a 64-bit bootloader (otherwise it is back to DSDT patching for 32-bit devices) and audio is flakey to the point of not being really usable on desktops like LXDE.

However it does mean that LMDE (Linux Mint Debian Edition) gains the critical functionality making it more useable.

At the same time I thought it would be interesting to be able to have a choice in what kernel an Ubuntu based ISO could be respun with. Not only does the Ubuntu Kernel Team release mainline builds but they also provide builds based on the 'drm' sub-tree and have recently (albeit again) included kernels built on 'drm-intel-nightly'. These are particularly useful is seeing what new functionality is further upstream than the mainline release candidate builds.

As well any Ubuntu mainline kernel build can be installed up to the latest release candidate.

I've included all this functionality into a new release of my 'isorespin' script. And as such I've had to change the invocation or syntax for running the script slightly.

Usage: ./isorespin.sh [-h|-v] | [-u | -k <kernel>] -i <ISO>

The (latest version of the) script can be downloaded from isorespin.sh and when executed as a command has new parameters that support both short and long form when defining options. Besides a '-h' or '--help' that shows the syntax and a '-v' or '--version' to display the version, the ISO that you want respun must now be specified by the '-i' or '--iso' flag. The ISO can either be an Ubuntu, Ubuntu flavour, Linux Mint or Debian ISO and can be modified by further options. The '-u' or '--update' flag will update an Ubuntu or Linux Mint ISO with the latest mainline kernel. But when used with a Debian or LMDE ISO it will update the kernel with the Canonical DKMS patches for HDMI audio and RTL8723BS wifi. The '-k' or '--kernel' option is only allowed for Ubuntu and Linux Mint ISOs and any of the Ubuntu Kernel Team kernel builds found at http://kernel.ubuntu.com/~kernel-ppa/mainline can be specified by using the directory/folder name without the trailing '/' (as in '--kernel v4.11-rc6'). Note that the earlier kernels will probably not work well if at all.

For anyone brave enough to try an installation of the non-Ubuntu ISOs you may encounter the usual issues with installing a bootloader. To help I've written an additonal script 'linuxium-install-bootloader.sh' that even if it doesn't work directly, the commands within the script should provide a good enough guide to allow you to install manually. Note that running the script requires an internet connection and the target Linux partition needs to be already mounted on '/target' with the target boot partition mounted on '/target/boot/efi'.

If you like the scripts then please donate using the following link http://goo.gl/nXWSGf as everything helps with development costs.

Wednesday 5 April 2017

isorespin.sh Now With Mint and Cinnamon

UpdateThis work is superseded by my 'isorespin.sh' script which can respin an official ISO suitable for use on Intel Atom devices.

Besides Ubuntu and flavours you can now respin Linux Mint ISOs and if required update them with the latest Canonical build of an upstream kernel and boot them on Cherry Trail and Bay Trail devices with either a 32-bit bootloader or a 64-bit bootloader.

To respin an existing Linux Mint ISO you will need to use a Linux machine with 'squashfs-tools' and 'xorriso' installed (e.g. 'sudo apt install -y squashfs-tools xorriso') and a working internet connection with at least 10GB of free space. Having downloaded an ISO (for example 'linuxmint-18.1-cinnamon-64bit.iso' simply download my (latest version of the) script isorespin.sh and run with the ISO as a parameter (e.g. './isorespin.sh linuxmint-18.1-cinnamon-64bit.iso'). Or to respin and update the kernel include a '-u' or '--update' option (e.g. './isorespin.sh --update linuxmint-18.1-cinnamon-64bit.iso').

For wifi with the RTL8723BS chipset you can either download and run linuxium-install-rtl8723bs.sh which will build the wifi and bluetooth drivers from source but requires a working internet connection or you can download and run linuxium-install-rtl8723bs-binaries-for-4.11.0-rc5.sh which installs prebuilt binaries from the same source but does not require the internet (Update: see more recent posts). This second script only works for the Canonical built upstream kernel defined by the script name.

For wifi using Broadcom chipsets you can download and run linuxium-install-broadcom-drivers.sh which should get wifi and bluetooth working for most of the popular Broadcom chipsets.

The bluetooth service requires 'systemd' to be installed which should be by default on recent ISOs otherwise a manual configuration is required to run the service.

If you have a headphone jack you can update the UCM files once you have an internet connection by downloading and running linuxium-install-UCM-files.sh which provides audio for devices with the 5640 and 5640 chipsets.

For more detailed information see my post Running Ubuntu with an Upstream Kernel on Intel Bay and Cherry Trail Devices and don't forget to donate using the following link http://goo.gl/nXWSGf as everything helps with development costs.