Here are some suggestions and instructions for how to customize a bootable flash drive (either USB drive or SDcard).
This document is rather old. Nowadays you can use gui tools such as usb-creator-gtk. It seems to produce usable results. The gory details discussed here may be useful if you want to modify some aspects of the process.
A particularly useful customization is to implement a persistent read/writeable “auxiliary” partition on the flash drive, so you can easily save stuff that you want to persist across reboots. This includes crucial things like random seeds for the random number generator, your personal SSH keys, perhaps a hostname and/or IP address, et cetera.
We start with an existing flash-bootable .iso image, then modify it slightly. Note the contrast:
| Our discussion here focuses on the Ubuntu Live distribution image, which is flash-bootable and customizable. | Not all .iso images are bootable. Even ones that are CD-bootable may not be flash-bootable. For details on this, see section 10.5. |
| FWIW, certain other objects such as memtest86+.elf and/or memtest86+.bin are flash-bootable. | They are not so easily customizable, not with the methods discussed here. |
Customization can be done using only standard Linux command-line tools.
Plug in the flash drive. Presumably it will show up as "/dev/sd?" or "/dev/mmcblk?". You can look through /dev to find the device you want, or you can use the handy script given in section 11.
:; ./find-attached-disk.pl
For safety sake, define an environment variable $DISK:
export DISK # for the most-recently attached disk :; find-attached-disk.pl :; eval $(blink)$(find-attached-disk.pl | tail -1) :; set | grep ^DISK # or by hand: :; export DISK=/dev/sdX # possibly /dev/sdb or whatever
The point is, the name $DISK is resistant to typos. Without these safety measures, you would be only a single-letter typo away from wiping out the system disk on your machine.
Look at the size of the Live .iso file, so we can make the partition slightly bigger than that:
:; du -s ubuntu-16.04.1-desktop-amd64.iso 1135684 ubuntu-16.04.1-desktop-amd64.iso
The du command gives the size in kiB; drop three digits to get a nice slight overestimate of the size in MiB. Unless you are super-short on space, round up a little bit, so that later you can replace the .iso with a bigger one without having to reformat the disk.
Rounding up from 1136 to 1500 Mib is reasonable. Use cfdisk or the like to set up partitions. If it asks what type of partition-table to use, choose “dos”. Within the parition table, create partition #1 of this size, with parition-type 0xef (’EFI’). Create partition #2 to take up the rest of the space, with partition-type 0x83 (’Linux’).
:; cfdisk ${DISK}
Be sure to Write the setup to disk, by giving the Capital W command to cfdisk. When it asks you to confirm, answer “yes” using all three letters of the word.
Don’t use gparted, for multiple reasons: (1) It refuses to run without root privileges, even if the permissions on $DISK would allow it. This is unbelievably stupid. It is the opposite of safety. (2) Gparted won’t let you select a partition-type unless it knows how to format it. In particular, even though it recognizes type 0xef, it won’t let you create a partition of that type (presumably because doesn’t know how to format it). (3) It requires a windowing environment, whereas cfdisk runs just fine on a dumb terminal (using ncurses).
Check the work so far:
:; fdisk -l ${DISK}
Disk /dev/sdc: 1.9 GiB, 2021654528 bytes, 3948544 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc9b83b63
Device Boot Start End Sectors Size Id Type
/dev/sdc1 2048 3074047 3072000 1.5G ef EFI (FAT-12/16/32)
/dev/sdc2 3074048 3948543 874496 427M 83 Linux
Another check:
:; sfdisk -d ${DISK}
# partition table of /dev/sdc
unit: sectors
/dev/sdc1 : start= 2048, size= 3072000, Id=ef
/dev/sdc2 : start= 3074048, size= 874496, Id=83
/dev/sdc3 : start= 0, size= 0, Id= 0
/dev/sdc4 : start= 0, size= 0, Id= 0
Create a file system, and assign it the “aux” label:
mkfs.ext4 ${DISK}2 -L aux
partprobe ${DISK}
Also, here’s a useful trick to guarantee room for grub. This matters if the device had previously been used with no partition table, just one filesystem taking up the whole device, such as you could easily get by trying dd if=*.iso of=$DISK or the like.
:; dd if=/dev/zero of=${DISK} bs=512 seek=1 count=2047
Choose an appropriate operating system .iso image. Download it from the provider if necessary. A good example is the "Ubuntu Live" image.
It is not necessary to modify the Ubuntu Live image, but other images may require hybridization.
>> : 1; isohybrid --partok foo.iso # not needed for Ubuntu >> # ignore warning message about size >1024 >> # observe foo.iso updated /in place/
This teaches the image to handle being chainloaded on a non-optical medium, as opposed to being loaded natively from an optical medium. The --partok option teaches it to handle things when it occupies a partition rather than occupying the whole disk.
Copy the bits:
:; time dd if=ubuntu-16.04.1-desktop-amd64.iso of=${DISK}1 bs=8K
# takes about 7 minutes
The copy (using dd) does a treeemendous amount of read-ahead and write-behind. The write-behind cannot be interrupted, not even by kill -9.
If desired:
:; mount ${DISK}2 /mnt/aux # if not already mounted
:; cp $path_to/memtest86+.bin /mnt/aux/boot/
:; cp $path_to/memtest86+.elf /mnt/aux/boot/
Give the new system some randomness of its own ... in a couple of plain files, and in the grub environment:
:; mount ${DISK}2 /mnt/aux # if not already mounted
:; dd iflag=fullblock if=/dev/random bs=512 count=4 of=/mnt/aux/random.true
:; dd iflag=fullblock if=/dev/urandom bs=512 count=4 of=/mnt/aux/random.pseudo
:; ./mk-grub-random.sh
This is an important step. It is a first step in a three-step process. The second step is to get grub to offer the randomness to the kernel. This is done using the grub environment, in mk-grub-cfg.sh with help from mk-grub-random.sh.
The third step is to get the Linux /dev/random driver to do something useful with the randomness that is being offered. This step is not yet completed.
You have two options:
One option is to use the simple 440-byte chainloader provided by the syslinux-common package. This has the advantage of simplicity.
:; dd bs=440 count=1 conv=notrunc if=mbr/mbr.bin of=/$drive
Another option is to install grub on the flash drive. This has the advantage of providing a possible route to provide randomness to the kernel.
:; install -d /mnt/aux # create the mount-point
:; mount ${DISK}2 /mnt/aux
:; grub-install --root-directory=/mnt/aux ${DISK}
# takes about half a minute
Create a grub.cfg to tell grub what to do, and where to find the various images. The script to do this is shown in section 11.
:; mount ${DISK}2 /mnt/aux # if not already mounted
:; ./mk_grub_cfg.sh
According to the web, the grub.cfg file is supposed to need things like:
insmod part_msdos
insmod ntfs
insmod iso9660
However, experiments indicate that none of those are necessary with current versions of grub (mid 2015).
Bad things happen if you forget to do this.
:; umount ${DISK}2
It is convenient to use qemu to test the flash drive. You can use the handy script, as spelled out in section 11. Invoke it as:
:; /boot-usb-test.sh
I am aware that grub has a nice loopback feature. It means you could copy the .iso image to the flash drive as a plain file (rather than using dd to create an ISO-9660 partition). The advantage is that this would simplify layout of the flash drive. The disadvantage is that it would increase the boot-time complexity. Possibly it would increase the run-time complexity as well; I don’t know.
In any case, I prefer to unpack the thing once and for all, creating an ISO-9660 partition.
If you want to see the structure inside the .iso image, you can use something like this:
:; mount -o ro,loop ubuntu-16.04.1-desktop-amd64.iso /mnt/play
What you see in “/mnt/play” is what the target machine will see in “/” when it runs.
If grub fails with error messages bout “no such device” or “no such partition” don’t panic ... but don’t try to debug it.
The only cure I know is to start over. That is: unmount any partions associated with $DISK*, unplug the drive, plug it back in, and re-do all the steps outlined in the main part of this document.
Hint: If you copy bits to the drive while the kernel has parts of it mounted, things will get badly screwed up. The sync command won’t fix it. As far as I can tell, the only way to get un-screwed is to start over.
Some non-Linux folks use 0x96 to represent ISO-9660; reference: https://en.wikipedia.org/wiki/Partition_type
You could theoretically change it via:
:; sfdisk --print-id ${DISK} 1
:; sfdisk --change-id ${DISK} 1 96
However, things seem to work OK if we set it to type 0xef, aka EFI, so let’s do that. I’m not sure it matters; I don’t think grub or any of our other friends care very much about partition-type (aka partition-ID). If there is a filesystem on the partition, there does not seem to be much of a requirement for partition-ID to match the filesystem-type.
USB sticks typically come from the factory with a partition table formatted onto them (unlike floppy disks). There is only one partition. Sometimes there is a ten- or fifteen-megabyte chunk of unused space before the start of the partition; other than that, the partition covers the whole disk (excluding the usual 2048-sector boot area).
Even if you find a machine that allows a flash drive to emulate a DVD, don’t expect it to work anywhere else.
Even if you find a machine that allows a flash drive to emulate a floppy, don’t expect it to work anywhere else.
Even if you could “dd” a .iso image onto the flash drive, it wouldn’t be a very nice solution. The result would be an ISO-9660 filesystem that Linux would treat as permanently read-only. This would leave you no place to store auxiliary files.
Then you get the following message:
error: version too old for 32-bit boot (try with ‘linux16‘).
In all probability, the error message is misleading. Probably the .img file is not a linux kernel at all ... even if the "file" command says it has the appropriate magic for a linux kernel.
Note: The memtest86+.iso image exhibits this phenomenon.
To be bootable from USB, the kernel file needs to contain layers upon layers of USB drivers. Not all of them do. On top of that, except in trivial cases, the kernel needs to be able to find its root on the boot medium. So it needs to understand about partitions and filesystems. Ditto for MMC drives.
Scenario: You boot from a flash drive that contains an "Ubuntu Live" image. You get the following error message:
Failed to load COM32 file gfxboot.c32 boot:
All is not lost. At the boot prompt you can type:
live
or
live-install
and proceed with whichever you want to do.
More generally, at the boot: prompt you can type a tab to see a list of options.
The last time I looked, the options included:
live live-install check memtest hd mainmenu help
However, not all of them work. The last time I tried it, the "help" option returned to the COM32 error.
#! /bin/bash
<<\EoF cat > /mnt/aux/boot/grub/grub.cfg
#begin grub.cfg
set timeout_style=menu
set timeout=10
load_env
echo setting random.seed=${randomseed}
sleep -v 5
menuentry "Debian Live" {
set root='hd0,msdos1'
linux /casper/vmlinuz.efi persistent boot=casper noeject noprompt splash toram random.seed=${randomseed} --
initrd /casper/initrd.lz
}
menuentry "memtest86+.bin" { # works
insmod ntfs
set root='hd0,msdos2'
linux16 /boot/memtest86+.bin
}
menuentry "memtest86+.elf" { # works
set root='hd0,msdos2'
knetbsd /boot/memtest86+.elf
}
#end grub.cfg
EoF
|
#! /bin/bash
# Possible usage:
# ALTROOT=/mnt/aux/ mk-grub-random.sh
# I hate redirecting stderr ... but dd gives me
# no other way to reduce verbosity.
randomseed=$(2>/dev/null dd iflag=fullblock \
if=/dev/urandom count=1 bs=24 | base64)
# Note the 0: here.
# It means zero bits of physics entropy,
# i.e. pseudorandom, not true random
grub-editenv ${ALTROOT}/boot/grub/grubenv set randomseed="0:$randomseed"
|
#! /usr/bin/perl -w
# Read syslog to find when each USB or MMC drive was attached
# If it gets removed and re-attached, only the last one counts.
# If it was attached so long ago that the event is
# no longer visible in syslog, this approach fails.
# If there are more than one such devices, each one
# is shown, in order of time of attachment.
use strict;
use Symbol 'gensym';
use DateTime;
my @moname = ("Jan","Feb","Mar","Apr","May","Jun",
"Jul","Aug","Sep","Oct","Nov","Dec");
my $tz = DateTime::TimeZone::Local->TimeZone();
main: {
my %map = ();
my $sequence = -1; # keep track of chronological order
# Read /dev/disk/by-id
# Fairly robust way of finding all usb drives.
foo: {
my $dh = gensym();
my $dir= '/dev/disk/by-id';
my $simple = $dir;
$simple =~ s'/$'';
opendir($dh, $dir) || die;
# typical thing we are interested in:
#> /dev/disk/by-id/usb-SanDisk_Ultra_4C531001640417112401-0:0 -> ../../sdb
#> /dev/disk/by-id/mmc-SL64G_0x436a2d8a -> ../../mmcblk0
#
# and in contrast, not interested in:
#> /dev/disk/by-id/usb-USB_2.0_USB_Flash_Drive_2cf51c40bc0343-0:0-part1 -> ../../sdc1
while(my $entry = readdir $dh) {
if ($entry =~ m'^(usb|mmc)' && $entry !~ m'-part[0-9]*$') {
my $fn = "$simple/$entry";
my $target = readlink $fn;
if (defined $target) {
#### print "$fn -> $target\n";
my $rslt = {};
my @status = stat $fn;
my $ctime = $status[10];
$target =~ s'[./]*'';
$rslt->{drive} = $target;
$rslt->{date} = format_date($ctime);
$rslt->{seq} = $ctime;
$map{$target} = $rslt;
} # else not a symlink, should never happen
}
}
closedir $dh;
}
# sort so that they come out in chronological order:
for my $drive (sort {${$map{$a}}{seq} <=> ${$map{$b}}{seq}} keys %map) {
my $rslt = $map{$drive};
my $special = "/dev/$rslt->{drive}";
if (-b $special) {
my $cmd = "blockdev --getsz $special";
my $pipe = gensym();
open ($pipe, '-|', $cmd)
|| die "Cannot pipe from '$cmd' : $!\n";
my $size = <$pipe>;
if (close $pipe) {
$size /= 2; # convert from sectors to kiB
$size /= 1024; # convert to MiB
} else {
$size = '???'; # probably "permission denied"
# when trying to do --getsz
}
my $s1 = "DISK=$special";
my $s2 = "DISKsize__MiB=$size";
my $s3 = "DISKdate='$rslt->{date}'";
printf ("%-20s %-20s %s\n", $s1, $s2, $s3);
}
}
}
sub format_date{
my ($epoch) = @_;
my $dt = DateTime->from_epoch( epoch => $epoch, time_zone => $tz);
my $year = $dt->year;
my $month = $dt->mon; # 1-12 - you can also use '$dt->mon'
my $day = $dt->day; # 1-31 - also 'day_of_month', 'mday'
my $dow = $dt->day_of_week; # 1-7 (Monday is 1) - also 'dow', 'wday'
my $hour = $dt->hour; # 0-23
my $minute = $dt->minute; # 0-59 - also 'min'
my $second = $dt->second; # 0-61 (leap seconds!) - also 'sec'
my $doy = $dt->day_of_year; # 1-366 (leap years) - also 'doy'
my $doq = $dt->day_of_quarter; # 1.. - also 'doq'
my $qtr = $dt->quarter; # 1-4
my $ymd = $dt->ymd; # 1974-11-30
$ymd = $dt->ymd('/'); # 1974/11/30 - also 'date'
my $hms = $dt->hms; # 13:30:00
# $hms = $dt->hms('|'); # 13|30|00 - also 'time'
return "$moname[$month-1] $day $hms";
}
|
#! /bin/bash
# Note that usermode networking supports TCP and UDP but not ICMP. In
# particular, ping will not work.
# Useful hint: Sometimes it's nice to pass the "-snapshot" argument
# Note: requires "modprobe kvm-intel"
# which loads "kvm" as well as "kvm-intel"
if test -z "${USB}" ; then
1>&2 echo "Please export USB=/dev/something"
exit 1
fi
mounted=$( awk '{
if (match($1, "^'"${USB}"'") ) printf "%s ", $1;
}' /proc/mounts )
if test -n "$mounted" ; then
1>&2 echo "You probably want to unmount some stuff:"
for part in $mounted ; do
1>&2 echo " :; umount $part"
done
exit 1
fi
if ! lsmod | grep -w -q '^kvm_intel' ; then
1>&2 echo 'Requires KVM modules.'
1>&2 echo 'Hint: sudo modprobe kvm_intel'
exit 1
fi
qemu-system-x86_64 \
-hda ${USB} \
-boot order=c \
-enable-kvm \
-m 1024M \
-rtc base=utc \
-object rng-random,filename=/dev/urandom,id=rng0 \
-device virtio-rng-pci,rng=rng0 \
"$@"
|