BeleniX

The Live CD Boot Sequence

• After the PC is powered on and the BIOS has completed it's work it loads the code located in the first sector of the boot device and passes control to it. In case of the hard disk this is known as the Master Boot Record and in case of a CD it is the Boot Sector. The x86 Bios uses a predefined order when searching for bootable devices with removable media typically being the first. So in my case the Bios loads the first sector from the CD and passes control to the code.


• The boot sector simply passes control to another larger program called the stage2 bootloader. Since OpenSolaris uses GRUB, I am using Grub's stage2_eltorito bootloader that is used to boot off cdroms and knows how to handle the High Sierra/iso9660 filesystem. This stage2 bootloader then loads and parses the /boot/grub/menu.lst file and displays the boot choices to the user.


• The menu entries in Grub when booting off a CD just specify the kernel to load and the initrd or the boot archive that contains the given kernel and basic necessary kernel modules. This kernel is expected to comply with Grub's multiboot specification. This is true of the Linux kernel. But in Solaris the multiboot implementation has been kept in a separate standalone program called /platform/i86pc/multiboot. This is essentially the secondary bootloader for Solaris. Once a Solaris menu entry is selected Grub loads the multiboot program and the boot archive into memory. The boot archive is a gzip-compressed file that contains a UFS filesystem image containing the barebones kernel/modules/drivers binaries. Grub then passes control to multiboot.


• The multiboot program is Solaris's secondary bootloader. Among other things it queries the boot args passed to Grub and initializes the memory allocator. The boot archive that was uncompressed and copied into RAM by Grub is a minimal Ramdisk having a UFS filesystem. Multiboot mounts this as the root filesystem and reads the kernel and krtld and performs a jump to the kernel's main. The multiboot source code can be viewed at cvs.opensolaris.org
  
  Before loading the kernel multiboot also reads the /boot/solaris/bootenv.rc file. On SPARC Solaris depends on the OBP (Openboot Prom) for a lot of configuration and device infomation. Since there is no OBP on x86, an OBP emulation is used and the configuration parameters are stored in /boot/solaris/bootenv.rc. One of the parameters "bootpath" holds the physical device node that contains the root filesystem. This is ultimately mounted by the kernel as "/". If the bootpath parameter is absent then the kernel assumes that the ramdisk created from the boot archive is the rootfs and mounts it under "/". This is the crucial bit of detail that enables us to ultimately have a LiveCD.


• Booting the kernel from the ramdisk is used in Solaris's install framework to load a basic filesystem into RAM and boot the OS off that. This is called the install miniroot that also contains all the binaries and utilities required by the installer. This enables the install process to eject CDROMS as and when their contents have been installed. This approach is also used to boot off the Failsafe miniroot and perform system recovery if needed.
  
  This approach also enables us to prepare a LiveCD. However the setup required for a LiveCD is slightly different than the Install/Failsafe miniroot. On a LiveCD we try to minimize the space used by the ramdisk and try to use as many of the binaries directly from the CD as possible.
  The approach is to mount the cdrom onto some directory and loopback mount other directories in the cd onto appropriate points in the minimal rootfs. This enables all the utilities and apps to run directly off the CD.


• In order to achieve the objective mentioned above the CD should have a proper rootfs layout and should be mounted as early as possible in the startup process. In OpenSolaris the CD has to be mounted in one of the initial services that SMF loads. In order to identify the appropriate service I needed to get a complete chart of service startup order in SMF. While discussing this with a co-worker I was pointed to this excellent resource on Dan Price's Blog: More on Bootchart for Solaris
  
  Looking at the bootchart one can clearly see that net-physical runs first and then fs-root runs that checks the rootfs and remounts it since the rootfs would have been mounted read-only initialy by the kernel. It also mounts an optimized platform-specific libc over /usr/lib/libc.so.1. So fs-root should be the appropriate place to mount the CDROM and perform the other lofs mounts.
  
  But the problem here is that though /devices is available at this point, devfsadm is only executed by devices-local that runs after net-loopback: fs-root->net-loopback->devices-local. Without devfsadm being run, we do not have the /dev symbolic links in order to identify the /dev/dsk/* entries. After pondering this for sometime I decided to write a tiny utility that walks through all the minor device nodes and lists out the devfs path for all block and raw cdrom nodes. This output is parsed and "/usr/lib/fs/hsfs/fstyp -v" if executed on the raw node. Among other stuff this also prints the cdrom Volume ID which I compare with the volumeid stored in a file in the ramdisk. If the two matches then we have located our LiveCD. It then performs the other lofs mounts of /usr and /var/svc/manifest. In future I'd also need to add an lofs mount for /opt.


• After /usr is mounted and all the processing in fs-root service is over we run a small perl script to ask the user to set the keyboard layout for the system. It is a pain to work if the keyboard is screwed up. This Perl script uses the handy Perl-Curses and Perl-Curses-UI modules to put interactive screens by way of ncurses.


• When devices-local is executed, it runs devfsadm to create the /dev links. At this point we have a fully populated /dev/dsk. This allows us to perform an useful task. We step through all the /dev/dsk/* nodes and try to identify using "/usr/sbin/fstyp" whether it contains a supported filesystem or not (taking care to ignore the LiveCD mounted earlier).A supported filesystem is mounted automatically under an appropriate directory under /mnt. Something that happens when you boot one of the many Linux LiveCDs.
  
  In addition the code here contains another tweak reminiscent of Knoppix. It will actually try to find out whether we have a Solaris swap slice on the harddisk, and use that as the swap device. This is achieved by using "/usr/sbin/prtvtoc -h". Once we have found a /dev/dsk entry that is a ufs slice, we run prtvtoc on that. The output of prtvtoc contains information for all the slices in the VTOC to which this slice belongs. The slice type or tag numbered 3 indicates a swap slice. So we extract the slice number corresponding to tag 3 and generate the device node name thus: /dev/dsk/cNdNs{slice num}


• After the initial jugglery it is pretty much the standard service startup process that follows. Apart from the bootenv.rc entries another crucial element of booting from a CD involves reconfiguration. The "/reconfigure" file must be present in the boot archive to enable reconfiguration every time the CD is booted to ensure that all devices are setup properly. I was faced with some trouble in the service startup due to a missing service manifest called "sysidtool.xml". There are a bunch of network services that depend n this. Sysidtool is part of the Solaris installer and is not included in OpenSolaris as the install framework is still closed.
  
  So I ended up writing a new service manifest called "sysconfigure" having two instances called "sysconfigure:net" and "sysconfigure:system" as is the case for sysidtool. I also changed the dependent services to depend on sysconfigure.
  
  Presently "sysconfigure:system" does nothing and "sysconfigure:net" brings up all network interfaces using "ifconfig -a plumb". Subsequently it initiates DHCP on all interfaces that have the running flag set. There is a script called /etc/dhcp/eventhook that is invoked by dhcpagent when either a DHCP bind is successful or a DHCP release is successful. Upon a successful binding to DHCP address this script will query DNS and NIS settings from the DHCP server and setup the network configuration appropriately.


• Apart from the above SMF also does a manifest-import to refresh the repository if it detects a reconfiguration reboot. However this delays the boot and is unnecessary since I am already providing a prebuilt /etc/svc/repository.db in the ramdisk. The simple solution is to add a check for the livecd environment at the beginning of the manifest-import service and do ?exit 0? if this is the case.


• After I was able to achieve full startup of all services, I got an issue with inetd. At first I noticed that all the inetd services were failing to start with the error ?Property exec for method inetd_start of instance svc:/application/.../...:default is invalid? It took me a while before I could figure out what was wrong. All the inetd services have a method definition in their manifest that is named ?inetd_start?. Here is an example:
  

              <exec_method type="method"
                 name="inetd_start"
                 exec="/usr/lib/inet/in.chargend -s"
                 timeout_seconds="0"
                 delete="false">
              ...
  

  It appears that the command string given to the parameter exec="..." can contain variables and meta-characters to be expanded by the shell. In order to do the shell expansion inetd calls the libc function "wordexp(3C)". A quick scan through the wordexp implementation shows that it either calls /usr/bin/ksh or /usr/xpg4/bin/sh with the parameters "-^EuN" and passes the string passed to it by the caller. The shell is supposed to treat these parameters specially and just echo the expanded string. Ordinarily /usr/lib/libc.so.1 will call /usr/bin/ksh to perform the expansion unless it was compiled for xpg4 compliance in which case /usr/xpg4/bin/sh will be called. Now the problem is /usr/bin/ksh or ksh88 as we all know is closed source and not even redistributable in binary form. So it is absent from OpenSolaris.
  
  The open-source ksh93 does not support these options/feature though it can be coaxed to perform word expansion only using a combination of other options. Initially I tried to do that using a wrapper shell script that checks for the "-^EuN" options and execs ksh93 with the appropriate options. However inetd started exiting without any error message when I did that. Finally I noted that /usr/xpg4/bin/sh has most of the features of ksh including, of course, word expansion. Thus the easiest solution is to copy it into /usr/bin as ksh.


• Finally a note about swap is not out of place here though we already mentioned swap earlier Since OpenSolaris is booting from a CD with only "/" mounted read/write on the ramdisk, there is no separate swap device configured, unless the harddisk contains a Solaris partition with swap which has been detected and used by the devices-local tweaks. In any case service fs-usr will add physical swap and /tmp and /var/run are mounted on swap. In actuality anonymous pages are alocated dynamically for swap in the absence of a swap device.

Note: For more on Solaris x86 booting and multibooting refer to Solaris x86 Multiboot Site.

Other Customizations Required for LiveCD

• The Math Library libm.so.2 is a critical component of official Solaris. Many daemons/utilities depend on this library to work. However this library has not been included as part of the OpenSolaris source release which is one of reasons why OpenSolaris by itself is not self-hosting. That is OpenSolaris by itself cannot be booted. The solution here is to use the libm from FreeBSD. FreeBSD's math library itself has been donated by SUN way back in 1993.
  
  In the FreeBSD source hierarchy it is available in the src/lib/msun directory. However the sources in this directory depend on headers in other directories. To make this libm self-contained, I wrote my own script src/lib/msun/src/mk and copied the system headers into subdirectories of src/lib/msun. In addition, this version of libm is missing some functions like nan, nanf and nanl. Fortunately there are functions in libc called strtod, strtof and strtol that accept string representations of NAN and return the standard numeric equivalent. Thus I implemented the nan functions as simple wrappers that call libc's strtod and other functions.
  
  At present the mk script only builds a 32-bit libm. However support exists for compiling this libm for amd64 as well so I will need to write a 64-bit version of this script. The modified sources can be downloaded from http://sarovar.org/projects/belenix/


• OpenSolaris's Service Management Framework also has a dependency on Mozilla NSS libraries. These had to be compiled manually since it is bit tricky to get these libraries compiled on Solaris. I do not yet have a proper Makefile but hope to create one soon.


• I had to add a modified version of usr/share/lib/termcap file to support the sun-color terminal type. This is the default terminal type for OpenSolaris's workstation console.


• Since I am adding GNU screen to the list of included software, I included a default screenrc file that enables some nify features including a status bar.


• For boot configuration I have a modified boot/solaris/bootenv.rc, boot/grub/menu.lst and the boot/grub/os-choice.xpm.gz file that adds a custom Grub splash screen.


• Since I am compiling FreeBSD's libm with gcc I had to include the redistributable libgcc_s.so.1 in /lib.


• The file /reconfigure must be present in the ramdisk to ensure that device probing and reconfiguration is done every time. In a LiveCD reconfiguration must be done every time. The /noautoshutdown file must be present to ensure that power management dosen't suddenly shutdown the system.


• The biggest set of customizations obviously go to files in the /etc directory. These are too numerous to list exhaustively however the imporatnt ones can be listed here. Of all the files, it is critical to have driver_classes, driver_aliases, name_to_major, minor_perm and security/device_policy files. The driver_aliases associates device driver names with the PCI devices that it should control (via vendorid and deviceid numbers). name_to_major contains major numbers assigned to each device driver. minor_perm lists permissions and ownership of all minor device nodes under /devices. security/device_policy indicates security policy restrictions for device access - generally network drivers are allowed raw access. It should be noted that /devices is pseudo filesystem maintained by devfs and the files in it are device nodes whose permissions are derived from minor_perm. Without these files being properly setup many of the devices on the system will not be recognised. So I had to copy contents of these files from a properly working system and augment them whenever I included new drivers in the LiveCD. On an installed Solaris system, these files are actually maintained automatically via usr/sbin/add_drv and usr/sbin/update_drv. They are not supposed to be modified manually. However this does not work when one is building a LiveCD. Nothing is stored or updated on the harddisk. Everything that you store on the CDROM must be pre-configured.
  
  For more on devfs permissions read this
  For more of other devfs features read this
  


• OpenSolaris includes a Volume Management Daemon that manages all the removable media on a system including SD cards and CDROMs. It will try ot mount a cdrom automatically once it is inserted. But our LiveCD environment seems to unsettle Volume Manager as tries to mount the CDROM again even though it is already mounted in /.cdrom. So it actually ends up forcibly ejecting the CD even though it is mounted !! So the best route for me was to disable cdrom handling in Volumen Manager by editing the configuration file etc/rmount.conf.


• Last but not the least is to properly setup the etc/vfstab file that ensures proper mounting of all filesystems. One of the things to note is that the root filesystem is mounted with ufs logging (remember ext3/reiserfs journalling ?) disabled. We not have any need for journalling in a ramdisk as there is nothing to recover. This saves about 1MB of space in the ramdisk.

Optimizing space usage on the RamDisk

As we know by now that the LiveCD installs the root filesystem in a disk image loaded in the RAM and accessed via the ramdisk module. This ramdisk shares the available RAM with swap space and space for running processes. Thus it is absolutely imperative to minimize the size of the ramdisk as much as possible. After much effort I have been able to attain a ramdisk size of around 60MB. So here are the things I did:

• Remove all unnecessary files and binaries. /usr need not be present on the ramdisk since it is mounted directly off the CD. However by experimentation I have found a few binaries in /usr/kernel and /usr/bin required to be present. Other things required in the ramdisk are /var, /etc, /lib, /kernel (obviously!), /platform and /sbin.
• The files within /var/svc/manifest are relatively static and not read every time. So that directory need not be present.
• Many of the grub files under /boot/grub are not required and have been removed
• During boot, Service Management Framework creates a backup copy of it's service repository database in /etc/svc/repository.db. While this is useful in an installed system, for a LiveCD this is unnecessary. So there is a check for this backup copy in one service script which then deletes it. This saves me 2MB.
• As mentioned before, mounting the UFS filesystem in the ramdisk with nologging option (disables journalling) saves 1MB of space approximately.
• UFS by default reserves about 10% of space on the filesystem for root-only access. This is to allow recovery if an runaway user process tries to fill up / . No user process will be able to access space in this reserve. For a LiveCD this is not of much use, if something fills up / just reboot! This reserve amount can be tuned using the command "newfs -m " that initializes the UFS filesystem to reserve n% of space. I initialize the UFS filesystem that goes in the ramdisk using "newfs -m 0". This saves me around 6MB.
• The initial LiveCD is 32 bit only so removing all amd64 directories also saves quite a bit of space.
• In addition there are lots of files that are not relevant to booting or using a LiveCD but should be present when the LiveCD is installed to harddisk. All these files are collected together into a compressed file on he CDROM called archive.tar.bz2 .