Setting up a disk "offline" that will be put in a different computer as primary disk

April 5, 2011

Warning: This article was written in 2011, and is based on using Debian 5 or 6. A lot has changed in Debian and Ubuntu Linux regarding disk devices, GRUB, HAL (gone), and certain special cases in the filesystem. So while this may still give some ideas, you will need to research the details yourself.

A convenient way to set up a disk for another computer is to attach it to a Linux host computer (either via a USB interface, or using hot-swappable SATA), and from there format the disk and copy a filesystem onto it. This can also all be automated with some scripts.

Here's how I did it and got around some issues.

First, I have a python script detect and list avaliable removable hard disks disks using HAL.

Next, I partition it using sfdisk. An advantage of sfdisk over e.g. fdisk is that you can run it unattended from a script and give it input in this form:

# partition table of /dev/sde
unit: sectors

/dev/sde1 : start=     2048, size= 74219520, Id=83
/dev/sde2 : start= 74221568, size=  3942400, Id=82
/dev/sde3 : start=        0, size=        0, Id= 0
/dev/sde4 : start=        0, size=        0, Id= 0

I generate this from a template where I can replace the disk device name based on what you select during the install. I plan to eventually have this completely generated with the partition sizes all calculated from the disk size as reported by HAL.

You can create the partition table using fdisk or other familiar tool and make such an input file for sfdisk using sfdisk -d. You may be used to always starting the first partition at cylider 1 (default in fdisk), but I found that on some disks (namely an SSD I was using), the disk geometry is such that Grub's bootstrap program won't fit at the beginning of the disk -- it gives an error like this (this is from grub-install 1.98 on Debian 6):

/usr/sbin/grub-setup: warn: Your embedding area is unusually small.  core.img won't fit in it..
/usr/sbin/grub-setup: error: embedding is not possible, but this is required for cross-disk install.

I don't know how much room you specifically need for Grub, and disk geometry stuff confuses me. I just started the first partition at cylinder 2 in fdisk and that way it will almost certainly have more than enough room on almost any disk. Easy.

Next use mke2fs as usual to create the filesystem. This will be slow over a USB interface if you let it do the bad blocks check, be forewarned. (It's probably much faster over SATA directly, so do that if speed is important.)

Next mount the new filesystem and copy the files to it. Make sure that when you are copying filesystems around (I rsync it from a server) that you deal with symlinks correctly (preserve them, not follow them), and that you preserve or fix file ownership, mode (permissions), including the suid bit, timestamps, etc. I use find to make a manifest file that includes all this info for each file, and to fix it, originally used awk to process this manifest file and chown and chmod the files (the right options to commands like rsync and cp allow them to preserve timestamps and other stuff).

This also takes a long time if you do it for every file. Therefore instead of downloading the filesystem directly to the disk, I instead download it to a local cache on the host computer (to which the new disk is attached), and created a python script that compares the definitive manifest (with all correct file information) to the output of rsync to determine which specific files were updated by downloading from the server to a local cache, and fixed permissions there. So we can just download any changed files from the server to the local computer, fix file attributes of any that changed (if any), and then copy it to the target disk using cp with the right arguments to preserve file permissions.

(I was setting up multiple disk installation stations, with various versions of filesystems uploaded from developers for testing or approved for production use on the server; if you don't have multiple installation stations, you maybe don't need the extra server download step.)

Once you have the filesystem on the host computer ready to copy to the target disk, you just need to use cp with the right arguments to copy the files, you don't need to also re-fix the permissions on the target disk. Make sure to use numeric owner and group IDs, not symbolic ones that your host system doesn't have.

If neccesary, one way to make changes or apply updates to the filesystem using that target system's own tools is to use chroot. You can use the --bind option to mount to bring the host's /dev and /proc directories into the chroot, as mentioned in this page.

If using a recent OS such as Debian 5+, you will want to omit a certain file: /etc/udev/rules.d/70-persistent-net.rules. If this file is missing at startup, it is regenerated. Included in this file is a mapping from ethernet devices identified by MAC address to network interface name (eth0, eth1, etc.), so it is actually machine specific.

You should also regenerate the target machine's ssh host keys (stored in /etc/ssh as well.

Later versions or other distributions likely have other similar files that must be unique to a specific host and should be regenerated (or omitted from the filesystem if they will be automatically generated on first boot).

Once the filesystem is on the disk you can install the Grub bootloader. Make sure to use the same version Grub tools as the target OS uses. I did this by just copying them into a "grub-0.97" directory, and modifying the grub-install script to allow prefix to be set externally. (Debian 5 uses Grub "Legacy" 0.97, while my host Ubuntu system uses Grub 2.) /mnt/target is where I mounted the target disk, which is /dev/sdb. (Again, HAL can tell you what the device name is.)

  GRUB_PREFIX=/opt/grub-0.97/usr /opt/grub-0.97/usr/bin/grub-install --no-floppy --recheck --root-directory=/mnt/target /dev/sdb

To install Windows, you need to create the partitions, then clear the first 512 bytes from the disk (Windows boot sector) and install a Windows bootloader. I just use dd with if=/dev/zero bs=1 count=512 to clear it, and use a copy of a Windows XP boot sector as input with count=446 to write the boot loader (grabbed the first 446 bytes from a Win XP disk using dd to get the boot loader). You then need to write 0x125 0x252 as bytes 511 and 512 at the end of the boot sector for it to work.

Then I use ntfsclone to "restore" an NTFS image I previously grabbed off a disk with a working Windows XP Embedded system on it. You could also try creating an NTFS or DOS filesystem and mounting it and copying the files on as with Linux above.