Project

General

Profile

Home » Software choices » Software installation »

5. Master Node Installation

The nodes will be headless, this configuration enables to :
- Install software and data on every node, and to reset all of them to a clean state in 2 commands
- Install, Upgrade or change the root filesystem (clean installation) of every node in 2 commands
- Install, Upgrade or change the boot filesystem (firmware) of every nodes in 2 commands

Initial Raspbian installation

On my Linux workstation, I download the last Raspbian Unattended Installer from [[https://github.com/debian-pi/raspbian-ua-netinst/releases/latest]] and write it to the future master's SD card :

xzcat /home/fcerbell/Documents/Raspberry/raspbian-ua-netinst-v1.0.2.img.xz > /dev/mmcblk0

I place the SD card in the master's SD slot, connect it to the network (to internet), the power and wait for the process to complete (20-30 minutes). At the end, I have a Raspbian installed, with the latest kernel and packages. Its hostname is pi, the root password is raspbian. It is connected to my network with a dynamic DHCP address and has an SSH server listening allowing root connections. As I love to copy/paste command lines (from this page), I need to have both this page and a terminal on the RPi from my workstation. I need the RPi IP address, so I prepared my internal network's DHCP to give a fixed address to this RPi, I already know its IP. If I did not, I would have to connect a display to the HDMI connector and a keyboard to connect as root and ask for the IP address. Anyway, now, I can connect to the RPi from my workstation, with SSH, as root and I can do the installation with copy/paste the command from this page... I'm so lazy...

Master & Slaves : Upgrade the system

First, I install aptitude because I am more used to aptitude than to apt-get, then I check that the system is up to date regarding the repositories, the package lists and the installed packages :

apt-get install -y aptitude && 
aptitude update && 
aptitude safe-upgrade -y && 
aptitude dist-upgrade -y && 
aptitude full-upgrade -y && 
aptitude purge ~c -y && 
aptitude clean

Master & Slaves : Generate root's SSH keypair

Then, I generate the root's SSH passwordless key pair and I allow key-authentication for root to root. I also relax the checkings regarding the Host Keys (to avoid the question asking to add or not the remote host key during the first connection) and the DNS (to avoid DNS checkings that could slower the connections). As I generate it before making the snapshot that will be used as a template for the node's filesystem, it means that all the nodes will have the same root's key pair and will all be able to ssh to the others passwordless.

ssh-keygen -N "" -f /root/.ssh/id_rsa
cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys
echo >> /etc/ssh/ssh_config
echo "    StrictHostKeyChecking no" >> /etc/ssh/ssh_config
echo "UseDNS no" >> /etc/ssh/sshd_config

Master & Slaves : SSH authorized keys

I want to be able to connect directly as root from my laptops, so I also add my public keys to root's authorized_keys file (useful when opening a terminal to every node with cssh) :

echo "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8QIqdBsSN+X7p0yHbhtz4rYylElQBZQuFvx4nVMAFLW5I1ibuahmOSHsh2FRSaA+W99yGpToyB6w2Tk5dBte6h5QGnPJXwFZMBK9Xyc9L+iQDPyu0oQljpaB4fky9jsCpjtk5i6FldlcXLAeH8mb79eERxXxISZP+3mxA2udbPiffADq20idiVnd7zA0Ekl24283cooPPJXO7EPPNCBVzCxk1OhcYI91Pbjs2fY08lu2zaUsP7Fndx/IwS4k+FOYAnxTwhcvCH0g3qWv4VN+uDHoTG5l69XaCgZoD6W8gukk6yhrP4fd0ZyKcsZ9w7mo9XpLYLk9U8+I/u5vo8PCB francois@redislabs.com" >> /root/.ssh/authorized_keys
echo "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAtM8LzekUr46wvVNWoYzxPuKVTv7yFp+Aa/a1vKAendFa3xsMZz6Pp0Xn8U5ZYbTpqqVeM8O+ETqjtpBVk+7+C516DwB+R/cKulTjy061fBPZvTp5pIKm4+NQXNBhwjmQs//nWJ54PlDS5mHuj9NalX07b2OBztrvLjPzf/m4sB0= Francois Cerbelle" >> /root/.ssh/authorized_keys

echo -n '192.168.2.1 ' > /root/.ssh/known_hosts
cat /etc/ssh/ssh_host_ecdsa_key.pub >> /root/.ssh/known_hosts
echo -n ' root@(none)' >> /root/.ssh/known_hosts

Master : External hard drive to store repository mirrors and data

I need a storage to store the cluster files. I use a small hard drive, with one big partition, formatted in ext4 and connected to the USB port.

echo /dev/sda1 /data auto defaults 0 1 >> /etc/fstab
mkdir /data
mount -a

Master & Slaves : Puppet agent

apt-get -y update && 
apt-get -y upgrade && 
apt-get -y dist-upgrade && 
apt-get -y install puppet facter ruby-hiera augeas-tools &&
apt-get -y autoremove && 
apt-get -y clean && 
apt-get -y autoclean

cat << EOF | augtool
set /files/etc/puppet/puppet.conf/main/server rpinode01.raspicluster.cerbelle.net
set /files/etc/puppet/puppet.conf/main/pluginsync true
set /files/etc/puppet/puppet.conf/main/report true
set /files/etc/puppet/puppet.conf/main/report_server rpinode01.raspicluster.cerbelle.net
set /files/etc/puppet/puppet.conf/main/certname "rpinode01.raspicluster.cerbelle.net" 
set /files/etc/puppet/puppet.conf/user/waitforcert 120
set /files/etc/default/puppet/START no
set /files/etc/hosts/01/ipaddr 192.168.2.1
set /files/etc/hosts/01/canonical rpinode01.raspicluster.cerbelle.net
set /files/etc/hosts/01/alias[1] rpinode01
save
EOF

Master & Slaves : Raspberry Pi CPU tuning

First, I overclock the RPi, using the maximum allowed without breaking the warranty. Other values might set the "void warranty bit" in the card. These are sufficient (1GHz) and should not bring the RPi to its max temperature (85°C), even in a compact case. If it should happen, the RPi has an internal security which will slow it down until the temperature fall. NEVER change the max temperature threshold, it would break the warranty.

echo '#arm_freq=1000' >> /boot/config.txt
echo '#core_freq=500' >> /boot/config.txt
echo '#sdram_freq=600' >> /boot/config.txt
echo '#over_voltage=6' >> /boot/config.txt
echo 'gpu_mem=16' >> /boot/config.txt

I also install a library which is configured to be automatically preloaded (LD_PRELOAD) by the dynamic linker with every binary and which overloads the memcpy and memmove functions to optimize them on the RPi.

apt-get install -y raspi-copies-and-fills

Master & Slaves Internationalization

I choose to compile the en_US.UTF8 and to use it as default locale. But I choose to locate the RPi in the Europe/Paris timezone.

#dpkg-reconfigure locales
sed -i 's/.*en_US.UTF-8/en_US.UTF-8/' /etc/locale.gen
echo en_US.UTF-8 > /etc/default/locale
locale-gen
dpkg-reconfigure tzdata

Master & Slaves : Kernel downgrade

I had to install the linux kernel 3.12, the last Raspbian precompiled version including aufs.

apt-get install -y linux-image-3.12-1-rpi
cp /boot/initrd.img-3.12-1-rpi /boot/initrd.img
cp /boot/vmlinuz-3.12-1-rpi /boot/kernel.img 
sed -i 's/^initramfs.*/initramfs initrd.img/' /boot/config.txt
sed -i 's/^kernel=.*/kernel=kernel.img/' /boot/config.txt
reboot
apt-get purge -y linux-image-3.18.0-trunk-rpi

Master : Snapshot the base system as the future slave's root filesystem

Before taking the snapshot, Iclean the downloaded packages stored in apt cache, it is useless to replicate them on the slaves, they will be obsoletes and it will use bandwith/time to transfer them through the cluster network.

aptitude clean

Then, I create the root and boot filesystems copy as templates for the slaves.

rm -Rf /data/slaves/{rootfs,bootfs}
mkdir -p /data/slaves/{rootfs,bootfs}
cp -ax / /data/slaves/rootfs/
cp /boot/*.{bin,dat,elf} /data/slaves/bootfs/

And I remove the files that were automatically created during the previous steps and which are irrelevant to the slave nodes. The slave's root account does not need my installation bash history and, more important, they will have a different ethernet (MAC) address, so I make the system forget the MAC mapping to the NIC number (eth0, eth1, ...) :

rm /data/slaves/rootfs/root/.bash_history
rm /data/slaves/rootfs/etc/udev/rules.d/70-persistent-net.rules

Slaves : Disk configuration

The slaves will not have any hard drive connected so I have to remove the related line in /etc/fstab and they will not need the / and /boot mount neither because they will be mounted during the initrd, so I disable (comment) all these lines from the file :

sed -i 's~^/.* / .*$~#&~' /data/slaves/rootfs/etc/fstab
sed -i 's~^/.* /boot .*$~#&~' /data/slaves/rootfs/etc/fstab
sed -i 's~^/.* /data .*$~#&~' /data/slaves/rootfs/etc/fstab
rmdir /data/slaves/rootfs/data

Slaves : Networking configuration

I cleanup the /etc/hosts file, no need for 127.0.1.1 to resolve its own name as each node will have a DHCP assigned IP address with a DNS server resolving this external IP to the right name. Furthermore, I add a custom script to update the files and the running system with the right hostname as soon as the DHCP assign an IP address to the node. Basically, the node will have no name when it boots, as there is none in the configuration files, but everything will be automatically updated in files and in memory from the DHCP answer.

cat > /data/slaves/rootfs/etc/hosts << EOF
127.0.0.1       localhost
EOF
cat > /data/slaves/rootfs/etc/dhcp/dhclient-exit-hooks.d/hostname << EOF
#!/bin/bash

if [ \$reason = "BOUND" ]; then
    oldhostname=\$(hostname -s)
    if [ \$oldhostname != \$new_host_name ]; then
        echo \$new_host_name > /etc/hostname
        hostname -F /etc/hostname
    fi
fi
EOF

Master : PuppetMaster

All the remaining configuration (common to each node, specific to master and specific to slaves) will be handled and managed by puppet. So, I need to install a first standalone puppet master. It will use the embedded WebRicks to serve the incoming connections. Even if WebRicks is mono-threaded, it is not an issue as it will have to serve the master's puppet agent first with a "master configuration". This master configuration will begin to manage the master's puppet master installation and will configure it with Apache and Passenger (mod_rails). So, this very first puppet master installation is only intended to be used only once.

apt-get install -y puppetmaster
/etc/init.d/puppetmaster stop
cat << EOF | augtool
set /files/etc/default/puppetmaster/START yes
set /files/etc/puppet/puppet.conf/master/autosign true
set /files/etc/puppet/puppet.conf/master/allow_duplicate_certs true

# Puppet 3.x
#set /files/etc/puppet/puppet.conf/main/modulepath /etc/puppet/modules:/etc/puppet/site-modules:/usr/share/puppet/modules
set /files/etc/puppet/puppet.conf/main/environmentpath /etc/puppet/environments
set /files/etc/puppet/puppet.conf/main/basemodulepath /etc/puppet/modules:/etc/puppet/environments/production/site-modules:/usr/share/puppet/modules
save
EOF
mkdir -p /etc/puppet/environments/production/manifests
apt-get install -y mercurial
hg clone --insecure https://www.cerbelle.net/hg/puppet-modules /tmp/puppet-modules
/etc/init.d/puppetmaster stop
mv /tmp/puppet-modules /tmp/puppet
mkdir -p /etc/puppet/environments/production/manifests
cp -r /tmp/puppet/modules /etc/puppet/environments/production
cp /tmp/puppet/manifests/default.pp /etc/puppet/environments/production/manifests/site.pp
cp -r /tmp/puppet/hieradata /etc/puppet/
cp /tmp/puppet/hiera.yaml /etc/puppet/
ln -s /etc/puppet/hiera.yaml /etc/                                                                                                                            
rm -Rf /tmp/puppet
/etc/init.d/puppetmaster start

Master : Network configuration

sed -i 's/.*dhcp.*/#&/' /etc/network/interfaces 
cat >> /etc/network/interfaces << EOF
iface eth0 inet static
    address 192.168.2.1
    network 192.168.2.0
    netmask 255.255.255.0
    gateway 192.168.2.254
    broadcast 192.168.2.255
EOF
echo 'rpinode01' > /etc/hostname
hostname -F /etc/hostname
cat > /etc/hosts << EOF
127.0.0.1       localhost
192.168.2.1 rpinode01.raspicluster.cerbelle.net rpinode01
192.168.2.2 rpinode02.raspicluster.cerbelle.net rpinode02
192.168.2.3 rpinode03.raspicluster.cerbelle.net rpinode03
192.168.2.4 rpinode04.raspicluster.cerbelle.net rpinode04
192.168.2.5 rpinode05.raspicluster.cerbelle.net rpinode05
192.168.2.6 rpinode06.raspicluster.cerbelle.net rpinode06
192.168.2.7 rpinode07.raspicluster.cerbelle.net rpinode07
192.168.2.8 rpinode08.raspicluster.cerbelle.net rpinode08
192.168.2.9 rpinode09.raspicluster.cerbelle.net rpinode09
192.168.2.10 rpinode10.raspicluster.cerbelle.net rpinode10
192.168.2.11 rpinode11.raspicluster.cerbelle.net rpinode11
192.168.2.12 rpinode12.raspicluster.cerbelle.net rpinode12
192.168.2.13 rpinode13.raspicluster.cerbelle.net rpinode13
192.168.2.14 rpinode14.raspicluster.cerbelle.net rpinode14
192.168.2.15 rpinode15.raspicluster.cerbelle.net rpinode15
EOF
cat > /etc/resolv.conf << EOF
domain raspicluster.cerbelle.net
search raspicluster.cerbelle.net
nameserver 192.168.2.254
EOF

Master : DHCP, DNS and TFTP server with DNSMasq

apt-get install -y dnsmasq
/etc/init.d/dnsmasq stop
echo 'conf-dir=/etc/dnsmasq.d' >> /etc/dnsmasq.conf
cat > /etc/dnsmasq.d/rpicluster << EOF
dhcp-range=192.168.2.20,192.168.2.250,12h
dhcp-host=b8:27:eb:cb:63:18,,rpinode01,192.168.2.1,1h
dhcp-host=b8:27:eb:c2:9f:19,,rpinode02,192.168.2.2,1h
dhcp-host=b8:27:eb:27:b6:53,,rpinode03,192.168.2.3,1h
dhcp-host=b8:27:eb:b9:61:6d,,rpinode04,192.168.2.4,1h
dhcp-host=b8:27:eb:b6:57:a0,,rpinode05,192.168.2.5,1h
dhcp-host=b8:27:eb:ef:75:38,,rpinode06,192.168.2.6,1h
dhcp-host=b8:27:eb:2b:41:91,,rpinode07,192.168.2.7,1h
dhcp-host=b8:27:eb:0c:25:82,,rpinode08,192.168.2.8,1h
dhcp-host=b8:27:eb:e5:83:f5,,rpinode09,192.168.2.9,1h
dhcp-host=b8:27:eb:c5:5e:e0,,rpinode10,192.168.2.10,1h
dhcp-host=b8:27:eb:f2:55:fb,,rpinode11,192.168.2.11,1h
dhcp-host=b8:27:eb:0c:dc:64,,rpinode12,192.168.2.12,1h
dhcp-host=b8:27:eb:22:97:40,,rpinode13,192.168.2.13,1h
dhcp-host=b8:27:eb:e3:00:59,,rpinode14,192.168.2.14,1h
dhcp-host=b8:27:eb:19:dc:95,,rpinode15,192.168.2.15,1h
dhcp-option=6,192.168.2.1,192.168.2.254 # dns-server
dhcp-option=3,192.168.2.254 # router
dhcp-option=15,raspicluster.cerbelle.net # domain-name
dhcp-option=66,192.168.2.1 # tftp-server
dhcp-option=69,192.168.2.1 # smtp-server
dhcp-option=119,raspicluster.cerbelle.net # domain-search
enable-tftp
tftp-root=/data/slaves/tftp
EOF
mkdir -p /data/slaves/tftp
chmod -R u+rw,g+r,o+r /data/slaves/tftp

Master : Mail server : XMail

echo "xmail xmail/daemonpasswd string postmaster" | debconf-set-selections
echo "xmail xmail/daemonuser string postmaster" | debconf-set-selections
echo "xmail xmail/domainname string /etc/mailname" | debconf-set-selections
apt-get install -y xmail

Master : Create and use repository mirrors with apt-mirror (optional)

Installing a single node from the internet server is fine, the other nodes will not download everything, they will use a copy of the master filesystem as a template but they will install the additional specific packages by themselves. Installing on one single node is fine too, but installing (even a single package without dependencies) from 15 of them is slow and unfair !

apt-get install -y apt-mirror
mv /var/spool/apt-mirror /data/
sed -i 's~set base_path.*~&\nset base_path    /data/apt-mirror~' /etc/apt/mirror.list
sed -i 's~set run_postmirror.*~&\nset run_postmirror 1~' /etc/apt/mirror.list
sed -i 's~^deb~#&~' /etc/apt/mirror.list
sed -i 's~^clean~#&~' /etc/apt/mirror.list
cat >> /etc/apt/mirror.list << EOF

deb-armel http://archive.raspberrypi.org/debian wheezy main untested
deb-armhf http://archive.raspberrypi.org/debian wheezy main untested

deb-armhf http://archive.raspbian.org/mate wheezy main
deb-src http://archive.raspbian.org/mate wheezy main

deb-armhf http://archive.raspbian.org/multiarchcross wheezy main
deb-src http://archive.raspbian.org/multiarchcross wheezy main

deb-armhf http://archive.raspbian.org/raspbian wheezy main contrib non-free rpi firmware
deb-src http://archive.raspbian.org/raspbian wheezy main contrib non-free rpi firmware
deb-armhf http://archive.raspbian.org/raspbian wheezy-staging main contrib non-free rpi firmware
deb-src http://archive.raspbian.org/raspbian wheezy-staging main contrib non-free rpi firmware

deb-armhf http://archive.raspbian.org/raspbian jessie main contrib non-free rpi firmware
deb-src http://archive.raspbian.org/raspbian jessie main contrib non-free rpi firmware
deb-armhf http://archive.raspbian.org/raspbian jessie-staging main contrib non-free rpi firmware
deb-src http://archive.raspbian.org/raspbian jessie-staging main contrib non-free rpi firmware

clean http://archive.raspberrypi.org/debian
clean http://archive.raspbian.org/raspbian
EOF

cat > /data/apt-mirror/var/postmirror.sh << EOF
#!/bin/sh
/data/apt-mirror/var/clean.sh
chmod -R 755 /data/apt-mirror/mirror
chown -R apt-mirror.apt-mirror /data/apt-mirror/
find /data/apt-mirror/mirror -type f -exec chmod 644 {} \;

EOF

I enable the automatic execution every night :

sed -i 's~^#0~0~' /etc/cron.d/apt-mirror
sed -i 's~/var/spool~/data~' /etc/cron.d/apt-mirror

apt-mirror has a bug in a search : first for "arm" and then for "armhf", the second is never found as the first also match... Here is my fix :

sed  -i 's~arm|armhf~armhf|arm~' /usr/bin/apt-mirror

I launch an immediate mirror update :
This step which can take several days to download the 175GB, so I plug directly the hard-drive to one of my other local servers to initialize the cluster mirror with an already existing raspbian mirror. And I can continue with the other chapters while this mirroring is running. As this chapter takes a very long time to be executed, first it was designed to be non blocking, so I can continue with the following chapters while updating the mirror.

At the end of the mirroring process, I update the master's sources.list file to use my local repository from the filesystem (if I choose to not install Apache in the next steps or if Apache crashes)

su - apt-mirror -c apt-mirror
echo 'deb file:///data/apt-mirror/mirror/archive.raspbian.org/raspbian wheezy main firmware' > /etc/apt/sources.list
echo 'deb file:///data/apt-mirror/mirror/archive.raspberrypi.org/debian wheezy main' >> /etc/apt/sources.list
apt-get update

Master : Share the repository mirrors over the network with Apache (optional)

Given the cluster design, the slaves will execute Puppet agent and the PuppetMaster will be on the master node. Rake, the default PuppetMaster HTTP server is mono-thread and will not be able to answer to 15 simultaneous inquiries. It will need a stronger HTTP server such as Apache associated with mod_passenger. As I will need Apache for PuppetMaster, I will also use Apache to serve the repository mirrors. I install Apache and configure it to serve the files located in /data/apt-mirror/mirror.

apt-get install -y apache2
cat > /etc/apache2/conf.d/mirror << EOF
    <Directory /data/apt-mirror/mirror/>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>
    Alias /mirror /data/apt-mirror/mirror
EOF
/etc/init.d/apache2 restart

TODO: Optimize the Apache configuration to use less memory and CPU by decreasing the number of processes and of threads configured in /etc/apache2/apache2.conf

Slaves : Update the sources.list

Use the shared mirrored repository, then update the package list :

echo 'deb http://192.168.2.1/mirror/archive.raspbian.org/raspbian wheezy main firmware' > /data/slaves/rootfs/etc/apt/sources.list
echo 'deb http://192.168.2.1/mirror/archive.raspberrypi.org/debian wheezy main' >> /data/slaves/rootfs/etc/apt/sources.list
chroot /data/slaves/rootfs/ /usr/bin/apt-get update

Slaves : Network enabled bootloader : Das U-Boot

I enable the frame buffer in the command line (bcm2708_fb.fbwidth=1280 bcm2708_fb.fbheight=800 bcm2708_fb.fbdepth=8) because I sometimes debug a failing node by connecting it to a video projector.

apt-get install -y build-essential python bc git
git clone git://git.denx.de/u-boot.git
cd u-boot/

Apply the following patch to make the boot faster

cat | patch -p1 << EOF
--- u-boot.orig/include/configs/rpi-common.h    2016-01-22 10:25:00.487730581 +0100
+++ u-boot/include/configs/rpi-common.h    2016-01-22 10:31:02.286654115 +0100
@@ -168,7 +168,8 @@
     "scriptaddr=0x02000000\0" \\
     "ramdisk_addr_r=0x02100000\0" \\

-#define BOOT_TARGET_DEVICES(func) \\
+#define BOOT_TARGET_DEVICES(func) func(DHCP, dhcp, na)
+#define DISABLED_BOOT_TARGET_DEVICES(func) \\
     func(MMC, mmc, 0) \\
     func(USB, usb, 0) \\
     func(PXE, pxe, na) \\
EOF


make rpi_defconfig
make
cp u-boot.bin /data/slaves/bootfs/
grep -v initramfs /boot/config.txt > /data/slaves/bootfs/config.txt
echo 'kernel=u-boot.bin' >> /data/slaves/bootfs/config.txt
cat > /data/slaves/tftp/boot.scr << EOF
tftp \${kernel_addr_r} vmlinuz-3.12-1-rpi
tftp \${ramdisk_addr_r} initramfs.igz.uimg
setenv bootargs "bcm2708_fb.fbwidth=1280 bcm2708_fb.fbheight=800 bcm2708_fb.fbdepth=8 dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 elevator=deadline root=/dev/mmcblk0p2 rootwait quiet smsc95xx.macaddr=\${usbethaddr} aufs=disk masterip=192.168.2.1" 
bootz \${kernel_addr_r} \${ramdisk_addr_r} 
EOF
/root/u-boot/tools/mkimage -A arm -O linux -T script -C none -n "U-Boot script" -d /data/slaves/tftp/boot.scr /data/slaves/tftp/boot.scr.uimg

No uEnv.txt (on local boot partition)

TODO: remove compilation dependencies packages

Slaves : Kernel and customized initial ramdisk

cp /boot/vmlinuz-3.12-1-rpi /data/slaves/tftp/

Ref : [[http://jootamam.net/howto-initramfs-image.htm]]
Ref : [[http://www.raspberrypi.org/forums/viewtopic.php?p=228099]]

apt-get install -y busybox-static e2fsck-static dosfstools
mkdir -p /data/slaves/tftp/initramfs/{bin,sbin,etc/udhcpc,proc,sys,rootfs,bootfs,aufs,rw,usr/bin,usr/sbin,lib/modules,root,dev}
cd /data/slaves/tftp
mknod /data/slaves/tftp/initramfs/dev/null c 1 3 
mknod /data/slaves/tftp/initramfs/dev/tty c 5 0 
touch /data/slaves/tftp/initramfs/etc/mdev.conf
cp /bin/busybox /data/slaves/tftp/initramfs/bin/busybox
chmod +x /data/slaves/tftp/initramfs/bin/busybox
ln -s busybox /data/slaves/tftp/initramfs/bin/sh
cp /usr/share/doc/busybox-static/examples/udhcp/simple.script /data/slaves/tftp/initramfs/etc/udhcpc/
chmod +x /data/slaves/tftp/initramfs/etc/udhcpc/simple.script

mknod -m 0444 /data/slaves/tftp/initramfs/dev/random c 1 8 
mknod -m 0444 /data/slaves/tftp/initramfs/dev/urandom c 1 9 
echo 'root:x:0:0:root:/root:/bin/bash' > /data/slaves/tftp/initramfs/etc/passwd
cp /lib/arm-linux-gnueabihf/libnss_files.so.2 /data/slaves/tftp/initramfs/lib/
PROG="/usr/bin/ssh";  for i in ${PROG} `ldd ${PROG} | cut -d/ -f2- | cut -d\  -f1`; do mkdir -p /data/slaves/tftp/initramfs/$i; rm -Rf /data/slaves/tftp/initramfs/$i; cp /$i /data/slaves/tftp/initramfs/$i; done
#PROG="/usr/bin/ssh-keygen";  for i in ${PROG} `ldd ${PROG} | cut -d/ -f2- | cut -d\  -f1`; do mkdir -p /data/slaves/tftp/initramfs/$i; rm -Rf /data/slaves/tftp/initramfs/$i; cp /$i /data/slaves/tftp/initramfs/$i; done
#chroot initramfs/ ./bin/sh -c 'ssh-keygen -N "" -f /root/.ssh/id_rsa'
#cat /data/slaves/tftp/initramfs/root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys
cp -r /root/.ssh /data/slaves/tftp/initramfs/root/
chmod 600 /data/slaves/tftp/initramfs/root/.ssh/id_rsa
cp /lib/modules/3.12-1-rpi/kernel/fs/aufs/aufs.ko /data/slaves/tftp/initramfs/lib/modules/
mknod -m 600 /data/slaves/tftp/initramfs/dev/watchdog c 10 130
cp /lib/modules/3.12-1-rpi/kernel/drivers/watchdog/bcm2708_wdog.ko /data/slaves/tftp/initramfs/lib/modules/
cat > /data/slaves/partitions.sfdisk << EOF 
# partition table of /dev/mmcblk0
unit: sectors

/dev/mmcblk0p1 : start=       16, size=    97712, Id= b
/dev/mmcblk0p2 : start=    97728, size=  2000000, Id=83
/dev/mmcblk0p3 : start=  2097728, size=  2000000, Id=83
/dev/mmcblk0p4 : start=  4097728, size=  3529024, Id=83
EOF
PROG="/sbin/sfdisk";  for i in ${PROG} `ldd ${PROG} | cut -d/ -f2- | cut -d\  -f1`; do mkdir -p /data/slaves/tftp/initramfs/$i; rm -Rf /data/slaves/tftp/initramfs/$i; cp /$i /data/slaves/tftp/initramfs/$i; done
PROG="/sbin/mke2fs";  for i in ${PROG} `ldd ${PROG} | cut -d/ -f2- | cut -d\  -f1`; do mkdir -p /data/slaves/tftp/initramfs/$i; rm -Rf /data/slaves/tftp/initramfs/$i; cp /$i /data/slaves/tftp/initramfs/$i; done
cp /sbin/e2fsck.static /data/slaves/tftp/initramfs/sbin/e2fsck.static
PROG="/sbin/mkdosfs";  for i in ${PROG} `ldd ${PROG} | cut -d/ -f2- | cut -d\  -f1`; do mkdir -p /data/slaves/tftp/initramfs/$i; rm -Rf /data/slaves/tftp/initramfs/$i; cp /$i /data/slaves/tftp/initramfs/$i; done
PROG="/sbin/dosfsck";  for i in ${PROG} `ldd ${PROG} | cut -d/ -f2- | cut -d\  -f1`; do mkdir -p /data/slaves/tftp/initramfs/$i; rm -Rf /data/slaves/tftp/initramfs/$i; cp /$i /data/slaves/tftp/initramfs/$i; done
PROG="/usr/bin/rsync";  for i in ${PROG} `ldd ${PROG} | cut -d/ -f2- | cut -d\  -f1`; do mkdir -p /data/slaves/tftp/initramfs/$i; rm -Rf /data/slaves/tftp/initramfs/$i; cp /$i /data/slaves/tftp/initramfs/$i; done
touch /data/slaves/tftp/initramfs/init
wget -O /data/slaves/tftp/initramfs/init --no-check-certificate https://www.cerbelle.net/redmine/attachments/download/50/init
chmod +x /data/slaves/tftp/initramfs/init

Copy the attached init script in */root/work/initramfs/init", rebuild the initramfs file and package it for u-boot

cd /data/slaves/tftp/initramfs && find . | cpio -H newc -o | gzip > /data/slaves/tftp/initramfs.igz && cd -
/root/u-boot/tools/mkimage -A arm -O linux -T ramdisk -C gzip -n "U-Boot Initial RamDisk" -d /data/slaves/tftp/initramfs.igz /data/slaves/tftp/initramfs.igz.uimg

Master : Clean the installation (optional)

Clean the downloaded packages stored in apt cache.

apt-get clean

Master : Halt

Now, the master is ready to serve. At the next boot, it will use its static cluster IP (192.168.2.1), will act as a DHCP, DNS and TFTP server to permit the automatic slave installation, updates and boot. So, you can halt the master :

halt

Connect it to the cluster network, with all the slaves and begin to prepare the slave's SD cards.

Slave's SD card

Create a partition table with only one vfat partition.
Mount the partition
Copy all the files from /data/slaves/bootfs/ in this partition
unmount the partition
Boot the slave with this SD card, it will :
- get an IP address (and the Master's IP) from DHCP
- get u-boot's configuration file (boot.scr.uimg) from master via tftp
- get the kernel from the master via tftp
- get the initramfs from master via tftp
- boot the kernel with initramfs

The initramfs will :
- get the Slave's partition table from master,
- if the partition table changed, it will apply it, format all the partitions, update the boot partition (firmware) from the master and reboot

If the partition table is OK, so it will :
- if the slave's root filesystem (system) changed (faster than a full filesystem comparison), it rsync it from the master via rsync (faster than a copy) to the slave
- it mounts the root filesystem
- if a working partition is requested, it remounts the root filesystem RO, create and format a RW partition (in memory if volatile or on disk), and overlay this RW partition on top of the system filesystem
- switch to this root partition and launch the standard init process

Also available in: PDF HTML TXT