5. Master Node Installation¶
The nodes will be headless, this configuration enables to :
- Install software and data on every node, and to reset all of them to a clean state in 2 commands
- Install, Upgrade or change the root filesystem (clean installation) of every node in 2 commands
- Install, Upgrade or change the boot filesystem (firmware) of every nodes in 2 commands
Initial Raspbian installation¶
On my Linux workstation, I download the last Raspbian Unattended Installer from [[https://github.com/debian-pi/raspbian-ua-netinst/releases/latest]] and write it to the future master's SD card :
wget -c https://github.com/debian-pi/raspbian-ua-netinst/releases/download/v1.1.7/raspbian-ua-netinst-v1.1.7.img.xz -O /tmp/raspbian-ua-netinst.xz xzcat /tmp/raspbian-ua-netinst.xz | sudo dd of=/dev/sdi
Create an unattended configuration file to customize a little bit the installation. I replaced the default "pi" hostname by "raspbian10-base", I created a default user "pi" with password "pi". The default installation does not have any regular user, only "root", and the default openSSH does not allow access to root. Quite embarrassing for an unattended installation without a local keyboard, mouse and display... Given that I need to install software as a regular user, I'll need to create him later, so I created him at installation, without any privileges. I also customized the boot parameters to have a better fsck (this will be an headless appliance).
sudo mount /dev/sdi1 /mnt cat << EOF | sudo tee /mnt/installer-config.txt hostname=raspbian10-base username=pi userpw=pi cmdline="dwc_otg.lpm_enable=0 console=tty1 elevator=deadline fsck.repair=yes" EOF sudo umount /mnt
I place the SD card in the master's SD slot, connect it to the network (to internet), the power and wait for the process to complete (20-30 minutes). At the end, I have a Raspbian installed, with the latest kernel and packages. Its hostname is pi, the root password is raspbian. It is connected to my network with a dynamic DHCP address and has an SSH server listening allowing root connections. As I love to copy/paste command lines (from this page), I need to have both this page and a terminal on the RPi from my workstation. I need the RPi IP address, so I prepared my internal network's DHCP to give a fixed address to this RPi, I already know its IP. If I did not, I would have to connect a display to the HDMI connector and a keyboard to connect as root and ask for the IP address. Anyway, now, I can connect to the RPi from my workstation, with SSH, as root and I can do the installation with copy/paste the command from this page... I'm so lazy...
Master & Slaves : Upgrade the system¶
First, I check that the system is up to date regarding the repositories, the package lists and the installed packages :
apt-get install -y aptitude && aptitude update && aptitude safe-upgrade -y && aptitude dist-upgrade -y && aptitude full-upgrade -y && aptitude purge ~c -y && aptitude clean
Master & Slaves : Heartbeat on external led¶
I connect an external white LED on each RPi card's GPIO4, through a resistor, to expose the activity (ACT) kernel LED. Thus, I need to tell the firmware to use the GPIO4 as Activity LED exposed to the kernel. Then, I can change the behavior (triggering) of this LED in the OS' userspace (`echo heartbeat > /sys/class/leds/led0/trigger` or `echo heartbeat > /sys/class/leds/ACT/trigger`, depending on the kernel version), but it occurs late in the boot process. I prefer to have heartbeat very early, initialized by the firmware.
echo 'dtparam=act_led_gpio=4' >> /boot/firmware/config.txt echo 'dtparam=act_led_trigger=heartbeat' >> /boot/firmware/config.txt
Master & Slaves : Generate root's SSH keypair¶
Then, I generate the root's SSH passwordless key pair and I allow key-authentication for root to root. I also relax the checkings regarding the Host Keys (to avoid the question asking to add or not the remote host key during the first connection) and the DNS (to avoid DNS checkings that could slower the connections). As I generate it before making the snapshot that will be used as a template for the node's filesystem, it means that all the nodes will have the same root's key pair and will all be able to ssh to the others passwordless.
ssh-keygen -N "" -f /root/.ssh/id_rsa cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys echo >> /etc/ssh/ssh_config echo " StrictHostKeyChecking no" >> /etc/ssh/ssh_config echo "UseDNS no" >> /etc/ssh/sshd_config
Master & Slaves : SSH authorized keys¶
I want to be able to connect directly as root from my laptops, so I also add my public keys to root's authorized_keys file (useful when opening a terminal to every node with cssh) :
echo "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAIEAtM8LzekUr46wvVNWoYzxPuKVTv7yFp+Aa/a1vKAendFa3xsMZz6Pp0Xn8U5ZYbTpqqVeM8O+ETqjtpBVk+7+C516DwB+R/cKulTjy061fBPZvTp5pIKm4+NQXNBhwjmQs//nWJ54PlDS5mHuj9NalX07b2OBztrvLjPzf/m4sB0= Francois Cerbelle" >> /root/.ssh/authorized_keys
echo -n '192.168.2.1 ' > /root/.ssh/known_hosts cat /etc/ssh/ssh_host_ecdsa_key.pub >> /root/.ssh/known_hosts echo -n ' root@(none)' >> /root/.ssh/known_hosts
Master : External hard drive to store repository mirrors and data¶
I need a storage to store the cluster files. I use a small hard drive, with one big partition, formatted in ext4 and connected to the USB port.
echo /dev/sda1 /data auto defaults 0 1 >> /etc/fstab mkdir /data mount -a
Master & Slaves : Puppet agent¶
apt-get -y update && apt-get -y upgrade && apt-get -y dist-upgrade && apt-get -y install puppetserver && apt-get -y autoremove && apt-get -y clean && apt-get -y autoclean cat << EOF | augtool set /files/etc/puppet/puppet.conf/main/server rpinode01.raspicluster.cerbelle.net set /files/etc/puppet/puppet.conf/main/pluginsync true set /files/etc/puppet/puppet.conf/main/report true set /files/etc/puppet/puppet.conf/main/report_server rpinode01.raspicluster.cerbelle.net set /files/etc/puppet/puppet.conf/main/certname "rpinode01.raspicluster.cerbelle.net" set /files/etc/puppet/puppet.conf/user/waitforcert 120 set /files/etc/default/puppet/START no set /files/etc/hosts/01/ipaddr 192.168.2.1 set /files/etc/hosts/01/canonical rpinode01.raspicluster.cerbelle.net set /files/etc/hosts/01/alias[1] rpinode01 save EOF
Master & Slaves : Raspberry Pi CPU tuning¶
First, I overclock the RPi, using the maximum allowed without breaking the warranty. Other values might set the "void warranty bit" in the card. These are sufficient (1GHz) and should not bring the RPi to its max temperature (85°C), even in a compact case. If it should happen, the RPi has an internal security which will slow it down until the temperature fall. NEVER change the max temperature threshold, it would break the warranty.
echo '#arm_freq=1000' >> /boot/config.txt echo '#core_freq=500' >> /boot/config.txt echo '#sdram_freq=600' >> /boot/config.txt echo '#over_voltage=6' >> /boot/config.txt echo 'gpu_mem=16' >> /boot/config.txt
I also install a library which is configured to be automatically preloaded (LD_PRELOAD) by the dynamic linker with every binary and which overloads the memcpy and memmove functions to optimize them on the RPi.
apt-get install -y raspi-copies-and-fills
Master & Slaves Internationalization¶
I choose to compile the en_US.UTF8 and to use it as default locale. But I choose to locate the RPi in the Europe/Paris timezone.
#dpkg-reconfigure locales sed -i 's/.*en_US.UTF-8/en_US.UTF-8/' /etc/locale.gen echo en_US.UTF-8 > /etc/default/locale locale-gen dpkg-reconfigure tzdata
Master & Slaves : Kernel downgrade¶
I had to install the linux kernel 3.12, the last Raspbian precompiled version including aufs.
apt-get install -y linux-image-3.12-1-rpi cp /boot/initrd.img-3.12-1-rpi /boot/initrd.img cp /boot/vmlinuz-3.12-1-rpi /boot/kernel.img sed -i 's/^initramfs.*/initramfs initrd.img/' /boot/config.txt sed -i 's/^kernel=.*/kernel=kernel.img/' /boot/config.txt reboot apt-get purge -y linux-image-3.18.0-trunk-rpi
Master : Snapshot the base system as the future slave's root filesystem¶
Before taking the snapshot, Iclean the downloaded packages stored in apt cache, it is useless to replicate them on the slaves, they will be obsoletes and it will use bandwith/time to transfer them through the cluster network.
aptitude clean
Then, I create the root and boot filesystems copy as templates for the slaves.
rm -Rf /data/slaves/{rootfs,bootfs} mkdir -p /data/slaves/{rootfs,bootfs} cp -ax / /data/slaves/rootfs/ cp /boot/*.{bin,dat,elf} /data/slaves/bootfs/
And I remove the files that were automatically created during the previous steps and which are irrelevant to the slave nodes. The slave's root account does not need my installation bash history and, more important, they will have a different ethernet (MAC) address, so I make the system forget the MAC mapping to the NIC number (eth0, eth1, ...) :
rm /data/slaves/rootfs/root/.bash_history rm /data/slaves/rootfs/etc/udev/rules.d/70-persistent-net.rules
Slaves : Disk configuration¶
The slaves will not have any hard drive connected so I have to remove the related line in /etc/fstab and they will not need the / and /boot mount neither because they will be mounted during the initrd, so I disable (comment) all these lines from the file :
sed -i 's~^/.* / .*$~#&~' /data/slaves/rootfs/etc/fstab sed -i 's~^/.* /boot .*$~#&~' /data/slaves/rootfs/etc/fstab sed -i 's~^/.* /data .*$~#&~' /data/slaves/rootfs/etc/fstab rmdir /data/slaves/rootfs/data
Slaves : Networking configuration¶
I cleanup the /etc/hosts file, no need for 127.0.1.1 to resolve its own name as each node will have a DHCP assigned IP address with a DNS server resolving this external IP to the right name. Furthermore, I add a custom script to update the files and the running system with the right hostname as soon as the DHCP assign an IP address to the node. Basically, the node will have no name when it boots, as there is none in the configuration files, but everything will be automatically updated in files and in memory from the DHCP answer.
cat > /data/slaves/rootfs/etc/hosts << EOF 127.0.0.1 localhost EOF cat > /data/slaves/rootfs/etc/dhcp/dhclient-exit-hooks.d/hostname << EOF #!/bin/bash if [ \$reason = "BOUND" ]; then oldhostname=\$(hostname -s) if [ \$oldhostname != \$new_host_name ]; then echo \$new_host_name > /etc/hostname hostname -F /etc/hostname fi fi EOF
Master : PuppetMaster¶
All the remaining configuration (common to each node, specific to master and specific to slaves) will be handled and managed by puppet. So, I need to install a first standalone puppet master. It will use the embedded WebRicks to serve the incoming connections. Even if WebRicks is mono-threaded, it is not an issue as it will have to serve the master's puppet agent first with a "master configuration". This master configuration will begin to manage the master's puppet master installation and will configure it with Apache and Passenger (mod_rails). So, this very first puppet master installation is only intended to be used only once.
apt-get install -y puppetmaster /etc/init.d/puppetmaster stop cat << EOF | augtool set /files/etc/default/puppetmaster/START yes set /files/etc/puppet/puppet.conf/master/autosign true set /files/etc/puppet/puppet.conf/master/allow_duplicate_certs true # Puppet 3.x #set /files/etc/puppet/puppet.conf/main/modulepath /etc/puppet/modules:/etc/puppet/site-modules:/usr/share/puppet/modules set /files/etc/puppet/puppet.conf/main/environmentpath /etc/puppet/environments set /files/etc/puppet/puppet.conf/main/basemodulepath /etc/puppet/modules:/etc/puppet/environments/production/site-modules:/usr/share/puppet/modules save EOF mkdir -p /etc/puppet/environments/production/manifests apt-get install -y mercurial hg clone --insecure https://www.cerbelle.net/hg/puppet-modules /tmp/puppet-modules /etc/init.d/puppetmaster stop mv /tmp/puppet-modules /tmp/puppet mkdir -p /etc/puppet/environments/production/manifests cp -r /tmp/puppet/modules /etc/puppet/environments/production cp /tmp/puppet/manifests/default.pp /etc/puppet/environments/production/manifests/site.pp cp -r /tmp/puppet/hieradata /etc/puppet/ cp /tmp/puppet/hiera.yaml /etc/puppet/ ln -s /etc/puppet/hiera.yaml /etc/ rm -Rf /tmp/puppet /etc/init.d/puppetmaster start
Master : Network configuration¶
sed -i 's/.*dhcp.*/#&/' /etc/network/interfaces cat >> /etc/network/interfaces << EOF iface eth0 inet static address 192.168.2.1 network 192.168.2.0 netmask 255.255.255.0 gateway 192.168.2.254 broadcast 192.168.2.255 EOF echo 'rpinode01' > /etc/hostname hostname -F /etc/hostname cat > /etc/hosts << EOF 127.0.0.1 localhost 192.168.2.1 rpinode01.raspicluster.cerbelle.net rpinode01 192.168.2.2 rpinode02.raspicluster.cerbelle.net rpinode02 192.168.2.3 rpinode03.raspicluster.cerbelle.net rpinode03 192.168.2.4 rpinode04.raspicluster.cerbelle.net rpinode04 192.168.2.5 rpinode05.raspicluster.cerbelle.net rpinode05 192.168.2.6 rpinode06.raspicluster.cerbelle.net rpinode06 192.168.2.7 rpinode07.raspicluster.cerbelle.net rpinode07 192.168.2.8 rpinode08.raspicluster.cerbelle.net rpinode08 192.168.2.9 rpinode09.raspicluster.cerbelle.net rpinode09 192.168.2.10 rpinode10.raspicluster.cerbelle.net rpinode10 192.168.2.11 rpinode11.raspicluster.cerbelle.net rpinode11 192.168.2.12 rpinode12.raspicluster.cerbelle.net rpinode12 192.168.2.13 rpinode13.raspicluster.cerbelle.net rpinode13 192.168.2.14 rpinode14.raspicluster.cerbelle.net rpinode14 192.168.2.15 rpinode15.raspicluster.cerbelle.net rpinode15 EOF cat > /etc/resolv.conf << EOF domain raspicluster.cerbelle.net search raspicluster.cerbelle.net nameserver 192.168.2.254 EOF
Master : DHCP, DNS and TFTP server with DNSMasq¶
apt-get install -y dnsmasq /etc/init.d/dnsmasq stop echo 'conf-dir=/etc/dnsmasq.d' >> /etc/dnsmasq.conf cat > /etc/dnsmasq.d/rpicluster << EOF dhcp-range=192.168.2.20,192.168.2.250,12h dhcp-host=b8:27:eb:cb:63:18,,rpinode01,192.168.2.1,1h dhcp-host=b8:27:eb:c2:9f:19,,rpinode02,192.168.2.2,1h dhcp-host=b8:27:eb:27:b6:53,,rpinode03,192.168.2.3,1h dhcp-host=b8:27:eb:b9:61:6d,,rpinode04,192.168.2.4,1h dhcp-host=b8:27:eb:b6:57:a0,,rpinode05,192.168.2.5,1h dhcp-host=b8:27:eb:ef:75:38,,rpinode06,192.168.2.6,1h dhcp-host=b8:27:eb:2b:41:91,,rpinode07,192.168.2.7,1h dhcp-host=b8:27:eb:0c:25:82,,rpinode08,192.168.2.8,1h dhcp-host=b8:27:eb:e5:83:f5,,rpinode09,192.168.2.9,1h dhcp-host=b8:27:eb:c5:5e:e0,,rpinode10,192.168.2.10,1h dhcp-host=b8:27:eb:f2:55:fb,,rpinode11,192.168.2.11,1h dhcp-host=b8:27:eb:0c:dc:64,,rpinode12,192.168.2.12,1h dhcp-host=b8:27:eb:22:97:40,,rpinode13,192.168.2.13,1h dhcp-host=b8:27:eb:e3:00:59,,rpinode14,192.168.2.14,1h dhcp-host=b8:27:eb:19:dc:95,,rpinode15,192.168.2.15,1h dhcp-option=6,192.168.2.1,192.168.2.254 # dns-server dhcp-option=3,192.168.2.254 # router dhcp-option=15,raspicluster.cerbelle.net # domain-name dhcp-option=66,192.168.2.1 # tftp-server dhcp-option=69,192.168.2.1 # smtp-server dhcp-option=119,raspicluster.cerbelle.net # domain-search enable-tftp tftp-root=/data/slaves/tftp EOF mkdir -p /data/slaves/tftp chmod -R u+rw,g+r,o+r /data/slaves/tftp
Master : Mail server : XMail¶
echo "xmail xmail/daemonpasswd string postmaster" | debconf-set-selections echo "xmail xmail/daemonuser string postmaster" | debconf-set-selections echo "xmail xmail/domainname string /etc/mailname" | debconf-set-selections apt-get install -y xmail
Master : Create and use repository mirrors with apt-mirror (optional)¶
Installing a single node from the internet server is fine, the other nodes will not download everything, they will use a copy of the master filesystem as a template but they will install the additional specific packages by themselves. Installing on one single node is fine too, but installing (even a single package without dependencies) from 15 of them is slow and unfair !
apt-get install -y apt-mirror mv /var/spool/apt-mirror /data/ sed -i 's~set base_path.*~&\nset base_path /data/apt-mirror~' /etc/apt/mirror.list sed -i 's~set run_postmirror.*~&\nset run_postmirror 1~' /etc/apt/mirror.list sed -i 's~^deb~#&~' /etc/apt/mirror.list sed -i 's~^clean~#&~' /etc/apt/mirror.list cat >> /etc/apt/mirror.list << EOF deb-armel http://archive.raspberrypi.org/debian wheezy main untested deb-armhf http://archive.raspberrypi.org/debian wheezy main untested deb-armhf http://archive.raspbian.org/mate wheezy main deb-src http://archive.raspbian.org/mate wheezy main deb-armhf http://archive.raspbian.org/multiarchcross wheezy main deb-src http://archive.raspbian.org/multiarchcross wheezy main deb-armhf http://archive.raspbian.org/raspbian wheezy main contrib non-free rpi firmware deb-src http://archive.raspbian.org/raspbian wheezy main contrib non-free rpi firmware deb-armhf http://archive.raspbian.org/raspbian wheezy-staging main contrib non-free rpi firmware deb-src http://archive.raspbian.org/raspbian wheezy-staging main contrib non-free rpi firmware deb-armhf http://archive.raspbian.org/raspbian jessie main contrib non-free rpi firmware deb-src http://archive.raspbian.org/raspbian jessie main contrib non-free rpi firmware deb-armhf http://archive.raspbian.org/raspbian jessie-staging main contrib non-free rpi firmware deb-src http://archive.raspbian.org/raspbian jessie-staging main contrib non-free rpi firmware clean http://archive.raspberrypi.org/debian clean http://archive.raspbian.org/raspbian EOF cat > /data/apt-mirror/var/postmirror.sh << EOF #!/bin/sh /data/apt-mirror/var/clean.sh chmod -R 755 /data/apt-mirror/mirror chown -R apt-mirror.apt-mirror /data/apt-mirror/ find /data/apt-mirror/mirror -type f -exec chmod 644 {} \; EOF
I enable the automatic execution every night :
sed -i 's~^#0~0~' /etc/cron.d/apt-mirror sed -i 's~/var/spool~/data~' /etc/cron.d/apt-mirror
apt-mirror has a bug in a search : first for "arm" and then for "armhf", the second is never found as the first also match... Here is my fix :
sed -i 's~arm|armhf~armhf|arm~' /usr/bin/apt-mirror
I launch an immediate mirror update :
This step which can take several days to download the 175GB, so I plug directly the hard-drive to one of my other local servers to initialize the cluster mirror with an already existing raspbian mirror. And I can continue with the other chapters while this mirroring is running. As this chapter takes a very long time to be executed, first it was designed to be non blocking, so I can continue with the following chapters while updating the mirror.
At the end of the mirroring process, I update the master's sources.list file to use my local repository from the filesystem (if I choose to not install Apache in the next steps or if Apache crashes)
su - apt-mirror -c apt-mirror echo 'deb file:///data/apt-mirror/mirror/archive.raspbian.org/raspbian wheezy main firmware' > /etc/apt/sources.list echo 'deb file:///data/apt-mirror/mirror/archive.raspberrypi.org/debian wheezy main' >> /etc/apt/sources.list apt-get update
Master : Share the repository mirrors over the network with Apache (optional)¶
Given the cluster design, the slaves will execute Puppet agent and the PuppetMaster will be on the master node. Rake, the default PuppetMaster HTTP server is mono-thread and will not be able to answer to 15 simultaneous inquiries. It will need a stronger HTTP server such as Apache associated with mod_passenger. As I will need Apache for PuppetMaster, I will also use Apache to serve the repository mirrors. I install Apache and configure it to serve the files located in /data/apt-mirror/mirror.
apt-get install -y apache2 cat > /etc/apache2/conf.d/mirror << EOF <Directory /data/apt-mirror/mirror/> Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny allow from all </Directory> Alias /mirror /data/apt-mirror/mirror EOF /etc/init.d/apache2 restart
TODO: Optimize the Apache configuration to use less memory and CPU by decreasing the number of processes and of threads configured in /etc/apache2/apache2.conf
Slaves : Update the sources.list¶
Use the shared mirrored repository, then update the package list :
echo 'deb http://192.168.2.1/mirror/archive.raspbian.org/raspbian wheezy main firmware' > /data/slaves/rootfs/etc/apt/sources.list echo 'deb http://192.168.2.1/mirror/archive.raspberrypi.org/debian wheezy main' >> /data/slaves/rootfs/etc/apt/sources.list chroot /data/slaves/rootfs/ /usr/bin/apt-get update
Slaves : Network enabled bootloader : Das U-Boot¶
I enable the frame buffer in the command line (bcm2708_fb.fbwidth=1280 bcm2708_fb.fbheight=800 bcm2708_fb.fbdepth=8) because I sometimes debug a failing node by connecting it to a video projector.
apt-get install -y build-essential python bc git git clone git://git.denx.de/u-boot.git cd u-boot/
Apply the following patch to make the boot faster
cat | patch -p1 << EOF --- u-boot.orig/include/configs/rpi-common.h 2016-01-22 10:25:00.487730581 +0100 +++ u-boot/include/configs/rpi-common.h 2016-01-22 10:31:02.286654115 +0100 @@ -168,7 +168,8 @@ "scriptaddr=0x02000000\0" \\ "ramdisk_addr_r=0x02100000\0" \\ -#define BOOT_TARGET_DEVICES(func) \\ +#define BOOT_TARGET_DEVICES(func) func(DHCP, dhcp, na) +#define DISABLED_BOOT_TARGET_DEVICES(func) \\ func(MMC, mmc, 0) \\ func(USB, usb, 0) \\ func(PXE, pxe, na) \\ EOF
make rpi_defconfig make cp u-boot.bin /data/slaves/bootfs/ grep -v initramfs /boot/config.txt > /data/slaves/bootfs/config.txt echo 'kernel=u-boot.bin' >> /data/slaves/bootfs/config.txt cat > /data/slaves/tftp/boot.scr << EOF tftp \${kernel_addr_r} vmlinuz-3.12-1-rpi tftp \${ramdisk_addr_r} initramfs.igz.uimg setenv bootargs "bcm2708_fb.fbwidth=1280 bcm2708_fb.fbheight=800 bcm2708_fb.fbdepth=8 dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 elevator=deadline root=/dev/mmcblk0p2 rootwait quiet smsc95xx.macaddr=\${usbethaddr} aufs=disk masterip=192.168.2.1" bootz \${kernel_addr_r} \${ramdisk_addr_r} EOF /root/u-boot/tools/mkimage -A arm -O linux -T script -C none -n "U-Boot script" -d /data/slaves/tftp/boot.scr /data/slaves/tftp/boot.scr.uimg
No uEnv.txt (on local boot partition)
TODO: remove compilation dependencies packages
Slaves : Kernel and customized initial ramdisk¶
cp /boot/vmlinuz-3.12-1-rpi /data/slaves/tftp/
Ref : [[http://jootamam.net/howto-initramfs-image.htm]]
Ref : [[http://www.raspberrypi.org/forums/viewtopic.php?p=228099]]
apt-get install -y busybox-static e2fsck-static dosfstools mkdir -p /data/slaves/tftp/initramfs/{bin,sbin,etc/udhcpc,proc,sys,rootfs,bootfs,aufs,rw,usr/bin,usr/sbin,lib/modules,root,dev} cd /data/slaves/tftp mknod /data/slaves/tftp/initramfs/dev/null c 1 3 mknod /data/slaves/tftp/initramfs/dev/tty c 5 0 touch /data/slaves/tftp/initramfs/etc/mdev.conf cp /bin/busybox /data/slaves/tftp/initramfs/bin/busybox chmod +x /data/slaves/tftp/initramfs/bin/busybox ln -s busybox /data/slaves/tftp/initramfs/bin/sh cp /usr/share/doc/busybox-static/examples/udhcp/simple.script /data/slaves/tftp/initramfs/etc/udhcpc/ chmod +x /data/slaves/tftp/initramfs/etc/udhcpc/simple.script mknod -m 0444 /data/slaves/tftp/initramfs/dev/random c 1 8 mknod -m 0444 /data/slaves/tftp/initramfs/dev/urandom c 1 9 echo 'root:x:0:0:root:/root:/bin/bash' > /data/slaves/tftp/initramfs/etc/passwd cp /lib/arm-linux-gnueabihf/libnss_files.so.2 /data/slaves/tftp/initramfs/lib/ PROG="/usr/bin/ssh"; for i in ${PROG} `ldd ${PROG} | cut -d/ -f2- | cut -d\ -f1`; do mkdir -p /data/slaves/tftp/initramfs/$i; rm -Rf /data/slaves/tftp/initramfs/$i; cp /$i /data/slaves/tftp/initramfs/$i; done #PROG="/usr/bin/ssh-keygen"; for i in ${PROG} `ldd ${PROG} | cut -d/ -f2- | cut -d\ -f1`; do mkdir -p /data/slaves/tftp/initramfs/$i; rm -Rf /data/slaves/tftp/initramfs/$i; cp /$i /data/slaves/tftp/initramfs/$i; done #chroot initramfs/ ./bin/sh -c 'ssh-keygen -N "" -f /root/.ssh/id_rsa' #cat /data/slaves/tftp/initramfs/root/.ssh/id_rsa.pub >> /root/.ssh/authorized_keys cp -r /root/.ssh /data/slaves/tftp/initramfs/root/ chmod 600 /data/slaves/tftp/initramfs/root/.ssh/id_rsa cp /lib/modules/3.12-1-rpi/kernel/fs/aufs/aufs.ko /data/slaves/tftp/initramfs/lib/modules/ mknod -m 600 /data/slaves/tftp/initramfs/dev/watchdog c 10 130 cp /lib/modules/3.12-1-rpi/kernel/drivers/watchdog/bcm2708_wdog.ko /data/slaves/tftp/initramfs/lib/modules/ cat > /data/slaves/partitions.sfdisk << EOF # partition table of /dev/mmcblk0 unit: sectors /dev/mmcblk0p1 : start= 16, size= 97712, Id= b /dev/mmcblk0p2 : start= 97728, size= 2000000, Id=83 /dev/mmcblk0p3 : start= 2097728, size= 2000000, Id=83 /dev/mmcblk0p4 : start= 4097728, size= 3529024, Id=83 EOF PROG="/sbin/sfdisk"; for i in ${PROG} `ldd ${PROG} | cut -d/ -f2- | cut -d\ -f1`; do mkdir -p /data/slaves/tftp/initramfs/$i; rm -Rf /data/slaves/tftp/initramfs/$i; cp /$i /data/slaves/tftp/initramfs/$i; done PROG="/sbin/mke2fs"; for i in ${PROG} `ldd ${PROG} | cut -d/ -f2- | cut -d\ -f1`; do mkdir -p /data/slaves/tftp/initramfs/$i; rm -Rf /data/slaves/tftp/initramfs/$i; cp /$i /data/slaves/tftp/initramfs/$i; done cp /sbin/e2fsck.static /data/slaves/tftp/initramfs/sbin/e2fsck.static PROG="/sbin/mkdosfs"; for i in ${PROG} `ldd ${PROG} | cut -d/ -f2- | cut -d\ -f1`; do mkdir -p /data/slaves/tftp/initramfs/$i; rm -Rf /data/slaves/tftp/initramfs/$i; cp /$i /data/slaves/tftp/initramfs/$i; done PROG="/sbin/dosfsck"; for i in ${PROG} `ldd ${PROG} | cut -d/ -f2- | cut -d\ -f1`; do mkdir -p /data/slaves/tftp/initramfs/$i; rm -Rf /data/slaves/tftp/initramfs/$i; cp /$i /data/slaves/tftp/initramfs/$i; done PROG="/usr/bin/rsync"; for i in ${PROG} `ldd ${PROG} | cut -d/ -f2- | cut -d\ -f1`; do mkdir -p /data/slaves/tftp/initramfs/$i; rm -Rf /data/slaves/tftp/initramfs/$i; cp /$i /data/slaves/tftp/initramfs/$i; done touch /data/slaves/tftp/initramfs/init wget -O /data/slaves/tftp/initramfs/init --no-check-certificate https://www.cerbelle.net/redmine/attachments/download/50/init chmod +x /data/slaves/tftp/initramfs/init
Copy the attached init script in */root/work/initramfs/init", rebuild the initramfs file and package it for u-boot
cd /data/slaves/tftp/initramfs && find . | cpio -H newc -o | gzip > /data/slaves/tftp/initramfs.igz && cd - /root/u-boot/tools/mkimage -A arm -O linux -T ramdisk -C gzip -n "U-Boot Initial RamDisk" -d /data/slaves/tftp/initramfs.igz /data/slaves/tftp/initramfs.igz.uimg
Master : Clean the installation (optional)¶
Clean the downloaded packages stored in apt cache.
apt-get clean
Master : Halt¶
Now, the master is ready to serve. At the next boot, it will use its static cluster IP (192.168.2.1), will act as a DHCP, DNS and TFTP server to permit the automatic slave installation, updates and boot. So, you can halt the master :
halt
Connect it to the cluster network, with all the slaves and begin to prepare the slave's SD cards.
Slave's SD card¶
Create a partition table with only one vfat partition.
Mount the partition
Copy all the files from /data/slaves/bootfs/ in this partition
unmount the partition
Boot the slave with this SD card, it will :
- get an IP address (and the Master's IP) from DHCP
- get u-boot's configuration file (boot.scr.uimg) from master via tftp
- get the kernel from the master via tftp
- get the initramfs from master via tftp
- boot the kernel with initramfs
The initramfs will :
- get the Slave's partition table from master,
- if the partition table changed, it will apply it, format all the partitions, update the boot partition (firmware) from the master and reboot
If the partition table is OK, so it will :
- if the slave's root filesystem (system) changed (faster than a full filesystem comparison), it rsync it from the master via rsync (faster than a copy) to the slave
- it mounts the root filesystem
- if a working partition is requested, it remounts the root filesystem RO, create and format a RW partition (in memory if volatile or on disk), and overlay this RW partition on top of the system filesystem
- switch to this root partition and launch the standard init process