Servers

Here, we will present our servers. We won't make the difference between physical and virtual servers, because some of the virtual machines we run are not managed by us.

duck

This server is hosted at Hetzner, in their Falkenstein, Germany datacenter. It is our main server, where basically everything runs. Its configuration is hosted here.

Hardware

The hardware configuration lives in two places:

  • shabka: nixos/hardware module: there is not much going on there, but we share that part of the configuration in shabka so other users with the same type of machine can reuse this part in their configuration.
  • hardware-configuration.nix: this is where everything is described as configuration.

Have a quick read through those two, and while you are reading the rest of this part, try to figure out where each part is described in the configuration.

Processor: Intel Xeon E5-1650V2 RAM: 64GB: 4x RAM 16384 MB DDR3 ECC reg. Storage: 2x HDD SATA 2,0 TB Enterprise Network: NIC 1 Gbit - Intel I350

Partitions

The two disks are in a RAID 1 array, managed by several tools. Actually, the whole disks are not directly put inside of RAID array. Instead, they have been separated into four partitions.

The system is booted through good'ol BIOS (yes, we know its bad but there is no support for PXE UEFI provided by Hetzner, so we have no choice), and so GRUB is installed on both disks, allowing us to boot without problem even in the event of a drive failure! As we are using GPT disks, we need a BIOS boot partition to store GRUB somewhere. This is the first partition of the disks, replicated after each installation of GRUB (basically when changing the system configuration). This is obviously unencrypted.

The second partition is where /boot lives. This partition is unencrypted, even if it can be, but we wouldn't be able to decrypt it remotely. The two second partitions, each on one disk, are replicated in a RAID 1 array, managed by mdadm:

# cat /proc/mdstat
md127 : active raid1 sdb2[1] sda2[0]
      523264 blocks super 1.2 [2/2] [UU]

The RAID array is composed of the two partitions number of the disks, each 1.8TB. This is where the rest of our filesystem lives, on top of LVM, but thats for later. It is also managed by mdadm:

# cat /proc/mdstat
md126 : active raid1 sda3[0] sdb3[1]
      1886273536 blocks super 1.2 [2/2] [UU]
      bitmap: 3/15 pages [12KB], 65536KB chunk

And then, for the fourth and last partitions, each disks have a 64GB partition of encrypted data, on top of which resides a swap partition. Those are not replicated. During normal operation, the system thus has 128GB of swap, and in the event of a drive failure, it still has the size of its RAM as swap.

Here is a quick recap of what we described so far:

                                     +------+
          +------------------------->+RAID 1+<------+
          |                          |/boot |       |
          |                          +------+       |
          |                                         |
+---------+----+-----------------------+  +---------+----+-----------------------+
|BIOS| Part of | Part of RAID|Encrypted|  |BIOS| Part of | Part of RAID|Encrypted|
|boot| RAID for| for LVM and |  SWAP   |  |boot| RAID for| for LVM and |  SWAP   |
|    | `/boot` |  filesystem |         |  |    | `/boot` |  filesystem |         |
+----+---------+--+----------+---------+  +----+---------+--+----------+---------+
                  |                                         |
                  |         +------------------------+      |
                  |         |RAID 1 for filesystem,  |      |
                  +-------->+on top of which resides +<-----+
                            |an encrypted volume, on |
                            |top of which resides LVM|
                            +------------------------+

Filesystem

The RAID partition used for the filesystem is encrypted. On top of that encrypted partition, resides an LVM volume group called vg0, composed of only one physical volume, that is the encrypted RAID partition. From this volume group, several logical volumes have been created:

# lvs
  LV                     VG  Attr       LSize
  home                   vg0 -wi-ao---- 100.00m
  home-diego             vg0 -wi-ao----   5.00g
  home-lewdax            vg0 -wi-ao----   5.00g
  home-risson            vg0 -wi-ao----   5.00g
  home-root              vg0 -wi-ao----   5.00g
  nix                    vg0 -wi-ao---- 100.00g
  opt                    vg0 -wi-ao---- 100.00m
  root                   vg0 -wi-ao----  10.00g
  srv                    vg0 -wi-ao---- 100.00g
  var                    vg0 -wi-ao----  10.00g
  var-cache              vg0 -wi-ao----   1.00g
  var-lib                vg0 -wi-ao----  10.00g
  var-lib-docker         vg0 -wi-ao----  20.00g
  var-lib-libvirt        vg0 -wi-ao----   1.00g
  var-lib-libvirt-images vg0 -wi-ao----  20.00g
  var-log                vg0 -wi-ao----  10.00g
  vm-lamacorp-ynh        vg0 -wi-ao---- 200.00g
  vm-lewdax-ynh          vg0 -wi-ao----  50.00g
  vm-wrongpasswd         vg0 -wi-a-----  20.00g

Here is a little explanation about all of them:

  • home: XFS, mounted at /home, for future use.
  • home-[user]: XFS, mounted at /home/[user]: one volume is created per-user.
  • home-root: XFS, mounted at /root: root's home.
  • nix: XFS, mounted at /nix, where the Nix store resides.
  • opt: XFS, mounted at /opt, for future use.
  • root: XFS, mounted at /, root of the filesystem.
  • srv: XFS, mounted at /srv, holds data for the services we run.
  • var: XFS, mounted at /var.
  • var-cache: XFS, mounted at /var/cache.
  • var-lib: XFS, mounted at /var/lib.
  • var-lib-docker: XFS, mounted at /var/lib/docker.
  • var-lib-libvirt: XFS, mounted at /var/lib/libvirt.
  • var-lib-libvirt-images: XFS, mounted at /var/lib/libvirt/images.
  • var-log: XFS, mounted at /var/log.
  • vm-[vm-name]: used as disk for the virtual machine named [vm-name].

Such a granular separation of the filesystem has been made in order to be able to snapshot only the folders we want to include. For instance, we want to snapshot everything in /var, except /var/cache, /var/log, /var/lib/docker and /var/lib/libvirt/images. This layout allows us to snapshot the logical volume named var, without including the contents of the directories listed previously, as they are not on the same logical volume.

You may also have noticed something about the boot.initrd.network option. This allows us to SSH one the server while it waits for the disks to be decrypted. A tutorial has been written about how to do that.