Posts on mapletree.moe

NixOS Part 1 - Introduction & Basic Setup

bakayuki — Sun, 04 Aug 2019 01:00:00 -0600

EDIT (2026/04/25): The remaining parts of this tutorial were never completed, but I do still hold a special place in my heart for NixOS.

Once again, various life stressors have resulted in me attempting to retreat into my little shell, and whenever that happens I always seem to find something new to work on. In this particular instance, the stress seemed to perfectly coincide with my discovery of a new Linux distribution, NixOS. I ended up becoming very fond of the distribution almost immediately, and after two weeks of living on it and learning a bit more about it, I recently redid the MacBook utilizing Nix as the sole operating system on it.

The next series of blog posts, assuming I manage to stay consistent, will be about my journey using NixOS, why I have become so fond of it so quickly, and other interesting things I come up with along the way.

Why NixOS

The most predominant form of package management in linux distributions tends to be imperative package management. Even in distributions like Gen/Funtoo, the make file is in general a static entity composed of variables that resolve how imperative commands, such as emerge system are executed. Essentially, we spend our time telling the system how we want it to accomplish what we actually want, and by following the list of imperative commands we (theoretically) end up with what we wanted.

The problem is, this doesn’t always work. To quote GlaDOS, “This former Linux Developer would like to remind you that Dependency Hell is a real place where you WILL be sent at the first sign of poor imperative order control.” Essentially, because we are so focused on how something will be accomplished, if anything along that chain breaks we end up in a genuinely unfortunate situation.

This is not an old problem, and many distributions have attempted to resolve this. Back in 2004/2005 when I was starting to get interested in Linux I spent some time talking to a friend about what I was using to tinker. While I cannot remember the specific distributions, I do know that RPM was the primary mode of package management on them. He essentially scoffed at me, and said while he was glad I was using Linux, he wished me luck when I inevitably discovered “RPM Hell,” perhaps one of the earliest forms of dependency hell.

When I inquired as to what he was using, he told me about Gentoo. This was a very very long time ago, where Stage 1/Stage 2/Stage 3 installations were a thing. The benefit, I was told, is that because literally every part of the package is imperatively defined ahead of time, you mitigate the chance of dependency hell, because you’re modifying the entire operating system as a single unit.

This began my extremely long and at times complicated history with the Gen/Funtoo community during my past life. I still hold an incredible soft spot for their ecosystem, but there are situations that can occur even within the portage subsystem, and the method of doing things imperatively often means that the problem isn’t discovered until the middle of rebuilding your system, for example with an emerge world. Not to mention there is a penalty, and it was even worse during the P3/P4 days, for building everything completely from source.

Various hybrid Binary/Source systems were developed over time, one that I remember offhand is Sabayon which started as a layer over Gentoo if I remember correctly, that would install unmodified packages from binary but anything modified would be built. However, this technique just solves the speed issue, it does nothing to resolve the imperative issues.

How does Nix Help

So Nix works by utilizing a declarative form of package management. Essentially, instead of focusing on telling Nix how we want our system, we tell it what system we want. We declare the Operating System we desire as a function, Nix evaluates that function, and then it decides how it will form that system. This is combined with transactional, atomic, procedures. Every “Generation” of Nix is a stand alone operating system. Installing/Un-installing packages never leaves any cruft behind because the packages are just dynamically linked upon boot. Removing the package from our declaration removes the request to symbolically link it to the system, which means it never existed.

Nix, in particular, is a functional package manager, an even more strict form of declarative package management. Instead of rehashing the explanations, I’ll let the wonderful nix webpage explain the difference. NixOS is simply an entire operating system built on top of Nix.

Beyond the benefits discussed on the official Nix website, there are other benefits to putting the entire operating system under functional control.

As an example, let us take the boot process. Since the system we are going to get to below utilizes encrypted swap and an encrypted root, you would expect that we need to handle a custom initrd, custom fstab, etc etc. You would be right! But instead of handling multiple files to get this done, we can actually do everything right in our configuration.nix.

Swap

swapDevices = [
  {
    device = "/dev/disk/by-uuid/f6533f92-baf2-4804-afda-880a7b5975ac";
    encrypted = {
      enable  = true;
      keyFile = "/mnt-root/root/swap.key"; # Believe it or not, this is correct.
      label   = "nixos-swap";
      blkDev  = "/dev/disk/by-uuid/6babbdb8-26ec-43ee-b7ab-76b43015acd3";
    };
  }
];

Root FS

boot = {
  initrd = {
    luks = {
      devices = {
        decrypted-disk-name = {
          device = "/dev/disk/by-uuid/0765a1fc-6045-45af-978e-db49609bc0e3";
          keyFile = "/root.key";
        };
      };
    };
  };
};

Additionally, while some distributions have opted to have several tools for building your grub.cfg, those still rely on modifying external files, for example under /etc/default or other directories. Instead, we add it right into our configuration.nix as well. We are just declaring what we want Nix to do, whatever else it decides it needs is up to it.

Grub Configuration

boot = {
  grub = {
    device                = "nodev";                # This isn't for BIOS.
    efiInstallAsRemovable = true;                   # Try to use Standard EFI.
    efiSupport            = true;                   # This IS for EFI.
    enable                = true;                   # Grub is needed for our weird shit
    enableCryptodisk      = true;                   # Add LUKS support
    extraInitrd           = "/boot/initrd.keys.gz"; # LUKS Key
    zfsSupport            = true;                   # Add ZFS support
  };
};

Some people may note right away that we’re building our key into the initrd and may worry about security issues, but we will get to that as well!

Needless to say, literally everything that is handled in configuration files throughout a normal linux distribution, is instead located in a centralized file. We can break that file up and import others as well, similar to any other programming language. Essentially, NixOS reduces the entire operating system to a series of Nix-Language source files, and we let Nix handle all the rest!

About Our System

So, for this initial article I will be discussing what I wanted out of my new daily driver operating system, and how I went about implementing it. You can look at my entire Nix-Configuration through the GitHub repository of the same name, but I won’t be referring to any specific files yet because the repository is going to change layouts several times throughout this series of articles, as I attempt to convert to a more functional form of creating my system.

So, without further ado, what are our requirements?

Encrypted Root Partition
ZFS Root Partition
Encrypted Boot(!) Partition, to protect our kernels and initrd
Encrypted Swap
Hibernation Support with encrypted swap(!)

There are actually more requirements, but these form the basis for this article. Before we get into it, it is worth noting that I used several different sources to compile all the steps needed to accomplish everything.

First, for Nix on ZFS, the NixOS Wiki Page of the same name was instrumental in solving the basic requirements of our work. We skip over ZFS Native encryption, because while it is not leaky in relevant ways, it is still slightly leaky. From man zfs:

zfs will not encrypt metadata related to the pool structure, including dataset
names, dataset hierarchy, file size, file holes, and dedup tables.

Next, we need to ensure that we can encrypt the boot directory. This blog post was instrumental in getting things to work. Had I not found this post, its possible that I would have forgone boot partition encryption, and I’m very glad I was able to get it done.

Last, but not least, encrypted swap with hibernation was enabled by following the first answer to this stack exchange question with some gentle modifications made.

Additionally, it is worth noting that the configuration.nix file was simply copied from my previous trial system, and tweaks made to it. The primary reason the repository is out of date is that while trying to get the system up and running, I ignored most standards of aesthetics, so I’d like to get it cleaned up proper before releasing it.

Let’s get started!

Part 1 - Live Environment Pre-Work

While installing the trial system, I simply used the minimal disk environment, without a GUI, and used my phone to access documentation. However, with the number of things I wanted to try this time, I figured it would be best to have a graphical environment to refer to the three sources mentioned above. This poses a unique problem on a Mac, as the proprietary NVIDIA driver is the only driver at the time of writing that will get an X-Session up and running.

Adding additional complexity is the fact that while preparing for this process, I disassembled my MacBook and somewhere along the way caused some sort of issue with the IO Board, which means I was limited to only one USB 3.0 port on the left side of the computer.

To start, I used macOS to install the new macOS beta. This was important because the only way to update MacBook firmware is through macOS. By installing a beta release, I was trying to get out ahead of any firmware updates to be released in the next six months. Fingers crossed, this is all that will be required, and we won’t need to figure out how to get macOS back because of a new firmware exploit.

Once the beta was installed, and everything was good to go, I downloaded a quick live-image of ElementaryOS (I knew the Nix live image would be a problem, and wanted to wait till the system was ready for installation to deal with it), and used Etcher to write it to a USB disk.

Rebooting into ElementaryOS, I ran a series of commands on the main SSD. Starting with blkdiscard, I initiated a manual TRIM on the disk to mark every sector as free of data. Next, I used an ATA Secure Erase command, per the Arch Wiki Memory Cell article to reset the drive to factory default write speed. For good measure, I ran the --security-erase-enhanced form of the wipe. Finally, I ran another blkdiscard on the drive, just to be really sure that everything was gone.

Next, we rebooted into ElementaryOS again, this time telling GRUB I wanted the whole live system to be stored in RAM. When this was done, I downloaded the NixOS Graphical Install CD, and burned the ISO to the USB drive.

Rebooting, I was presented with the boot menu, and I made sure to load NixOS entirely into RAM as well. The NixOS live CD does not contain any non-free firmware or software, which means the Mac Broadcom WIFI chipset will not be detected. I already needed to deal with one small issue with the live system, and it would be easier to simply unplug the USB drive and use a USB->Ethernet dongle to connect for the majority of the installation.

So, now we are inside the NixOS Live System, at a command prompt, and we have an internet connection. Attempting to boot into a GUI, as expected, results in a failure to find a valid display device. This isn’t as much of an issue as it could be on other systems, we simply need to edit /etc/nixos/configuration.nix on the live-system to include services.xserver.videoDrivers = [ "nvidia" ]; and then run nixos-rebuild switch. Once this completes, we run the given command to start the X-Session, and viola, it works. Checking all of our networking areas, we see that we have a proper internet connection, and we can move on to the more fun things.

Part 2 - Disk Configuration

This part is fairly straight forward. We use gdisk to set up three partitions. The first partition is for our EFI Boot Partition, the second will be for our Swap partition, made large enough to handle hibernation plus a little extra, and the third is our new root partition, where ZFS will live.

$ gdisk /dev/sda
GPT fdisk (gdisk) version 1.0.4

Partition table scan:
  MBR: not present
  BSD: not present
  APM: not present
  GPT: not present

Creating new GPT entries in memory.

Command (? for help): o
This option deletes all partitions and creates a new protective MBR.
Proceed? (Y/N): Y

Command (? for help): n
Partition number (1-128, default 1): 1
First sector (34-2097118, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-2097118, default = 2097118) or {+-}size{KMGTP}: +200M
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI System'

Command (? for help): n
Partition number (1-128, default 2): 2
First sector (34-2097118, default = 2048) or {+-}size{KMGTP}:
Last sector (2048-2097118, default = 2097118) or {+-}size{KMGTP}: +20G
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300): 8200
Changed type of partition to 'Linux swap'

Command (? for help): n
Partition number (2-128, default 3): 3
First sector (34-2097118, default = 411648) or {+-}size{KMGTP}:
Last sector (411648-2097118, default = 2097118) or {+-}size{KMGTP}:
Current type is 'Linux filesystem'
Hex code or GUID (L to show codes, Enter = 8300):
Changed type of partition to 'Linux filesystem'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): Y
OK; writing new GUID partition table (GPT) to /dev/sda.
The operation has completed successfully.

Now we can take a look at what we have:

$ gdisk /dev/sda
GPT fdisk (gdisk) version 1.0.4

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): p
Disk /dev/sda: 977105060 sectors, 465.9 GiB
Model: APPLE SSD SM0512
Sector size (logical/physical): 512/4096 bytes
Disk identifier (GUID): C35223A0-E004-474E-8B79-230B64658AB0
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 977105026
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048          411647   200.0 MiB   EF00  EFI System
   2          411648        42354687   20.0 GiB    8200  Linux swap
   3        42354688       977105026   445.7 GiB   8300  Linux filesystem

Command (? for help): q

So, everything is now set up on disk, it is time to build our filesystems.

Part 3 - Filesystems

With our disk structures on place, let’s talk about our filesystems. There will be three “main” ones, but it gets a bit more complex than that. First, let’s start by setting up our new EFI System Partition

mkfs.vfat /dev/sda1

That solves that issue, next we need to set up our two encrypted partitions. Despite the fact that we are going to use keyfiles, we should still establish a typed passphrase in the event we need to tweak the partitions from outside the operating system built on it. After setting up the encryption, we open each encrypted container and assign it a friendly name to work with.

cryptsetup luksFormat /dev/sda2

Enter passphrase:
Verify passphrase:
Command successful.

cryptsetup luksFormat /dev/sda3

Enter passphrase:
Verify passphrase:
Command successful.

cryptsetup luksOpen /dev/sda2 nixos-swap

Enter passphrase:
Command successful.

cryptsetup luksOpen /dev/sda3 nixos-root

Enter passphrase:
Command successful.

Okay, so now we have our containers. Next step is to form the basic filesystems inside of each one. Swap on our swap device, ZFS on our ZFS device. It is important that we use the /dev/disk/by-id/ entry with the UUID specification and the friendly name of the device. This makes identifying things easier when we work with them, and helps ZFS to understand what exactly is going on.

mkswap /dev/disk/by-id/dm-uuid-CRYPT-LUKS1-deadbeef-nixos-swap

Before we get to the ZFS setup, I’d like to explain the options I am using. While the explanations are available on the wiki page, they are restated here for brevity.

-O compression=lz4 - Disk space on an SSD is more valuable than CPU time. Using LZ4 will not impact the user experience to any discernible degree.
-O normalization=formD - Our whole filesystem will be in Unicode with this. While not really required, it could let you do some interesting things, and in general I like to use Unicode wherever possible.
-O xattr=sa - Boost performance with certain file attributes, this could become useful if I ever attempt system hardening (I likely will at some point)
-O acltype=posixacl - Required for systemd-journald
-O mountpoint=none - This turns off ZFS’ automount machinery. In certain instances, ZFS’ and NixOS’ boot time automounting machinery could trigger a race condition and prevent the system from booting. This allows us you bypass that potential completely.
-o ashift=12 - Force 4K sectors. It is very likely ZFS would have done this anyways, but instead of risking the chance that it would read the hardware incorrectly, I just manually declare it.

So, with that out of the way, here is what our nice bulky zpool creation command looks like:

zpool create -O compression=lz4 -O normalization=formD -O xattr=sa -O acltype=posixacl -O mountpoint=none -o ashift=12 zroot /dev/disk/by-id/dm-uuid-CRYPT-LUKS1-deadbeef-nixos-root

With our zpool initialized, next we need to form the filesystems under it. We will do three things. First, we will create a separate dataset for our /home directory, so that user data is kept somewhere separate from the root partition. Next, we will define a root dataset, and within that, a nixos dataset. This means, should we ever want to, we could run multiple distributions off of the same ZPool by nesting them under the root dataset, and pointing their /home at our home dataset. Is it likely we will ever do this? No, but it would be nice to have if we ever decided to try it!

Once again, we set our mount points to legacy to ensure that the automount machinery has absolutely nothing to go on, preventing it from firing during boot.

zfs create -o mountpoint=none zroot/root
zfs create -o mountpoint=legacy zroot/root/nixos
zfs create -o mountpoint=legacy zroot/home

Where does this leave us? As of right now, we have our EFI System Partition, freshly created with nothing on it. We have our Swap Partition, nested within a LUKS encrypted volume, and we have our three ZFS datasets, within our zroot zpool, within a LUKS encrypted volume.

Not yet done, we need to mount everything to the proper locations, and then do some additional work to make sure everything will boot as we want it to. To begin with, we will set up the easy ZFS mount points. Next, we need to mount our EFI System Partition to /mnt/efi. We do this because it will allow GRUB to write to the EFI partition, and have that point to our actual, encrypted boot directory, which we also create here. Lastly, for some additional work we will do here in a moment, we manually create the /root directory.

mount -t zfs zroot/root/nixos /mnt
mkdir /mnt/home
mount -t zfs zroot/home /mnt/home
mkdir /mnt/efi
mount /dev/sda1 /mnt/efi
mkdir /mnt/boot
mkdir /mnt/root

Now, we mentioned above that we would like to have the system automatically unlock. This is secure, because before we can even access GRUB directly, we will have to type in our decryption passphrase for our nixos-root partition. Essentially, everything we are about to do will be encrypted based on that master passphrase anyways, so there’s no real chance of a leak occurring.

To do this, we will create two binary keyfiles. swap.key will be the binary key for the Swap partition, and root.key the binary key for the root partition. We will use LUKS to assign those keys to their respective LUKS encrypted volumes, allowing the volumes to be decrypted both with a binary keyfile and a passphrase. The root.key file will then be packaged into a CPIO archive, and GRUB will append this to the initrd image made by NixOS.

During boot, we will type in our master password to unlock GRUB, select our boot entry, and then GRUB will hand over control to the initrd after appending our CPIO archive. During boot, the initrd will unlock /dev/sda3 using root.key and then hand over control to systemd. SystemD will continue the boot, and then load swap.key to unlock the swap partition. Since the swap partition is not encrypted randomly each time, this process can be repeated, thus enabling hibernation to function properly.

The end result of all of this is that a single master passphrase only needs to be entered once to allow the system to boot properly. Without this method, we would have to re-enter the nixos-root passphrase twice, and the nixos-swap passphrase once. I am not sure, but this also might break our hibernation capabilities.

Let’s get started. First we will create our binary keyfiles from /dev/urandom, then assign them to the volumes, then create the CPIO archive and stash it where it needs to be.

dd count=4096 bs=1 if=/dev/urandom of=/mnt/root/root.key
dd count=4096 bs=1 if=/dev/urandom of=/mnt/root/swap.key
cryptsetup luksAddKey /dev/sda2 /mnt/root/swap.key

Enter passphrase:
Command successful.

cryptsetup luksAddKey /dev/sda3 /mnt/root/root.key

Enter passphrase:
Command successful.

cd /mnt/root
echo ./root.key | cpio -o -H newc -R +0:+0 --reproducible | gzip -9 > /mnt/boot/initrd.keys.gz

With that last string of commands, we are all set. To recap what was accomplished in this section:

We created the FAT filesystem for our EFI System Partition
LUKS formatted /dev/sda2 and /dev/sda3 with a passphrase
Opened /dev/sda2 and /dev/sda3 and assigned them to nixos-swap and nixos-root, respectively
Created a swap FS on nixos-swap
Created a zpool on nixos-root
Created 3 datasets on nixos-root
Mounted everything correctly
Generated 2 binary keyfiles
Assigned each binary keyfile to its respective partition
Generated a CPIO archive for our initrd.

It’s time to move on to installing NixOS and configuring it to make use of our work.

End of Part 1

While I had intended for us to have a system up and running by the end of part one, this post is close to breaking 600 lines, and to be completely honest, this is the most I have written in quite a while. Part 2 will cover getting the system up and running, as well as a little preview of what our configuration.nix file will look like. Expect that installment to be quite a bit smaller than this one. Finally, part 3 will deal with the initial declarative configuration of our /home directory.

The end goal is that all user data will end up kept in a ~/.library directory, similar to the nix-store, and upon login, a symbolic-link farm will be built according to our declarative home-management system. In this way, everything, from /etc/nixos to various dotfiles will be kept in an easy to understand layout, and just linked to their less-easy to understand directories by the derivation created by home-management.

But, for another time.

More Blog Updates

bakayuki — Mon, 10 Dec 2018 01:00:00 -0600

There are yet more updates to the blog. The first of which is that we now have an actual domain name, which is [archived]. In keeping with my tradition of complete transparency, the domain was acquired through Gandi.net, after I found out that IWantMyName was unable to accept Discover cards. While I am still supportive of IWMN as a company, if they don’t accept my card it leaves me unable to use them.

Next, the DNS for this site is now handled through Cloudflare, which means that this site is now fully available via HTTPS with a valid SSL certificate. So, small victories.

While running through the process of updating the blog, I noticed several things were broken and went ahead and fixed those:

The “Site Version” link in the sidebar now properly links to the GitHub source repository.
~~A long standing issue with pagination has been corrected by updating to the jekyll-paginate-v2 gem, and rewriting the appropriate liquid blocks.~~
- Github-Pages does not support the v2 gem. Therefore, the site has been downgraded back to the v1 gem, and the liquid blocks were cleaned up based on trial and error.
Related posts are now actually related! This is accomplished by iterating through tags at compile time and creating a list of related posts. While this may not always be accurate, it is far more accurate than the time based system jekyll uses by default.
A small issue has been corrected with the header file used across pages. There was a typo that was generating invalid HTML. It didn’t cause any visible issues, but it was a problem all the same.
The archive page now uses a new Liquid code block. This is to resolve the long standing

problem, where the code would generate trailing closing tags.

HTTPS links have been enforced across the board. I cannot promise the site that you visit will have a valid SSL certificate, but we will certainly try to redirect the connection over SSL now.

~~HTML proofer is still throwing a few errors related to my consistent use of the Introduction and Conclusion headers, but these are not actual errors.~~

Even these errors have been fixed. HTMLProofer now returns a completely safe site.

I’m also in the process of going back through previous posts and cleaning up the YAML front matter. While this front-matter previously had very little impact on the site, it now can matter quite a lot with the way the related posts system works.

ThothBackup: Part 3

bakayuki — Sun, 09 Dec 2018 17:00:00 -0600

ThothBackup Project Series

EDIT (2026/04/25): As part of bringing the contents of the old blog back online, some things needed to be moved around. ThothBackup was originally a series of blog posts plus a dedicated project on the blog that was supposed to be completed, and never was. As much as could be preserved, was.

So, another week has gone, and it is time to update this blog with what I have learned. Unfortunately, experiments were not able to be run this week in the realm of data transfer. I decided to revisit the base system to focus on encrypting backup data while it is at rest on the system. This was one of the remaining security vulnerabilities with this process. While end-users still have to trust me, they can at least be assured the data is encrypted at rest.

Essentially, if the system was ever stolen, or our apartment door was broken down, we would just have to cut power and the data would be good. With that previous statement, please keep in mind that this week’s post only refers to the root drive. I didn’t make much progress because of things happening at work, but this is a nice, strong, foundation to build upon.

Many of the steps in this post were cobbled together from various sources across the internet. At the bottom of this post you can find a works cited that will show the posts that I used to gather the appropriate information.

End Goal

The end goal is to ensure that the operating system’s root drive is encrypted at rest. Full Disk Encryption is not an active security measure, it is a passive one. It is primarily there to ensure that should the system ever be stolen, it would not be readable. The root partition will not host any user data, so the encryption should be transparent and seamless.

In short, we will utilize a USB key to provide a Keyfile which will then be combined with LUKS encryption to unlock the LVM array to allow the initramfs to hand over control to the operating system.

Notes

Because we are using a solid state drive, and we will be filling the drive with data, it was important for me to over-provision the drive. The SSD we’re using comes with 240GB of space. We can assume that there is some form of manufacturer over-provisioning in play to get that number, if I had to guess I would assume there is actually 256GB of NAND memory on the drive, but only 240GB are made available to the user. This is a fairly reasonable level of over-provisioning.

However, with us planning to fill the drive with pseudorandom data in order to obfuscate the amount of data actually in use, this 16GB could potentially be used quite quickly. SSDs cannot actually rewrite sectors on the fly, they have to run a READ/ERASE/WRITE cycle. This is typically done by writing the new block to an over-provisioned area and then pointing the drive’s firmware at that block. In this way we avoid the ERASE penalty, which can be on the order of 0.5 seconds per block.

Essentially then, every single write to the drive will require a READ/ERASE/WRITE cycle, so padding the over-provisioning is a very good idea. It will help with wear leveling and prevent severe write amplification, while also making the drive “feel” faster.

Prior Work

Before we get into the new installation, we need to prepare the drive for its new role. Unless the flash cells are at their default state, the firmware will regard them as holding data and will not utilize them for wear leveling, thus rendering the over-provisioning useless.

To begin, boot the system via a Debian Live-CD and open up a root prompt using sudo.

If you, like me, prefer to work remotely, you will then need to run a sequence of commands to prep the system for SSH access. We need to add a password to the liveCD user, then install openSSH, and finally start the service. Once all of this is complete, you can log in from a more comfortable system.

apt-get update
apt-get install openssh-server
passwd user
systemctl start sshd

We will need to install one last software package, hdparm. Run apt-get install hdparm to grab it. Once you have done so, run hdparm -I /dev/sda. Under “Security” you are looking for the words “not frozen”. If it says frozen, and you are working remotely, you will need to access the physical console to suspend/resume the machine. This should unfreeze access to the ATA security system.

The first thing we need to do is to run an ATA Enhanced Erase. After this is done, I still like to run blkdiscard just to make sure every sector has been marked as empty. Finally, we will use hdparm to mark a host-protected-area, which the drive firmware will be able to use as an over-provisioning space. To calculate the HPA size, figure out what size you want to be available to you. Convert that into bytes, and divide by 512, which is the sector size. This will give you the number to pass to hdparm.

hdparm --user-master u --security-set-pass Eins /dev/sda
hdparm --user-master u --security-erase-enhanced Eins /dev/sda
blkdiscard /dev/sda
hdparm -Np390625000 --yes-i-know-what-i-am-doing /dev/sda
reboot

Once this is done reboot immediately. There is a lot that can go wrong if you fail to reboot. At this point, I swapped out my disk for the Debian installer. If you are doing this on your own 2006-2008 MacMini, you may want to use the AMD64-mac ISO that the Debian project provides.

From here, we just have to confirm that the drive shows up how we want in the installer (200GB in size, in my case), and we can proceed with the installation.

Installation

Most of the Debian installation process is self explanatory. The only point where I will interject is partitioning. Because of the way the MacMini2,1 boots, it is important that we use an MBR based grub installation. You can do a 32bit EFI installation, but it is very fragile, and I’m not a fan of fragile things. That being said, I still wanted the ability to use GPT partitions. I like being able to label everything from the partition up to the individual filesystems.

Accomplishing this is actually fairly easy anymore. You just need to create a 1MB grub_bios partition as part of your scheme and you’re good to go. To get the level of control we need, we will select manual partitioning when prompted to set up our partitions in the installer.

Create a new partition table (This will default to GPT), and then lay out your initial partition layout. It will look something like this:

                
#1        1MB     BIOS_PARTITION  none          grub_bios
#2        1GB     BOOT_PARTITION  ext4          bootable
#3        199GB   ROOT_PARTITION  crypto        crypto

When you select “Physical Volume For Encryption” it will prompt you to configure some features. You can customize the partition there, but I actually wanted more options than the GUI provided, so I accepted the defaults and planned to re-encrypt later. Please make sure to allow the installer to write encrypted data to the partition. Since we have already set up a customized HPA, a potential attacker already knows the maximum amount of cipher text that can be present, and if the HPA is disabled they would likely be able to gain access to more. Therefore, it is important that we take every possible precaution.

Once this is done, you should scroll to the top where it will say “Configure Encryption” or something similar. Select this option, then select the physical volume we just set up, and it should drop you back to the partitioning menu. This time, however, you will be able to see the newly unlocked crypto partition as something that we can further customize.

Select that volume and partition it like so:

                
#1        199GB                   none          lvm

The LVM option will show up in the menu as “Physical Volume for LVM.” From here, we go back up to the top of our menu and select “Configure Logical Volume Manager.” You will then be taken to a new screen where it should show that you have one PV available for use. Create a new volume group that fills the entire PV and name it as you would like. For this project, I named it djehuti-root and completed setup.

Next we need to create a Logical Volume for each partition that you would like to have. For me, this looked like the following:

    
#1                30GB    root-root
#2                25GB    root-home
#3                10GB    root-opt
#4                05GB    root-swap
#5                05GB    root-tmp
#6                10GB    root-usr-local
#7                10GB    root-var
#8                05GB    root-var-audit
#9                05GB    root-var-log
#10               05GB    root-var-tmp

Your layout may be similar. Once this is done, you can exit out and you will see that all of your logical volumes are now available for formatting. Since I wanted to stick with something stable, and most importantly resizable (more on why later), I picked ext4 for all of my partitioning. We will tweak mount options later. For now, the end product looked like the following:

                            
/dev/sda2                         ext4    /boot         defaults
/dev/djehuti-root/root-root       ext4    /             defaults
/dev/djehuti-root/root-home       ext4    /home         defaults
/dev/djehuti-root/root-opt        ext4    /opt          defaults
/dev/djehuti-root/root-swap       swapfs  none          defaults
/dev/djehuti-root/root-tmp        ext4    /tmp          defaults
/dev/djehuti-root/root-usr-local  ext4    /usr/local    defaults
/dev/djehuti-root/root-var        ext4    /var          defaults
/dev/djehuti-root/root-var-audit  ext4    /var/audit    defaults
/dev/djehuti-root/root-var-log    ext4    /var/log      defaults
/dev/djehuti-root/root-var-tmp    ext4    /var/tmp      defaults

Once everything is setup appropriately, follow through the installation until you get to the task-sel portion. You really only want to install an ssh server and the standard system utilities pack. Once the installation completes, reboot into your server and make sure everything boots appropriately. We’re going to be doing some offline tweaking after this point, so ensuring that everything is functioning as is will save you a lot of headache.

Once you are satisfied the initial installation is functioning and booting correctly, it is time to move on to re-encrypting the partition with our own heavily customized parameters.

Re-Encryption

This process isn’t so much difficult as it is simply time consuming. Go ahead and reboot your system to the boot media selection screen. You will want to swap out your Debian Installation CD for the Debian LiveCD that we used earlier. Once the disks have been swapped, boot into the live environment and then bring up a shell. We will first need to install the tools that we will use, and then run the actual command. The command is actually fairly self explanatory, so I won’t explain that, but I will explain the reasoning behind the parameters below.

apt-get update
apt-get install cryptsetup
cryptsetup-reencrypt /dev/sda3 --verbose --use-random --cipher serpent-xts-plain64 --key-size 512 --hash whirlpool --iter-time

So, onto the parameters:

cipher - I picked Serpent because it is widely acknowledged to be a more “secure” cipher. Appropriate text from the above link is as follows: “The official NIST report on AES competition classified Serpent as having a high security margin along with MARS and Twofish, in contrast to the adequate security margin of RC6 and Rijndael (currently AES).” The speed trade-off was negligible for me, as the true bottleneck in the system will be network speed, not disk speed.
key-size - The XTS algorithm requires double the number of bits to achieve the same level of security. Therefore, 512 bits are required to achieve an AES-256 level of security.
hash - In general, I prefer hashes that have actually had extensive cryptanalysis performed to very high round counts. The best example of an attack on whirlpool, with a worst case situation where the attacker controls almost all aspects of the hash, the time complexity is still 2^128th on 9.5 of 10 rounds. This establishes a known time to break of over 100 years.
iter-time - The higher your iteration time, the longer it takes to unlock, but it also makes it harder to break the hash function. So if we combine what we know above with a large iteration time, we gain fairly strong security at the expense of a long unlock time when using a passphrase.

Once these specifications have been entered, you simply need to press enter and sit back and relax as the system handles the rest. Once this process is complete, you should once again reset and boot into the system to verify that everything is still working as intended. If it is, you are ready for the next step, which is automating the unlock process.

Auto-Decryption

There are a few ways to handle USB key based auto-decryption. The end goal is to actually use a hardware security module to do this, and I don’t anticipate the FBI busting down my door any time soon for hosting the data of my friends and family, so I opted for one that is easily extendable.

Essentially, the key will live on an ext4 filesystem. It will be a simple hidden file, so nothing extremely complex to find. This shouldn’t be considered secure at this point, but it is paving the way to a slightly more secure future.

The first thing that I did, though it isn’t strictly necessary, is write random data to the entire USB stick. In my case, the USB drive could be found at /dev/sdb.

dd if=/dev/urandom of=/dev/sdb status=progress bs=1M

Once this is done, we’ve effectively destroyed the partition table. We will recreate a GPT table, and then create a partition that fills the usable space of the drive.

apt update
apt install parted
parted /dev/sdb
(parted) mklabel gpt
(parted) mkpart KEYS ext4 0% 100%
(parted) quit

Now we just create the filesystem, a mount point for the filesystem, and make our new LUKS keyfile. Once the file has been created, we just add it to the existing LUKS header.

mkfs.ext4 -L KEYS /dev/sdb1
mkdir /mnt/KEYS
mount LABEL=KEYS /mnt/KEYS
dd if=/dev/random of=/mnt/KEYS/.root_key bs=1 count=4096 status=progress
cryptsetup luksAddKey /dev/sda3 /mnt/KEYS/.root_key

After this point, the setup diverges a bit depending on what guide you follow. We will stick close to the guide posted to the Debian mailing list for now, as that guide got me a successful boot on the first try. The others are slightly more elegant looking, but at the expense of added complexity. As such, they may end up being the final configuration, but for this prototyping phase they are a bit excessive.

We have to modify the crypttab file to enable the keyfile to be loaded off of our freshly set up key drive.

sda3_crypt  UUID="..."  /dev/disk/by-label/KEYS:/.root_key:5  luks,initramfs,keyscript=/lib/cryptsetup/scripts/passdev,tries=2

At this point, we need to repackage our startup image, update grub, and reboot to test the whole package.

update-initramfs -tuck all
update-grub
reboot

At this point the system should boot automatically, but you will notice a weird systemd based timeout that happens. This is mentioned in the guide posted to the Debian Stretch mailing list, and is fairly easy to solve. We just need to create an empty service file to prevent systemd from doing it’s own thing.

touch "/etc/systemd/system/systemd-cryptsetup@sda3_crypt.service"
reboot

At this point, everything should boot correctly and quickly. You may notice a few thrown errors, but it shouldn’t be anything severe, more services loading out of order.

It used to be possible to allow for the creation of a fallback in the event that the key drive wasn’t present, but that seems to have been removed. I plan to look into it further when I have more time.

Conclusion

This concludes the first part of the Operating System setup process. The next step was originally planned to be thin-provisioning the partitions inside the djehuti-root volume group, but there seems to be some problems in getting the system to boot from a thin-provisioned root. I’m looking into a weird combined system, where the root is static but all the accessory partitions are thinly provisioned, but it will take time to tinker with this and report back.

Thin Provisioning isn’t strictly required, but it is a rather neat feature and I like the idea of being able to create more partitions than would technically fit. I’m not sure when this would be useful, but we will see.

Once all of this is finalized, we will move on to hardening the base system, and last but not least creating the Stage 1 Project page. Then it is back to experiments with data synchronization. This is a fairly large step back in progress, but I am hopeful it will result in a better end product, where security can be dynamically updated as needed.

Works Cited

The following sources were invaluable in cobbling this process together. I sincerely thank the authors both for figuring the process out and documenting the process online.

ThothBackup - Part 2

bakayuki — Sun, 02 Dec 2018 20:30:00 -0600

ThothBackup Project Series

Hello! It’s that time of the week again, where I update everyone on my latest work. This episode is far less technical and focuses more on the concept of a “One and Done” backup solution, aka the holy grail of data maintenance.

It fucking sucks.

Introduction

This entry is slightly unidirectional. The concept of a simple, easy to implement, catch everything you might ever need solution is quite literally the holy grail, yet it has never honestly been implemented. Sure, user data is generally scooped out, but in the day and age of game mods, and with some development projects taking place outside of the User directory, it seemed prudent to at least attempt the full backup. Well, I’ve been attempting it for seven days. Here’s what I’ve found.

Focus

We will not be focusing on the space impact of a complete backup. This is actually fairly negligible. With out-of-band deduplication, only one set of operating system files would ever be stored, so server side storage would reach a weird type of equilibrium fairly quickly. Instead, I’ll talk about three things:

Metadata Overhead
Metadata Processing
Initial Synchronization

There may be another post tonight talking about additional things, but this deserves it’s own little deal.

Metadata Overhead

A fully updated Windows 10 partition of your average gamer, aka my fiancé, is composed of 479,641 files and 70,005 directories which comprise a total data size of ~216 GiB. This is actually just the C drive and typical programs. If you factor in the actual game drive in use by our test case, that drive contains 354,315 files and 29,111 directories which comprise a total of ~385 GiB of space.

In summation, an initial synchronization of what is typically considered a “full system backup” comprises 833,956 files and 99,116 directories comprising ~601GiB which results in an average filesize of ~755KiB and an average directory size of ~9 files.

SyncThing creates a block store that is comprised of, by default, 128KiB blocks. This means that for our system, assuming the data is contiguous, we need 4923392 Metadata Entries. Assuming the files are NOT contiguous, this is probably closer to about 5 Million metadata entries. As of right now, the server side metadata storage for the testing pool is at 1.7 GiB and initial syncronization is not yet complete. Extrapolating a bit, we can assume that 2.0 GiB would not be an unreasonable size for a final server side data store.

The client side store, at the time of writing, is approximately 1 GiB and may grow slightly larger. However, I will use 1 GiB. This means that there is a plausible total of 3GiB of metadata overhead representing an overhead percentage of ~0.5% across the pool. Scaling up, this means 10 clients with 1TB of data each would require 51.2GB of Metadata.

Should anything happen to the metadata store, it would need to be rebuilt by data reprocessing. This introduces a potentially massive liability, as scanning frequency would need to be reduced to not impact the rebuild operation.

Metadata Processing

The server is capable of a hash rate of 107MB/s. I am picking the server’s hash rate because it is both the slowest hash rate of the pool and would have the most metadata that would need to be rebuilt.

For a complete rebuild of the data of our current cluster, it would take the server ~96 Minutes during which no data synchronization could occur. This equates to a minimum of 1 Missed Hourly Update and could potentially result in up to 2 missed hourly updates if the timing was unfortunate enough.

For a complete rebuild of the data of our theoretical cluster, we will allow for a hash rate of 300MB/s. The total data needed to be rebuilt would be 10TB. This would result in a database rebuilt time of ~10 Hours which could result in up to 11 missed synchronization attempts.

Initial Synchronization

The initial syncronization is composed of three primary parts. First, the client and host must agree on what folders to syncronize. Second, the client must build a database of the content hosted locally. Next, utilizing a rolling hash algorithm, data is entered into the metadata cache and transmitted to the server.

Per the developer of SyncThing, millions of small files are the worst case scenario for the backup system. As of my independent, albeit anecdotal testing, After 7 days the synchronization process is still in effect. This represents a very poor user experience and would not be ideal for a widespread rollout.

Conclusion

The primary goal of a backup utility is to synchronize files and achieve cross system consistency as quickly as possible. While it is true that eventually consistent systems are utilized in large scale operations, this type of consistency is allowable only, in my opinion, at data sizes over 10TB. The current testing set is approximately 1TB at most, and thus this is unacceptable.

Either the backup paradigm must change, or the utility used to implement it must change. While I do not expect to find any faster utilities for performing the backup process, I do plan to continue to experiment. At this time, however, it seems that the most likely way to make the process as friendly as possible would be the implementation of a default backup subset, with additional data added upon user request, and after the high priority synchronization had been completed.

Blog Updates

bakayuki — Fri, 30 Nov 2018 01:30:00 -0600

Yes, it’s that time of year again. I have updated the blog! You can find more information below.

Analytics

Google Analytics are back. I understand that some people may not like being tracked, but at that point you should have an add or tracking blocker installed. I recommend looking at the Brave Browser, which is what I personally use, or installing uBlock Origin. The reason I have added this back to the blog is that I have noticed links to this blog appearing in various places over the web, and I would like to be able to detect how much traffic I am getting from these links.

I wholeheartedly understand if you disapprove of the use of Google Analytics. If anyone is able to suggest a better service, that collects less user data, please open an issue in the GitHub repository for this site. I will gladly change providers as long as the new one is also free.

Theme Updates

I forked the Lanyon repository and applied all the currently pending pull requests. This should keep everything up to date with the latest version of Jekyll. To make it easier for anyone else looking for that information, I created a new PR in the lanyon repository to my mergers. You can also find them under my GitHub site.

Fonts

Somehow the fonts on the site got nuked during the upgrade. They are back in place now.

Page Speed

The new updates have not yet been optimized. Previously I used Google’s page load speed thingy to optimize the site. Maintaining that by hand is one of the reasons the upgrade was so painful to implement. I’m looking for a way to automate the process the same way that I have currently automated the link tests that I run prior to a push. This will likely involve writing a new process in the Rakefile, so it will take some time. In the meantime, the only thing that is really being pulled is a few font files and the analytics script, so the load impact shouldn’t be too bad.

Organization

There is a weird issue between rendering the site on my local machine and the way GitHub pages renders it on their side. To solve this I created a new template and appended it to all the pages that should not appear in the sidebar. Hopefully this strikes a good balance between being able to use standardized templates, as well as ease of use. The new template simply imports the existing page template under a new name.

Images

While I am primarily focused on text on this blog, I have recently included a few images. These are also hosted on GitHub, so I cleaned up the way the image directory is laid out. This has impacted all of one image, but should make any expansion in the future much easier.

Liquid Changes

There was some issue with the way the Liquid on the Tags page was written. This has been corrected accordingly.

About Page

The about page has been correctly updated! My view on some of the listed issues has evolved in recent years. You will now find those things struckthrough with comments added underneath.

Future Improvements

There is still minifying, javascript inlining, and font work to be done to make the page run faster. Additionally, the tags page is simply a disaster. Between all of that and the project pages, there is still a lot left to be done. Hopefully, all will be accomplished in time.

Conclusion

Hopefully all of these changes make the site a bit more enjoyable to use. I do understand if the use of analytics bothers you. Please make an issue in the tracker, or if someone already has, comment accordingly. If there are enough people using this site that honestly care, I might consider removing the analytics while I do further research.

ThothBackup: Part 1

bakayuki — Wed, 28 Nov 2018 16:30:00 -0600

ThothBackup Project Series

As many people may have guessed, this backup system very quickly got much larger than I initially expected. Because of the size of the backup project, the number of people interested, and how quickly things are changing along the way, I’ve decided to approach this project in a new way.

In the sidebar to the left you will notice there is a new link to a “Projects” directory. Here you will be able to find all my larger works. The project is now called ThothBackup, and what follows is a list of things I have learned along the way. All of this data will be consolidated and entered in a more coherent fashion into the project pages, so keep an eye out for those to update.

But for now, we have a lot of ground to cover, so let’s get to work.

Part 1 - Cross Platform is hard

One of the biggest parts of the backup project was its ability to be cross-platform. I want the system to be easy enough to use that anyone and everyone could grab a client, get it configured, and get going. To facilitate this, the initial idea was to use tools that were built into the operating system. On Linux and macOS this is easy enough, as rsync is installed on most distributions by default, and if it isn’t installed it is just a quick package manager installation away.

Then however, entered Windows. Initially I assumed that it would be easy to use with windows as well. After all, rsync.net has a nice little guide explaining how to set it up. However, their client can detect when you’re not using rsync.net servers (which is totally fair, there’s no hate from me on that) and limits using the program to 30 days. The other alternative is cwRsync, which was initially freeware, but has since changed to being a paid product. Obviously asking someone to play for a program to even be able to start to use the backup isn’t a great selling point.

The first idea that I had was to write something on my own. Maybe have shell scripts on all platforms check for required code and fetch anything that is needed. However, shell scripts are hard for many people to debug, and the sight of a command prompt can strike fear into the hearts of many windows users.

The second iteration of the idea was to write something in Python. However at that point the client is becoming a software project in its own right, and I didn’t start this to develop software, I started it because I wanted to set up a neat little backup service for my friends and family.

Thankfully, there are many other software suites that are both cross platform and useful for this task. We ended up going with SyncThing. SyncThing is an open-source (MPL2, which is a permissive form of copyleft) synchronization library that is cross-platform and written in Go. I’m a huge fan of Go even though I don’t actually write it myself, as it is a fantastic language for exactly this type of thing. Even better, SyncThing comes with easy to use and easy to understand GUIs, and is capable of NAT and Firewall punching via relays, and makes device configuration dependent on acceptance from both the server and the client. The protocol it uses is open source, and based on the usage reports at least one person is using it on 30 million files with 2,000 peers. Last, but most certainly not least, traffic is encrypted with 128 bit AES, and the protocol maintains perfect forward secrecy.

All of this (and a whole lot more, it really is an awesome bit of software) makes SyncThing perfect for our use case. This may not always remain the case, but it gives me somewhere to start. Even if we end up moving beyond SyncThing in the future, you really should give it a look. It is a phenomenal piece of software.

Part 2 - Changes in Sync Methods

As hinted above, the original plan to synchronize systems wasn’t going to work without more work than I was willing to put in to a single component of the system. Once we threw out the initial way the system was supposed to work, we had to retool the way things worked on the operating system too.

The original way the sync process was meant to work was that every user’s operating system would get its own Server Account, and rsync or some other synchronization system would be tunneled through SSH. I wasn’t sure if authentication would be handled by system accounts or LDAP, because I never got that far. But I did specifically pick the operating system (OpenSUSE) because of that distribution’s system configuration manager (YaST).

Now with the use of SyncThing, a daemon process would run under a single user, and all clients would then connect to that daemon process which would then write to disk using that daemon’s permission set. Thus, no need to worry about ACLs or anything of the like. It was interesting to work with ACLs though. You can see some of my old code if you browse through the commits history of the ThothBackup GitHub repository.

Part 3 - Filesystem Considerations

When I was testing the original synchronization strategy, I had everything being deposited onto BTRFS subvolumes that were mounted with the compress option. To be entirely honest, I wasn’t that impressed with the way the compression was working.

In the new system, BTRFS subvolumes are still being used (User, System Name, Operating System, Drive Name, Backup Client, Archive Client, etc) except now the subvolumes are mounted with compress-force option. Additionally, I have learned about out-of-band BTRFS deduplication and plan to play around with that at this stage in the project as well.

Part 4 - Operating Systems

I really, really like OpenSUSE. Like, a whole whole lot. It may very well be my favorite binary distribution, and I’ve used quite a few. I think the whole way it works is simply phenomenal, I like the company behind it, and it honestly boils down to just that: I like it.

But, after all the changes above I began to consider if I shouldn’t change distributions. Originally I thought of changing to BSD, but I was concerned about software availability. I know FreeBSD tends to have a very well maintained ports collection, but I was still.. concerned. Most tools in this arena seem to cater toward Linux, and if I was already changing multiple systems to avoid having to write new software, did I really want to run the risk of needing to write server side software?

After much deliberation, I ended up settling on Debian Stable with backports. The initial installation is extremely lean, and there is a truly massive amount of documentation available for Debian. It paid off well too. The initial install of Debian stable clocked in at 60MB of ram used, where as OpenSUSE was running around 200MB after reboot.

Conclusion

There is honestly still quite a bit more that needs to be discussed. One of the most amusing things that the past week or so has taught me is that Sydney’s computer is as good of a backup test as a normal single family household. His System has 4 drives, over one million files, a quarter of a million directories, and about a terabyte of used storage on it. Combining his single computer with my mac and a windows virtual machine, and we have as much testing as we could need.

Notes

I’m going to start including a little section at the bottom of each post to remind me what I need to work on. Hopefully having this publicly viewable will encourage me to actually follow through on writing more than one blog post every 18 days.

stage 1 project page
stage 2 project page
talk about security improvements that can be done
rewrite the server side new client script
talk about specific SyncThing configuration options used
write utility script to keep server config files up to date in git

ThothBackup: Stage 0 - Environment & Hardware Setup

bakayuki — Tue, 27 Nov 2018 16:30:00 -0600

ThothBackup Project Series

ThothBackup: Part 0
ThothBackup: Stage 0 (You are here.)
ThothBackup: Part 1
ThothBackup: Part 2
ThothBackup: Part 3

Environment & Hardware Setup

Introduction

This section will be divided into several parts. In the first part, we will talk about the environment that the server exists in. This will handle things like power and networking, basically everything that exists outside the server. Next, we will take a look at the server itself, and any changes that have been made to it. We will then discuss the currently used storage devices. Finally, we will talk about things that could be improved with this setup, and any potential changes that could be made to increase security.

Environment

Location

Everything lives in a TV-stand-esque cabinet inside the living room of our 800ft^2 apartment. The Mac Mini and the storage devices sit on a little shelf above the cabinet portion so they have ample access to fresh air, and the other devices sit inside the closed portion of the cabinet. The cabinet is actually right next to my bed, so in the event of a power outage the beep would theoretically be loud enough to wake me up.

Networking

The Mac Mini is connected to an Ubiquiti EdgeRouter Lite via a Netgear GS180v4 Unmanaged Switch. The WAN connection to the router is a 100mbps symmetrical fiber connection that will eventually be upgraded to a 1Gbps symmetrical fiber connection. The fiber is coverted to ethernet via a GPON to ethernet conversion jack. Cabling from each point is provided by LinkUp Fluke Certified Cat7 S/FTP Cables. While I don’t actually think these cables are any better than any other cable, I like the design of them and the way they feel. Besides, having an included Fluke report with each cable certainly can’t hurt.

Power

The Mac Mini is connected to a small battery backup. The battery wouldn’t be able to power it for the entire duration of the outage, but the theory is that it would allow time to get everything successfully and safely shut down.

Additionally, the router that the mini connects to the internet through is on the same battery. This is for the same reasons, as the router is an EdgeRouter and sudden shut off could potentially cause corruption to the boot image.

Finally, any and all storage devices that the server uses are connected to the battery as well. This ensures that any writes will be able to complete successfully.

The battery is connected to mains power a short distance away.

Server

Base Model

The Mac Mini that is being used is an A1176 Mac Mini with an Intel Core 2 Duo processor. You can see the original specifications of the Mac by following the link above. The Mac was purchased by me second hand for about $100 and some change and came with a whole host of accessories that are not in use at this time. Unfortunately, this Mac was one of the last ones before Apple switched to using a 64-bit EFI for the system firmware. You will read more about why that is unfortunate during stage one.

Upgrades & Modifications

EDIT(2026/04/25): Picture lost to time.

The Mac has been extensively modified. To begin with, many of the components inside have been removed. You can see from the above picture some of the modifications. The sound card, bluetooth card, wireless card, any errant wires, anything that is non-essential was stripped out and thrown away. The server has no ability to communicate wirelessly at this point. The only thing that remains is the IR sensor, but that is strange to me because I removed the actual receiver.

The CPU heatsink was removed, very carefully, and the thermal pad was replaced with Artic Silver thermal compound. The entire system was completely disassembled and meticulously cleaned with 99% Isopropyl alcohol. In addition, the fan was replaced with a new OEM fan to ensure longer survival.

The hard drive of the Mac was originally a 5400rpm Hitachi 160GB SATA drive that was white labeled for Apple specifically. This has been replaced with an OWC Mercury Electra 3G 240GB SSD and the RAM was upgraded with a matched set of OWC PC5300 DDR2 2GB memory modules. While it is likely I could have got the RAM and SSD from pretty much anywhere, I like OWC and their prices, for these upgrades at least, were reasonable.

Peripherals

The only peripheral worth listing in this section is a DVI EDID Dummy Plug which simulates a 1920x1200 monitor being attached to the Mac. Normally I would not bother with an EDID Dummy Plug because the server won’t need to render any graphics for this use case, but in their infinite wisdom Apple decided that this model of Mini should only be able to function as a Personal Computer, and the damn thing won’t load past the EFI unless it detects a monitor.

Other than power, networking, and storage, this is the only peripheral attached to the Mini.

Future Plans

It is possible more upgrades may be done on the Mini in time. The most significant issue preventing any further upgrades is that I cannot seem to get the damn computer to detect a USB drive for a boot option. Should I manage to get it to the point that the system will boot from USB, there are a few additional updates I would make:

Replace the Internal DVD-ROM Drive with a PATA to SATA converter from iFixit
Install a 1TB Western Digital RED data drive
Replace the top cover of the mini with the cover from an A1278 Mini Server
Possibly max out the CPU to an Intel Core2 T7600.
- Doing literally anything involving the CPU is a tedious endeavor because Apple used one way plastic plugs to mount the heatsink. I was lucky when I was changing the thermal compound, and I do not think I will get lucky again. This means that I would need to use nylon bolts to remount the HS and that solution isn’t particularly elegant to me.

Storage

Drive

Right now storage is provided by a single external drive. The drive is a G-Drive 4TB USB-C external drive, which internally is just a white labeled Western Digital 4TB RED drive. I bought the USB-C model even though the Mini only has USB 2.0 Ports because based on some preliminary research, USB-3 drives on USB-2 ports are still faster than USB-2 drives on USB-2 ports. Additionally, should I ever upgrade the server, the drive will just get faster instead of slower. Additionally, the USB-C port is just more elegant than the way they added USB3 onto existing USB2-B connections. I just like the look of them more. I understand that that is trivial, but that’s just the way it is.

Cable

While I am sure that the standard cable would have been more than sufficient, I have this weird thing about using manufacturer provided cables, even from a reputable manufacturer. To this end, I replaced the original cable with an Anker PowerLine+ USBC to USBA. It’s red, it’s pretty, and it has a pretty decent warranty on it.

Improvements

Honestly the biggest improvement that could be made would be swapping the server for something much more powerful. Right now I am looking at second hand Mac Pro towers, with Xeon processors and ECC Ram. Once I purchase one, it is likely that I would max it out and do some internal modifications to it before switching it out with the current Mini.

Beyond that obvious one though, other upgrades could be made as noted in the server section. The most realistic one that could happen relatively soon is the addition of a second external drive. This drive would allow for dedicated drives for each part of the data process, or more likely, mirrored drives to increase performance. That being said, the USB-2 connection isn’t actually the bottleneck in the system at this time. That discussion, however, will take place during Stage One.

Security Considerations

The biggest thing that could be done with respect to increasing security is the addition of a Hardware Security Module. Currently I am considering implementing the NitroKey HSM due to its reasonable price, open nature, and multiple interesting features. All of this is secondary, however, to the far more likely progression of upgrading the entire server with a new one. Once that is done, then it is reasonable that we may focus on security.

That said, it is possible that I will be unable to find a reasonable deal on secondhand towers, and in that case I may, instead, choose to continue working with this little Mini.

Conclusion

This concludes the first stage, or rather, the zeroth stage, of the backup project. We itemized the various components that form the foundation for the software aspects of this project, iterated over potential improvements, and listed security features that could be implemented at a later date. After this, we move on to Stage 1 where we talk about the chosen operating system and how to make it more secure for it’s use as a hosting environment.

ThothBackup: Part 0

bakayuki — Sat, 10 Nov 2018 12:00:00 -0600

ThothBackup Project Series

Introduction

I’ve been wanting to get back into blogging, or at least writing more consistently, and I’ve also been looking for projects to work on. One of my favorite things to do, oddly enough, is to work on systems that will manage large amounts of data. So, I’ve decided to start a new project.

This is the first in a series of posts detailing the creation of a multi-tiered backup system for me, some of my friends, and potentially (but not likely) others as well. I will warn people ahead of time that I’m not building any hardware that could be called “resilient,” instead I’m using things that I already have around the house, plus a few extra cheap purchases, and a lot of software to make things work.

Speaking of software, almost everything used will be cross-platform and open source. I will likely be writing a lot of “glue,” in the form of shell scripts or potentially even some ruby. I will also be trying to focus on keeping things secure on untrusted servers, and secure in transit, to sometimes ridiculous levels. That being said, this is NOT a zero-knowledge system. Since it is for me, family, and friends, they are made aware of this ahead of time. In time, it is possible that this setup will grow and evolve to make a zero-knowledge system that functions similarly to this initial design idea, but that’s far down the line.

None of these blog posts will likely be as refined as my Icinga2 tutorials. One of the things I am trying to do is just get myself into the habit of documenting my projects. If I keep having to revise, edit, and source, I’ll eventually give up. So if there is anything you are interested in learning more about, Google will likely be your best friend.

Essentially, these are meant to explain how I built the system, and why I did what I did. Sometimes that may be as simple as “because I like them more” and other times it may have to do with more in depth research.

Environment

The server will exist in an environment with a battery backup system provided for the server, any data drives, and the core router that the server will be connected to. Connection to the internet is provided with a 100mbps symmetrical fiber connection that can be upgraded instantly to 1Gbps symmetrical if needed. Connections between the server and the fiber node is provided via CAT-7A cabling. Routing is handled by a Ubiquiti EdgeRouter Lite, with 3gbps line rate and one million packet per second routing. IPv6 and IPv4 are supported.

Eventually the router will be able to provide routes both to the public internet and to the dn42 darknet and Tor services.

Hardware

The hardware isn’t exactly great, but it is the best that can be done at this time. The server consists of a Mac Mini Model 2,1 made circa early 2007. It contains a dual core Intel Core2 Duo processor running at 2.0 GHz. /proc/cpuinfo is provided here:

processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 CPU         T7200  @ 2.00GHz
stepping        : 6
microcode       : 0xd1
cpu MHz         : 2000.000
cache size      : 4096 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon
pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm pti tpr_shadow dtherm
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 3999.68
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

processor       : 1
vendor_id       : GenuineIntel
cpu family      : 6
model           : 15
model name      : Intel(R) Core(TM)2 CPU         T7200  @ 2.00GHz
stepping        : 6
microcode       : 0xd1
cpu MHz         : 1000.000
cache size      : 4096 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
apicid          : 1
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 10
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon
pebs bts rep_good nopl cpuid aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm lahf_lm pti tpr_shadow dtherm
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips        : 3999.68
clflush size    : 64
cache_alignment : 64
address sizes   : 36 bits physical, 48 bits virtual
power management:

The server has been modified substansially from it’s factory state. The sound card has been removed, as has the WiFi card and Bluetooth card. IR functionality has been disconnected. Any extraneous wires have been removed. The original 160GB Hitachi hard drive has been replaced with a 240GB solid state drive from OWC. The ram has been maxed out with 4GBs, again provided from OWC. The system is not capable of addressing all 4GB despite being a 64-bit system due to limitations imposed in the EFI. The CMOS battery was replaced, and the processor was unseated, cleaned, and thermal material replaced. The fan was also replaced to head off any potential issues caused by old age. You can see a picture of the internals of the system below.

EDIT(2026/04/25): Picture lost to time.

The system is connected to the router with a Fluke-Certified CAT-7A S/FTP dual shielded cable. The interface it is connected to supports a 1Gbps full-duplex connection.

The primary data disk is a G-Disk USB-C 4TB external drive. This drive utilizes a white label Western Digital Red WD40EFRX. It is connected to the Mac Mini via an Anker PowerLine+ USB-C to USB-A nylon cable.

The older Mac Mini’s would refuse to boot if the system could not detect that a monitor was connected to the system. Apple claims that this is because the Mac Mini was intended to be a true personal computer, which would generally prohibit the system’s operation without a display device. Strangely, no such detection is present to test for either a keyboard or a mouse.

To bypass this issue, a DVI Dummy Plug is installed in the Mini’s DVI port. This plug uses EDID to tell the system that a 1920x1200 capable display is connected. This has also been reported to be useful in accelerating VNC Remote Administration, however no GUI is present on the system.

Operating System

The operating system used on the server is OpenSUSE Leap 15 x64. You may remember that I mentioned above that the EFI used on the system is 32-bits. This limitation is bypassed by formatting the internal server drive to utilize a legacy MBR partition table. From this, GRUB2 loads in legacy mode, which is 32-bits. This provides the shim that allows the 32-bit system firmware to load a 64-bit operating system. Over 72 hours of stress testing revealed no instability with the operating system booted via this method.

OpenSUSE was installed using the Server profile. Originally the Transactional-Server profile was used, however the early morning reboots could cause issues, and the installation did not seem to play well with LVM.

Hardware Stress Testing

The internal solid state drive was tested via an OpenSUSE Live distribution. The computer was first suspended and resumed to bypass the system’s EFI’s proclivity for freezing the internal drive’s security interface. Once the interface was unlocked, smartctl was used to trigger the drives internal SMART short test, conveyance test, and extended test, in that order. All results were nominal. The internal drive was then wiped via hdparm by sending an ATA Enhanced Secure Erase Command with a non-null password.

A quick installation of OpenSUSE Leap 15 was then performed.

The hardware was then tested from inside the new installation utilizing the Stress-NG test suite for over 24 hours. During this time, four virtual memory test workers iterated over 100% of the available memory to force swapping as well as check for ram errors. Two CPU test workers were ran on the aggressive settings, and an IO worker was deployed to check the SSD interface. All systems passed the stress test perfectly, with benchmarks appropriate for a system from 2007.

The external drive was then connected to the system. A battery of tests were performed. First, the drive was tested via smartctl. As the drive is connected via USB, the -d sat,16 option was passed to allow communication directly with the USB-SATA bridge. The drive was tested out of the box using the SMART short self test, conveyance test, and extended self test. This first round of testing took 10 hours to execute. After the inital round of tests, the SMART attributes table and its corresponding values was recorded.

Then the badblocks command was ran to write to and read from every sector on the drive. The full command was badblocks -wsv -t random -b 4096 -c 4096 -p 4 -o /root/sdb.log /dev/sdb. This performed four read-write-verify passes. Each pass took approximately 50 hours to complete.

Once the badblocks battery was complete, smartctl was once again used to run the same testing sequence of short, conveyance, and extended self tests. The attribute table was once again dumped, it’s values recorded, and the values then compared to those from the table dumped prior to running badblocks.

As these tests are actually still in progress, the results are not yet definitive.

Conclusion

This first post simply outlines the primary variables in this project. Hardware, operating system, and environment should all be assumed to remain static for the remaining project entries. Should anything change, I will likely update this entry versus notating it in the respective entry where the change was made. It will be made clear that this entry has been edited, should that come to pass.

Extensive testing was performed to detect any chance of infant-mortality in the hardware. I have previously had negative experiences utilizing hardware right out of the box.

A Gift for a Friend's Den

bakayuki — Sun, 21 Oct 2018 16:00:00 -0600

EDIT (2026/04/25): This project was done long before current LLMs and the animosity around them. It was also all compiled locally, and everything was open source. For various reasons, this project is no longer available in source form.

Once again it has been quite a while since I updated this blog. Strangely, the last time it happened was also in October. There must be something in the air around this time of year. Like cold. There’s cold in the air.

Anyways this post is due to a gift I am working on for UrbanFriendDen, aka Ruben. While I don’t talk to Ruben often, he is an absurdly talented writer and all around Good Dude(TM). He recently tweeted that tomorrow is his birthday. Technically, he asked people to overthrow the American Imperialist Regime, but I am very tired and still in bed at 4PM, so I proposed something a little different.

The following blog post will detail two separate projects, one of which Ruben already knows about and the other he does not. In the first project, we will create a corpus of his DICKHARDBOILED Series and the Communist Manifesto. We will then train a TensorFlow network on this corpus following the wonderful article written by Max Woolf. While I would normally like to roll my own network, that takes an absurd amount of time and Ruben’s birthday is tomorrow.

In the second project, we will using the same training guide as used for project one. The difference is that we will first be scraping Ruben’s [wordpress site] and downloading all of his writings. Once done, they will be assembled into a corpus which the network will be trained on. We should end up with an interesting little network that sounds quite a bit like his fantastic writing style.

As a side note in which I gush, I’m such a huge fan of Ruben that I’ve planned on commissioning him to write corpus texts to train another neural network I’m building as a hobby. If anyone reading this remembers the original version of Seara, a sassy little helper robot/chat bot that did moderation on the Collegi Pixelmon server, this would be version 2.0 of her. Her new model will have a generative text backend that responds when her retrieval model doesn’t have a programmed solution, but to do so it requires a LOT of text. I’m hoping to be able to commission Ruben to write me text in her voice. I reached out to him once before, but alas, money is tight, and artists should always be compensated.

Anyways, off we go to work.

Important Considerations

While all of my writing is available under CC0/Public Domain (See the “No Rights Reserved” link in the sidebar), Ruben’s writing remains HIS OWN. Nothing in this article should be construed as sub-licensing his work or making it available in the public domain. I have asked for permission to release the training data, but even if the data is released it still remains his intellectual property. Don’t be an ass, credit artists for their work.

Additional Update

While I was writing this, Ruben granted permission for me to upload the data sets so that they are publicly available. I will be posting the link to the github repository once I’ve finished this article. These files are made available with permission from Ruben. If you attempt to use them to produce any derivative works that you intend to publish, please contact him before doing so. He’s fairly chill, but again this is his intellectual property.

Project One - Workers of Neo Noir Dark Noir City… UNITE

This one is actually fairly simple for us to get started on. I created a new directory inside my development workspace named “Ruben_Present”, and opened that up inside the Atom editor. Once I had defined it as a project directory, I created a subdirectory named “Project_1.”

Then I copied all of the text from Project Gutenberg’s copy of the Communist Manifesto that was linked to above and saved it to “manifesto.txt.” Once done, we just had to go through and remove extraneous information. Everything inside the file will be used to train the neural network, and it is incapable of distinguishing between headers, sub-headers, or body text. Since we write headers differently, they need to go. Formatting doesn’t honestly matter.

During that stage I also updated anything the spellchecker flagged to modern day spelling. Part of training a neural network is related to vectorization, and different spellings of the same word would be treated as different words, introducing unneeded complexity into the model. If there was no equivalent word, I left it alone.

After doing this, I took a look at the total size of manifesto.txt. We had a 72KB file, which meant that I would need at least 72KB of Ruben’s writing to be able to cross balance the two.

First I pulled down the original story. I applied the same process that was applied to the manifesto, checking for text marked by the spell checker, removing extraneous formatting (legs woman’s atypical font, for example), and reducing everything to 80 characters per line. (I feel it is worth noting that during this process I attempted to smoke two vaporizers simultaneously – it seemed appropriate for the subject matter – but my significant other frowned at me until I went back to just a single one. I still made sure that smoke was leaving my face at the maximum possible velocity. It’s worth noting that I choked at “inject an epi-pen of smoke directly into my lungs.” I really need to quit.)

After processing the original story, I took a look at the file size. We only had 10KB of data versus the 72KB of the manifesto. Luckily, Ruben has written more than one story for Mr. Hardboiled. Puffing furiously on my nicotine delivery device, Noir-inspired inspiration inspired me. I smoked the– sorry, I added the additional stories. You can find the first additional story here. The second one is located here.

After working on this, I once again checked our total data quantity. We had 22KB of Dick Hardboiled. 50KB short of what we would need to have a semi-balanced model. I spent a long time thinking about what to do. I could scrape some of Ruben’s other writing, but that would infringe on surprise number two. I decided that I would scrape Ruben’s twitter and put a piece of himself into the engine. The easiest way it seemed to do this was a python library called twitter-scraper. So I needed to create a throwaway python environment, install twitter scraper, and then pull down as many of Ruben’s tweets as I could. This is somewhat of a process so you can watch an ASCIInema of the process below. Remember, you can directly copy from this playing video if you wanted to do this yourself. You can find my comments in the video stream.

This is about a 12 minute video, and isn’t really needed unless you want to see how I obtained the twitter data.

The initial scrape yielded approximately 57KB of data. So I got to be a bit selective with what I kept. I spent a period of time processing the tweets by hand according to the same rules as listed above.

While doing this however, I realized that I also had scraped retweets. Part of the idea in doing this was to ensure that Ruben’s voice would stay at the for front. Also, you can tell this article was a stream of work from beginning to end and I am so sorry. Anyways, after looking at the data I had collected, I decided to pull down some additional stories from Ruben’s website to get to the amount of data I needed. I discarded the twitter scrape.

The most appropriate story to scrape to flesh out the data requirement was “A Cyber Punk.” To ensure hardboiled wouldn’t be drowned out, I duplicated the text twice within the hardboiled file. This may come back to bite me. The end result is that I needed approximately 30KB.

I ended up pulling down parts 1-6 to get a sufficient quantity of data. I then began the process of formatting the text to 80 characters per line, correcting any spelling “mistakes” and generally removing extraneous formatting. Around this time I decided I would actually end up making two models for this first project.

The first model will be JUST the manifesto and Hardboiled, which will lead to a… neo-noir inspired Karl Marx, and the second will be the manifesto with hardboiled and A Cyberpunk. This should give us half/half karl marx and Ruben.

Both models will be made available and I will post the resulting 1000 words from each model in this article.

I’m also out of nicotine at this point. My blood is more blood than nicotine.

It’s fuckin terrible.

(As a fun side note, I’m processing the text as I write this, which means this article also has a bizarre number of stream of consciousness interactions. One neat thing I’ve noticed is Ruben’s texts lend well to division by 80 Characters. This implies that he frequently writes with words that are a factor of 80. Neat! )

When the github repository goes live you will be able to find the raw files used to assemble the corpus under Project_1/Raw-Data. They are labeled text files according to their contents.

Now that we have all the needed plain text, it’s time to assemble them. Corpus-1.txt contains the Communist Manifesto and Dick Hardboiled. Corpus-2 contains the Communist Manifesto, Dick Hardboiled, and A Cyber Punk.

Now that we have the assembled data, we just need to follow the directions in the aforementioned article to generate our network. The reason we are using that notebook is because it gives us access to free compute power that will dramatically speed up the learning process. It isn’t as cool as doing it by hand, but time is getting away from me, and having access to google’s servers for free is pretty chill.

I ran corpus 1 over 10 epochs. Took approximately 15 minutes to fully execute.

The dataset is so small that the system needs longer to learn. The following output shows the process with only 10 epochs of iteration.

Training new model w/ 4-layer, 128-cell Bidirectional LSTMs
Training on 91,072 character sequences.
Epoch 1/10
88/88 [==============================] - 34s 389ms/step - loss: 3.6865
Epoch 2/10
88/88 [==============================] - 31s 355ms/step - loss: 2.5044
####################
Temperature: 0.2
####################
e the of the the at ont the the the the the the the the that the the the the the ant the the the the the the the the an the the tit the the ant the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the allint the t

he ant the atist the ther the tit at the the the the the the the the that the the the the the the ond the the the the an the the the the the the the the the the the inge the thes the the the the the the the the the and the the the the the the ant the the the the the the the the ant ant the the the t

he the the to the the the the the at the the the at the the the the the the that the the an the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the at the the the alle the the the the the

####################
Temperature: 0.5
####################
r tande the praly ont to tha tre anses atse taint at the the at ins the ang tit the alet tinge therre theat the at the of
ride ter pere ind tians thetterat pe iret there annt ant an tilouritis the ond thee theast tit the ante hal thatt tite to the the thationt pret, atitinge iteve dorectit thes
cess

thers ato tourer the toant the antiald the ang the tas the ang inge, catralationt ses all ont the the letet to the tuerte the tater itare ant at thiond I an stes thed indetins the thes the thed the and al thes acaren theres ind the te then thitast the the a anta the re on the
cke tar of the ter al t

end to pe ingentis the theins ther pent thestitint as issit alllind and thexat the the thet the the ant ant at thete erorin ta ofr tallse ithe the tint ant atiallind thet ants, allen the the to of the theeis thin thess thary the is the ancet an pay te the then mis fate olind tourind thel the thed as

####################
Temperature: 1.0
####################
a.

betess dete an ta. I of loto te Saved,, The iny ono tiousge
Acet mt moweve
I lI is tiherlpet BeLtay, onctly batitysmpemdt kut, da Fre thalt ta buls larisstam staliase rins sothestin the cheith itimet,
 do featialyin ancer the eptl plaoclouty fo aiteieboc she. Siis
ank.

I trours tthe aboumawatl

nd toe ive of
ctm
“Wes, oftian atamel, iy PI mut die, ar sad.
It this couleicol wie tarinten
bomabulawa tra piitio pan tareroftisttt I tret sithienk
Ivase
ant
vurs I IxEcjushat as. mole
tid atane las se thancitind, tho caporinnskin worre.
plabed grery ther iomat at oveang an.

ketres ave I Thied ap

Hen mtory anticogeied, ad, ediss tecitho ithereby tis
buto
Belkenst alte ties aiy ste ase timy, mtatehemingighe, mes, there way,
ither thiss.

coont cand ito e, twy groas.

eda iewpter,
antal ic tf t.
 lied med, talariont,
rs.
 mtat sas tadse
The Yarre cte semptoropahat ialptotse, to spaprans… wacay

Epoch 3/10
88/88 [==============================] - 31s 355ms/step - loss: 2.3817
Epoch 4/10
88/88 [==============================] - 31s 353ms/step - loss: 2.2756
####################
Temperature: 0.2
####################
he the the the the the the the the the and an and an and an and the and and an the the an the the and and and and an the the the the an and the the the the the the the the the and an the the the an the the the and an the an an the the the the the the the the the the the the the the the the the the a

e the the the an and and the an the the the the the the the the of the soce and an the the the the an the and the the the and and an the the the the the the the the the the the the the the and the the an the the the the the the and an the the the the the the the the the the the al the the and and an

of the and the the the the the the and and and an and al and the the an the the of an an the an of the and and the an the the an the the the the the the the the the al an the the an an the the the the the the and an and the and an the and an an the the the the the the the the the the the the the the

####################
Temperature: 0.5
####################
he in the and than polly and at on and an ang of the of an the thing manercall an pan an cond ad laris ald the of we the the the of the the con in the an the the the the an the the the and to an the socend the beve the the and in and the an the the the ape se the of ach and the the the the the is ay

on fin cans and and as sitry the an as of an of to the the trall the an and the cond at the and thit the, the the the far of as to abore an sed the as to popectan at abere to the an the the the so pen the to and ind of al of socing and an the ferty and all e of the the the the in an the antent ond f

ith the the an the in and in an the hang ant an to ande cand sontion acand in, the of al the ond the fin the ind the che an the the nes the the pre in and and and and all to the an alivermas ant efran aban of the the
ans the land al the in ton of of the of and bolly cant and to and an the coris inte

####################
Temperature: 1.0
####################
at a ince, aspon of cllitlal
I sen, the shatarsiasis, halin, itth berer, ducles nom.”
The fmads yo mewarily kinty an ond of peraterif henst worevisoss aldakel ior acs, ith kurgel opronge, on ag
the eprnif to an fatke poud the farissir collith-reain: sretacats ow she th Gerle
be any thanis alf crangh

rer. I rusap, aby a and banod foverrey wich ind wont me boled waby a inditicte bolf by giations ghaprpiticaser siltian as the the ad a tais of land on reut athes and gugne a them inucthendonng love yats, ton th is forked.

I ta the wie the ebome sorg of lory ftoicim and ectage.  Onuiss the ton er vu

cas ond batie, an so os and liciablery
ccicit ad ave th at’, and uved inkent smpeceve ughe whe fis don tat er thate ant sud efforestisingion, axpest lans fordetals
anme.
Th’n fin, tot diss ofmbald the an e and dis becoreryials a lagiat, them
cacey the ROiof ous a on feby, pnath Nin th goen exsht of

Epoch 5/10
88/88 [==============================] - 31s 355ms/step - loss: 2.1740
Epoch 6/10
88/88 [==============================] - 31s 353ms/step - loss: 2.0598
####################
Temperature: 0.2
####################
ll of the propertions of the bourgeoising of the proletered of the bourgeoisingeoising and are and in the bourgeoisingeoising of the prolered and and and of the prolered of the condionged of the prolertare and and and and and of the proletaricall and and of the sociation it of the bourgeoising of th

all and and the bourgeoising of the socially of the prolered and of the proletaricall and of the bourgeoising and and and and of the bourgeoising of the ware and social and and and the proletarial of the and and and the proletaries of the the conding and and the berection, of the proletaricall of th

e bourgeoising of the proletaricall the proletered of the bourgeoisingeoising of the bourgeoising and of and of the condere at of the bere of the contered and of the prolere and the sociall of the proletaricall of and and of the conderers and of the the prolentaring the conding of the proletaricall

####################
Temperature: 0.5
####################
iatingsicall
the nenther of the poluctioningeoiss.

The pracentry of the
goling the leargeois of the countion the momeral all
condernote of the nasertensens and and are as the socation.

The and age itsence of and courgeoisically of
the condernolition the of
mourgeoisingeoises of the
recainers the t

.  The prole, the condif lether the and the prerentorer the proletenticall the co manderaction
con and contered the socout of that beresetion in the and to ginto the bourgeoisinalially for at and in that the seace at bourgeoisition to and bere proletaly
con ragen pourgeoisition in the clation
gelath

d an sepistorial the ine on of the the ine in to mourgeoising cand the contung apreroped and mourgeoising the the andernere arestions abourgeois of meacentungeoision, a sule of bertorgeoisie. I prole the more safling the caring of approlopre at of and compilartion.

The bourgeoinition
the and with a

####################
Temperature: 1.0
####################
er conder of
graigion
sindo iar for ougllly. Thithe listlizatall.  Thoudsh DAllllg
nanke courne ce retill atser. It I in forgatings to
than furnerich-poivaingel, vasist ackeander.  Freack abardeing igneave to
acher mase of
a
moverc of
thee orelve, fin marecoret. Ind eay’t I’maef
a efrae the a yolved

onating wintsh pures the prolever themed pre an owst engaved curadecten. I fore ay letachiated of ital the free loithar.

“rominat, da deck!”
Incosetion" of by reegent gera weremominioust reeader.
 The parke of clacaderered the bugtgheiren wapionual hquin rithe mopestoracully cotiond dould’s ho clar

thole meantoretion. ’vend wheses hemomerlize.
E’s blolll and offichetery.

The madterainduing aganiby horefaricatal the niabusical are andernevend, dave reomenc
to
pentary ith rive, rig andemens an jus worm vare us phan’se alition.
 Therys then they ore bourgeoinsimales of iner casterepreteoas..”, e

Epoch 7/10
88/88 [==============================] - 31s 354ms/step - loss: 1.9494
Epoch 8/10
88/88 [==============================] - 31s 353ms/step - loss: 1.8507
####################
Temperature: 0.2
####################
n the working and the proletarial and in the proletaries of the proletaries of the proletaries of the conding of the proletaries and mover the proletaries the sellice of the proletaries of the proletaries and and the proletaries and in the modern of the proletaries of the wat and and and the and in

the proletaries of the conding of the proletaries of the bourgeoisie of the for in the proletaries of the modern of the proletaries of the modern of the proletaries of the competion of the proletaries of the proletariat of the proletaries of the proletariat and and the modentry and and the proletari

e modern of the proletaries society and the with and and and in the proletaries of the conding of the bourgeois in the proletaries of the proletaries of the proletaries of the conding of the proletariat and the proletaries of the proletaries of the proletaries of the proletariat of the modern of the

####################
Temperature: 0.5
####################
 the clolution and in the condings of the down the now proletaries introductions she proleved in social and of the and in the ore mover the bourgeoisies, the property and rastion of the condine worne.

And the mounder the she proletariat sitered the society, in the nevernoption the and and in a hise

ry histred cits of mench ine conderne the do class with the larling of the proletaring and the proletaring in the modern on the wat of my the wast of her the clansss of the leverty on the mere the in bourgeois and the ine productions of the
movent of the destrence nother the class, when and the into

ore in the pastion the country the prolutions of the reepentions of the whis clas for cleass of the proletarion of the conding in the class class achered society society it apprers, and lition mence of the proletariate, society factions the smenetion and by
modern the commention of the production pr

####################
Temperature: 1.0
####################
t sepres tingh proda,
They the isncited – I lollicazal seations and leath herrover of hizder hathed’tion, wat selyre ourde of thee sunn’in retently steansy.

The coundend and and and ove cind-alod.
 The
proletaringé
fat adident in labe tot and for pate. The agelopede the clallf the hallibal the clod

 tacel, an whis Fen the moabile her, the gected-veren an freforen the lacinticalilly. In jentalizally feat the well.  Thion you apond her prises of the smostiver of it iny
bearthing awituall purolever the stanbleciatly bentor feous, upe the naze out in antivent. Ho dion. Mand sins.” I lew nistay her

thee anal bourgeoismings the at pelode proder an
but
shear
be reash indietion the wactes, im?
 I heand whis Stas ust. I when’sel my treea sepre of the Socuce esindy.  In the haver forsered sou’ly and of an
the pornations withs cinsh wornountive no. Hhe
Ger the senc the ame of sother form and
dithh w

Epoch 9/10
88/88 [==============================] - 31s 354ms/step - loss: 1.7696
Epoch 10/10
88/88 [==============================] - 31s 352ms/step - loss: 1.7040
####################
Temperature: 0.2
####################
iat of the working of the proletariat of the counding of the proletariat of the bourgeoisie of the social of the proletariat of the proletariat the proletariat of the modern of the proletariat of the compintence of the proletariat of the comment of the proletariat of the proletariat of the proletari

e proletariat in the conditions of the proletariat of the proletariat of the proletariat of the working of the proletariat of the working of the proletariat of the proletariat of the proletariat, the proletariat of the modern of the modern of the proletariat of the proletariat the proletariat of the

rking the proletariat of the proletariat of the proletariat of the selfice of the proletariat of the countriations of the proletariat the proletariat of the working on the bourgeoisie of the proletariat of the social of the proletariat in society of the proletariat of the bourgeois the proletariat o

####################
Temperature: 0.5
####################
ordaw.

I class of the proletariat and it the classs of the proletariat has the cas. The itsurs of mans and becases, in the popcetent of the proletariat up class bourgeois condition the socialisty, in the to the workers of inderation the proletariat and of the modern and of the complete of the repan

he socialism of the bourgeoisie, the proletariat a society.

The country, and barling the carst the the working and the rever of ont the bourgeoisie of the prentence of the lother production, the bourgeoisie of the comment condition of the ore have sead inderation of and no her for its with mange th

n onterant of the modern the
French of all and in the proletariat of the proletarian of the the proletariat of the politial in the shanke shourgeois head of wall with inding of
standing of the complosition to was the down you existion of the canists of production of inting condernation the groletari

####################
Temperature: 1.0
####################
ongual moust, mut the Cohm more whom belopred that, the bourgeois ine of impepencant wivicums and
and
more to bourgeed us of the not like in the coin cramborer, but fir and polterians, with the useving one the suringain frieas, ce on the revolking pullivale predevenct in the worting a cocorets, no g

dms it the sonsing widss, the the fromen hich peconce of the onalilutions
crose dive and feudtory poterte,
fell bess mave stomence in heroms,
class and
miss of
untam, indeed had the clich but is drestroopres her, and
gontername musep-codustrie, the the wag.  They Oust when for by lasucts a socust "P

f mand to shaire sup the chifaped of but bemaning and of voing.

Neo lonje circtious yrair?
Gerss a’lly theypp foure on make fifferent slucks its, of abrongior its inuce,
der.

I a Neir
dis shey lorks you polition, raking where noger of her I moter squelling sacke?
The beemed
of the prolutical conpr

The higher the temperature the more creative the model is allowed to be. Around this time I began to panic that there might not be enough data to properly train such an advanced network.

I quickly decided to make some changes to the model. To begin with, I had been training the model to learn how to spell. With such a small dataset, it would likely be a better idea to teach it how to construct sentences. To do so, I set the neural network to learn at the “word level” versus the character level.

To account for this learning change, I also modified the maximum number of previous words the model would be allowed to consider before generating a new word from 40 to 8. This is approximately the number of words in a basic sentence.

Second, looking at our data reveals that it follows three very different schemas, or formats. I had initially tried to train the model so that it would learn forwards and backwards, but without more concrete and stable formatting this was inefficient. I disabled bidirectional learning for the LSTM.

Finally, in QUINTUPLED the number of epochs to 50. I then executed the changes against corpus 1. While this also produced highly amusing output, I spent some additional time tweaking the neural network.

Some time later

After playing with a bunch of settings, I ended up modifying a lot. The first thing I did was increase the Hardboiled:Manifesto ratio in Corpus 1. Once I increased the ratio, I duplicated the resulting text to get over 10,000 lines.

I then did the same thing and duplicated all the content in Corpus 2 until I also had over 10,000 lines. Finally, I did a bunch of customization to the neural network. At one point I had gone so focused that it was only capable of learning the word of. Here’s the resulting configuration:

model_cfg = {
    'rnn_size': 128,
    'rnn_layers': 4,
    'rnn_bidirectional': True,
    'max_length': 10,
    'max_words': 10000,
    'dim_embeddings': 100,
    'word_level': True,
}

train_cfg = {
    'line_delimited': False,
    'num_epochs': 700,
    'gen_epochs': 50,
    'batch_size': 1024,
    'train_size': 0.75,
    'dropout': 0.0,
    'max_gen_length': 300,
    'validation': True,
    'is_csv': False
}

The model is currently trianing, and I expect it to take approximately 3-4 hours. I plan to update this post with new information as I compile it, but I do work tomorrow so I have to be responsible. :(

Collegi Backup - Work Notes

bakayuki — Fri, 14 Oct 2016 03:00:00 -0500

EDIT (2026/04/25): Collegi has been discontinued for quite a while now, the below information is still broadly useful, however, so it is maintained. However, I have switched from borg to another software called Kopia.

Collegi Minecraft Server Series

Collegi Pixelmon Server Backup
Collegi Backup System: Part 2
Collegi Backup Work Notes (You are here.)

The following post is an extreme rough draft. In fact, it isn’t even actually a post. These are my development notes from my refactoring of the collegi data infrastructure. As such, they’re arranged in no real sensible order besides having been written chronologically. Additionally, these have not been proofread, grammar checked, copyedited, or spell checked, as i write them in an IDE and not an actual text editor. As such, please don’t judge my writing ability off of them. More importantly, however, these do not have the standardized links that i provide to new concepts or commands in my blog posts, as embedding links to things I already know or have access to in a developer log that on average no one else sees just seems silly.

So, if you have questions, use google, and expect these to be updated over time.

The logs as of this posting run from 10/13/2016 to 10/16/2016, so over three days of work. There is a -LOT- more to be done.

They are broken down into the following format. Each list is a set of specific actions I took, and sometimes the list ends up with notes in it because, again, no one generally sees these, but under the task list is the space reserved for notes on the above list. Then a new task list is declared, then notes, then tasks, and so on and so forth. Generally each new task heading would signify a new blog post, talking about the tasks and the notes, so keep that in mind.

These were requested by Kan, a player on our server. Enjoy!

Tasks - Part 1

Made a backup of the repository as it stood on 2016-10-13 in the event anything breaks too badly during this.
Removed all existing submodules from the git repository. Committed the removal.
Ran the previous backup script to make sure that 10/13 was backed up. This included new additions to git annex.
Forced git annex to drop the old SHA256E key-value backend files that were made obsolete by the conversion to SHA512E key-value backend.

Notes 1: During this time, and while watching the way the version 1.0 backup script ran, I noticed there is a significant performance penalty for moving the location of the local mirror. Borg uses the entire path as the file name, so any deviation in the path spec causes it to treat the files as brand new. Note that this does not cause any issues with de-duplication, but the process of adding these files causes a massive performance hit. This made me start thinking about including the local mirror in the git annex so that as long as the annex was kept in tact in regards to metadata, the paths would remain the same as all additions to Borg would take place from the same root directory.

The problem with this would be the fact that annex keeps everything as symlinks. As such, I am looking into the unlock feature of version six repositories.

Notes 2: Dropping unused from a local area goes -much- faster than dropping from remote. Who knew, right?

Tasks - Part 2

git-Annex drop completed, but Finder isn’t showing a reduction in used drive space, but I think this is more an error on the side of finder than something with git annex, as du -h showed the directory was down to the size it should have been. Once I manage to get this finder thing figured out, I’ll move on to the next part.
Finder is taking too bloody long to figure its shit out, so I moved on to the next step in cleaning up the repository. I’m rewriting the commit history to completely remove files I don’t need from the actual git repo. In theory this shouldn’t touch git-annex at all, but that remains to be seen.
Ran BFG Repo Cleaner on the following directories and files:
- collegi.web
- collegi.pack
- collegi.git
- .DS_Store
- .gitmodules
- collegi.logs (Just for a moment, and we made backups.)
- collegi.configs
Ran filter branch to purge any empty commits left after the above.
Expired original ref-logs, repacked archive.

Notes 3: At this point we had gone from 230 commits to 102 commits. We were also left with the original envisioning of what this repo would be, which was a simple git annex to push files to Backblaze b2 from the Borg repository. Now to verify that all of our data is still 100% ok.

Tasks - Part 3

Ran git fsck
Ran git annex fsck

Notes 4: Wow this is going to take a long fucking time. Who woulda thunk it.

Notes 5: So apparently the current version of git-annex is using the old mixed hashing method, which is a format that “we would like to stop using” according to the wiki. Might need to migrate. Need to figure out how.

Notes 6: From the wiki: “Initial benchmarks suggest that going from xX/yY/KEY/OBJ to xX/yY/OBJ directories would improve speed 3x.” It’s worth migrating.

Tasks - Part 4

Run git annex uninit
Reading through the git-annex-init man page to see what else we should change now since we’re already migrating. Post Uninit we’re going to have to run a full borg data consistancy check.

Notes 7: Ugh. The document I found was actually an theoretical one, and while it is true that git-annex does use the new hashing format in bare repositories there is no actual way to move to the new one in a regular repo. So I am running an uninit for basically no reason. The only good thing about this that I can think of is that I will be able to reform the final git-annex repo in a much saner fashion. The bad news is that I have lost the log files, unless git-annex is going to bring those back for me. I am annoyed.

Notes 8: Good news! I just remembered that I had made a rsynced backup of the repository before I started fucking with it. So I didn’t actually lose the log files, I just went ahead and pulled them out of the git-annex backup.

Tasks - Part 5

After the git annex had uninitialized, I decided that if I was going to do this whole damn thing over again I was going to do it right.
Started a new borg repository in new-collegi. Pulled out contents from the original borg repository, using backups to restore any files that got hit in the above clusterfuck, then recompressed with maximum LZMA compression.
During this period I also standardized how the borg create paths would work. The server would exist within a collegi.mirror directory, and the entire directory would be added to borg upon each run of the backup script. This effectively means we never have to worry about the LZMA penalty discussed below again after the first re-add, unless we do major server restructuring, because paths will remain stable between commits.

Notes 9: The initial speed penalty for using LZMA is absolutely jaw dropping. One borg create took eight hours to complete. Eight. However, I quickly noticed that due to Borg’s de-duplication mechanism, the add times got faster the more data I added, and gzip-9 to lzma-9 did actually yield some improvement. It also reduces the incentive for me to do this fucking disaster again, because of how much it absolutely fucking sucks.

Notes 10: As an example of what I mean by the above, the initial adding of 1.8.9 took six hours with LZMA-9. When the map was changed from NewSeed over to Collegi, it took another four hours just to update the paths and what not, even though the data hadn’t updated, just the paths have changed. (This is indicated by the fact that the total repository size barely increased, all the size that changed could be explained by new metadata.) However, when the paths are kept the same, adding 100GB of data takes 13 to 15 minutes. So, the benefit of LZMA-9 is worth the initial startup, imho.

Notes 11: borg extracting from the GZIP-9 archives takes about 40 minutes, and that’s from highly de-duplicated and GZIP-9 archives. What this means is that pulling from an lzma-9 is probably going to take about an hour, depending on just how de-duplicated the archive is (as in, how many different chunk files contain parts needed to reassemble the original content).

Notes 12: Have hit the series of backups where things have moved into the Users path, and I’m restructuring them. It made me think about how I will handle the mirror directory in the future. I think I am going to do a few new things with respect to the new setup. The mirror directory will be a part of the git-annex repository, so there will be a new folder inside it called collegi.mirror or something similar, and then I can move the new backup script to be ran from the root directory, which will be beneficial. That way everything is neatly packaged. the issue becomes mirroring this, because uploading that much constantly changing data to backblaze would be literally stupid, and not at all within our budget. What I will likely do is initialize a “bare repository” on my time machine drive, and mirror the entirity of the git-annex repository to that.

Mandatory Break Notes

You need to run borg info to make sure the latest creation thingy is the proper size, and a borg check might not be a bad idea either as you fell asleep and closed the mac during work on the repo.
Cleaned the time machine volume of the repeated backups of the new repository because it doesn’t make any sense to have 20 versions of it.
Moved the repo to the time machine drive as temporary storage using rsync.

Tasks - Part 6

Restarted the transfer process starting on the 8th of October

Notes 13: Not a huge shock but running some of these commands across USB 2.0 can add anywhere from 10 to 30 minutes. Doing them cross device gets even worse, with some transactions taking almost an hour.

Notes 14: I’ve been going back and forth on what filesystem I would like to deploy since I am redoing the collegi drive as a whole. Now the interesting thing to note here is that by the time I get this thing fully ready to deploy, the drive I have here may not be the drive it ends up on, but this is as good of a testbed as any. I’m really thinking I will go with apfs. Most of the gripes I have with it are easily resolved through borg and git annex.

Notes 15: In a highly amusing turn of events, it is bigger in lzma 9 than it was with gzip 9. weird.

Notes 16: While it would likely be prudent to go back to the previous compression method, the benefits that I have made to the directory structure while redoing the borg repository are worth the few extra gigabytes of overhead especially concerning with Backblaze B2 it barely costs a penny.

Tasks - Part 7

Use JHFSX for the new drive. I would have really liked to use APFS but I am still worried about the data loss considering there is almost a year till it will ship. JHFSX is reasonable enough for right now, while still being safe to unplug.
I went round and round on using encryption on the new drive. did it.
using rsync to bring the data to its final resting location.
OK started setting things up
Defined gitlab as the metadata backup again
created a bare repository on skaia
set up prefered content so skaia requires everything in the main repo
set the main repo to require a –force to drop content via preferred content
Set the backend to SHA512E
began the long process of adding the data to the git-annex
Set up bin directory to not be tracked by git-annex but instead by git
added backblaze remote, not encrypted, with a proper prefix
started to sync to backblaze
noticed an issue with how the sync was going to gitlab, will correct.
- corrected the issue