I use a lot of object storage. Not a lot from the perspective of (insert your favorite example tech company here), but for a private user/small community manager, a good amount. Looking back over things, my spend this year peaked at about 35 euro in a 30 day period, which at my current pricing is about 5.8TB.

Most of that goes toward the “Tokisaki Memory Engine” I run, which is really just a fancy name for kopia plus a dedicated repo server, and some special sauce that is usually lost in the weeds, like object locking, restricted keys, etc. We even have automated daily testing of restores to /dev/null. Neat!

The main goal is to let someone get all the benefits of kopia with just an username and password, while only giving up the same thing they give up with most other cloud storage providers (Mainly the fact that if I really really wanted to, I suppose I could look through your stuff, but it’d be damn obnoxious to do because of how the server works, and its built that anyone could catch me at any time, so, why bother).

I migrated last year to Hetzner’s Object Storage after B2 became VERY UNHAPPY with how many API calls my little harness around kopia uses. The migration was kind of in a rush, and I forgot some things!

This won’t detail all the fun that I have had growing tokisaki-v4 along-side Hetzner’s Object Storage (which has been having its own growing pains), but a fun little oops I just found.

Huh, that’s weird

So after fixing some earlier corruption this year and reaching out to Hetzner and making everything kosher, I’ve been trying to keep a close eye on things and I noticed today that while kopia was reporting 75272 blobs (1.7TB) under management, the hetzner dashboard was showing 174782 blobs (2.5TB) (!!) under that bucket.

That’s weird!

So I go to list any “incomplete multipart uploads” or “delete markers” or any file with a version that isn’t current.

Huh I only have 86 unfinished multipart uploads. That’s not too bad! Wait.. there’s 86 delete markers too? Weird, but that could make sense… …there’s 86 old version files…

Hey, did you know that hetzner paginates massive responses into groups of 86? Me neither!

But why

This shouldn’t be a thing. See when you build an object locked bucket on top of kopia, it’s a little weird because its actually kopia that manages the object locking. It’s really not advised that you have the bucket do it, because kopia’s maintenance wants to touch those flags (Side note: this urge to touch all the flags was a source of GREAT SUFFERING for me earlier this year). So what you do instead is designate the object locking to kopia and write a policy to manage object lifecycle.

You did write a policy to manage object lifecycle right?

What’s a lifecycle

Excellent question. A lifecycle is the cycle of life of an object.

Now that that’s cleared up…

(A lifecycle is a set of policies that tell the backend processes running an object storage provider when it should clean things up. If you enable versioning, but forget to write a lifecycle policy, you’ve basically created a world with aging and disease but missing the grim reaper. There’s excellent fiction written about this. NONE OF THEM END WELL)

A quick migration

Apparently when I was moving over from B2, I set everything up correctly but forgot to write a lifecycle policy and apply it to the bucket. This means that for OVER A YEAR, kopia has been dutifully pronouncing files deceased, shoving them in a body bag, tagging them with a 90 day tag that says “only send to the mortuary after this date, just in case we need to do anything else with the body” then shoving them in the file freezer.

Except the hearst has just.. never pulled up. What, you think that’s kopia’s job? Kopia’s job ends after chucking the file into the body bag and putting it away. It’ll keep piling the dead up inside that freezer all day long, likely while complaining about the terrible service from the Hetzner funeral home, and then will cheerily report “All’s well!”

A quick back of the envelope calculation reveals this has likely cost me well over 70 euro throughout the past twelve months.

So yeah!

Lifecycles are important!

Japan is perhaps the only country I have lived in where sleep promoting drugs are more readily controlled than stimulants (I can get 60mg of methylephedrine otc without even needing to talk to a pharmacist, but melatonin is a prescription). They also love what we would call “kitchen sink” drugs in US pharmacy practice. The flu medicine I am taking has 6 active ingredients, 3 of which are straight up stimulants. Yes, you are supposed to take it three times a day even in the evening.

So, as I lay awake at 2AM desperate for sleep that simply won’t come, my mind was going back through a conversation I had with some friends earlier about creativity in coding. This is an attempt to explain why I don’t see what I do as creative, even though I do agree creativity is very much a part of coding. It is also an attempt to make me tired enough that I fall asleep.

I don’t expect this to convince anyone, just explain my thinking on it.

Creativity vs. Engineering/Technical Work

One of my favorite youtubers is “Wristwatch Revival.” As you can undoubtedly tell from the name, he’s a watchmaker who does videos on restorations of old watches. Shockingly, most mechanical watches need very little restoration work to get running again, outside of a “service.”

A service is very simply: fully disassemble the watch down to its smallest components, clean them (either with a special machine or in an ultrasonic), inspect the parts for any rust or breakage, put the watch back together.

While this is extremely skilled work, and Marshall demonstrates an incredible amount of patience, I struggle to call a plain service creative work. Every mechanical watch is made up of the same standard components, arranged in well defined ways, and the goal is just to swap out any bad ones, and then lubricate and re-assemble them. This is technical work, bordering on engineering work.

Where Wristwatch Revival becomes creative is when there is major damage to a watch. There have been episodes where he needs to restore a dial without causing further damage, or needs to reloom or reblue the hands, and bluing in particular requires an eye for what colors match and the skill to get the metal to that temperature without exceeding it. There are also cases where a standard service my go awry, and that requires creative thinking to resolve.

I dislike creative solutions

In about 90% of my projects I ruthlessly cut out any solution that I come back to later and think “that was clever.” It’s not because I hate the feeling of being clever, it’s because of a concept called “innovation tokens.”

In Choose Boring Technology, Dan McKinley writes about the benefits of choosing boring technology. Boring is not to be conflated with “bad,” though boring bad technology does exist. Boring is defined as a technology which is clearly understood, has been around for a while, and is “good enough” for the problem you are facing. Every time you decide to go with something new, or fast moving, or still in the process of being understood, you are spending an “innovation token” and depending on the size of your company or project, you have a finite number of those and they don’t come back for a very long time.

My biggest project to date is yukari and I have spent about two years of my life building it after getting burned by multiple “creative” solutions to self-hosting. yukari is boring, and I don’t really consider any part of it to be creative. It is unusual, however, because it is so boring that most companies or home labbers just build their own version of yukari that does only what they want it to do, and then keep it to themselves. The unusual part of yukari is that I’m building it to be used by probably 10 other people on the planet, max, and when they find it I want them to have a nice experience.

The first component of yukari is yukari-canon. yukari-canon is able to manage an arbitrary number of servers, either physical or cloud based, as a single distributed system.

All of yukari-canon, save for two areas so far, is boring technology:

• It is written in ansible, which was started in 2012 and is now owned by RedHat, arguably the most boring example of Linux. That’s why IBM bought them.
• It expects RedHat Enterprise Linux, or a clone.
• It expects podman, the open source alternative to docker that is mature.
• The software selection for containers is all boring, well used, FOSS software.
• The mesh network I selected was nebula which was developed by Slack before being fully open sourced.
• Security is based on the CIS Level 2 Standard, which again, is ancient and well understood.
• Instead of inventing the wheel, I used fedora.linux_system_roles, which is the upstream of RedHat Enterprise Linux’s ansible roles, so its used in prod.
• Notifications are sent via email, using postfix as a null-relay.

There are two places that I spent my innovation tokens:

• The ran helper, which is written in python (despite me hating python), because python is boring and needed for ansible. ran is a series of scripts that automates the most tedious parts of getting up and running, like secret generation.
• The logic for configuring the *arr stack, since every software I looked at that did that was essentially abandoned.

90% of the 20k lines of code is just “glue.” Connect this standard component to that standard component.

When I ran into issues that seemed like they needed a clever solution, I avoided the clever solution by a mile:

Backups

Every person on the planet has written some kind of homelab backup software and open-sourced it for docker/podman containers. Essentially all of them have “untar and copy” as the restore step, and no way of validating the data on recovery is the same as what went into it. Alternatively, they have ballooned wildly in scope.

yukari-backup is just a 600 line bash script that uses pg_dump or tar combined with sha512sum, zstd, age, and rclone for uploading to the cloud. None of this is revolutionary. I’m still honestly annoyed it doesn’t already exist. It has a full test harness that is centered around testing restores on an actual container running actual software, and intentionally injecting crap everywhere to make sure the assurances yukari-backup makes are tested.

Everyone wants retention, or deduplication, or incremental backups. I like those things, but if you’re using S3 you have object lock and compliance mode for free. That’s your retention settings. For deduplication, to me, you’re immediately needing to introduce checking, because one bad deduplicated block ruins every backup. For a sane retention window in homelabs/containers, you are likely spending under a dollar or two for hosting unless god forbid you’re on AWS.

A backup is just a bash pipe:

pg_dump/tar -> zstd -> age -> rclone

A restore just reverses the process. There is a manifest file that records the sha512 of the original, pre-compression/encryption data and the filesize to detect any issues. The manifest is written last. Something fails during the upload? A manifest isn’t written, the backup can’t be used without a manifest, the restore doesn’t even see it. The errored backup is rotated out by your compliance settings.

It’s very dumb. In a good way. Its also done. It doesn’t need any new features. It doesn’t need to keep growing. Its tested, I did a real fire drill with it, it worked perfectly. It will live as version 0.1 indefinitely. I guess I should bump that to 1.0, but that would feel weird and arbitrary.

File Downloads

There are so many solutions available to containerized file downloaders for a server. Copyparty is well known, and very creative! SFTP2Go is closer to what I wanted, but again has its own user management system.

Every downloader container wants to be a downloader and a user manager. Or a downloader and a cloud storage system. Or a downloader and .

I have a user database, I have a cloud storage system. This is just meant to allow a registered user a way to log in and pipe things from a server. That’s it.

There is basically no container that satisfies that. Which is crazy to me.

I created a container that is a mini alma-linux container, that runs an SFTP server. It also has PAM and SSSD running on it. The container then uses RFC2307bis to authorize users from an ldap database. This is not a creative solution. It boggles my mind that this doesn’t exist as a ready made container, but the main reason is most companies probably just built it themselves, because it’s an obvious solution.

RFC2307 is from 1998. SFTP is arguably as old. Boring technology.

Is any of this creative

IDK. Maybe? To me, it feels more like servicing a watch. Maybe it’s just that I’ve been in linux for 25 years now, but all of these parts were already there. There’s no need to re-invent the wheel for any of these. Hilariously, I think that leads to basically every company that needs one of these solutions reinventing the wheel, because it doesn’t exist pre-shaped.

Essentially all of my projects, even what I’m doing at work follows the same pattern:

1. Read as much of the relevant research as I can
2. Identify any relevant RFC/ISO standards that define parts of the problem
3. Read the example code and the standards
4. Follow the standards.
5. If a software package doesn’t exist, or if it exists but some company has monopolized it or is charging an insane fee (Looking at you, translation memory systems), just.. build that myself, usually with the most bog standard UI you can imagine.
6. Get feedback, especially on UI/UX
7. Iterate

Most of my design decisions, either in a personal project or a work project, come down to saying no:

• “no, we don’t need that.”
• “no, that’s out of scope.”
• “no, that’s solving a problem we don’t have.”

(N.B. I am not immune to this. The first version of ran demo created a whole virtual network running a virtual runner and a virtual server, with a router and a whole bunch of other stuff. Then I came back to it the next day, wondered what the hell I was doing, and deleted 70% of it and the remaining 30% became the single VM demo that exists now. “no” is important to tell yourself too.)

I don’t think I have ever written a completely novel piece of software, or even an original piece of software. I’d rather glue together pieces made by people much more competent than me.

I like creative software, as in I like using creative software. I don’t personally want to create anything that gets called “clever”. Because then I would need to be clever to maintain it.