Oops

I use a lot of object storage. Not a lot from the perspective of (insert your favorite example tech company here), but for a private user/small community manager, a good amount. Looking back over things, my spend this year peaked at about 35 euro in a 30 day period, which at my current pricing is about 5.8TB.

Most of that goes toward the “Tokisaki Memory Engine” I run, which is really just a fancy name for kopia plus a dedicated repo server, and some special sauce that is usually lost in the weeds, like object locking, restricted keys, etc. We even have automated daily testing of restores to /dev/null. Neat!

The main goal is to let someone get all the benefits of kopia with just an username and password, while only giving up the same thing they give up with most other cloud storage providers (Mainly the fact that if I really really wanted to, I suppose I could look through your stuff, but it’d be damn obnoxious to do because of how the server works, and its built that anyone could catch me at any time, so, why bother).

I migrated last year to Hetzner’s Object Storage after B2 became VERY UNHAPPY with how many API calls my little harness around kopia uses. The migration was kind of in a rush, and I forgot some things!

This won’t detail all the fun that I have had growing tokisaki-v4 along-side Hetzner’s Object Storage (which has been having its own growing pains), but a fun little oops I just found.

Huh, that’s weird

So after fixing some earlier corruption this year and reaching out to Hetzner and making everything kosher, I’ve been trying to keep a close eye on things and I noticed today that while kopia was reporting 75272 blobs (1.7TB) under management, the hetzner dashboard was showing 174782 blobs (2.5TB) (!!) under that bucket.

That’s weird!

So I go to list any “incomplete multipart uploads” or “delete markers” or any file with a version that isn’t current.

Huh I only have 86 unfinished multipart uploads. That’s not too bad! Wait.. there’s 86 delete markers too? Weird, but that could make sense… …there’s 86 old version files…

Hey, did you know that hetzner paginates massive responses into groups of 86? Me neither!

But why

This shouldn’t be a thing. See when you build an object locked bucket on top of kopia, it’s a little weird because its actually kopia that manages the object locking. It’s really not advised that you have the bucket do it, because kopia’s maintenance wants to touch those flags (Side note: this urge to touch all the flags was a source of GREAT SUFFERING for me earlier this year). So what you do instead is designate the object locking to kopia and write a policy to manage object lifecycle.

You did write a policy to manage object lifecycle right?

What’s a lifecycle

Excellent question. A lifecycle is the cycle of life of an object.

Now that that’s cleared up…

(A lifecycle is a set of policies that tell the backend processes running an object storage provider when it should clean things up. If you enable versioning, but forget to write a lifecycle policy, you’ve basically created a world with aging and disease but missing the grim reaper. There’s excellent fiction written about this. NONE OF THEM END WELL)

A quick migration

Apparently when I was moving over from B2, I set everything up correctly but forgot to write a lifecycle policy and apply it to the bucket. This means that for OVER A YEAR, kopia has been dutifully pronouncing files deceased, shoving them in a body bag, tagging them with a 90 day tag that says “only send to the mortuary after this date, just in case we need to do anything else with the body” then shoving them in the file freezer.

Except the hearst has just.. never pulled up. What, you think that’s kopia’s job? Kopia’s job ends after chucking the file into the body bag and putting it away. It’ll keep piling the dead up inside that freezer all day long, likely while complaining about the terrible service from the Hetzner funeral home, and then will cheerily report “All’s well!”

A quick back of the envelope calculation reveals this has likely cost me well over 70 euro throughout the past twelve months.

So yeah!

Lifecycles are important!