Skip to content
Skip to main content

Writing

Why destructive updates leave your machines out of date

When the only way to repair a machine is to wipe it, nobody updates. I split update from wipe and made the restore cheap.

The Linux machine I work on every day had not been updated in a long time. Not because I had not noticed. Because the only supported path to repair it was to wipe it and start over.

That is worth stating plainly. The platform had one operation that covered both "apply the latest fixes" and "destroy everything I set up". The operation was a reinstall. My files and logins survived it. Every package I had installed by hand did not, so repairing the machine came with reinstalling each of those tools one at a time afterwards.

So the price of a small update was a wipe and a rebuild. I did what anyone does with a price like that. I skipped it. Then I skipped it again.

The trap

This is not one platform's problem. It is arithmetic.

If the only repair path is destructive, the repair path is expensive. If the repair path is expensive, maintenance stops being maintenance and becomes a project. Projects get scheduled for later. Later turns into months.

In those months nothing looks broken. That is the part that makes it hard to see. The machine boots. The old bugs are quiet ones: a fixed crash that still happens on a rare input, a patch that closes something nobody has tried to exploit yet. The work of shipping those fixes is done, and none of it reached me.

A platform whose only repair path is destructive will be under-updated. Not because its users are careless, but because maintenance has been priced above what they will pay for it on a normal day.

What an update should be

Two properties matter more than speed.

The first is that update and wipe are different actions. An update should carry my installed software, my settings, and my files forward. A wipe should be something I choose on a day I set aside, with the intent to start clean. When one button does both, the safe choice and the destructive choice cost the same, and the destructive one wins by default.

The second is that restore has to be cheap and scripted. A plain list of the packages I added, in the order they need, is a shell script that runs in minutes. It is not clever, and that is why I would trust it. I want to read the whole thing, run it, and know what it does. When restore is that cheap, an update stops being an event and becomes a thing I do on a Friday afternoon.

Two smaller things follow. The machine should tell me what an update will change before it runs, so a new major version of something I depend on is not a surprise. And it should record the package set as part of the update, so the list I restore from is generated instead of maintained from memory.

The harder half

The update problem was the easy half. The other half was a background process, and from the outside it looked like a hung machine.

The process walked the disk to sync a workspace. It ran unbounded: no cap on how many files it would visit, no ceiling on the memory it could hold, no pause. It ran on the same host that served the desktop sessions and the jobs I was running. My scratch tree had grown very large. A pass would walk all of it, and the next pass would do it again.

While it ran, nothing else worked. Typing lagged. Commands that usually finish in a second took a minute. The machine was not hung. It was busy with work I had never bounded, and from the front of the screen those two look the same.

The change that did most of the work was one ignore rule. The scratch tree was never meant to be mirrored, and taking it out of the sync removed the bulk of the work in a few lines of configuration. It prunes about seventy directories a cycle. Most passes finish in about a minute and a half over roughly 885,000 entries. A few larger syncs ran about three to four minutes on real work. One failed outright on a write timeout after a little over five minutes. The longest passes of all are not slow walks: the clock kept running while the box was suspended mid-pass, which is a host artifact rather than work. The errors the completed passes did report were in packing and uploading, not in the walk.

The rest of the fixes were unglamorous, and none of them was a bigger machine.

  • Stop after a fixed number of entries or a fixed amount of time, then resume from where it stopped on the next pass.
  • Give the process a memory ceiling, so it recycles instead of growing without limit.
  • Run work like this where it cannot starve an interactive session.

Those are changes I asked for at the layer that owns the sync, because a process that walks user data needs them by default, not after an incident.

The lesson I actually drew

I did not need a bigger machine. I needed a non-destructive update tier and a restore script I trust enough to run without planning my week around it.

Those change the arithmetic. When the safe path is cheap, I take the safe path. When it costs a wipe and a rebuild, I take the shortcut, and the shortcut is why the machine was old in the first place.

The bound matters too, and it is easy to forget because it is invisible when it works. Unbounded work on a shared host does not present as a slow job. It presents as a broken machine, and the person waiting on it has no way to tell the difference.

Separating update from wipe is not the kind of feature that goes on a slide. It is the difference between a machine that stays current and one that does not.