Data and reliability engineeringDone
Deleting a terabyte safely
Moved irreplaceable media and data to cloud storage and deleted local copies only after per-file hash proof.
- Role
- Designer of the safety policy
- Period
- Aug 30 to Oct 8, 2026
- Team
- Solo, with Claude Code
- Stack
- rclonePythonzshGoogle Drive APIddrescue
Receipt Verified 2026-10-09Deleted 1,044.9 GiB across 9,395 files, each only after a fresh per-file hash proof against the cloud copy.Period2026-09-13 to 2026-10-06MethodSum of 13 deletion-gate logs.SourceDeletion-gate logsdeleted only after hash proof
Receipt Verified 2026-10-09When DNS failed mid-job, the gate refused 459 deletes and made 0, then the job resumed and finished.Period2026-10-04MethodRefusal log of the affected run.SourceGate refusal logrefusals / deletes in a DNS outage
Receipt Verified 2026-10-09A cache-evicted re-read held back 2 of 34 video clips that a checksum copy had already passed.Period2026-09-25MethodPage cache evicted (msync MS_INVALIDATE), then re-hashed from the medium.SourceJob logbad reads caught after a checksum copy passed
Goal
Free a nearly full 1 TB laptop without losing a single irreplaceable file.
Obstacles
- The disk sat at 96 to 99% full, so every job ran with almost no headroom.
- Checksum copies can pass bad reads: macOS serves re-reads from memory, not from the card.
- Flaky cards, a DNS outage, a hidden API quota and an exit code that reported success over 1,114 failures.
Approach
- A fail-closed gate: exactly one cloud object, a fresh size + MD5 + SHA-256 match, no live writers, the expected count, and a dry run by default.
- A true second read: evict the page cache, then re-hash from the medium, with a self-test that refuses to run if the cache wasn't cleared.
- Negative controls: every gate is shown failing against a wrong destination before it is trusted.
- Overlapped, resumable pipelines: upload batch N+1 while verifying batch N.
- Measure the bottleneck before tuning: uplink, API quota, USB link or card.
Lift (results)
- Receipt Verified 2026-10-09Deleted 1,044.9 GiB across 9,395 files, each only after a fresh per-file hash proof against the cloud copy.Period2026-09-13 to 2026-10-06MethodSum of 13 deletion-gate logs.SourceDeletion-gate logs deleted, each file only after a fresh hash proof against its cloud copy.
- During a DNS outage the gate logged Receipt Verified 2026-10-09When DNS failed mid-job, the gate refused 459 deletes and made 0, then the job resumed and finished.Period2026-10-04MethodRefusal log of the affected run.SourceGate refusal log (refused / deleted), then the job resumed and finished.
- A cache-evicted re-read caught Receipt Verified 2026-10-09A cache-evicted re-read held back 2 of 34 video clips that a checksum copy had already passed.Period2026-09-25MethodPage cache evicted (msync MS_INVALIDATE), then re-hashed from the medium.SourceJob log clips that a checksum copy had already passed.
- An independent check matched Receipt Verified 2026-10-09An independent check matched 6,547 of 6,547 uploaded files by size, MD5 and SHA-256.Period2026-10-03MethodPre-transfer manifest vs cloud metadata.SourceVerification table uploaded files by size, MD5 and SHA-256.
- Hidden API quota errors: Receipt Verified 2026-10-09Found 403 quota errors visible only in verbose logs (210 in one job) and took them to 0 with a dedicated OAuth client.PeriodSep to Oct 2026MethodError counts in comparable upload logs.SourceUpload logs after switching to a dedicated OAuth client.
Next
- A sampled restore audit.
- Open-source the gate (prune-gate) and the re-read hasher (truecheck).