How a silent TypeError drained our cloud credits and what we learned

We set up pooled ephemeral DigitalOcean droplets to run ZAP active scans cheaply — batch free-tier scans into shared 45-minute sessions so we'd pay pennies instead of dollars per scan. The design was solid. The implementation had a bug that cost us all of our free credits in three days.

Here's exactly what happened, why it happened, and what we changed.

#The setup

Vergate runs ZAP (OWASP's active security scanner) in Docker containers. For free-tier scans, we batch them into shared sessions on DigitalOcean droplets instead of spinning up a fresh one per scan. A background thread — the "pool manager" — polls every 30 seconds to drain idle pools and destroy droplets that have outlived their 45-minute billing window.

The pool manager keeps an in-memory store of active pools. When a droplet's session ends, the manager calls droplet.destroy() to stop billing. Simple.

#The bug: three compounding failures

#Bug 1 — The smoking gun

python-digitalocean returns created_at as an ISO 8601 string — something like "2026-08-13T11:28:36Z". Our orphan reaper did this:

age = (datetime.now(UTC) - created).total_seconds()

Subtracting a string from a datetime raises TypeError. The reaper caught it silently and set age = 0.0. A 0-second-old droplet is obviously not an orphan — so the reaper skipped it. Every single time.

# What we wrote (broken)
try:
    age = (datetime.now(UTC) - created).total_seconds()
except Exception:
    age = 0.0  # ← silent failure = droplet looks brand new

if age >= ORPHAN_MIN_AGE:
    destroy(droplet)

# What we should have written
if isinstance(created, str):
    created = datetime.fromisoformat(created.replace("Z", "+00:00"))
age = (datetime.now(UTC) - created).total_seconds()

This alone would have been a $5 lesson. But two more bugs compounded it.

#Bug 2 — Destroy failure orphaned the droplet permanently

When a pool's session ended, the manager called droplet.destroy() and then deleted the pool record from the in-memory store. If destroy() failed (network error, API timeout), the pool was still deleted — the droplet existed on DigitalOcean but had no tracking record. No record = no future cleanup attempts.

#Bug 3 — No startup sweep after restart

When the worker process restarted (deploy, crash, reboot), the in-memory pool store was wiped empty. Every existing droplet on DigitalOcean became an untracked orphan. The reaper wouldn't see them for 45 minutes (its age threshold), and even then, Bug 1 meant it would never see them at all.

The lifecycle was:

1. Worker starts, creates pools, boots droplets
2. Worker restarts (deploy) → pool store wiped → all droplets orphans
3. Reaper runs, Bug 1 silently caught, age = 0, skip
4. Droplet runs for 3 days, billing the whole time
5. Repeat on every deploy

#The proof

We wrote a script that queries the DigitalOcean API for all droplets named vergate-zap-*. It found one that had been alive since August 13 — three days past its 45-minute intended lifetime. We destroyed it manually and watched the credits stop draining.

Total damage: $5 — the entirety of DigitalOcean's free signup credits. Not catastrophic, but completely preventable.

#The fix

Three changes, all deployed the same day:

1. Parse the string before subtracting

if isinstance(created, str):
    created = datetime.fromisoformat(created.replace("Z", "+00:00"))
age = (datetime.now(UTC) - created).total_seconds()

2. Don't delete the pool on destroy failure

# Before: delete regardless → orphan
pool_store.delete(pool_id)

# After: only delete when destroy succeeds → retry next tick
if not destroy_droplet(droplet):
    return  # try again in 30s
pool_store.delete(pool_id)

3. Startup sweep as a safety net

On the first tick after a restart, if the pool store is empty, sweep all vergate-zap-* droplets on DigitalOcean and destroy them. This catches anything orphaned by a previous restart regardless of whether the reaper works.

#What we learned

Silent error handling is worse than crashing. If the TypeError had propagated, we would have seen it in logs immediately. Instead, setting age = 0.0 in the except block made the bug invisible — the reaper ran every 30 seconds, looked healthy, and did nothing. Crash loudly in cleanup code. A dead reaper is better than a sleeping one.

In-memory state plus restarts equals data loss. The pool store was in-memory for low latency and simplicity. That's fine for a cache — but we were using it as the source of truth for which droplets to clean up. When the process restarted, that truth disappeared. If your cleanup logic depends on state that can be wiped, you need a safety net that doesn't.

Cloud costs are proportional to time, not size. A $0.07/hour droplet doesn't sound dangerous. But 72 hours of it is $5 — all of our free credits. At scale, the same class of bug (orphaned instances, unattached volumes, leaked load balancers) is the #1 source of surprise cloud bills. Every startup that's been shocked by an AWS bill knows this feeling.

Always have a kill switch that doesn't depend on your code running. The startup sweep is our kill switch. Even if every other cleanup mechanism fails, a fresh process will find and destroy anything that shouldn't exist. Cloud providers have their own cleanup tools too — DigitalOcean has droplet auto-destroy on expiration, AWS has instance lifecycle hooks. Use them.

#The receipt

This is the kind of bug that doesn't show up in unit tests because the unit tests don't restart the process. It only shows up in production, after a deploy, when the in-memory state disappears. If you're running ephemeral cloud resources with in-memory tracking, check your cleanup code for silent except blocks. Especially around datetime parsing. Especially if the datetime came from an API that returns strings instead of objects.

We're sharing this because we think postmortems should be public. Not because $5 is a crisis, but because the pattern — silent failure in cleanup code + in-memory state + restarts — is universal. It happens at every scale. The only difference is how many zeros are on the bill.