kodebeat / papers / cost

Stopping an AWS account's spend without losing anything

A reversible spend killswitch with a recorded restore path.

Halts spend by cutting ingress first and then draining compute, deleting nothing and recording exactly how to put it all back. plan is read-only and safe against production, restore replays the snapshot, and verify fires, reads the account back, restores and reads it back again, because the API returning no error is not evidence that anything stopped.

The problem

There are two moments when an organisation needs an AWS account to stop spending immediately, and neither has a good answer.

Credentials are compromised. A key has leaked. Something is spinning up GPU instances in regions nobody uses. The bill is climbing by the minute, and the response options are all bad: revoke every credential and take down the production service along with the attacker, delete resources and lose data, or click through the console for two hours while the meter runs.

A non-production environment is idle and expensive. A development account runs at full capacity through nights and weekends because switching it off means somebody has to remember how to switch it back on: the desired counts, the ASG bounds, the Lambda concurrency limits. Nobody records those.

Both cases share the same missing capability: a reversible stop. The available operations sit at the extremes. Deleting resources stops the spend and destroys the environment. Leaving them running preserves the environment and the spend. What is needed is the middle (drain the compute, keep the storage, and record the restore path), and that middle does not exist as a button.

There is a second problem underneath, which is why the tool has a verify command. A killswitch nobody dares press does not get pressed. Any team that builds one faces the same question at the moment of truth: has anyone tested this? Testing it means testing the restore, in an account, against real resources, which few teams do, so the switch sits there untrusted.

The method

Ingress before compute, and say why in the output. The plan's section headers carry the reasoning. From the README:

$ aws-killswitch plan
plan ks-20260802-021500 — account 123456789012, regions eu-west-1

INGRESS — stop the traffic that drives the spend, before draining what serves it
  alb-listener   api-prod:443                     return a fixed 503 instead of forwarding to the target group

COMPUTE — drain compute; EBS is kept, nothing is deleted
  lambda         image-resize                     set reserved concurrency to 0
  ecs-service    checkout-api                     set desired count to 0
  asg            workers-prod                     set min/desired/max to 0

NOT TOUCHED (4)
  ec2-instance   build-cache (i-0a1b2c)           has instance-store volumes, which a stop erases; set allow_instance_store_loss to accept that
  rds-instance   orders-prod                      database: excluded unless include_databases is set
  lambda         cron-billing                     tagged killswitch:protect
  ecs-service    checkout-staging                 Env=staging, scope wants prod

4 resources would change

Nothing has changed. Run `fire --yes` to apply.

Draining compute while traffic still arrives produces autoscaling fighting the drain, retries amplifying load, and a slower stop. Cutting ingress first removes the demand signal, so the drain is monotonic. The cost: a fixed 503 is an outage for every in-scope service, by design.

Nothing is deleted, with one opt-in exception. Listeners return a fixed 503 rather than being removed. Compute is scaled to zero rather than terminated. EBS volumes are kept. Each operation has an inverse, so the restore is a replay rather than a rebuild. The exception is delete_nat_gateways: NAT gateways bill by the hour and cannot be stopped, so opting in deletes them, and restore recreates them with a new address and repoints the route tables.

Say what was not touched, and why. The NOT TOUCHED list above gives four different reasons, each named. A tool that silently skipped these would leave the operator believing spend had stopped when four resources were still running. Instance-store volumes are the sharp one: stopping such an instance erases them, so the tool refuses by default and names the flag that accepts the loss. Databases are excluded unless explicitly included, because a database is where the state is. The cost of scoping by tag: an untagged resource keeps running and keeps billing.

The snapshot is the restore path, and it is protected. When state_uri is configured, snapshots go to S3 and a local copy, and both must accept the write; without it the tool warns that snapshots are local only and proceeds. S3 is in the never-touch set, so the killswitch cannot destroy its own restore record, and the README recommends bucket versioning for the same reason.

Progressive confirmation. plan is read-only. fire is still a dry run. fire --yes acts. A plan larger than confirm_above resources (default 25) also needs --force, as does a plan carrying an instance-store, NAT or CloudFront warning. The design assumes the operator is under pressure and may not be reading carefully.

The headline results

  • The correctness argument is the test suite: 77 Go test functions across the planner, engine, state store, verify, trip and CLI packages, run in CI.
  • What the tests cannot establish is that the AWS API calls do what the in-memory account says they do.

The full results, with the tables and the reasoning behind each figure, are in the PDF.

What is in the PDF

  • The problem — what goes unanswered without it, and who notices first.
  • Why the obvious alternative falls short — stated plainly, including where it is the better choice.
  • How it works — the method, not a feature list.
  • Concrete use cases — with console output quoted from the repository, never reconstructed.
  • The methodology behind any number it emits — every term shown, so the figure survives a question.
  • What it deliberately does not do — the section most papers leave out.

Part of the Cost & FinOps theme.

Get the PDF

This paper already selected; the download starts when you send the form. No follow-up sequence.

Get the PDF