Context

The portal exposes a self-service catalog: a small set of one-click actions (backed by an automation controller running real playbooks) that let an authorized user spin up disposable infrastructure without asking me to do it by hand. Today it offers two items: a web server, and a PostgreSQL database with a lightweight web admin UI in front of it.

The problem this had to solve

Building a working self-service item means solving several things at once:

• Provisioning a full VM, not just configuring an existing one: clone a template, assign it an address, register it in DNS, put a reverse proxy with HTTPS in front of it — several distinct systems orchestrated by one click.
• Least privilege: the automation account that runs this must be able to do only this — not touch unrelated infrastructure.
• Running in an isolated execution environment: the automation runs inside a container with its own toolset, not on my own workstation, so it can't lean on files or dependencies that happen to exist locally.
• Not letting throwaway VMs pile up forever: every self-service VM carries an expiry tag, and a scheduled job — deliberately generic, not tied to any specific item type — cleans up anything past its expiry: reverse proxy entry removed, DNS entry removed, and only then is the VM itself destroyed, strictly in that order, so a failure never leaves an orphaned DNS record pointing at nothing.

What makes it worth telling

• A constraint discovered by testing, not by reading code. An early VM-numbering range chosen on paper turned out to be structurally invalid the moment it collided with how addresses are derived in this lab — the real API rejected it outright. Lesson: test early against the real system rather than trusting a plan that looks reasonable on paper.
• An isolated execution environment is a genuinely different world. The container that runs these automations doesn't have the same libraries installed as a normal control machine. Rather than maintaining a custom image just to keep using convenient libraries, the integrations were rewritten as plain HTTP calls against each system's REST API — more verbose, but with zero hidden environment assumptions.
• A safety net proven, not just designed. During testing of the cleanup job, a real bug in one of the downstream APIs caused the DNS-removal step to fail — and the VM was correctly kept rather than destroyed, exactly as the ordering rule was supposed to guarantee. Seeing the safeguard actually catch a real failure, instead of only working on paper, is what gave me confidence in the design.
• A found permission you don't know you need until you hit it. A hypervisor permission required for a modern feature only showed up as a hard failure on the first real end-to-end test — not documented anywhere obvious beforehand. Some things really only reveal themselves at the point of contact with the real system.

The pattern, generalized

Every new catalog item reuses the same three ingredients: a scoped automation account with only the permissions that one item needs, a Survey-style form so a non-technical user can fill in a few parameters safely, and the same generic expiry-tag contract so the cleanup job never needs to know about a new item type to handle it correctly. Adding a second catalog item required zero changes to the cleanup logic — a good sign the abstraction was drawn in the right place the first time.