Engineers across the company started buying Mac minis and running their own AI agents at home. Unmanaged hardware the company had no control over, personal API keys, subscription tokens sitting on kitchen-table hardware, and company data flowing through it all. Banning it would have driven the shadow further underground. Providing a sanctioned path was the only viable move.
I built the sanctioned path on ECS. Same agents, better home. Per-agent EFS access points isolate persistent state. Per-agent IAM task roles isolate AWS resource access. No public ingress. Controlled egress. An audit trail the security team can actually use. Engineers get to keep the productivity, the company gets its blast radius back.
The shadow started small. One engineer bought a Mac mini, installed an AI agent stack on it, wired it up to Slack and their inbox and a couple of internal APIs. It worked. Word spread. Two months later, half the engineering org was running some flavour of personal agent on hardware nobody in IT had ever seen.
The problems, in the order they became obvious:
The productivity gap was too big. Every engineer who used their agent for even ordinary tasks (triaging Slack, summarising a doc, drafting a PR description) had proof it saved them real time. Prohibition doesn't beat that math. The tools would move to encrypted personal machines nobody at the company would ever see, and the visibility problem would get worse, not better.
The only viable move was to offer a legitimate path that met engineers where they were. Same agent, same behaviour, run somewhere the company controls.
I wrote the requirements as a checklist the security team could sign off on:
ECS was the right size for this. Kubernetes would have been overkill for a fleet of 3-10 agents. Lambda's execution model does not fit long-lived agents that hold sessions open for hours. ECS is boring, cheap, well understood, and every AWS-native team already runs on it.
The whole design hangs on these three, applied to every agent equally.
One shared EFS filesystem, one access point per agent. The access point maps the agent's container view of /data to a scoped sub-directory of the filesystem with a specific UID and GID. The container sees only its own view. Two agents on the same filesystem cannot see each other's files at all.
What lives in /data:
EFS is not fast, but it is durable, POSIX, and cheap. For OAuth state and session tokens it is the right primitive.
Every agent gets its own IAM task role. The task role is scoped to exactly the S3 prefixes, Secrets Manager keys, and other AWS resources that specific agent needs. Agent A cannot read agent B's secrets. Agent A cannot enumerate agent B's S3 objects. Agent A cannot call arbitrary AWS APIs it does not have permission for.
The task role is separate from the execution role. Execution role is what ECS itself uses to pull the image and write logs. Task role is what the running container's application sees. Keeping them separate is the difference between "compromised agent" and "compromised platform."
One ECR repo per agent. One execution role per agent that can only pull from that repo. A compromised build pipeline for agent A cannot poison agent B's image. Image scans run before promotion. Nothing runs in the cluster that has not been through the pipeline.
Agents live in a private subnet. No ALB. No public IP. No security group rule that allows inbound traffic from anywhere on the internet. When agents need webhook-style callbacks, they poll instead of listen. Slower, but there is no anonymous entry point into the runtime.
Egress is where the interesting rules live. VPC endpoints for AWS services keep that traffic on the AWS backbone and out of the public internet. Vendor API endpoints (LLM providers, Slack, GitHub, whichever integrations the agent needs) are allow-listed at the security group and NAT level. Anything not on the list is refused.
The practical outcome: if an agent tries to exfiltrate data to random-suspicious-domain.example, the connection dies before it leaves the VPC. The security team gets an alert. The blast radius stays inside the runtime.
Different agents have different credential shapes. Some authenticate with per-token API keys, which live in Secrets Manager and get injected as environment variables at container start. Some authenticate with subscription-shaped session tokens, which need to persist on disk between task restarts and cannot be safely handed out through Secrets Manager as a static value.
Both shapes fit the isolation model. API keys live in Secrets Manager entries the task role can read. Session-shaped tokens live on the per-agent EFS access point where nothing else can see them. Neither type of credential is visible to the wrong agent.
Agents often need config or credential rotation without dropping in-flight work. Signal-based restart handles this: SIGUSR1 to the main agent process triggers a graceful reload of config and credentials. Existing sessions finish, new work uses the new config. The ECS task stays up, the health check smooths over the transition, no cold start.
For a hard restart, ECS task lifecycle handles it. Bring up the new task, drain the old one, done.
EFS latency. Fine for OAuth state and session token files. Not fine for high-throughput working data. Choose the primitive that fits the data shape.
Task definitions get verbose. Every isolation control is another line in the task definition. Templating helps. Terraform modules or a small internal generator keeps the noise manageable.
IAM role sprawl. Adding agents faster than you audit their task roles is a recipe for a slow, silent expansion of blast radius. Regular review is part of the ongoing cost.
Vendor terms of service. Some subscription-backed authentication surfaces have terms that expect a single human user. Running them under a controlled runtime for company work needs a conversation with legal, not a shortcut around it.
Some workflows do not fit. Agents that need broad local disk access or a full desktop environment cannot move to a container without rework. Rare, and worth naming as "the exception, not the rule."
The Mac mini purchases stopped. The runtime is where employee agents run. Engineers get the productivity they were chasing, without the personal hardware, personal credentials, and personal-router-punched-holes that came with the shadow version. Security gets a real audit story, real revocation controls, and real blast-radius limits.
The pattern also gave the org a starting point for the harder conversation about AI governance. It is easier to write a policy about "AI agents used for company work must run on the sanctioned runtime" than to enforce a policy about "no AI agents anywhere." The runtime made the policy possible.
Shadow IT was the actual problem. The runtime was the answer. If I had built this as an efficiency play, it would have been a solution looking for a problem. Because it was a security response to a real threat that was actively growing every week, it landed and the org adopted it fast.
Same pattern applies anywhere an unmanaged consumer trend (AI agents, personal Kubernetes clusters, developer VMs) starts eating into a company's control plane. Give people the thing they want on hardware you control before someone gets breached.