AI Design Platform
An internal tool where marketing designers generate ad creatives with AI. It started as a prototype with file storage and no deployment; I took it to production and kept it alive there.
Watch the explainer: AI Design Platform, 74 secondsWatch the 74-second narrated explainer. English captions on.Read the transcript
TL;DR
Async jobs to get past a 100-second CDN cap, and two unrelated root causes found behind one flapping connection indicator.
01
Context
The interesting problems arrived once real designers depended on it daily: image generations taking 60 to 110 seconds behind a CDN that hard-caps requests at 100 seconds, and a connection indicator that kept flapping red for what turned out to be two entirely unrelated root causes.
02
What I built
- Productionised in the first fortnight: SQLite to Postgres with pooling and SSL, CI/CD with Docker, Helm and Kubernetes, and single sign-on replacing custom JWTs.
- An async job architecture to beat the 100-second wall: submit returns a job ID in under a second, clients poll, and a staleness-based reaper self-heals after pod restarts.
- WebSocket forensics, cause one: cached expired tokens silently killed sockets; shipped forced re-auth plus a reconnect state machine fixing four independent leaks, including multi-tab retry storms.
- Cause two, found separately: each AI generation spawned a 200 to 300 MB agent subprocess and concurrent runs blew the pod memory limit; fixed with a per-pod concurrency gate reading cgroup memory, the number the OOM killer actually sees.
- A wire-level Playwright suite that counts real ping and pong frames on real sockets: a green dot with reconnect churn behind it fails the build.
03
Engineering judgment
Reliability work followed a strict evidentiary discipline: a correct fix that does not fully resolve a symptom is treated as evidence of a second independent fault, not a failed hypothesis. That discipline separated two unrelated root causes behind a single symptom, and produced a verification suite that asserts real connection behavior rather than surface indicators.
04
Outcome
A stable daily tool for the design team, with every discovered failure mode pinned by an automated test that talks to a real backend over real sockets.
05
Transcript
Read the 74-second explainer as text
The image was ready, but the request was gone. Generations ran 60 to 110 seconds, behind a CDN that cuts every request at 100. An AI design platform for marketing designers, taken from rapid prototype to production.
So generation went asynchronous. Submit returns a job ID in under a second, the client polls, and a staleness-based reaper self-heals after pod restarts. One flapping connection dot, two unrelated causes. Expired cached tokens silently killed sockets. And separately, agent runs of 200 to 300 MB each blew pod memory. Now a gate reads cgroup memory, the number the OOM killer sees.
A wire-level Playwright suite counts real ping and pong frames. A green dot with reconnect churn behind it fails the build. React, Node, WebSockets, OpenAI Images, Cognito and EKS. Productionised in a fortnight, now a stable daily tool for the design team.
One symptom. Two root causes.