Muhammad Tayyab
Senior AI Engineer
Duration:
6 months
Role:
Production Owner & Reliability Engineer
Stack:
React · Node · WebSockets · OpenAI Images · Cognito · EKS
An internal platform where marketing designers generate ad creatives with AI: image generation and editing, an agent-powered canvas assistant and a searchable asset library. It began life as a rapid prototype with file-based storage and no deployment story; I owned taking it to production and keeping it alive there.
The interesting problems arrived once real designers depended on it daily: image generations taking 60 to 110 seconds behind a CDN that hard-caps requests at 100 seconds, and a connection indicator that kept flapping red for what turned out to be two entirely unrelated root causes.
What I Built
Productionised in the first fortnight: SQLite to Postgres with pooling and SSL, CI/CD with Docker, Helm and Kubernetes, and single sign-on replacing custom JWTs
An async job architecture to beat the 100-second wall: submit returns a job ID in under a second, clients poll, and a staleness-based reaper self-heals after pod restarts
WebSocket forensics, cause one: cached expired tokens silently killed sockets forever; shipped forced re-auth plus a reconnect state machine fixing four independent leaks, including multi-tab retry storms
Cause two, found separately: each AI generation spawned a 200 to 300 MB agent subprocess and concurrent runs blew the pod memory limit; fixed with a per-pod concurrency gate reading cgroup memory, the number the OOM killer actually sees
A wire-level Playwright suite that counts real ping and pong frames on real sockets: a green dot with reconnect churn behind it fails the build
Engineering Judgment
Reliability work followed a strict evidentiary discipline: a correct fix that does not fully resolve a symptom is treated as evidence of a second independent fault, not a failed hypothesis. That discipline separated two unrelated root causes behind a single symptom, and produced a verification suite that asserts real connection behavior rather than surface indicators.
Outcome
A stable daily tool for the design team, with every discovered failure mode pinned by an automated test that talks to a real backend over real sockets.