Load test program for an open enrollment window

The platform's traffic multiplies by roughly twenty for the six-week open enrollment window, and the year before I joined, day one had brought a brownout: not an outage, which would at least have been dramatic, but forty-minute page loads and a support queue that took a month to drain. Leadership wanted a load testing program with teeth — one that could say 'ready' or 'not ready' before the window opened, and be believed.

QA engineer embedded with the platform team, one of two people on performance, working remote from Quezon City on a US-hours-adjacent schedule. The program runs on a quarterly cycle building toward each year's window, and I've now been through three enrollment seasons with it.

I own the scenario scripts — the k6 models of what real users actually do — the environment parity checklist that decides whether a test run means anything, and the run reports that go to engineering leadership. The Grafana dashboards that show test load and system behaviour side by side are mine as well; the infrastructure under test belongs to the platform engineers, which is a boundary that keeps the findings honest.

The foundational decision was modeling scenarios from production access logs rather than from anyone's intuition about user behaviour — the logs showed enrollment traffic is dominated by plan comparison pages being refreshed obsessively, not by the submission flow everyone assumed, and that single correction redirected two months of optimization work. After one round of testing against a scaled-down environment and extrapolating, we found the extrapolation had lied about a connection pool ceiling, and I pushed to only test at production scale in scheduled weekend windows from then on — expensive, and worth it. I also added soak tests to the program over the objection that spikes were the risk; the soak is what caught a slow memory leak that would have surfaced in week four of a six-week window.

No production data in test, ever — the compliance rules around health information are absolute, so I built a generator that produces realistic synthetic households, and 'realistic' took three iterations before the data's shape stopped flattering the caches. Cloud cost caps meant full-scale runs were scheduled and budgeted like small events, and the last test window each season closes three weeks before enrollment opens, after which findings are academic.

The last two enrollment day-ones have held p95 page times under 800 milliseconds at full load — against forty-minute loads the year before the program existed. The connection pool ceiling and a cache stampede on the plan comparison page were both found and fixed in test windows, not in the news. The run report format has been adopted as the template for the company's other performance work, which I take as the quiet version of a compliment.

The other performance engineer owns the infrastructure side of every finding — I say where it breaks, she says why. Platform engineers triage findings with us in a standing Friday call, and an SRE taught me to read the database's behaviour under load properly, which upgraded every report I've written since.