shri.cx what should exist but can't?
>

Twelve years inside a moving system

In 2003, Stagecoach needed a ticketing system for a new coach service called megabus.com. I quoted 33 days, built it in six weeks and charged £13,200. The support contract was £350 a month — for one day.

On the first day, megabus.com sold 200 orders.

What was built

I chose PHP and PostgreSQL for the stack. MySQL was the obvious choice at the time — fast, popular, and everywhere. Under sustained load it had a habit of giving up. PostgreSQL slowed to a crawl but kept going. This was an e-commerce system for a transport operator. It needed transactions and it needed to stay up.

The core of it was a dynamic pricing engine — yield management for buses, years before this was a product category. Seat availability tracking. Worldpay payment integration. Financial reporting. Admin loading sheets iterated until they fitted on A4 landscape for depot drivers. A four-level access system.

Six weeks from first commit to go-live.

What it became

Routes multiplied. Traffic spiked unexpectedly. Servers were added overnight. I wrote replication by hand — PostgreSQL had no native solution in 2004, and wouldn’t until 2010. Built from a trigger-based log table, replicated synchronously within the PHP request thread. A surprisingly small change, for a mighty difference.

Rate limiters baked into the search algorithm, running live on a production system, because not having them was more expensive than having them. Search logic rewritten from stored procedures to PHP — offloading CPU from the database tier to the application tier. Five hundred percent improvement in search performance.

The platform grew to serve megabus.com, Oxford Tube, Coach USA, and Coach Canada — six ticketing systems across three countries, running concurrently, sharing a single PHP library. Near-100% uptime across eight years on the simpler systems.

The rebuild

In 2008, I made the decision to rebuild in Java — distributed, scalable, a system that could grow without limits. Costed at £650k and a year. Stagecoach wanted it for £500k in six months.

It wasn’t a clean build. Budget shortfalls, contractors cycling out, corners cut under pressure. Canada first — that wasn’t too bad. Then the US. Then the UK.

The stability problems were exactly the kind that were hard to replicate and hard to test. The system struggled to stay up under load for extended periods. The more nodes, the worse it was.

On megabus, most failures meant a restart. There was an on-call rota. I still took 156 of 266 out-of-hours incidents.

I worked through a hit list methodically — JVM garbage collection, timezone corruption, index bloat, JBoss clustering. Each one found, fixed, documented.

What finally pushed it over the line was staggered nightly automated restarts of each node. There were servers on the network that hadn’t been restarted in years. It was a hail Mary. It worked.

The migration tooling — ~100GB of data across three countries, parallelised across a hand-built dependency graph — was written from scratch, because nothing off the shelf did what was needed.

Polskibus

In 2011, the same stakeholder network — Souter Investments, the investment arm of Sir Brian Souter — launched Polskibus: megabus in Eastern Europe, serving Poland and most of its neighbours.

This time, kraya was the lead contractor, managing external suppliers directly.

The same Java ticketing system, adapted. Feature flags separated behaviour between the two systems within a shared codebase. The same schemas ran across both — which occasionally meant running migrations on a system where a given feature wasn’t relevant. The accepted trade-off for keeping the codebase sane.

Polish tax law required a VAT receipt printed for every booking — via VPN, to a thermal printer at Polskibus HQ. Receipt printing was asynchronous. Booking completion never depended on a successful print. Retry-until-success for compliance. In practice: the printer ran out of paper regularly. It never once affected an order.

Traffic was spikier than megabus. Route launches produced sharp, visible demand spikes. The rate limiter was owned directly this time — on megabus, that had been a third party’s responsibility. The difference mattered.

10 December 2012

Polskibus ran a promotional campaign. Fifty thousand 1 PLN tickets.

A bank of screens on the wall, all the key stats for the system. It looked cool — a bit like being on a TV show.

32,513 bookings in a single day. A booking every 2.7 seconds, sustained. 44,487 passengers. 168,000 searches per hour at peak. 14,920 concurrent sessions — seven times the average. The previous single-day record was 7,543.

Watched closely, all day.

Nothing broke. Nothing screamed.

That result was the product of a database indexing change four months earlier that dropped front-end CPU load from 95.8% to 2.5%, documented capacity thresholds with action runbooks for each level, and years of knowing exactly where the system would crack under pressure.

Zero emergency tickets. Zero downtime.

What it was

A £13,200 system became a platform serving millions of passengers across North America, the UK, and Eastern Europe. I built and held it under conditions that would have broken most systems — because every architectural decision was made by whoever would be fixing it at 3am if it went wrong .

SC890602 forged in Scotland, for the world