The 16-Server Architecture That Streams 15 Petabytes a Day: How Tom Killalea Rebuilt Amazon Prime Video's Monolith — And Made 'Distributed First' Engineers Delete Half Their Code

It was March 2023. A blog post titled "Scaling up the Prime Video audio/video monitoring service and reducing costs by 90%" appeared on the Amazon Prime Video Tech blog. Within hours, it was the most controversial engineering article of the year.

The reason? Amazon — the company that invented microservices, the company whose CTO Werner Vogels wrote the famous "Everything Fails All The Time" mantra — had just admitted that they deleted their distributed serverless architecture and went back to a monolith.

The post was calm, technical, measured. But the subtext was explosive: distributed systems aren't always the answer. Sometimes, putting everything on one server is the right move.

The internet lost its mind. Hacker News erupted. Engineering Twitter fractured into warring camps. DHH from Basecamp declared victory for the "majestic monolith." Microservices advocates scrambled to explain why this didn't mean everything they'd preached was wrong.

But the real story wasn't about ideology. It was about Tom Killalea, Amazon's VP of Engineering, and a team of engineers who looked at a system processing 15 petabytes of video data every single day — and realized that the architecture designed to scale had become the bottleneck.

This is the story of how Amazon Prime Video's monitoring system went from 1,000+ Lambda functions and Step Functions orchestrating distributed workflows across AWS services — to 16 EC2 instances running a single monolithic process. And why the lesson they learned matters more than any framework war.

The Problem: When 'Serverless' Means You're Paying for the Network

Prime Video streams to over 200 million subscribers. Every single video — every frame, every audio track, every subtitle file — needs to be checked for quality defects before it reaches viewers. Sync issues. Color corruption. Audio dropouts. Frozen frames.

In 2020, Prime Video's quality control team built a monitoring service using AWS's serverless stack: Lambda functions for compute, Step Functions for orchestration, S3 for storage, DynamoDB for state. It was exactly what the AWS Well-Architected Framework recommended. Loosely coupled. Event-driven. Infinitely scalable.

And it worked. Sort of.

The system processed thousands of videos every day. It detected defects. It scaled up and down automatically. But it had a problem that nobody anticipated: the architecture was spending most of its time and money moving data between services.

Here's how it worked:

1. A video file would trigger a Lambda function
2. Lambda would download chunks of the video from S3 (network call #1)
3. Lambda would process frames and detect defects
4. Lambda would write intermediate results to S3 (network call #2)
5. Step Functions would orchestrate the next stage (network call #3)
6. Another Lambda would read those results from S3 (network call #4)
7. Lambda would write final results to DynamoDB (network call #5)

For every 10 seconds of video analyzed, the system was making 5+ network calls between AWS services. Each call cost money. Each call added latency. Each call had to serialize and deserialize data.

The AWS bill for this system was hitting $100,000+ per month. And it couldn't scale past a few thousand concurrent videos because Step Functions has a hard limit on state transitions.

Prime Video's senior principal engineer, Marcin Kolny, looked at the architecture diagram and asked the question that no one wanted to hear: "What if we just... put all of this on one server?"

The Heresy: Collocating Everything in RAM

The team ran an experiment. What if they took the entire distributed pipeline — video ingestion, frame extraction, defect detection, result aggregation — and collapsed it into a single process running on a single EC2 instance?

No Lambda. No Step Functions. No S3 intermediate storage. No DynamoDB coordination. Just:

• ECS Fargate running a single container
• In-memory state instead of S3 writes
• Direct function calls instead of Step Functions orchestration
• Local disk for temporary frame buffers

The results were staggering.

Cost dropped by 90%. The $100,000/month serverless architecture became a $10,000/month monolith.

Throughput increased by 10x. Without network overhead, a single instance could process far more video than dozens of Lambda invocations.

Latency dropped to milliseconds. Passing data between functions in RAM is 1,000x faster than serializing to S3 and deserializing back.

The monolith could process the same workload with 16 EC2 instances that previously required 1,000+ concurrent Lambda executions.

But here's the part that broke everyone's brains: this wasn't a failure of distributed systems. It was a failure of applying distributed systems to the wrong problem.

The Lesson: Know Your Data Flow

The Prime Video monitoring service had a fundamental characteristic that made distribution harmful: high data transfer, low fan-out.

Each video is large (gigabytes). Each video flows through the same sequential pipeline (extract frames → analyze frames → aggregate results). There's no parallelism to exploit across videos — you're just processing one stream at a time.

In this scenario, network costs dominate compute costs. Moving gigabytes of video data between Lambda, S3, and Step Functions costs more than just running the entire pipeline in one process.

Contrast this with Prime Video's actual streaming service:

• Millions of concurrent viewers
• Each viewer is independent (high fan-out)
• Each viewer streams a relatively small chunk at a time
• Content is cached globally on CloudFront CDN
• Origin servers coordinate via distributed databases (DynamoDB, Aurora)

That architecture is perfectly suited for microservices. Each component scales independently. Each viewer's request is isolated. The data flow is optimized for low-latency, high-concurrency reads.

But the monitoring service? It's a batch job in disguise. And batch jobs don't need microservices — they need collocated compute and data.

The Architecture: How 16 Servers Handle 15 Petabytes

Here's how the final monolithic architecture works:

1. ECS Fargate Containers
Each instance runs a single containerized application. The container orchestrates the entire pipeline: video download, frame extraction, defect detection, result persistence.

2. Vertical Scaling Over Horizontal
Instead of spinning up 1,000 Lambda functions, they run 16 large EC2 instances (c5.9xlarge: 36 vCPUs, 72GB RAM). Each instance processes multiple videos concurrently using threads, not network calls.

3. In-Memory State Management
Intermediate results (extracted frames, partial defect lists) are kept in RAM. No S3 writes until the final result. A video that previously required 5+ S3 round-trips now requires 1.

4. Local SSD for Frame Buffers
Video frames are temporarily written to the instance's local NVMe SSD (provisioned at 3.5GB/s read speed). This is 10x faster than writing to S3, and costs nothing since it's ephemeral.

5. Direct Database Writes
Final results go straight to DynamoDB. No intermediate Step Functions. No orchestration tax.

6. Built-in Retry Logic
If an instance crashes mid-processing, the video is re-queued. Simple dead-letter queue pattern. No distributed transaction coordination needed.

The system still uses AWS services — but only where distribution adds value:

• S3 for durable video storage (the source of truth)
• DynamoDB for storing final defect reports (globally replicated for low-latency reads)
• CloudWatch for monitoring and alerting
• SQS for work queue management

But the core processing pipeline? Pure monolith.

The Backlash: When Your Blog Post Breaks the Internet

When Marcin Kolny and his co-author Marcin Lawnik published the blog post, they expected a few hundred views. They got millions.

The microservices community was defensive. "This doesn't mean microservices are bad!" "They chose the wrong tools!" "Serverless wasn't designed for this!"

They were right — but they missed the point.

The monolith community was triumphant. "We told you so!" "Death to microservices!" "DHH was right all along!"

They were also right — and also missed the point.

The real lesson wasn't about monoliths vs. microservices. It was about architecture as a function of data flow.

Martin Kleppmann, author of Designing Data-Intensive Applications, tweeted: "This is why you need to understand your data access patterns before choosing an architecture. Not all problems are the same shape."

Tom Killalea, who had overseen the transformation, gave a talk at AWS re:Invent later that year. His message was simple:

"Use the right tool for the job. Serverless is incredible for spiky, unpredictable workloads with low data transfer. But if you're moving gigabytes of data between functions, you're paying a network tax that no amount of scale will fix. Sometimes, the best distributed system is one that doesn't distribute."

The Turning Point: When 'Best Practices' Become Dogma

The real villain of this story isn't microservices or serverless. It's cargo-culting best practices without understanding the trade-offs.

For years, Amazon's own internal engineering culture had pushed "service-oriented architecture" as gospel. Jeff Bezos' famous 2002 mandate — "All teams will expose their data and functionality through service interfaces" — had created a generation of engineers who defaulted to distribution.

But Bezos' mandate was about organizational boundaries, not technical ones. It was about letting teams ship independently without coordinating deploys. It wasn't about splitting every function into a Lambda.

Prime Video's team had accidentally conflated two ideas:

1. Organizational decoupling (good: teams own services, ship independently)
2. Runtime decoupling (sometimes good, sometimes catastrophically expensive)

They'd applied #2 to a problem that didn't need it — and paid $90,000/month for the privilege.

The fix wasn't to abandon distributed systems. It was to use them where they add value:

• API Gateway + Lambda? Perfect for user-facing APIs with millisecond response times and unpredictable load.
• Step Functions + Lambda? Great for long-running workflows with branching logic and human approval steps.
• Serverless data pipelines? Ideal for low-volume, high-latency ETL jobs that run on a schedule.

But high-throughput video processing? That's a job for a monolith.

The Legacy: A New Religion of Pragmatism

Today, the Prime Video monitoring service runs on 16 servers. It processes 15 petabytes of video data every day. It costs $10,000/month instead of $100,000.

The team published the source architecture in follow-up blog posts. Other AWS teams started auditing their own systems. Twitter's engineering team cited the article when they re-architected their timeline service. Shopify referenced it when consolidating their checkout flow.

The phrase "monolith first" entered the lexicon — not as a rejection of microservices, but as a reminder that premature distribution is expensive.

Martin Fowler updated his famous "Microservices" article with a new section: "Monolith First." He wrote:

"Even if you're certain your application will benefit from a microservices architecture, it's worth building it as a monolith first. You'll learn where the real boundaries are. You'll learn what needs to scale independently. And you'll avoid the Prime Video problem: paying for distribution you don't need."

The Prime Video case study became required reading at AWS training programs. Werner Vogels himself referenced it in keynotes: "Everything fails all the time — including your assumptions about architecture."

But the most important legacy wasn't the technical lesson. It was the cultural one.

For years, distributed systems had been sold as a default choice — the "mature" architecture, the "scalable" architecture, the architecture that "real engineers" build. The Prime Video story punctured that mythology.

It proved that the best engineers aren't the ones who build the most complex systems. They're the ones who build the simplest system that solves the problem.

Sometimes that's 1,000 microservices.

Sometimes it's 16 servers running a monolith.

And the difference between the two is worth $90,000 a month.