Throughput is a measure of how much work a system gets done in a given amount of time. For an AI model it usually means how many tokens it generates per second, or how many requests it can serve at once before performance starts to drop.
Picture a supermarket checkout. One cashier can scan items quickly, but with a long queue what really matters is how many shoppers leave the store per minute across all the registers. That total flow is throughput. A single fast checkout is about inference speed for one customer, while throughput is about serving the whole crowd. Push too many people through one lane and everyone waits.
Throughput and cost are tightly linked. A system with high throughput squeezes more useful work out of the same hardware, which lowers the cost per token and keeps the experience smooth when traffic spikes. Batching is the big lever here: instead of running each request alone, the GPU processes a group together and gets far more done per cycle. The trade-off is that one user might wait a few milliseconds longer while their request joins a batch.
So you tune for the workload. A chat feature with bursty traffic leans on batching and a streaming response so the first words appear fast even under load. A nightly job that summarises ten thousand documents cares only about total tokens per hour, not how any single one feels. Watch the wrong number and you either over-buy hardware or hit a wall on launch day.
At TopDevs we size and tune AI systems for real throughput, not demo numbers, so the feature still feels fast on the busiest day instead of only when one person is testing it.
