Over on the 37signals blog, DHH writes Mr. Moore gets to punt on sharding. His argument is basically that if you continually delay fixing your data storage and retrieval layer, Moore’s Law will be there to save our ass–over and over again.

Bzzzt. Wrong answer.

Depending on future improvements to fix your own bad planning is a risky way to build an on-line service–especially one you expect to grow and charge money for.

It’s easy to forget history in this industry (as Paul pointed out in the comments on that post). There was a point a few years ago when people still believed the clock speed of CPUs would be doubling roughly every 18 months for half the cost. Putting aside that Moore’s Law is really about transistor density and not raw speed, we all ended up taking a funny little detour anyway.

Until recently, the sweet spot (in terms of cost and power use) was probably a dual CPU, dual core server with 16 or 32GB of RAM. But soon that’ll be dual quads with 32 or 64GB of RAM. And then it’ll be quad eight core CPUs with 128GB or whatever.

But notice that nowadays we’re not all running 6.4GHz CPUs in our servers. Instead we’re running multi-core CPUs at slower clock speeds. Those two are definitely not equivalent.

A funny thing happens as you add cores and CPUs. You begin to find that the underlying software doesn’t always… get this… scale. That’s right. Software designed in a primarily single or dual CPU world starts to show its age and performance limitation in a world where you have 8, 16, or 32 cores per server (and more if you’re running one of those crazy Sun boxes).

You see, David is talking specifically about MySQL (and probably InnoDB), which is currently being patched by outside developers precisely because it has multi-core issues . Its locking is expensive and not granular enough to utilize all those cores. It’s expensive in terms of memory use too. And there are assumptions built into the I/O subsystem that don’t scale well in today’s world of fast multi-disk RAID units, SSDs, and SANs. People are hitting these issues in the real world and it’s definitely becoming a serious bottleneck.

See Also: The New MySQL Landscape.

Moore’s Law is no silver bullet here. A fundamental change has occurred in the hardware platform and now we’re all playing catch-up in one way or another.

I’ll discuss this a bit in my upcoming MySQL Conference Talk too. The world is not nearly as clear or simple as DHH is suggesting. Perhaps they can get by with constantly postponing the work of sharding their database, but that doesn’t mean you should follow their lead.

(comments)