dot-coms' newest secret weapon doesn't have name

by Lenny Liebmann
<[email protected]>
Lenny Liebmann is a consultant and writer specializing in the application of computing and communications technologies to real-world business challenges. His work appears in Computerworld, Internet Week, Network Magazine, and other leading IT publications.

Editor's Note: I am pleased to reprint this article by permission of the quarter-million circulation ComputerWorld magazine, from whose January 31, 2000, issue it came. This article is important because it is among the first to recognize in a large public forum that a huge disparity exists among the skills of system administrators and that those who are at the high end of the scale can make dramatic improvements to business's bottom line. He references the interesting term "über-geek" and uses the term "holistic" in reference to senior administrators' abilities — I really liked that. It is articles like these that show that systems administration is finally being recognized as a true profession unto itself. — RK

There's an absolutely critical new skill that most IT staffs sorely lack. It's a skill that clearly separates successful dot-coms from e-commerce wanna-bes. It's also a skill that doesn't really have a name yet. For now, let's just call it Web-application scalability engineering. It's part science and part art, and it's crucial to dealing with the new economics of e-commerce.

I know what you're thinking: "We have Java programmers. We have UNIX systems administrators. They know how to build systems that scale." But do they? Or do they just know how to throw hardware at a problem?

I've seen dot-coms achieve 160 times the capacity of a corporate Web site with a budget one-fifth as big. How do they do it? They're sure not gonna tell you, because it's at the core of their competitive advantage.

But what I can tell you is the dot-coms have some smart technical people on their teams who take a very cross-disciplinary approach to achieving scalability. These people understand networking issues like Internet service provider peering (how their providers interconnected with other ISPs) and payload-to-header ratios (a factor that determines how efficiently they use their available bandwidth). They understand systems issues like processor utilization and caching. They understand software engineering issues like database connection pooling (which reduces the strain on back-end resources) and intelligent agents. Their holistic view of how these components affect a customer's ability to get a fast response when they click on a Web-page button makes them formidable competitors.

These new IT alchemists know how to stretch a budget, too. They have zero tolerance for software licensing schemes that penalize them for success. They'd rather write a database from scratch than shell out six figures for the privilege of running a vendor's solution across 48 commodity Intel servers. They understand that scale isn't just about MIPS and megabits per second. It's about dollars and cents.

You see, you can't achieve scale when you're still taking a stovepipe approach to the problem, divvying up infrastructure responsibilities and application developers. The new breed of technosavant is a polymath who understands that to achieve end-to-end performance you need to end-to-end perspective. That end-to-end mentality is something to which most IT shops still only give lip service. But the über-geeks who ride shotgun on the Web's top sites have it scoped. If you don't think your company's success hinges on Web-application scalability engineering, just look at outfits like DoubleClick, Amazon.com, and Motley Fool. They've been able to rise as far and as fast as they have because they've cracked the code of scalability. And they did it before the cash started pouring in.

The Web operates under a set of technical and economic principles vastly different from the world of client/server application development. At the heart of that difference is scale. Once your site starts attracting heavy traffic, scalability is what makes the difference between revenue and ridicule. Recognize it as a distinct discipline. Cultivate it among the members of your staff with the right competencies and disposition. Or prepare yourself to become the next Web scandal. Those are about your only choices in the megahit per second world of the Web.