The Best Tool for the Job

I spent some time this month learning more about the LexisNexis Data Analytics Supercomputer (or HPCC) system. This is a great tool for building and deploying lightning fast Content Services with high quality content enrichment to turn commodity content in to a valuable information product that professionals will pay for. It's purpose built by a team of very smart technologists who have been turning out content-based products for a long time. For re-engineering a large scale data processing system with hundreds or thousands of input files running on a variety of maxed out Unix servers or mainframes HPCC is a great fit.

The system design reminded me a lot of the notion that when you are doing something over and over again, the right approach is not to just get better and faster at repeating the specific task - but to find a better tool to eliminate the task. If you're a professional carpenter and putting in a lot of nails you may be tempted to look for a better hammer. The best hammer money can buy will certainly help you hammer in nails better. And that hammer will feel great - like an extension of your arm. It will have perfect balance and enable you to bang in nails all day long without feeling tired. You could be recognized as a true black belt hammering expert able to pound on those nails as long as anyone.

But if you spend the time to find & use a better tool (or even invent a new tool) -- say a nail gun! -- you won't be 5% or 10% faster at hammering - you won't be hammering at all. For the first hour the new nail gun will feel klunky, the tool will be inelegant and ungainly compared to that perfectly balanced hammer you're used to. You may resist because the hammer you're used to has worked so well for so long - you can't count how many hours you've used it to bang in the same nail over and over and over again.

But once you grok the new nail gun and get to used to the new way of accomplishing your task you'll see how much faster the new tool lets you put in nails. In fact, you'll be so fast at putting in nails, you'll stop measuring how long it takes to put in a nail and start thinking about the fact that you're 1,000% faster at putting up wood framing, which is the actual goal. This insightful leap requires you to realize that continuing down your existing path and perfecting your use of your current tool is not the best approach - optimization will not win over innovation.

In software development we are blessed that it is so easy to become 5 or 10% more productive. There is always a new trick to learn, always a new pattern to understand, always one more tip for a language to master, always a faster hotkey or way to do that same task again with fewer clicks, always a piece of code to copy & paste from a previous module or an internet example, always a faster way to repeat the same solution from yesterday over again today. Hmm... maybe "blessed" should really be "cursed".

And maybe the real blessing is that it is also very feasible to invent or find completely new approaches and new tools that make ourselves orders of magnitude more productive - to deliver more business value. The key is to recognize when using a new tool will not just save a few minutes here and there, but will actually save weeks or months of effort!

If you're working on an information supply chain that is processing terabytes of content or billions of rows of data, the LexisNexis DAS system and the ECL language is definitely one of those orders of magnitude improvement tools. It may take a while to stop thinking about SQL and good old RDBMS design, but once you get used to the power of your new nail-gun, you love it!