The Best Tool for the Job

I spent some time this month learning more about the LexisNexis Data Analytics Supercomputer (or HPCC) system. This is a great tool for building and deploying lightning fast Content Services with high quality content enrichment to turn commodity content in to a valuable information product that professionals will pay for. It's purpose built by a team of very smart technologists who have been turning out content-based products for a long time. For re-engineering a large scale data processing system with hundreds or thousands of input files running on a variety of maxed out Unix servers or mainframes HPCC is a great fit.

The system design reminded me a lot of the notion that when you are doing something over and over again, the right approach is not to just get better and faster at repeating the specific task - but to find a better tool to eliminate the task. If you're a professional carpenter and putting in a lot of nails you may be tempted to look for a better hammer. The best hammer money can buy will certainly help you hammer in nails better. And that hammer will feel great - like an extension of your arm. It will have perfect balance and enable you to bang in nails all day long without feeling tired. You could be recognized as a true black belt hammering expert able to pound on those nails as long as anyone.

But if you spend the time to find & use a better tool (or even invent a new tool) -- say a nail gun! -- you won't be 5% or 10% faster at hammering - you won't be hammering at all. For the first hour the new nail gun will feel klunky, the tool will be inelegant and ungainly compared to that perfectly balanced hammer you're used to. You may resist because the hammer you're used to has worked so well for so long - you can't count how many hours you've used it to bang in the same nail over and over and over again.

But once you grok the new nail gun and get to used to the new way of accomplishing your task you'll see how much faster the new tool lets you put in nails. In fact, you'll be so fast at putting in nails, you'll stop measuring how long it takes to put in a nail and start thinking about the fact that you're 1,000% faster at putting up wood framing, which is the actual goal. This insightful leap requires you to realize that continuing down your existing path and perfecting your use of your current tool is not the best approach - optimization will not win over innovation.

In software development we are blessed that it is so easy to become 5 or 10% more productive. There is always a new trick to learn, always a new pattern to understand, always one more tip for a language to master, always a faster hotkey or way to do that same task again with fewer clicks, always a piece of code to copy & paste from a previous module or an internet example, always a faster way to repeat the same solution from yesterday over again today. Hmm... maybe "blessed" should really be "cursed".

And maybe the real blessing is that it is also very feasible to invent or find completely new approaches and new tools that make ourselves orders of magnitude more productive - to deliver more business value. The key is to recognize when using a new tool will not just save a few minutes here and there, but will actually save weeks or months of effort!

If you're working on an information supply chain that is processing terabytes of content or billions of rows of data, the LexisNexis DAS system and the ECL language is definitely one of those orders of magnitude improvement tools. It may take a while to stop thinking about SQL and good old RDBMS design, but once you get used to the power of your new nail-gun, you love it!

Forcing new google gadget to load, not from cache

One common challenge when working with Google Gadgets is forcing a new development version to load, not reusing a cached version. A quick search on finds several suggestions and conversations about how to do this, but they are not complete. Renaming the gadget for each save is overkill and not worth it. The developer gadget does not appear to correctly skip the cache in Safari. A little playing reminded me that the key is to edit the bogus querystring parameter to make the URL unique each time you re-add the gadget.

So, as the first suggestion above says, add a parameter like ?nocache=1 to the end of your gadget moduleurl when adding it.

AND... when you need to make another edit, change the parameter! The 2nd time you need for force a reload, set the url to ?nocache=2

Next time, ?nocache=3

And so on...

Google Gadgets

I started experimenting with Google Gadgets today - and I am very impressed with how easy the iGoogle and OpenSocial frameworks are to use. This is a great example of how a good framework, combined with some on-demand hosting services and mashups of REST-based data sources can make lightweight (or "rowboat") application development very simple. Mixing data from multiple sources via easy-to-integrate URL's is a great development paradigm for creating operational dashboards or quick glance BI reports.

In my particular case, I was looking to be able to quickly glance at a page of stock charts to see whether any stocks I'm currently following are at an interesting point. FinViz.com is a wonderful freemium site for simple analysis, but to follow 4 or 5 stocks required just too many searches and clicks. But FinViz follows the good Web 2.0 pattern of making their charts publishable and linkable through a URL, so you can combine them to make a dashboard or mashup very easily. (More proof that REST-based web services are better for distributing your data or enabling partners to easily integrate. SOAP is ok for heavy weight system integration, but if your goal is to get your data used, use REST!)

A few hours of reading the docs and scripting and now I've got a configurable stock dashboard integrated with my iGoogle home page. I just have to decide how to handle the charts when the gadget is not maximized -- any suggestions? :-)

Integrating Content in Customer Workflow Applications

As I discussed in my last post, it is critical for information firms to distribute and deliver their content where their customers want to purchase, read, and use it. In addition to needing to support more platforms and mobile devices, publishers need to integrate their content in to their customer's workflow applications. Typical customers do not purchase data, reports, articles, or analyses for fun - they purchase these content pieces to help them accomplish their larger business objectives. This might mean checking the credit history of a loan applicant, scouring scientific literature for a grant application, or analyzing the legal precedents of an upcoming case. In each situation, the goal of the customer is not to get a document to display on her bookshelf, but to get access to the information in the course of performing a larger task.

Obviously, the key to make this easy is to not force the customer to leave their workflow tools and go to a discrete content website but to INTEGRATE the content in to the workflow tool. In many cases Information Providers are moving up the value chain and selling those workflow enabling tools - for example:

• LexisNexis Litigation Workflow Tools
• D&B Risk Management Tools
• Thomson Reuters Healthcare Clinical Evidence Solutions

In each case, the Information Provider adds value to the raw underlying content by providing a business process focused (or Knowledge Worker focused) workflow tool to help the knowledge worker turn the content from "information" in to "knowledge". Of course, this value-add is in addition to the direct content value-add through aggregation, classification, entity recognition & linking, and analytics applied in the Information Factory.

By building these Workflow solutions on top of standard REST or WebServices API's, the provider is able to leverage the existing content repositories, search, and enrichment capabilities without having to create duplicate product stacks. And, as I mentioned last time, the same content could be directly integrated with a customer's proprietary workflow tools or data. A well thought out and designed API enables many additional distribution channels and revenue generation options with a high ROI by enabling both provider built workflow tools and customer integration. And of course those same API's can support the new mobile applications that customers want.

There is also a very fast growing and productive middle ground for Information Providers to integrate with - Microsoft Sharepoint. This is one of the fastest growing products in Microsoft history, and used (admittedly to varying extents) by many, or even most, Corporate customers. The ability to provide pre-built Web Parts that customers can easily install in to their Sharepoint portal to integrate a provider's search and document retrieval capabilities are very powerful and provide a low-cost, very easy way to move beyond a generic internet content web site to a more integrated Enterprise application. Providing additional Sharepoint workflow enabled components can enable an information provider to provide the full range of on-premise, Enterprise capability without the cost and complexity of Enterprise application development and maintenance.

Sell Where Your Customers Buy

Many marketing or business planning discussions eventually hit on the seemingly self-evident notion that you should "Sell What People Are Buying." (For a humorous discussion of this, see this blog) In addition to this basic truth, it is also important to "Sell Where Your Customers Buy." Think of the sidewalk entrepreneurs in New York City selling ice cold sodas or bottled water out of a cooler on the corner in lower Manhattan on a hot summer day. These vendors are able to charge a premium for a cold drink because they are right where the people are walking and thirsty - they are selling Where their customers want to buy.

For Information Providers, this truth is becoming very important. Not only is the Information industry moving heavily from a print to an all-digital delivery model, and not only are the roles of intermediaries and centralized procurement "locations" (such as corporate libraries) being diminished, and not only do Information Consumers have more choices and more access mechanisms than ever before, but in addition the easy access to information is changing the notion of information gathering from a stand alone task to simply one small (and hopefully automated) step within a larger process. Let me explain what I mean in a little more detail.

We all know that Print media is turning Digital. See the news stories about declining Newspaper readership, Amazon's (and BN's, and Border's, and...) push for digital books, the frequent discussion about the "Google Effect" on content providers, and the Open Access Initiative. The digital content is available on many platforms - the web, your mobile phone, smartphones, tablet computers, and e-readers. Information Consumers expect to be able to read the articles, analyze the data, and do their jobs on any of these devices, anytime during the day, on the train, in the airport, or wherever they are. This is what I mean by "Sell WHERE Your Customers Buy" - you need to be able to sell the article, or provide the data visualization, or the drill-down analysis on any device connected to the internet. If your valued, long-term Customer is sitting in the airport, curious about something she is not going to wait to get back to her office to use her subscription service to order a new report - she wants to download it to her phone, right now. If your service does not provide that, she will go to your competitor's app and get her Information right now. Your valued, long-term, trusted relationship goes right out the window because your Web 1.0 Information Delivery website cannot meet her expectation of instant information access.

The effect of this is to require every Information Provider to need an iPhone app. Well an iPhone app and an Android app. Oh, and an iPad app. And probably a Galaxy and Playbook app. And of course, a mobile friendly version of the web site. You can see, this quickly adds up.

Early attempts to create mobile apps usually involved creating a separate application stack, content collection, or Information Delivery Channel. This was the quickest, and least expensive approach to get the first mobile app out and does work. But the cost curve is not pretty, and you soon need to maintain and support 10 different applications, delivery channels, etc at 10x the cost for only marginal revenue increases.

The better approach is to build an API. Not just a dirty hack of a url that can be used to access content, but an actual, thought-out Service Oriented API that provides billing, metering, search, retrieval, and easy integration for apps. Now those 10 apps are just thin layers reusing the core SOA enterprise services and can be built and maintained cheaply. Better yet - if you have a loyal community who truly values your content, they will wind up building innovative apps on every platform you have never heard of to access and embed your content. This is because your customers are empowered to scratch their own itch, and build an application to solve their own personal or particular corporate need. And it turns out that a lot of times your customers know how to use your content better than your own product development team -- so the apps they build and the embedded data usage they put together are more useful to your customers than the pre-built website or stand alone applications you might design. Your customers become your distributors and open up entire new retail channels you were not even aware of.

So... you get more sales channels, at a cheaper delivery cost, with more customer value from enabling your content through an API. Sounds like a Win-Win-Win!

The New CMOS - Cloud Mobile Open-source Social

CMOS is a well-known acronym in chip manufacturing meaning "Complementary Metal-Oxide-Semiconductor". According to Wikipedia:

"CMOS" refers to both a particular style of digital circuitry design, and the family of processes used to implement that circuitry on integrated circuits (chips). CMOS circuitry dissipates less power than logic families with resistive loads. Since this advantage has increased and grown more important, CMOS processes and variants have come to dominate, thus the vast majority of modern integrated circuit manufacturing is on CMOS processes.

In other words - because this way of designing and building chips has distinct advantages in improving efficiency and enabling new features, it has become the dominant approach to building chips. Any mainstream (non-experimental) chip manufacturer has settled on this process because of the inherent benefits.

There is a new CMOS taking over the software development industry - "Cloud Mobile Open-source Social." New applications, whether they are commercial software products, internal enterprise applications, or B2C web applications (think "Web 2.0", Linked Data and related) are - by default - leveraging the combination of Cloud, Mobile, Open-source, and Social technologies to enable rich feature, scalable, always on applications that are developed faster and cheaper.

Users expect 24/7 access. Consumers expect to be able to share and tag and link content. Readers expect to be able to access articles and applications on any computer, any device, anywhere. This is a fundamentally different mindset than "traditional" application development and traditional publishing models. It requires new ways of architecting applications and new ways of thinking about data and content. It's a heck of a good place to work!

Ten Steps to Agile Software Development Process Improvement

In a previous blog entry, I mentioned 10 Steps to Improve Software Development Process. If you read these and pause for a minute, you'll notice that many of these are actually principles of Agile development. I want to expand on a few of these thoughts here.

1. Focus on the top 20% of features:
This is one of the primary drivers of value in Agile custom application developmentpractices. By prioritizing and rank-ordering every item in the feature request backlogs, only the most important ones are developed. By focusing on these Top 20%, you can often satisfy 80% of what end users want, and they can start using the system sooner, to add to the profitability of your company. (If the most important 20% of the features do not add to your company profitability, you should probably cancel the project!)

2. Break things up into smaller projects:
Big projects turn in to huge projects. And miss deadlines. And run over budget. Reading the Standish Group CHAOS report figures on failed IT projects always makes me wonder why more people don't follow the simple advice to "not bite off more than you can chew"?

Again, the Agile concept of smaller, more frequent releases echoes through this item. Getting a working system in to the hands of users is always a good thing - that's why you are implementing the system to begin with! Giving an important and useful portion of functionality early is even better. This has many organization benefits - from psychological ones like proving that the overall program can work and the enthusiasm from a successful launch, to risk mitigation benefits like the ability to redirect spending on an investment after the first release if priorities change, to the practical one that it is a lot easier to notice a project in trouble if a specific release is over budget or late than if fuzzy 'milestones' are being missed.

5. Obtain user feedback
During the implementation process, keep the end users constantly in the picture. Show them early versions of the system - for instance at each Sprint Demonstration or at least with small, frequent releases. Let them give you feedback, and above all, let them change their minds without being punished. Trust your end users; they know what works and what does not - and they are the ones who are going to use the system every day! When they see working features, they may be able to better prioritize other changes or features to be able to complete an important business process or simplify many steps.

7. Mistakes are a way of learning
Remove the blame culture. Let people make mistakes quickly - failing early is much better than missing a delivery date. These so-called mistakes are part of the overall discovery process and will help lead and evolve to the eventual solution. If you blame people for mistakes (picking the wrong feature for a Sprint, not seeing a bug, very bad color choices) they will react to the blame and change their own behavior. Rather than being an active participant in making the project a success, they will become "followers", just doing what they are told - no way to get blamed there - and punching the clock. The good people will hate the blame culture enough to look for work somewhere better.

10. Something has to give in the Iron Pyramid (Quality, Time, Cost, Features)
The old Iron Triangle, which I've always thought should be an Iron Pyramid with Quality as an explicit dimension, still holds true today. In an agile development process you try to "bake in" high Quality by using unit tests, refactoring, engaged people, and frequent review processes. You fix the Time or at least the time cycles - every two weeks you have a Sprint release; and the Cost is essentially fixed based on the size and members of the team. The Features dimension is what gives - and that is where the prioritization comes in. By putting the most important features first, you complete as many in each Sprint as you can and know that you are achieving the most important features at any given time.

Over a longer time horizon, say a Release of 5 to 6 Sprints, you can adjust the Time and Cost dimensions, by letting the process run for more Sprints or deciding that enough Features are ready to stop this project.

I focus on the Iron Pyramid because we all know the truth. If you push people hard enough, they will relax some of the quality checks, and they will get one extra feature in. But you have lowered the quality level - maybe not enough to notice today, but you will pay for it in the future. Whether it is the performance testing that is skipped, leading to a slow web application, or "smelly code" that costs more and more to maintain over time you pay for the lapse in quality.