Set up Hadoop on OSX

Let's say you're looking to get you MacBook Pro all set up with a local Hadoop instance to play with for that long flight across the Atlantic (or some other time when you really want to be running locally).

The following steps work for OSX 10.7 Lion

• Pull down the latest stable release (as of today, 1.0.4) from http://hadoop.apache.org
• Un tar, and move to your favorite projects directory
• Following the Single Node Setup is a great way to get going

The following tips may be useful the first time you're setting this up on a new computer:

1. Turn on SSH (System Preferences => Sharing => Remote Login => "On")
2. For the JAVA_HOME variable, you could put in the current full path: 
   • export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Home
   • or, you could do it the smart way:
   • export JAVA_HOME=`/usr/libexec/java_home`
3. If you get this error: 

   ERROR org.apache.hadoop.hdfs.server.datanode.DataNode: java.io.IOException: Incompatible namespaceIDs in /Users/jchen/Data/Hadoop/dfs/data: namenode namespaceID = 773619367; datanode namespaceID = 2049079249

   
   It's because you've formatted the namenode twice. Happens when you're walking through tutorials. The answer is well spelled out here (which is another good setup tutorial). The summary is: either start over - delete the datanode directory and then reformat the name node - or manually fix the version file in the datanode to match the name node.

Enabling X11 Forwarding to Ubuntu on AWS

If you're using Amazon Web Services EC2, you probably discovered that the RightScale Linux images are very good starting points. The Ubuntu server image is a good starting point for many standard dev or demo purposes. Usually the ssh session is good enough, but recently I wanted to run JConsole on the server to access some MBeans published by the graph database Neo4J.

Here are the changes I made to my basic server set up script (you do automate all the steps you like to apply to any new EC2 image, right?). Add 2 X-related packages and switched from the headless OpenJDK to the full OpenJDK:

apt-get install xauth -y

apt-get install x11-apps -y

#need full jdk, not headless in order to run jconsole as UI

#apt-get install openjdk-6-jre-headless -y

apt-get install openjdk-6-jdk -y

And remember to update your SSH command to include trusted X11 forwarding

ssh -i root@.compute-1.amazonaws.com -Y -C

Content Access Becomes King

The old mantra was that "Content is King" but that's changing quickly and it's now more true to say that "Content Access is King." The software applications that put the content in front of the user is what determines the winner.

In the overall evolution of the mobile app market, the open web, and the semantic web one irony is that while content is becoming more accessible, more accessed, and more widely used the seeming importance of the actual content is going down. Consumers have always thought of the content access tool as the product, not the content itself. When it was difficult to get to the content repositories, the Bloomberg terminal, LexisNexis green screen, and the CompuServe dial-up software were the product and there was no way to get to the content without going through the access platform.

The internet and search engines started the trend - consumers did a search and got results and never knew that the content was not owned by Google or Yahoo - and did not think about the quality, accuracy, or completeness of the search results because they were much more focused on the ease of access. This trend accelerates with newer internet applications and the Mobile Internet in particular. Users, consumer and professionals, will buy the apps that give them access to content - regardless of the content source. Yelp! and Urban Spoon replaced Zagat because it was easy to find and contribute reviews on your mobile device, not because the content was better or more complete than Zagat.

To compete in this industry today information services companies must:

1. Continue to make your content better - try to maintain the Accuracy, Completeness, and Timeliness gap between proprietary content in captive repositories and the Free Content so that the upstarts stay at 80% solutions (or go down, not up)
2. Innovate and deliver the world class applications and access points that consumers want. Incumbent information providers need to create rich mobile, always on, easy-to-use apps that make it easy to find, read, and contribute content from any device at any time that consumers like to use and promote to their friends. Rich User Experience, cutting edge software development.
3. Open up your content repositories through APIs to get your content used by new innovators. Embedding your content in the largest social media applications, the new local apps, and every entrepreneur's crazy new idea gets your content in front of new users and new customers. By making it easy to use your content in new ways and new ideas, you also forestall the creation of even more competitive content sources and make your content the preferred choice for every new innovator.
4. Overhaul your content supply chain. Make it nimble, make it flexible. New content sources, new content enrichment, new content integration. Faster, cheaper, and better.

More sophisticated customers may realize that free or smaller content providers do not meet the very high Accuracy, Completeness, and Timeliness hurdles that Professional information workers have. But they will still buy and use the easy-access mobile solutions to get access to the 80% content anywhere and all the time. As the 80% solutions get better (85%, 90%, ...), the professionals may decide they don't need to pay full price either.

Most ironically, the more content providers try to lock up their content, the more it will become commoditized -- startups who make great software products using a free content source because it's available and easy to use convince customers that the free content is good enough. There goes your proprietary brand and content differentiation.

Don't Fire QA - Embed QA in to Agile teams

Mike Gualtieri recently wrote a post about how 1 financial data company was able to improve software quality by firing their QA team and making developers responsible for testing the software. This is a good concept - have the team of engineers responsible for creating the software also be responsible for proving the software works. I agree that you should break down the barrier between a development team and a QA team. I disagree that you should fire your testers. You should Embed them in to Agile teams.

I met with a client today who described doing weekly releases to a very high volume online advertising system. This system has hundreds of servers around the globe and is directly revenue generating software with complex algorithms. His approach was not to fire the QA but to embed them in to the development scrum teams to help the developers make even better software.

This is one of the basic tenets of typical Agile software development approaches. To quote Janet Gregory, Agile teams have "Blurred Lines Between Roles":

• Agile developers are "test infected"
• Agile testers and programmers collaborate
• Agile testers and customers collaborate
• "Whole Team" responsibility for testing
• Everyone understands the business

When you follow these approaches, you get Unit tests from your developers. You get acceptance tests written for user stories BEFORE the code is written. You get developers who try to prove their code works, not who just try the happy path and throw the code over the wall to QA and wait to see what bugs get reported back.

When you have developers who are responsible for executing regression tests, you get automated unit and regression tests to validate the software in every build. And when you have developers who are responsible for successfully deploying the software to production they think about how to automate the deployment so it works every time. This is all good!

But you still need people who are trained to think about testing, requirements, and user story completeness, and who are used to looking for corner cases, and thinking like a user, and who have seen bad data, funny encodings, and the hundred other oddities that a skilled tester can find in software.

This is why it's better to embed the QA testers in to the scrum teams. The TEAM is responsible for the software quality. The 1 or 2 engineers on the team with a QA background go about ensuring quality differently - they work with the users and customers more to ensure the team really understands how the system is going to be used. They build automated tests for parts of the system that are not easy to automate with unit tests. They double check the consistency of the look & feel. They run stress tests. They manage sample data. They ensure that 2 user stories do not conflict with each other. They do all the things that a developer focused on a single user story might miss or not realize was important, because after all the developer's code passed the unit test and passed the acceptance tests.

If you want great software, you need more than just coders. You need testers - but you need Agile testers who are part of the software creation process, not a separate team given an impossible task of "finding all the bugs" in code casually written by the coders. Don't fire, EMBED QA!

Custom app dev is DEAD. Long live the Agile Business Platforms.

Custom application development is dead. Over the next 3 years Agile Business Platforms development like force.com and Mendix will replace custom development for 90% of business applications. The ability to rapidly prototype business requirements and deploy scalable, working applications in a fraction of the time of traditional Enterprise application development processes is a game-changing business advantage. No one who understands the ROI and business value benefit will hire a Java or .Net developer to build a new business application from scratch. Anyone looking to reduce costs and improve business agility by reinventing their legacy systems needs to look at a tool like Mendix that can deliver immediate business applications and continuous Agile business improvements.

The traditional 2- or 3-year Enterprise application development process run in the traditional way by the IT team is a waste of money and time and sacrifices key business agility. In today's hyper competitive and fast moving world, no business can afford to wait that long to introduce new capabilities, integrate with new supply chain partners, or automate existing costly manual processes. Agility, flexibility, and lower cost are the name of the game.

These Agile Business Platforms can be either on-premise of cloud based Platforms-as-a-Service (PaaS) options. The key is to be able to have a business analyst sit with users and business people and turn requirements in to prototypes immediately. This way the business people can "touch and feel" the application and see how their business process will work. They can provide feedback and iterate through processes, problems, and ideas in a matter of days not months. This is the definition of an Agile business and it is the promise of on-demand IT services that require a minimum of custom coding and maintenance. The companies who embrace and benefit from these cloud platforms will be able to out innovate and out compete their competitors by trying new business ideas, improving business processes, and leveraging the global supply chain of partners to produce the best products, services, and customer experience. IT must be the enabler, not the bottleneck to this true Business Agility.

Long live the new Agile Business Platforms.

Cross Platform Mobile App Development

In order to avoid headaches, reduce time, and reach a broader audience it is critical to have a good cross platform (or is that cross-device?) mobile application development framework to enable a "Write Once, Run Anywhere" experience without having your dev team try to learn five different SDK's and a zillion different libraries. With the plethora of different mobile platforms and operating systems, to reach the largest audience you would need to target at least 3 separate major SDK's - Apple's iOS, Google's Android, and RIM's Blackberry.

And then let's not forget the other smaller (but still significant) players, HP/Palm webOS, Microsoft's Win Phone 7, and Symbian.

So that's at least 6 separate SDK's and versions of mobile apps your team would have to build. Oh boy, that quick mobile app you wanted to build just got a lot harder.

Or did it? What if you could use a standard application development toolkit, maybe something a lot of developers already have experience with that worked across all the major mobile devices? That would suddenly cut your 6 separate SDK's back down to 1, plus some wrappers to get the native app's built and deployed on each platform.

Sounds pretty good - right?

Well - it's here, and it's HTML5. That's right, your favorite good old fashioned web development toolkit is also the best mobile development toolkit for building cross device mobile applications.

One of the best tools for packaging your HTML5 based app for each mobile platform is PhoneGap, an open source tool that uses each major SDK to provide a native mobile app for each platform. These HTML5 mobile apps have full access to native features and look like all the other apps you are already using. Heck, a lot of apps you are using are already developed using PhoneGap. They are working on additional enhancements to make automated build processes so that even the work of setting up and build five different flavors of your mobile app is automated.

Then there are various javascript libraries available to make your app shine and give you full development tools for building that killer business logic you need. Some of the ones AGS uses in our application development are:

1. JQuery Mobile - open source jQuery plug-in with great mobile app theming support
2. Sencha - ExtJS based commercial desktop & mobile app library
3. Rhomobile - a set of products for full enterprise mobile application development

By leveraging these tools and techniques, we are able to build full-featured mobile apps that work on multiple platforms at the same speed (or faster in some cases) as traditional web applications.

Dynamic Named Ranges in Excel

I want to share a tip I read about (and used) today for Dynamic Named Ranges in Excel. I've used Excel for a long time, and spent many many many hours building spreadsheets that mimic little databases because a client or user couldn't support a database but really "wanted" a simple spreadsheet. I often use named ranges for data validation to substitute for lookup tables and foreign key constraints. A common problem is when the users go to add a new value to the bottom of the lookup table the new value would fall outside the named range and not be used in the lookup table.

I've tried teaching users, writing instructions on how to expand the range, and had fallen in to the habit of making the last element be a row of dashes ("-----") with instructions to "add a new entry to the list by inserting a row ABOVE the dashes." Not very elegant at all, but it did usually work.

The nice folks at ozgrid.com gave a good, simple explanation for how to create Dynamic Named Ranges that will automatically expand to include new values added at the end of the list. Great time saver!

Cloud Architecture Realizes 33% Savings and Faster Time To Market

In a previous post, I noted that Economics - specifically cost savings and moving from up-front capital purchases to on-demand usage - was a key reason to adopt cloud based services. Recently, Alliance helped a publishing and information services client design a cloud based deployment for a new information portal web application that saved 33% in infrastructure costs and saved months on deployment time. This is a typical content based online publishing application, with large content repositories, search indexes, innovative user experience, and community feedback widgets like recommendations and filtering.

The previous version of this web application was built and hosted in their corporate data center - which is a large and fairly efficient internal hosting operation. They follow ITIL standards, leverage bulk buying discounts, try to automate where possible, and many large billion dollar revenue web applications. But the internal hosting center is still in a craftsman mode - each new hosting application is custom designed, with very specific hardware specs and design, specific hardware ordered and configured for just this one application, and custom software installation followed for each and every machine in the environment. This type of craftsman approach is fairly typical in hosting (and software development) and drives a MUCH longer timeline as well as higher costs.

Cloud IaaS providers (like Amazon, Rackspace, or GoGrid) take a different approach - they give you a fixed set of virtual machines to choose from with defined compute specs and standard connectivity and let you turn them on at will. Instantly. Zero-delay. With no long planning, ordering, and set up. As you can imagine, this approach drastically reduces the time required to deploy an application to production. And it eliminates many of the error prone and manually intensive (and expensive) steps in planning, ordering, installing, and configuring hardware for the specific application. All of this is a clear win for IaaS over in-house hosting data centers. The Per Server savings are dramatic...

Combining a mix of on-premise database servers, data transfer costs, load balancers, and data storage costs gets the typical total cost savings to 30-40%. When you consider that by using a cloud IaaS solution you can scale instantly and incrementally to meet demand as you grow your product, you can easily add disaster recovery (DR), and you follow best practices like automated deployment it's obvious why cloud based application architectures are becoming the norm, even for typical "enterprise applications."

Iterative Development Key for New Product Development

There is a lot of discussion about Agile vs. Plan Driven software development methodologies - usually driven by advocates of one or the other. Certainly, it's a more complex topic and there is a wide spectrum of extremely structured, plan-driven approaches to extremely agile, change embracing approaches. One well thought-out and nuanced explanation with advice for choosing where on the continuum to try to be can be found here.

Two concepts are fairly easy to accept and understand - one is that when you need to coordinate many different parties a more planned approach with more formal communication between the parties is essential. This comes in to play when coordinating marketing campaigns, customer roll-outs, contracts with vendors, and corporate funding approval processes. In all of these cases you need to have an idea of how long an effort will take and how much it will cost - and all of the stakeholders involved need to be formally updated to changes in the overall timeline or cost . This can form the basis of the overall planned Program at a high level.

The second is that at a lower level of detail, and especially with new product development, it is impossible to formally plan the software development with any degree of accuracy regarding how long each feature will take or whether every scalability and performance metric will be achieved by the first, second, or third attempt. Additionally, the new product development implementation process is really a R&D process, not a defined engineering approach. You need to explore whether customers can understand the new product, whether the new UI makes sense, whether the most anticipated features are actually used, and on and on. As part of your new product conceptualization, you need to prototyp, build, and let customers work with the software to learn what is really needed in the product. Josh Green, the CEO of Panjiva, explained it well during his presentation at the Information Industry Summit this week:

When we showed Panjiva to our first customers, we expected them to be blown away - we were giving them more data, more easily to use than they ever had before. But they kept asking for more data or asking for the data to be shown differently. After a while I realized that my customers were TEACHING US HOW THE PRODUCT SHOULD REALY WORK!

This is the essential point about truly new product development - it is an exploration. No one can completely specify in advance how the product should be implemented. Users will teach you how to make the software better. But users can only teach you when they react to something that you put in front of them. Iterative development, and especially early prototyping and mock-ups are key to enabling this customer feedback loop. It is imperative to learn from your customers and deliver the next version quickly so that they continue to see the improvements and continue to help you make your product better.

At the same time, you probably need a Program level plan - how much funding do you have? How many iterations can you afford? What conferences or industry events make sense to demo your software at?

So you see, it's not always either Planned or Agile, it's often "both."

Networked Content Manifesto

Yesterday Temis distributed print outs of their newly released "Networked Content Manifesto" at the SIIA conference in New York. Ignoring the irony of a firm focused on digital content processing printing copies of their content and handing them out (or maybe that's an accurate reflection of the current state of online publishing?), the manifesto is a good introduction to the concepts surrounding Semantic technologies and "Content Enrichment". In Alliance's Information Services industry focus, wework frequently with clients on implementing taxonomies, ontologies, classification systems, and other tools to automate the "Enrichment" portion of the information supply chain processing pipeline. Combining these sophisticated tools with good Master Data or Master Entity repositories and linking with other internal content or the public Linked Data initiative provides a much richer experience for researchers and content users.

By providing more meaning - more semantic information - about the concepts, people, and entities that are in the document and providing easy ways to navigate through the overall content space we can create a richer experience for the end user and make it easier to discover the information she is looking for. For the publisher this translates in to increased usage which means easy subscription renewals, so that's a good thing too!

Automated Functional Tester

This week saw the roll-out of Alliance's new "Automated Functional Tester" (AFT) framework. This is a very comprehensive and powerful automated testing framework enabling the end-to-end testing of complex web applications. It's driven by easy to write business requirements, handles rich AJAX interactions, integrated Windows security and file operations, enables fast regression suite runs, and rich reporting. This framework is built on and integrates a number of open-source testing frameworks to provide full capability and more comprehensive testing than any individual tool. And by using standard test execution environments, like Selenium, it's possible to take tests generated by the framework and execute them in Cloud testing platforms for performance testing or large-scale cross browser testing.

We use automated testing very extensively, from developer written unit tests to fully automated test suites. Sometimes clients balk at the high license fees associated with very full featured commercial testing tools like QTP and this open source testing framework is a great capability to provide our clients with custom software development backed by fully automated testing at a lower price point!

Great job to the AFT team!

Why use the Cloud?

A lot of people ask "What is Cloud Computing?" There are good answers for that, and I'm sure I'll expand on it more in this blog as well. SaaS, PaaS, IaaS, Public, Private, Hybrid, Virtualization, Storage, Compute, Developer Clouds, Production Clouds, etc. Lots and lots of definitions of "What is Cloud."

But there's a fundamental question I want to answer - Why use the Cloud? What's the business value for Cloud Computing?

Well - it's all about 3 things:

1. Agility
2. Capability
3. Economics

Agility - using on demand software and infrastructure enables you to be more flexible and achieve a given result faster. If you're looking to implement a CRM package, turning on Salesforce.com or NetSuite takes a day or two to start and a week or two to get rolling - much quicker than the usual months-long implementation effort for on premise package installation. If you need to turn on some developer or testing lab server instances you can sign up with an Skytap, GoGrid or Amazon.com, configure a virtual instance and be completely done by lunch instead of the usual few weeks procurement process to buy a new server, get the IT department to install and configure it, and be able to start using it productively. Using cloud based platforms for development, like Azure, Long Jump, Force.com or App Engine enable much faster software development and implementation and let you start realizing the business benefit of your application much sooner.

Capability - Using proven, scalable services to build on enables much richer functionality and incredible scalability that you would be very hard pressed to achieve from scratch even if you could afford to build out all of the infrastructure and functionality. You can add amazing rich features to your web application, whether it's web analytics, photo sharing, data entry forms, data visualization and other business functions so much faster by integrating existing software service like Google Analytics, Flickr, Caspio, or Birst than you ever could by trying to write requirements, design, build and test the software, deploy the new feature and worry about scalability. When you're looking to deploy robust, scalable applications that support thousands of users around the world, deliver fast performance to every corner of the globe, and meet modern expectations for responsiveness and user experience using Cloud based Content Delivery Networks, Cloud based in-memory caching or databases, cloud-based load balancing between data centers and across tiers, and instantaneously available storage and compute capacity enables companies to have world-class capabilities that they could never hope to achieve by building out their own internal data centers.

Economics - Like many revolutions, by "doing things differently" Cloud based solutions are able to provide very large cost savings. For individual services, whether it's reliable storage or scalable web servers, a huge IaaS provider like Rackspace can provide much lower cost per unit than an internal data center could. These large Cloud providers make an investment in automation that enables them to efficiently manage orders of magnitude more capacity per engineer. They are able to buy in bulk from server and equipment vendors, site data centers next to low-cost electricity sources, and operate their data centers like a modern "factory" instead of the "cottage industry" of the internal corporate data centers - with all the cost savings you would expect. The same is true of SaaS providers, who are able to invest massive amounts in software engineering, user experience and design to produce world class software and share the costs across dozens, thousands, or millions of subscribers. An individual corporation will need to invest much much more to build their own version of a software package or service than it would cost to subscribe to a well implemented service.

For the Cloud subscriber, there is another benefit to subscribing to a cloud service rather than buying or building your own internal version - the switch from a large up-front capital investment to a monthly operating cost. This On Demand pricing model enables you to start small and incrementally increase your investment as users move to the service or your customer base grows, rather than requiring you to over-provision for your hoped-for 1 or 2 year projections. This subscription model also ensures you have continued flexibility to evaluate whether the deployed solution is meeting the needs of your customers - the cost to increase the service, switch to a different solution, or turn off the service completely if the business model is not working no longer hinges on huge up-front investments that must be managed and written off - you simply stop paying the monthly subscription and move on.

As you can see, using Cloud based services provide very real, concrete business value by enabling more Agility, more Capability, and better Economics. There are also numerous other intangible benefits to moving to Cloud services. You need to automate your deployment, you can test more easily, you need to define real service boundaries and interfaces, you can prototype integration and new services quickly, you have access to a wider range of technology options, and on and on.

That's Why you SHOULD use the Cloud!!

The Best Tool for the Job

I spent some time this month learning more about the LexisNexis Data Analytics Supercomputer (or HPCC) system. This is a great tool for building and deploying lightning fast Content Services with high quality content enrichment to turn commodity content in to a valuable information product that professionals will pay for. It's purpose built by a team of very smart technologists who have been turning out content-based products for a long time. For re-engineering a large scale data processing system with hundreds or thousands of input files running on a variety of maxed out Unix servers or mainframes HPCC is a great fit.

The system design reminded me a lot of the notion that when you are doing something over and over again, the right approach is not to just get better and faster at repeating the specific task - but to find a better tool to eliminate the task. If you're a professional carpenter and putting in a lot of nails you may be tempted to look for a better hammer. The best hammer money can buy will certainly help you hammer in nails better. And that hammer will feel great - like an extension of your arm. It will have perfect balance and enable you to bang in nails all day long without feeling tired. You could be recognized as a true black belt hammering expert able to pound on those nails as long as anyone.

But if you spend the time to find & use a better tool (or even invent a new tool) -- say a nail gun! -- you won't be 5% or 10% faster at hammering - you won't be hammering at all. For the first hour the new nail gun will feel klunky, the tool will be inelegant and ungainly compared to that perfectly balanced hammer you're used to. You may resist because the hammer you're used to has worked so well for so long - you can't count how many hours you've used it to bang in the same nail over and over and over again.

But once you grok the new nail gun and get to used to the new way of accomplishing your task you'll see how much faster the new tool lets you put in nails. In fact, you'll be so fast at putting in nails, you'll stop measuring how long it takes to put in a nail and start thinking about the fact that you're 1,000% faster at putting up wood framing, which is the actual goal. This insightful leap requires you to realize that continuing down your existing path and perfecting your use of your current tool is not the best approach - optimization will not win over innovation.

In software development we are blessed that it is so easy to become 5 or 10% more productive. There is always a new trick to learn, always a new pattern to understand, always one more tip for a language to master, always a faster hotkey or way to do that same task again with fewer clicks, always a piece of code to copy & paste from a previous module or an internet example, always a faster way to repeat the same solution from yesterday over again today. Hmm... maybe "blessed" should really be "cursed".

And maybe the real blessing is that it is also very feasible to invent or find completely new approaches and new tools that make ourselves orders of magnitude more productive - to deliver more business value. The key is to recognize when using a new tool will not just save a few minutes here and there, but will actually save weeks or months of effort!

If you're working on an information supply chain that is processing terabytes of content or billions of rows of data, the LexisNexis DAS system and the ECL language is definitely one of those orders of magnitude improvement tools. It may take a while to stop thinking about SQL and good old RDBMS design, but once you get used to the power of your new nail-gun, you love it!

Forcing new google gadget to load, not from cache

One common challenge when working with Google Gadgets is forcing a new development version to load, not reusing a cached version. A quick search on finds several suggestions and conversations about how to do this, but they are not complete. Renaming the gadget for each save is overkill and not worth it. The developer gadget does not appear to correctly skip the cache in Safari. A little playing reminded me that the key is to edit the bogus querystring parameter to make the URL unique each time you re-add the gadget.

So, as the first suggestion above says, add a parameter like ?nocache=1 to the end of your gadget moduleurl when adding it.

AND... when you need to make another edit, change the parameter! The 2nd time you need for force a reload, set the url to ?nocache=2

Next time, ?nocache=3

And so on...

Google Gadgets

I started experimenting with Google Gadgets today - and I am very impressed with how easy the iGoogle and OpenSocial frameworks are to use. This is a great example of how a good framework, combined with some on-demand hosting services and mashups of REST-based data sources can make lightweight (or "rowboat") application development very simple. Mixing data from multiple sources via easy-to-integrate URL's is a great development paradigm for creating operational dashboards or quick glance BI reports.

In my particular case, I was looking to be able to quickly glance at a page of stock charts to see whether any stocks I'm currently following are at an interesting point. FinViz.com is a wonderful freemium site for simple analysis, but to follow 4 or 5 stocks required just too many searches and clicks. But FinViz follows the good Web 2.0 pattern of making their charts publishable and linkable through a URL, so you can combine them to make a dashboard or mashup very easily. (More proof that REST-based web services are better for distributing your data or enabling partners to easily integrate. SOAP is ok for heavy weight system integration, but if your goal is to get your data used, use REST!)

A few hours of reading the docs and scripting and now I've got a configurable stock dashboard integrated with my iGoogle home page. I just have to decide how to handle the charts when the gadget is not maximized -- any suggestions? :-)

Integrating Content in Customer Workflow Applications

As I discussed in my last post, it is critical for information firms to distribute and deliver their content where their customers want to purchase, read, and use it. In addition to needing to support more platforms and mobile devices, publishers need to integrate their content in to their customer's workflow applications. Typical customers do not purchase data, reports, articles, or analyses for fun - they purchase these content pieces to help them accomplish their larger business objectives. This might mean checking the credit history of a loan applicant, scouring scientific literature for a grant application, or analyzing the legal precedents of an upcoming case. In each situation, the goal of the customer is not to get a document to display on her bookshelf, but to get access to the information in the course of performing a larger task.

Obviously, the key to make this easy is to not force the customer to leave their workflow tools and go to a discrete content website but to INTEGRATE the content in to the workflow tool. In many cases Information Providers are moving up the value chain and selling those workflow enabling tools - for example:

• LexisNexis Litigation Workflow Tools
• D&B Risk Management Tools
• Thomson Reuters Healthcare Clinical Evidence Solutions

In each case, the Information Provider adds value to the raw underlying content by providing a business process focused (or Knowledge Worker focused) workflow tool to help the knowledge worker turn the content from "information" in to "knowledge". Of course, this value-add is in addition to the direct content value-add through aggregation, classification, entity recognition & linking, and analytics applied in the Information Factory.

By building these Workflow solutions on top of standard REST or WebServices API's, the provider is able to leverage the existing content repositories, search, and enrichment capabilities without having to create duplicate product stacks. And, as I mentioned last time, the same content could be directly integrated with a customer's proprietary workflow tools or data. A well thought out and designed API enables many additional distribution channels and revenue generation options with a high ROI by enabling both provider built workflow tools and customer integration. And of course those same API's can support the new mobile applications that customers want.

There is also a very fast growing and productive middle ground for Information Providers to integrate with - Microsoft Sharepoint. This is one of the fastest growing products in Microsoft history, and used (admittedly to varying extents) by many, or even most, Corporate customers. The ability to provide pre-built Web Parts that customers can easily install in to their Sharepoint portal to integrate a provider's search and document retrieval capabilities are very powerful and provide a low-cost, very easy way to move beyond a generic internet content web site to a more integrated Enterprise application. Providing additional Sharepoint workflow enabled components can enable an information provider to provide the full range of on-premise, Enterprise capability without the cost and complexity of Enterprise application development and maintenance.

Sell Where Your Customers Buy

Many marketing or business planning discussions eventually hit on the seemingly self-evident notion that you should "Sell What People Are Buying." (For a humorous discussion of this, see this blog) In addition to this basic truth, it is also important to "Sell Where Your Customers Buy." Think of the sidewalk entrepreneurs in New York City selling ice cold sodas or bottled water out of a cooler on the corner in lower Manhattan on a hot summer day. These vendors are able to charge a premium for a cold drink because they are right where the people are walking and thirsty - they are selling Where their customers want to buy.

For Information Providers, this truth is becoming very important. Not only is the Information industry moving heavily from a print to an all-digital delivery model, and not only are the roles of intermediaries and centralized procurement "locations" (such as corporate libraries) being diminished, and not only do Information Consumers have more choices and more access mechanisms than ever before, but in addition the easy access to information is changing the notion of information gathering from a stand alone task to simply one small (and hopefully automated) step within a larger process. Let me explain what I mean in a little more detail.

We all know that Print media is turning Digital. See the news stories about declining Newspaper readership, Amazon's (and BN's, and Border's, and...) push for digital books, the frequent discussion about the "Google Effect" on content providers, and the Open Access Initiative. The digital content is available on many platforms - the web, your mobile phone, smartphones, tablet computers, and e-readers. Information Consumers expect to be able to read the articles, analyze the data, and do their jobs on any of these devices, anytime during the day, on the train, in the airport, or wherever they are. This is what I mean by "Sell WHERE Your Customers Buy" - you need to be able to sell the article, or provide the data visualization, or the drill-down analysis on any device connected to the internet. If your valued, long-term Customer is sitting in the airport, curious about something she is not going to wait to get back to her office to use her subscription service to order a new report - she wants to download it to her phone, right now. If your service does not provide that, she will go to your competitor's app and get her Information right now. Your valued, long-term, trusted relationship goes right out the window because your Web 1.0 Information Delivery website cannot meet her expectation of instant information access.

The effect of this is to require every Information Provider to need an iPhone app. Well an iPhone app and an Android app. Oh, and an iPad app. And probably a Galaxy and Playbook app. And of course, a mobile friendly version of the web site. You can see, this quickly adds up.

Early attempts to create mobile apps usually involved creating a separate application stack, content collection, or Information Delivery Channel. This was the quickest, and least expensive approach to get the first mobile app out and does work. But the cost curve is not pretty, and you soon need to maintain and support 10 different applications, delivery channels, etc at 10x the cost for only marginal revenue increases.

The better approach is to build an API. Not just a dirty hack of a url that can be used to access content, but an actual, thought-out Service Oriented API that provides billing, metering, search, retrieval, and easy integration for apps. Now those 10 apps are just thin layers reusing the core SOA enterprise services and can be built and maintained cheaply. Better yet - if you have a loyal community who truly values your content, they will wind up building innovative apps on every platform you have never heard of to access and embed your content. This is because your customers are empowered to scratch their own itch, and build an application to solve their own personal or particular corporate need. And it turns out that a lot of times your customers know how to use your content better than your own product development team -- so the apps they build and the embedded data usage they put together are more useful to your customers than the pre-built website or stand alone applications you might design. Your customers become your distributors and open up entire new retail channels you were not even aware of.

So... you get more sales channels, at a cheaper delivery cost, with more customer value from enabling your content through an API. Sounds like a Win-Win-Win!

The New CMOS - Cloud Mobile Open-source Social

CMOS is a well-known acronym in chip manufacturing meaning "Complementary Metal-Oxide-Semiconductor". According to Wikipedia:

"CMOS" refers to both a particular style of digital circuitry design, and the family of processes used to implement that circuitry on integrated circuits (chips). CMOS circuitry dissipates less power than logic families with resistive loads. Since this advantage has increased and grown more important, CMOS processes and variants have come to dominate, thus the vast majority of modern integrated circuit manufacturing is on CMOS processes.

In other words - because this way of designing and building chips has distinct advantages in improving efficiency and enabling new features, it has become the dominant approach to building chips. Any mainstream (non-experimental) chip manufacturer has settled on this process because of the inherent benefits.

There is a new CMOS taking over the software development industry - "Cloud Mobile Open-source Social." New applications, whether they are commercial software products, internal enterprise applications, or B2C web applications (think "Web 2.0", Linked Data and related) are - by default - leveraging the combination of Cloud, Mobile, Open-source, and Social technologies to enable rich feature, scalable, always on applications that are developed faster and cheaper.

Users expect 24/7 access. Consumers expect to be able to share and tag and link content. Readers expect to be able to access articles and applications on any computer, any device, anywhere. This is a fundamentally different mindset than "traditional" application development and traditional publishing models. It requires new ways of architecting applications and new ways of thinking about data and content. It's a heck of a good place to work!

Ten Steps to Agile Software Development Process Improvement

In a previous blog entry, I mentioned 10 Steps to Improve Software Development Process. If you read these and pause for a minute, you'll notice that many of these are actually principles of Agile development. I want to expand on a few of these thoughts here.

1. Focus on the top 20% of features:
This is one of the primary drivers of value in Agile custom application developmentpractices. By prioritizing and rank-ordering every item in the feature request backlogs, only the most important ones are developed. By focusing on these Top 20%, you can often satisfy 80% of what end users want, and they can start using the system sooner, to add to the profitability of your company. (If the most important 20% of the features do not add to your company profitability, you should probably cancel the project!)

2. Break things up into smaller projects:
Big projects turn in to huge projects. And miss deadlines. And run over budget. Reading the Standish Group CHAOS report figures on failed IT projects always makes me wonder why more people don't follow the simple advice to "not bite off more than you can chew"?

Again, the Agile concept of smaller, more frequent releases echoes through this item. Getting a working system in to the hands of users is always a good thing - that's why you are implementing the system to begin with! Giving an important and useful portion of functionality early is even better. This has many organization benefits - from psychological ones like proving that the overall program can work and the enthusiasm from a successful launch, to risk mitigation benefits like the ability to redirect spending on an investment after the first release if priorities change, to the practical one that it is a lot easier to notice a project in trouble if a specific release is over budget or late than if fuzzy 'milestones' are being missed.

5. Obtain user feedback
During the implementation process, keep the end users constantly in the picture. Show them early versions of the system - for instance at each Sprint Demonstration or at least with small, frequent releases. Let them give you feedback, and above all, let them change their minds without being punished. Trust your end users; they know what works and what does not - and they are the ones who are going to use the system every day! When they see working features, they may be able to better prioritize other changes or features to be able to complete an important business process or simplify many steps.

7. Mistakes are a way of learning
Remove the blame culture. Let people make mistakes quickly - failing early is much better than missing a delivery date. These so-called mistakes are part of the overall discovery process and will help lead and evolve to the eventual solution. If you blame people for mistakes (picking the wrong feature for a Sprint, not seeing a bug, very bad color choices) they will react to the blame and change their own behavior. Rather than being an active participant in making the project a success, they will become "followers", just doing what they are told - no way to get blamed there - and punching the clock. The good people will hate the blame culture enough to look for work somewhere better.

10. Something has to give in the Iron Pyramid (Quality, Time, Cost, Features)
The old Iron Triangle, which I've always thought should be an Iron Pyramid with Quality as an explicit dimension, still holds true today. In an agile development process you try to "bake in" high Quality by using unit tests, refactoring, engaged people, and frequent review processes. You fix the Time or at least the time cycles - every two weeks you have a Sprint release; and the Cost is essentially fixed based on the size and members of the team. The Features dimension is what gives - and that is where the prioritization comes in. By putting the most important features first, you complete as many in each Sprint as you can and know that you are achieving the most important features at any given time.

Over a longer time horizon, say a Release of 5 to 6 Sprints, you can adjust the Time and Cost dimensions, by letting the process run for more Sprints or deciding that enough Features are ready to stop this project.

I focus on the Iron Pyramid because we all know the truth. If you push people hard enough, they will relax some of the quality checks, and they will get one extra feature in. But you have lowered the quality level - maybe not enough to notice today, but you will pay for it in the future. Whether it is the performance testing that is skipped, leading to a slow web application, or "smelly code" that costs more and more to maintain over time you pay for the lapse in quality.

Data Visualization and Web 2.0

I love data visualization techniques. From my early days as a operations data analyst and all of my software development career, finding patterns in data and finding an easy way to convey the pattern through a graph or other visualization has always been fun. Working on custom application development projects that provided a picture of how the business was doing, where customers were spending, etc is fun. Now working with our Business Information Services clients to help create innovative approaches to information discovery and data analysis is fun. It really is true that often "a picture is worth 1,000 words."

I vividly recall a stubborn memory leak my team had been trying to track down for several weeks. This was a long time ago, in the days of VB6 COM dll's running inside ASP web pages, and we were pretty sure our code was not leaking. The team had found memory leaks before, and tracked every single one of them down to circular references in our COM object model so that the automatic release never occurred. Historically, it had been easy to find a leak by running a simple load script, executing each page thousands of time in isolation and watch to see which page shows the memory leak. But not this time. We had run the load tests several times and never found the leak. We scanned the code thoroughly. We added as many "set obj = nothing" safety lines as we could. But still the production web servers kept leaking memory, and we were forced to move the automatic restarts of the servers from weekly to daily and hope our band-aid would hold.

One day, I had some down time and decided to see if I could find a correlation between the memory used and what pages were invoked on the production system. An hour or two later, I had pulled all the IIS log files, gotten dumps of memory traces from the systems team and started my analysis. A bit of awk, grep, Access, and other quick and dirty processes later to pull out the data I wanted, adjust for timezones, aggregate hits to cumulative 15 minute buckets and otherwise line up the datasets and I was ready to plot the data.

Instantly, the answer of where the leak was was obvious. The two lines, cumulative hits to a particular URL and memory in use were nearly on top of each other. The correlation jumped out, completely overwhelming the noise of other URL's, pages, etc. This is the power of a good visualization. (Of course, it turned out that the leak was coming from a web services API proxy URL, not a page in the website that everyone had focused on! Since the proxy was not 'in' the website it had been ignored for weeks as the team hunted for the answer.)

Recently, some colleagues and i were discussing what areas Alliance Global Services provides solutions to clients in. This is a pretty broad topic, and we talked about the types of industries we serve (including our focus on Business Information Services), the geographies we serve (mostly North East US, from about Virginia to Boston), the types of services we provide (Custom Software Development,Application Architecture Analysis). And we talked about the easiest way to visualize our coverage areas.

Well, today I had a little downtime before the holidays. So I took a list of our client locations used some simple geocoding tools, and put together two quick samples of mapping in the Web 2.0 world - one using a Yahoo! map through batchgeocode.com and the other using the Google visualization API.

Batchgeocode.com made it very easy to process the first set of data and create a map but then you were stopped. Google was a different story - getting the map running required coding, but then I had full control. To see the first map, visit this blog on Alliance Global Services.

Obviously it's not perfect, but lots of fun for a quick afternoon's work!