Enabling X11 Forwarding to Ubuntu on AWS

If you're using Amazon Web Services EC2, you probably discovered that the RightScale Linux images are very good starting points. The Ubuntu server image is a good starting point for many standard dev or demo purposes. Usually the ssh session is good enough, but recently I wanted to run JConsole on the server to access some MBeans published by the graph database Neo4J.

Here are the changes I made to my basic server set up script (you do automate all the steps you like to apply to any new EC2 image, right?). Add 2 X-related packages and switched from the headless OpenJDK to the full OpenJDK:

apt-get install xauth -y

apt-get install x11-apps -y

#need full jdk, not headless in order to run jconsole as UI

#apt-get install openjdk-6-jre-headless -y

apt-get install openjdk-6-jdk -y

And remember to update your SSH command to include trusted X11 forwarding

ssh -i root@.compute-1.amazonaws.com -Y -C

Content Access Becomes King

The old mantra was that "Content is King" but that's changing quickly and it's now more true to say that "Content Access is King." The software applications that put the content in front of the user is what determines the winner.

In the overall evolution of the mobile app market, the open web, and the semantic web one irony is that while content is becoming more accessible, more accessed, and more widely used the seeming importance of the actual content is going down. Consumers have always thought of the content access tool as the product, not the content itself. When it was difficult to get to the content repositories, the Bloomberg terminal, LexisNexis green screen, and the CompuServe dial-up software were the product and there was no way to get to the content without going through the access platform.

The internet and search engines started the trend - consumers did a search and got results and never knew that the content was not owned by Google or Yahoo - and did not think about the quality, accuracy, or completeness of the search results because they were much more focused on the ease of access. This trend accelerates with newer internet applications and the Mobile Internet in particular. Users, consumer and professionals, will buy the apps that give them access to content - regardless of the content source. Yelp! and Urban Spoon replaced Zagat because it was easy to find and contribute reviews on your mobile device, not because the content was better or more complete than Zagat.

To compete in this industry today information services companies must:

1. Continue to make your content better - try to maintain the Accuracy, Completeness, and Timeliness gap between proprietary content in captive repositories and the Free Content so that the upstarts stay at 80% solutions (or go down, not up)
2. Innovate and deliver the world class applications and access points that consumers want. Incumbent information providers need to create rich mobile, always on, easy-to-use apps that make it easy to find, read, and contribute content from any device at any time that consumers like to use and promote to their friends. Rich User Experience, cutting edge software development.
3. Open up your content repositories through APIs to get your content used by new innovators. Embedding your content in the largest social media applications, the new local apps, and every entrepreneur's crazy new idea gets your content in front of new users and new customers. By making it easy to use your content in new ways and new ideas, you also forestall the creation of even more competitive content sources and make your content the preferred choice for every new innovator.
4. Overhaul your content supply chain. Make it nimble, make it flexible. New content sources, new content enrichment, new content integration. Faster, cheaper, and better.

More sophisticated customers may realize that free or smaller content providers do not meet the very high Accuracy, Completeness, and Timeliness hurdles that Professional information workers have. But they will still buy and use the easy-access mobile solutions to get access to the 80% content anywhere and all the time. As the 80% solutions get better (85%, 90%, ...), the professionals may decide they don't need to pay full price either.

Most ironically, the more content providers try to lock up their content, the more it will become commoditized -- startups who make great software products using a free content source because it's available and easy to use convince customers that the free content is good enough. There goes your proprietary brand and content differentiation.

Don't Fire QA - Embed QA in to Agile teams

Mike Gualtieri recently wrote a post about how 1 financial data company was able to improve software quality by firing their QA team and making developers responsible for testing the software. This is a good concept - have the team of engineers responsible for creating the software also be responsible for proving the software works. I agree that you should break down the barrier between a development team and a QA team. I disagree that you should fire your testers. You should Embed them in to Agile teams.

I met with a client today who described doing weekly releases to a very high volume online advertising system. This system has hundreds of servers around the globe and is directly revenue generating software with complex algorithms. His approach was not to fire the QA but to embed them in to the development scrum teams to help the developers make even better software.

This is one of the basic tenets of typical Agile software development approaches. To quote Janet Gregory, Agile teams have "Blurred Lines Between Roles":

• Agile developers are "test infected"
• Agile testers and programmers collaborate
• Agile testers and customers collaborate
• "Whole Team" responsibility for testing
• Everyone understands the business

When you follow these approaches, you get Unit tests from your developers. You get acceptance tests written for user stories BEFORE the code is written. You get developers who try to prove their code works, not who just try the happy path and throw the code over the wall to QA and wait to see what bugs get reported back.

When you have developers who are responsible for executing regression tests, you get automated unit and regression tests to validate the software in every build. And when you have developers who are responsible for successfully deploying the software to production they think about how to automate the deployment so it works every time. This is all good!

But you still need people who are trained to think about testing, requirements, and user story completeness, and who are used to looking for corner cases, and thinking like a user, and who have seen bad data, funny encodings, and the hundred other oddities that a skilled tester can find in software.

This is why it's better to embed the QA testers in to the scrum teams. The TEAM is responsible for the software quality. The 1 or 2 engineers on the team with a QA background go about ensuring quality differently - they work with the users and customers more to ensure the team really understands how the system is going to be used. They build automated tests for parts of the system that are not easy to automate with unit tests. They double check the consistency of the look & feel. They run stress tests. They manage sample data. They ensure that 2 user stories do not conflict with each other. They do all the things that a developer focused on a single user story might miss or not realize was important, because after all the developer's code passed the unit test and passed the acceptance tests.

If you want great software, you need more than just coders. You need testers - but you need Agile testers who are part of the software creation process, not a separate team given an impossible task of "finding all the bugs" in code casually written by the coders. Don't fire, EMBED QA!

Custom app dev is DEAD. Long live the Agile Business Platforms.

Custom application development is dead. Over the next 3 years Agile Business Platforms development like force.com and Mendix will replace custom development for 90% of business applications. The ability to rapidly prototype business requirements and deploy scalable, working applications in a fraction of the time of traditional Enterprise application development processes is a game-changing business advantage. No one who understands the ROI and business value benefit will hire a Java or .Net developer to build a new business application from scratch. Anyone looking to reduce costs and improve business agility by reinventing their legacy systems needs to look at a tool like Mendix that can deliver immediate business applications and continuous Agile business improvements.

The traditional 2- or 3-year Enterprise application development process run in the traditional way by the IT team is a waste of money and time and sacrifices key business agility. In today's hyper competitive and fast moving world, no business can afford to wait that long to introduce new capabilities, integrate with new supply chain partners, or automate existing costly manual processes. Agility, flexibility, and lower cost are the name of the game.

These Agile Business Platforms can be either on-premise of cloud based Platforms-as-a-Service (PaaS) options. The key is to be able to have a business analyst sit with users and business people and turn requirements in to prototypes immediately. This way the business people can "touch and feel" the application and see how their business process will work. They can provide feedback and iterate through processes, problems, and ideas in a matter of days not months. This is the definition of an Agile business and it is the promise of on-demand IT services that require a minimum of custom coding and maintenance. The companies who embrace and benefit from these cloud platforms will be able to out innovate and out compete their competitors by trying new business ideas, improving business processes, and leveraging the global supply chain of partners to produce the best products, services, and customer experience. IT must be the enabler, not the bottleneck to this true Business Agility.

Long live the new Agile Business Platforms.

Cross Platform Mobile App Development

In order to avoid headaches, reduce time, and reach a broader audience it is critical to have a good cross platform (or is that cross-device?) mobile application development framework to enable a "Write Once, Run Anywhere" experience without having your dev team try to learn five different SDK's and a zillion different libraries. With the plethora of different mobile platforms and operating systems, to reach the largest audience you would need to target at least 3 separate major SDK's - Apple's iOS, Google's Android, and RIM's Blackberry.

And then let's not forget the other smaller (but still significant) players, HP/Palm webOS, Microsoft's Win Phone 7, and Symbian.

So that's at least 6 separate SDK's and versions of mobile apps your team would have to build. Oh boy, that quick mobile app you wanted to build just got a lot harder.

Or did it? What if you could use a standard application development toolkit, maybe something a lot of developers already have experience with that worked across all the major mobile devices? That would suddenly cut your 6 separate SDK's back down to 1, plus some wrappers to get the native app's built and deployed on each platform.

Sounds pretty good - right?

Well - it's here, and it's HTML5. That's right, your favorite good old fashioned web development toolkit is also the best mobile development toolkit for building cross device mobile applications.

One of the best tools for packaging your HTML5 based app for each mobile platform is PhoneGap, an open source tool that uses each major SDK to provide a native mobile app for each platform. These HTML5 mobile apps have full access to native features and look like all the other apps you are already using. Heck, a lot of apps you are using are already developed using PhoneGap. They are working on additional enhancements to make automated build processes so that even the work of setting up and build five different flavors of your mobile app is automated.

Then there are various javascript libraries available to make your app shine and give you full development tools for building that killer business logic you need. Some of the ones AGS uses in our application development are:

1. JQuery Mobile - open source jQuery plug-in with great mobile app theming support
2. Sencha - ExtJS based commercial desktop & mobile app library
3. Rhomobile - a set of products for full enterprise mobile application development

By leveraging these tools and techniques, we are able to build full-featured mobile apps that work on multiple platforms at the same speed (or faster in some cases) as traditional web applications.

Dynamic Named Ranges in Excel

I want to share a tip I read about (and used) today for Dynamic Named Ranges in Excel. I've used Excel for a long time, and spent many many many hours building spreadsheets that mimic little databases because a client or user couldn't support a database but really "wanted" a simple spreadsheet. I often use named ranges for data validation to substitute for lookup tables and foreign key constraints. A common problem is when the users go to add a new value to the bottom of the lookup table the new value would fall outside the named range and not be used in the lookup table.

I've tried teaching users, writing instructions on how to expand the range, and had fallen in to the habit of making the last element be a row of dashes ("-----") with instructions to "add a new entry to the list by inserting a row ABOVE the dashes." Not very elegant at all, but it did usually work.

The nice folks at ozgrid.com gave a good, simple explanation for how to create Dynamic Named Ranges that will automatically expand to include new values added at the end of the list. Great time saver!

Cloud Architecture Realizes 33% Savings and Faster Time To Market

In a previous post, I noted that Economics - specifically cost savings and moving from up-front capital purchases to on-demand usage - was a key reason to adopt cloud based services. Recently, Alliance helped a publishing and information services client design a cloud based deployment for a new information portal web application that saved 33% in infrastructure costs and saved months on deployment time. This is a typical content based online publishing application, with large content repositories, search indexes, innovative user experience, and community feedback widgets like recommendations and filtering.

The previous version of this web application was built and hosted in their corporate data center - which is a large and fairly efficient internal hosting operation. They follow ITIL standards, leverage bulk buying discounts, try to automate where possible, and many large billion dollar revenue web applications. But the internal hosting center is still in a craftsman mode - each new hosting application is custom designed, with very specific hardware specs and design, specific hardware ordered and configured for just this one application, and custom software installation followed for each and every machine in the environment. This type of craftsman approach is fairly typical in hosting (and software development) and drives a MUCH longer timeline as well as higher costs.

Cloud IaaS providers (like Amazon, Rackspace, or GoGrid) take a different approach - they give you a fixed set of virtual machines to choose from with defined compute specs and standard connectivity and let you turn them on at will. Instantly. Zero-delay. With no long planning, ordering, and set up. As you can imagine, this approach drastically reduces the time required to deploy an application to production. And it eliminates many of the error prone and manually intensive (and expensive) steps in planning, ordering, installing, and configuring hardware for the specific application. All of this is a clear win for IaaS over in-house hosting data centers. The Per Server savings are dramatic...

Combining a mix of on-premise database servers, data transfer costs, load balancers, and data storage costs gets the typical total cost savings to 30-40%. When you consider that by using a cloud IaaS solution you can scale instantly and incrementally to meet demand as you grow your product, you can easily add disaster recovery (DR), and you follow best practices like automated deployment it's obvious why cloud based application architectures are becoming the norm, even for typical "enterprise applications."

Iterative Development Key for New Product Development

There is a lot of discussion about Agile vs. Plan Driven software development methodologies - usually driven by advocates of one or the other. Certainly, it's a more complex topic and there is a wide spectrum of extremely structured, plan-driven approaches to extremely agile, change embracing approaches. One well thought-out and nuanced explanation with advice for choosing where on the continuum to try to be can be found here.

Two concepts are fairly easy to accept and understand - one is that when you need to coordinate many different parties a more planned approach with more formal communication between the parties is essential. This comes in to play when coordinating marketing campaigns, customer roll-outs, contracts with vendors, and corporate funding approval processes. In all of these cases you need to have an idea of how long an effort will take and how much it will cost - and all of the stakeholders involved need to be formally updated to changes in the overall timeline or cost . This can form the basis of the overall planned Program at a high level.

The second is that at a lower level of detail, and especially with new product development, it is impossible to formally plan the software development with any degree of accuracy regarding how long each feature will take or whether every scalability and performance metric will be achieved by the first, second, or third attempt. Additionally, the new product development implementation process is really a R&D process, not a defined engineering approach. You need to explore whether customers can understand the new product, whether the new UI makes sense, whether the most anticipated features are actually used, and on and on. As part of your new product conceptualization, you need to prototyp, build, and let customers work with the software to learn what is really needed in the product. Josh Green, the CEO of Panjiva, explained it well during his presentation at the Information Industry Summit this week:

When we showed Panjiva to our first customers, we expected them to be blown away - we were giving them more data, more easily to use than they ever had before. But they kept asking for more data or asking for the data to be shown differently. After a while I realized that my customers were TEACHING US HOW THE PRODUCT SHOULD REALY WORK!

This is the essential point about truly new product development - it is an exploration. No one can completely specify in advance how the product should be implemented. Users will teach you how to make the software better. But users can only teach you when they react to something that you put in front of them. Iterative development, and especially early prototyping and mock-ups are key to enabling this customer feedback loop. It is imperative to learn from your customers and deliver the next version quickly so that they continue to see the improvements and continue to help you make your product better.

At the same time, you probably need a Program level plan - how much funding do you have? How many iterations can you afford? What conferences or industry events make sense to demo your software at?

So you see, it's not always either Planned or Agile, it's often "both."

Networked Content Manifesto

Yesterday Temis distributed print outs of their newly released "Networked Content Manifesto" at the SIIA conference in New York. Ignoring the irony of a firm focused on digital content processing printing copies of their content and handing them out (or maybe that's an accurate reflection of the current state of online publishing?), the manifesto is a good introduction to the concepts surrounding Semantic technologies and "Content Enrichment". In Alliance's Information Services industry focus, wework frequently with clients on implementing taxonomies, ontologies, classification systems, and other tools to automate the "Enrichment" portion of the information supply chain processing pipeline. Combining these sophisticated tools with good Master Data or Master Entity repositories and linking with other internal content or the public Linked Data initiative provides a much richer experience for researchers and content users.

By providing more meaning - more semantic information - about the concepts, people, and entities that are in the document and providing easy ways to navigate through the overall content space we can create a richer experience for the end user and make it easier to discover the information she is looking for. For the publisher this translates in to increased usage which means easy subscription renewals, so that's a good thing too!

Automated Functional Tester

This week saw the roll-out of Alliance's new "Automated Functional Tester" (AFT) framework. This is a very comprehensive and powerful automated testing framework enabling the end-to-end testing of complex web applications. It's driven by easy to write business requirements, handles rich AJAX interactions, integrated Windows security and file operations, enables fast regression suite runs, and rich reporting. This framework is built on and integrates a number of open-source testing frameworks to provide full capability and more comprehensive testing than any individual tool. And by using standard test execution environments, like Selenium, it's possible to take tests generated by the framework and execute them in Cloud testing platforms for performance testing or large-scale cross browser testing.

We use automated testing very extensively, from developer written unit tests to fully automated test suites. Sometimes clients balk at the high license fees associated with very full featured commercial testing tools like QTP and this open source testing framework is a great capability to provide our clients with custom software development backed by fully automated testing at a lower price point!

Great job to the AFT team!

Why use the Cloud?

A lot of people ask "What is Cloud Computing?" There are good answers for that, and I'm sure I'll expand on it more in this blog as well. SaaS, PaaS, IaaS, Public, Private, Hybrid, Virtualization, Storage, Compute, Developer Clouds, Production Clouds, etc. Lots and lots of definitions of "What is Cloud."

But there's a fundamental question I want to answer - Why use the Cloud? What's the business value for Cloud Computing?

Well - it's all about 3 things:

1. Agility
2. Capability
3. Economics

Agility - using on demand software and infrastructure enables you to be more flexible and achieve a given result faster. If you're looking to implement a CRM package, turning on Salesforce.com or NetSuite takes a day or two to start and a week or two to get rolling - much quicker than the usual months-long implementation effort for on premise package installation. If you need to turn on some developer or testing lab server instances you can sign up with an Skytap, GoGrid or Amazon.com, configure a virtual instance and be completely done by lunch instead of the usual few weeks procurement process to buy a new server, get the IT department to install and configure it, and be able to start using it productively. Using cloud based platforms for development, like Azure, Long Jump, Force.com or App Engine enable much faster software development and implementation and let you start realizing the business benefit of your application much sooner.

Capability - Using proven, scalable services to build on enables much richer functionality and incredible scalability that you would be very hard pressed to achieve from scratch even if you could afford to build out all of the infrastructure and functionality. You can add amazing rich features to your web application, whether it's web analytics, photo sharing, data entry forms, data visualization and other business functions so much faster by integrating existing software service like Google Analytics, Flickr, Caspio, or Birst than you ever could by trying to write requirements, design, build and test the software, deploy the new feature and worry about scalability. When you're looking to deploy robust, scalable applications that support thousands of users around the world, deliver fast performance to every corner of the globe, and meet modern expectations for responsiveness and user experience using Cloud based Content Delivery Networks, Cloud based in-memory caching or databases, cloud-based load balancing between data centers and across tiers, and instantaneously available storage and compute capacity enables companies to have world-class capabilities that they could never hope to achieve by building out their own internal data centers.

Economics - Like many revolutions, by "doing things differently" Cloud based solutions are able to provide very large cost savings. For individual services, whether it's reliable storage or scalable web servers, a huge IaaS provider like Rackspace can provide much lower cost per unit than an internal data center could. These large Cloud providers make an investment in automation that enables them to efficiently manage orders of magnitude more capacity per engineer. They are able to buy in bulk from server and equipment vendors, site data centers next to low-cost electricity sources, and operate their data centers like a modern "factory" instead of the "cottage industry" of the internal corporate data centers - with all the cost savings you would expect. The same is true of SaaS providers, who are able to invest massive amounts in software engineering, user experience and design to produce world class software and share the costs across dozens, thousands, or millions of subscribers. An individual corporation will need to invest much much more to build their own version of a software package or service than it would cost to subscribe to a well implemented service.

For the Cloud subscriber, there is another benefit to subscribing to a cloud service rather than buying or building your own internal version - the switch from a large up-front capital investment to a monthly operating cost. This On Demand pricing model enables you to start small and incrementally increase your investment as users move to the service or your customer base grows, rather than requiring you to over-provision for your hoped-for 1 or 2 year projections. This subscription model also ensures you have continued flexibility to evaluate whether the deployed solution is meeting the needs of your customers - the cost to increase the service, switch to a different solution, or turn off the service completely if the business model is not working no longer hinges on huge up-front investments that must be managed and written off - you simply stop paying the monthly subscription and move on.

As you can see, using Cloud based services provide very real, concrete business value by enabling more Agility, more Capability, and better Economics. There are also numerous other intangible benefits to moving to Cloud services. You need to automate your deployment, you can test more easily, you need to define real service boundaries and interfaces, you can prototype integration and new services quickly, you have access to a wider range of technology options, and on and on.

That's Why you SHOULD use the Cloud!!