The New CMOS - Cloud Mobile Open-source Social

CMOS is a well-known acronym in chip manufacturing meaning "Complementary Metal-Oxide-Semiconductor". According to Wikipedia:

"CMOS" refers to both a particular style of digital circuitry design, and the family of processes used to implement that circuitry on integrated circuits (chips). CMOS circuitry dissipates less power than logic families with resistive loads. Since this advantage has increased and grown more important, CMOS processes and variants have come to dominate, thus the vast majority of modern integrated circuit manufacturing is on CMOS processes.

In other words - because this way of designing and building chips has distinct advantages in improving efficiency and enabling new features, it has become the dominant approach to building chips. Any mainstream (non-experimental) chip manufacturer has settled on this process because of the inherent benefits.

There is a new CMOS taking over the software development industry - "Cloud Mobile Open-source Social." New applications, whether they are commercial software products, internal enterprise applications, or B2C web applications (think "Web 2.0", Linked Data and related) are - by default - leveraging the combination of Cloud, Mobile, Open-source, and Social technologies to enable rich feature, scalable, always on applications that are developed faster and cheaper.

Users expect 24/7 access. Consumers expect to be able to share and tag and link content. Readers expect to be able to access articles and applications on any computer, any device, anywhere. This is a fundamentally different mindset than "traditional" application development and traditional publishing models. It requires new ways of architecting applications and new ways of thinking about data and content. It's a heck of a good place to work!

Ten Steps to Agile Software Development Process Improvement

In a previous blog entry, I mentioned 10 Steps to Improve Software Development Process. If you read these and pause for a minute, you'll notice that many of these are actually principles of Agile development. I want to expand on a few of these thoughts here.

1. Focus on the top 20% of features:
This is one of the primary drivers of value in Agile custom application developmentpractices. By prioritizing and rank-ordering every item in the feature request backlogs, only the most important ones are developed. By focusing on these Top 20%, you can often satisfy 80% of what end users want, and they can start using the system sooner, to add to the profitability of your company. (If the most important 20% of the features do not add to your company profitability, you should probably cancel the project!)

2. Break things up into smaller projects:
Big projects turn in to huge projects. And miss deadlines. And run over budget. Reading the Standish Group CHAOS report figures on failed IT projects always makes me wonder why more people don't follow the simple advice to "not bite off more than you can chew"?

Again, the Agile concept of smaller, more frequent releases echoes through this item. Getting a working system in to the hands of users is always a good thing - that's why you are implementing the system to begin with! Giving an important and useful portion of functionality early is even better. This has many organization benefits - from psychological ones like proving that the overall program can work and the enthusiasm from a successful launch, to risk mitigation benefits like the ability to redirect spending on an investment after the first release if priorities change, to the practical one that it is a lot easier to notice a project in trouble if a specific release is over budget or late than if fuzzy 'milestones' are being missed.

5. Obtain user feedback
During the implementation process, keep the end users constantly in the picture. Show them early versions of the system - for instance at each Sprint Demonstration or at least with small, frequent releases. Let them give you feedback, and above all, let them change their minds without being punished. Trust your end users; they know what works and what does not - and they are the ones who are going to use the system every day! When they see working features, they may be able to better prioritize other changes or features to be able to complete an important business process or simplify many steps.

7. Mistakes are a way of learning
Remove the blame culture. Let people make mistakes quickly - failing early is much better than missing a delivery date. These so-called mistakes are part of the overall discovery process and will help lead and evolve to the eventual solution. If you blame people for mistakes (picking the wrong feature for a Sprint, not seeing a bug, very bad color choices) they will react to the blame and change their own behavior. Rather than being an active participant in making the project a success, they will become "followers", just doing what they are told - no way to get blamed there - and punching the clock. The good people will hate the blame culture enough to look for work somewhere better.

10. Something has to give in the Iron Pyramid (Quality, Time, Cost, Features)
The old Iron Triangle, which I've always thought should be an Iron Pyramid with Quality as an explicit dimension, still holds true today. In an agile development process you try to "bake in" high Quality by using unit tests, refactoring, engaged people, and frequent review processes. You fix the Time or at least the time cycles - every two weeks you have a Sprint release; and the Cost is essentially fixed based on the size and members of the team. The Features dimension is what gives - and that is where the prioritization comes in. By putting the most important features first, you complete as many in each Sprint as you can and know that you are achieving the most important features at any given time.

Over a longer time horizon, say a Release of 5 to 6 Sprints, you can adjust the Time and Cost dimensions, by letting the process run for more Sprints or deciding that enough Features are ready to stop this project.

I focus on the Iron Pyramid because we all know the truth. If you push people hard enough, they will relax some of the quality checks, and they will get one extra feature in. But you have lowered the quality level - maybe not enough to notice today, but you will pay for it in the future. Whether it is the performance testing that is skipped, leading to a slow web application, or "smelly code" that costs more and more to maintain over time you pay for the lapse in quality.

Data Visualization and Web 2.0

I love data visualization techniques. From my early days as a operations data analyst and all of my software development career, finding patterns in data and finding an easy way to convey the pattern through a graph or other visualization has always been fun. Working on custom application development projects that provided a picture of how the business was doing, where customers were spending, etc is fun. Now working with our Business Information Services clients to help create innovative approaches to information discovery and data analysis is fun. It really is true that often "a picture is worth 1,000 words."

I vividly recall a stubborn memory leak my team had been trying to track down for several weeks. This was a long time ago, in the days of VB6 COM dll's running inside ASP web pages, and we were pretty sure our code was not leaking. The team had found memory leaks before, and tracked every single one of them down to circular references in our COM object model so that the automatic release never occurred. Historically, it had been easy to find a leak by running a simple load script, executing each page thousands of time in isolation and watch to see which page shows the memory leak. But not this time. We had run the load tests several times and never found the leak. We scanned the code thoroughly. We added as many "set obj = nothing" safety lines as we could. But still the production web servers kept leaking memory, and we were forced to move the automatic restarts of the servers from weekly to daily and hope our band-aid would hold.

One day, I had some down time and decided to see if I could find a correlation between the memory used and what pages were invoked on the production system. An hour or two later, I had pulled all the IIS log files, gotten dumps of memory traces from the systems team and started my analysis. A bit of awk, grep, Access, and other quick and dirty processes later to pull out the data I wanted, adjust for timezones, aggregate hits to cumulative 15 minute buckets and otherwise line up the datasets and I was ready to plot the data.

Instantly, the answer of where the leak was was obvious. The two lines, cumulative hits to a particular URL and memory in use were nearly on top of each other. The correlation jumped out, completely overwhelming the noise of other URL's, pages, etc. This is the power of a good visualization. (Of course, it turned out that the leak was coming from a web services API proxy URL, not a page in the website that everyone had focused on! Since the proxy was not 'in' the website it had been ignored for weeks as the team hunted for the answer.)

Recently, some colleagues and i were discussing what areas Alliance Global Services provides solutions to clients in. This is a pretty broad topic, and we talked about the types of industries we serve (including our focus on Business Information Services), the geographies we serve (mostly North East US, from about Virginia to Boston), the types of services we provide (Custom Software Development,Application Architecture Analysis). And we talked about the easiest way to visualize our coverage areas.

Well, today I had a little downtime before the holidays. So I took a list of our client locations used some simple geocoding tools, and put together two quick samples of mapping in the Web 2.0 world - one using a Yahoo! map through batchgeocode.com and the other using the Google visualization API.

Batchgeocode.com made it very easy to process the first set of data and create a map but then you were stopped. Google was a different story - getting the map running required coding, but then I had full control. To see the first map, visit this blog on Alliance Global Services.

Obviously it's not perfect, but lots of fun for a quick afternoon's work!

10 Steps To Software Development Process Improvement

I spend a lot of time workingwith customers and talking with colleagues about how to build better software,faster, and cheaper. Yes, "better, faster, cheaper" is hard,maybe impossible, but boy is it an enticing goal in custom application development. Ourdiscussions often look at various technologies that promise improvements -Silver Bullets. Every few months someone launches a new technology withpromises of huge efficiency gains, and a lot of these do offer productivitygains, after an appropriate learning curve - of course, first the productivityactually drops, as the team learns the new technology, spends time oninfrastructure setup, etc. The better, smaller options have very easylearning curves and quickly add to overall productivity, but there is always some initial drop before the the benefits begin to accrue.

We are technologists, and weare always looking for interesting new frameworks, tools, and ideas to improvethe technology we use to deliver our projects. Whether it is a newautomated static analysis tool, like CAST Software to help with application architecture analysis or abetter IDE for Java application development (likeIntelliJ) or new testing tools and frameworks to make our software testing outsourcing serviceseven better we often get excited about a "shiny new toy" that we canuse to be more productive and deliver better software every day.

Of course, these SilverBullets usually turn out the same way - a good tool, with some definitebenefit, but definitely not a game changing advance that will provide aorder-of-magnitude improvement in the quality of software developed. Themost effective way to improve the the overall software delivered - making itbetter, faster, and cheaper - is to improve the SoftwareDevelopment Process. When we sit down to work with clients, weoften talk about a list such as the following as a starting point for ensuringsuccess -- measured in terms of Business Value in the delivered software -- forour projects.

1. Focuson the top 20% of features

2. Breakthings up into smaller projects

3. Noneed for minuscule details

4. Letthe system evolve

5. Obtainuser feedback

6. Empoweryour users

7. Mistakesare a way of learning

8. Lesspeople in meetings

9. Smallerteams

10. Somethinghas to give in the Iron Pyramid (Quality, Time, Cost, Features)

It's no surprise that this list closely embodies many of the principles that the Agile development movement espouses. Agile development concepts are rooted in basic idea of trying to produce more value for an organization. In software development terms this means better software for the same cost (or faster, or cheaper, or all 3!). By having the business customer prioritize the features in the backlog, showing incremental progress each week or two, and being focused on measuring Running, Tested, Features the investment in software development is always focused on providing the best value for the business customers for the given investment.

In future posts I'll expand each of these topics further.

IntelliJ Open Source -- Good for Java Application Development

I was excited to see the announcement that JetBrains is making the core IntelliJ platform available under an Apache 2.0 Open Source license. IntelliJ is a great IDE, and one of the main ones we use for Java application development. Having a community edition that makes the platform available more widely is great and hopefully will enable more developers to take advantage of the great quality checking and productivity tools IntelliJ has.

During a recent discussion with a client, we brainstormed ways to customize or add on to the open source IntelliJ platform to improve the software development process and improve the quality of the code produced. We quickly came up with a number of ideas for custom facets - templates for common types of classes, custom analysis rules, and specific refactorings for best practices in this particular code base. All good ideas that could make the web application development faster and result in higher quality. A good win!

Application Risk Assessment, Code Quality, and C#

I have the opportunity to review and analyze a lot of different application code bases, across a number of difference technology stacks. Some of these are custom software applications that Alliance is building or maintaining for our clients. Some of these are open source packages we are using in our work. Others are analyzed for our clients as part of our Application Assessment and software testing outsourcing solution.

One of the things that continues to surprise me is the wide variance in code quality. After seeing so many different applications, created by so many different development teams of different skill levels Iknow I shouldn't be surprised at some of the things that I see. Every now and then a particular issue jumps out as so obvious, why didn't the original developers write the code better?

One recent example involves a Application Architecture Analysiswe did for a government client using the CAST Application Intelligence tool. The application was a medium sized .NET web application connecting to an Oracle database. The application was a port of an existing Powerbuilder desktop app, and exhibited a lot of classic problems with simplistic porting. Each screen in the desktop application was directly mapped to a web screen, without regard for whether the type of navigation and state management - not to mention browser round trips - made sense in a web application.

At the start of the engagement, the client identified that they had concerns around the correct handling of database connections. CAST is a great tool for finding problems like this in custom .NET applications, as the .NET analyzer is able to identify specific methods in which a database connection is opened but not closed. It's certainly much faster and more accurate to browse an easy-to-use Dashboard pointing to the exact 18 locations in the 400,000 Lines of Code (400 kLOC) that should be checked rather than trying to manually search ASP.NET pages and code behind files, dozens of VS.NET projects, and hundreds of C# classes, to find the needle in this haystack.

After identifying and fixing the specific problems, the database connection leaks were fixed, and the application was able to proceed through user testing. A perfect quick win for Application Risk Assessment & Code Quality analysis.

One of the surprising (yeah, I know, I shouldn't be surprised!) things in this case was that in many parts of the code base the developers had taken advantage of the "using" keyword in C# to automatically perform resource management of the database connection. This is a much simpler approach than trying to enforce correct usage of finally{} blocks for resource clean up and has been available for several versions and many years. Yet many developers are not aware of it, and do not use it regularly! This is a simple best practice that would enable higher quality code with less effort and fewer defects.

And that means higher productivity. Sounds like an obvious win to me!

These are the types of best practices we teach to our teams as part of our RightWare Software Development Process.

Automation in Software Development

There are many discussions about productivity and ways to increase quality in software development. There is no single magic bullet, but by far the most important overall technique I've ever seen is to aggressively automate the software development process and overall lifecycle.

This is a broad topic and covers many different specific areas such as automated unit testing that can also serve as the foundation for automated regression testing. Automation includes the build & deployment process. Automation includes functional testing and acceptance testing. It includes monitoring and error alerting. It includes code quality analysis and compliance checks. It includes providing self-test harnesses to prove an environment is configured correctly. It can also include automatically generating code or portions of an application as part of the software development process.

Without fail, when we look across our custom application development projects the ones that have implemented a significant amount of automation are able to deliver more value (in terms of running, tested features) per time or per unit cost than those that have skimped on automation. This holds across technologies and types of systems and holds across various team sizes.

Why does automation play such a big role in improving Application Quality and Productivity? It's fairly simple and relates to key ideas in Lean Development and other "manufacturing" optimization strategies. Automation enables the whole software development team (including the development engineers, testers, operations, and business sponsors) to focus on adding specific value and avoid waste by:

• Eliminating repetitive, low-value tasks - freeing time for knowledge creating work
• Lowering QA and bug fixing effort by finding errors sooner through executing test suites more frequently
• Enabling more complete test coverage by executing test suites automatically not through brute force
• Enabling faster, moge agile, development by providing a robust safety net to catch problems sooner
• Eliminating wasted time chasing configuration problems by reliably producing builds and deploying to all environments consistently
• Enabling new team members to contribute value sooner by speeding the creation of new environments and providing a framework to show how the system works
• Producing higher quality and lower maintenance systems by automating redundant code or module development
• Converting time spent on routine monitoring to value-add investigations in to problems
• Quickly alerting operations and development support teams to problems with running systems

Teams that focus on keeping the unit tests Green, look at their code coverage metrics, and know the current production system performance characteristics can spot problems before customers do. Teams that continually look for ways to streamline their tasks and ensure that any common process can be done through one command or one click have fewer problems and spend much more of their time building new features rather than solving the same silly problems over and over. Teams that ensure the basic smoke testing can be run by every engineer every day know that many fewer bugs will creep in to the testing environments and less time will be wasted with back and forth rework. These are the traits of successful teams that provide more value to the business and are truly successful!