10 Steps To Software Development Process Improvement

I spend a lot of time workingwith customers and talking with colleagues about how to build better software,faster, and cheaper. Yes, "better, faster, cheaper" is hard,maybe impossible, but boy is it an enticing goal in custom application development. Ourdiscussions often look at various technologies that promise improvements -Silver Bullets. Every few months someone launches a new technology withpromises of huge efficiency gains, and a lot of these do offer productivitygains, after an appropriate learning curve - of course, first the productivityactually drops, as the team learns the new technology, spends time oninfrastructure setup, etc. The better, smaller options have very easylearning curves and quickly add to overall productivity, but there is always some initial drop before the the benefits begin to accrue.

We are technologists, and weare always looking for interesting new frameworks, tools, and ideas to improvethe technology we use to deliver our projects. Whether it is a newautomated static analysis tool, like CAST Software to help with application architecture analysis or abetter IDE for Java application development (likeIntelliJ) or new testing tools and frameworks to make our software testing outsourcing serviceseven better we often get excited about a "shiny new toy" that we canuse to be more productive and deliver better software every day.

Of course, these SilverBullets usually turn out the same way - a good tool, with some definitebenefit, but definitely not a game changing advance that will provide aorder-of-magnitude improvement in the quality of software developed. Themost effective way to improve the the overall software delivered - making itbetter, faster, and cheaper - is to improve the SoftwareDevelopment Process. When we sit down to work with clients, weoften talk about a list such as the following as a starting point for ensuringsuccess -- measured in terms of Business Value in the delivered software -- forour projects.

1. Focuson the top 20% of features

2. Breakthings up into smaller projects

3. Noneed for minuscule details

4. Letthe system evolve

5. Obtainuser feedback

6. Empoweryour users

7. Mistakesare a way of learning

8. Lesspeople in meetings

9. Smallerteams

10. Somethinghas to give in the Iron Pyramid (Quality, Time, Cost, Features)

It's no surprise that this list closely embodies many of the principles that the Agile development movement espouses. Agile development concepts are rooted in basic idea of trying to produce more value for an organization. In software development terms this means better software for the same cost (or faster, or cheaper, or all 3!). By having the business customer prioritize the features in the backlog, showing incremental progress each week or two, and being focused on measuring Running, Tested, Features the investment in software development is always focused on providing the best value for the business customers for the given investment.

In future posts I'll expand each of these topics further.

IntelliJ Open Source -- Good for Java Application Development

I was excited to see the announcement that JetBrains is making the core IntelliJ platform available under an Apache 2.0 Open Source license. IntelliJ is a great IDE, and one of the main ones we use for Java application development. Having a community edition that makes the platform available more widely is great and hopefully will enable more developers to take advantage of the great quality checking and productivity tools IntelliJ has.

During a recent discussion with a client, we brainstormed ways to customize or add on to the open source IntelliJ platform to improve the software development process and improve the quality of the code produced. We quickly came up with a number of ideas for custom facets - templates for common types of classes, custom analysis rules, and specific refactorings for best practices in this particular code base. All good ideas that could make the web application development faster and result in higher quality. A good win!

Application Risk Assessment, Code Quality, and C#

I have the opportunity to review and analyze a lot of different application code bases, across a number of difference technology stacks. Some of these are custom software applications that Alliance is building or maintaining for our clients. Some of these are open source packages we are using in our work. Others are analyzed for our clients as part of our Application Assessment and software testing outsourcing solution.

One of the things that continues to surprise me is the wide variance in code quality. After seeing so many different applications, created by so many different development teams of different skill levels Iknow I shouldn't be surprised at some of the things that I see. Every now and then a particular issue jumps out as so obvious, why didn't the original developers write the code better?

One recent example involves a Application Architecture Analysiswe did for a government client using the CAST Application Intelligence tool. The application was a medium sized .NET web application connecting to an Oracle database. The application was a port of an existing Powerbuilder desktop app, and exhibited a lot of classic problems with simplistic porting. Each screen in the desktop application was directly mapped to a web screen, without regard for whether the type of navigation and state management - not to mention browser round trips - made sense in a web application.

At the start of the engagement, the client identified that they had concerns around the correct handling of database connections. CAST is a great tool for finding problems like this in custom .NET applications, as the .NET analyzer is able to identify specific methods in which a database connection is opened but not closed. It's certainly much faster and more accurate to browse an easy-to-use Dashboard pointing to the exact 18 locations in the 400,000 Lines of Code (400 kLOC) that should be checked rather than trying to manually search ASP.NET pages and code behind files, dozens of VS.NET projects, and hundreds of C# classes, to find the needle in this haystack.

After identifying and fixing the specific problems, the database connection leaks were fixed, and the application was able to proceed through user testing. A perfect quick win for Application Risk Assessment & Code Quality analysis.

One of the surprising (yeah, I know, I shouldn't be surprised!) things in this case was that in many parts of the code base the developers had taken advantage of the "using" keyword in C# to automatically perform resource management of the database connection. This is a much simpler approach than trying to enforce correct usage of finally{} blocks for resource clean up and has been available for several versions and many years. Yet many developers are not aware of it, and do not use it regularly! This is a simple best practice that would enable higher quality code with less effort and fewer defects.

And that means higher productivity. Sounds like an obvious win to me!

These are the types of best practices we teach to our teams as part of our RightWare Software Development Process.

Automation in Software Development

There are many discussions about productivity and ways to increase quality in software development. There is no single magic bullet, but by far the most important overall technique I've ever seen is to aggressively automate the software development process and overall lifecycle.

This is a broad topic and covers many different specific areas such as automated unit testing that can also serve as the foundation for automated regression testing. Automation includes the build & deployment process. Automation includes functional testing and acceptance testing. It includes monitoring and error alerting. It includes code quality analysis and compliance checks. It includes providing self-test harnesses to prove an environment is configured correctly. It can also include automatically generating code or portions of an application as part of the software development process.

Without fail, when we look across our custom application development projects the ones that have implemented a significant amount of automation are able to deliver more value (in terms of running, tested features) per time or per unit cost than those that have skimped on automation. This holds across technologies and types of systems and holds across various team sizes.

Why does automation play such a big role in improving Application Quality and Productivity? It's fairly simple and relates to key ideas in Lean Development and other "manufacturing" optimization strategies. Automation enables the whole software development team (including the development engineers, testers, operations, and business sponsors) to focus on adding specific value and avoid waste by:

• Eliminating repetitive, low-value tasks - freeing time for knowledge creating work
• Lowering QA and bug fixing effort by finding errors sooner through executing test suites more frequently
• Enabling more complete test coverage by executing test suites automatically not through brute force
• Enabling faster, moge agile, development by providing a robust safety net to catch problems sooner
• Eliminating wasted time chasing configuration problems by reliably producing builds and deploying to all environments consistently
• Enabling new team members to contribute value sooner by speeding the creation of new environments and providing a framework to show how the system works
• Producing higher quality and lower maintenance systems by automating redundant code or module development
• Converting time spent on routine monitoring to value-add investigations in to problems
• Quickly alerting operations and development support teams to problems with running systems

Teams that focus on keeping the unit tests Green, look at their code coverage metrics, and know the current production system performance characteristics can spot problems before customers do. Teams that continually look for ways to streamline their tasks and ensure that any common process can be done through one command or one click have fewer problems and spend much more of their time building new features rather than solving the same silly problems over and over. Teams that ensure the basic smoke testing can be run by every engineer every day know that many fewer bugs will creep in to the testing environments and less time will be wasted with back and forth rework. These are the traits of successful teams that provide more value to the business and are truly successful!

Expert Knowledge Model and Custom Application Development

I recently read Andy Hunt's book "Pragmatic Thinking and Learning." It's a very interesting book combining ideas about cognitive science, management, software development, and personal development. (It's also well written and fun to read)

The 2nd chapter discusses the Dreyfus skill model, and the "Journey from Novice to Expert" along five stages of skill development. This is relevant for any skill, but especially for technical skills requiring years of practice to achieve a suitable level of mastery. Certainly different people are capable of learning at different rates, and some people are able to actively learn and apply new concepts while others seem to repeat the exercise but not learn the concept (the classic interviewing criteria of actually having 10-years of experience, rather than 1-year of experience 10 times over), but overall when looking at a community or organization (or team) recognizing the different skill levels and capabilities associated with them is critical.

For instance, many agile teams have individual developers designing the implementation for a specific user story (or more accurately, a task as part of a user story). By recognizing that a particular developer may be a Novice and a different team member may be Proficient the team can change the dynamic around the design effort when these two are paired together. The Proficient team member can act as a mentor for the Novice, more carefully reviewing and enhancing any designs the Novice developer creates and fully explaining any designs the Proficient developer creates as a teaching exercise. This enables the peer review process to work as both a technical review exercise and as a learning process. Similarly, the team may recognize that it does not make sense for a Novice developer to peer review an architecture that is created by the Expert member of the team - it would be more appropriate for someone who is Competent or Proficient to peer review, as they will more easily understand the design and be able to more easily offer suggestions to improve the design rather than feeling overwhelmed or lost when reviewing the design.

Additionally, when establishing the engineering practices for a custom application development effort, a team of all Novices or even mostly Advanced Beginners should not be expected to realize the best approach to continuous integration, automated unit testing, and test coverage. A more knowledgeable mentor should help the team implement the best practices, teaching as they go, but insisting on the appropriate integration and automation techniques to ensure high quality.

And of course, the skill model applies equally to the technical and business domain aspects. It is important for both the individual developers, managers, testers, and architects to realize that although they might be Expert at their technical craft, they may be Novices in the business domain of their current application. Once this realization occurs, the team members should seek out additional information on the business domain - through web sites, books, the Product Owner or business sponsor, or other more knowledgeable members of their organization. This type of self realization that you "know what you don't know" is a key step towards ensuring you are always delivering high quality, useful,valuable software.