Integrate Button: Deliver Software at the Click of a Button

Creating Software Assets

In my last blog entry, I discussed Software Quality for Acquisition Valuation and identified that acquiring companies can negotiate lower valuations for software companies who do not follow early testing practices.

The positive corollary to how these practices affect acquisition valuation is that early testing practices have material impact of the value of software assets. In fact, early software quality practices add critical components that dramatically increase the value of software assets for an organization.

In software engineering, we think mostly about problems and solutions, leaving the questions of business value to our fellow executives. A simple illustration of what developers tend to do is shown below:

In traditional manufacturing, the business value of “process” is well understood. Even lay people can appreciate that the value of an automobile manufacturer is not in the ability to produce a single instance of a car. It is in the reliable and repeatable ability to create high-quality cars at a predictable cost. The actual business value of source code is related to its ability to reliably produce quality executable programs as requirements evolve. A primary factor in valuing source code as an asset is its independence from the developer who authored it. Early testing practices have appreciable positive impact on the quality of executable programs. But, much more than that, they add artifacts to the source repository, which help build independence from authorship. Using continuous integration practices and ensuring that code complexity, test coverage, source dependency and other metrics are maintained at reasonable levels helps increase the value of the source code assets. Many software organizations discuss strategies that include a sale of their organization to a larger entity. Shouldn’t the engineering team consider the impact of their process decisions on the evolving value of its source code?

Originally authored by Stelligent at testearly.com

Software Quality for Acquisition Valuation

A client recently acquired a company with an embedded software product that, according to all customer comments and due diligence reports, was of high quality. It was the plan of our client to integrate the acquired product with its larger enterprise application. Several months into the project, an esoteric measurement of software quality raised its head: Change Tolerance.

Change Tolerance describes the ability of software to survive change. Rather than an objective metric, it is a subjective measurement that is rooted in a complicated combination of code complexity, code dependency, test coverage, build processes and automated testing frameworks.

Other composite measurements, such as Defect Resistance and Quality Risk also require specific knowledge regarding not only the code that is currently compiling, but also the path and processes used to bring that code to fruition.

Rarely do the history, processes and measured code quality result in an acquirer walking away from a deal...too many other factors influence a decision to acquire. But objective and knowledgeable assessment of the true quality of existing software provides excellent negotiating leverage.

We worked with our client to eliminate the issues that affected the Change Tolerance of the product. But those costs could have – and should have – been passed onto the seller, rather than absorbed by the acquirer.

A unique offering we provide to organizations who are pursuing acquisitions is a historic quality assessment; where we assess code and process quality both in present terms and as a trend over time in the organization. These assessments can show the impact of the loss of a “star” performer, the drop in quality due to aggressive release cycles, negative impacts from change of technology or – for the best companies – a demonstrated pattern of consistent and well-applied quality practices.

Other subjective measures such as accounting records, corporate values, quality of management team, competition and intellectual property can affect valuation. But, in the case of software technology acquisitions, software quality is unique in its ability to be supported by meaningful metrics and its direct correlation to the value of the acquired assets.

Engaging in a software acquisition without an outside, objective assessment of core product quality can leave millions on the table in terms of unaddressed valuation concerns.

Originally authored by Stelligent at testearly.com

Quality Concept: Defect Resistant Code

For most people, quality concepts such as code complexity, test coverage, and source duplication are easy to comprehend. But the idea of making code “defect resistant” might not be implicitly clear.

Perhaps it is because the phrase “defect resistant code” really addresses more than just the code itself: it is a collection of practices, processes, coding metrics and technology infrastructure. These are all combined to make it difficult for a developer to introduce a defect into a code base.

At Stelligent, we often talk about the “lifetime” of a defect. All software professionals understand that defects will be introduced during development. The question we like to ask is, “How long will that defect live in your source management system?”

Effective organizations find ways to shorten that lifetime of a defect by using a combination of developer testing, automated analysis frameworks (that extract and report on factors such as code complexity and test coverage) and an automated testing and build framework (that builds and exercises unit tests with every material source code change).

The result is that code containing a defect will trigger a series of events that immediately communicate to the developer (and team) that a defect has been introduced.

Maybe it uses a little literary license to declare the “code” itself to be defect resistant. But the result of this collection of practices, coding techniques and automated processes does have the effect of making it difficult to add new defects to the existing code base.

All organizations that develop software should view their source code as an asset that should be protected. A great way to protect code assets is to prevent defects from impacting their stability and tolerance to change.

Organizations that make their code “defect resistant” improve their ability to efficiently deliver reliable, effective and high-quality software solutions.

Originally authored by Stelligent at testearly.com

Quality is Now Development Job One

In his encouraging article, "Quality is Now Development Job One," technology editor Peter Coffee reports the highlights of a roundtable discussion organized by eWEEK Labs about the importance of quality assurance, over time deadlines, throughout the software development life cycle. Shipping today, with unknown defects, just to meet a release date is only leading to harm for the company, the consumers and its partners.

The article describes that fixing defects consumes 80% of software development costs. Why not monitor the quality of software now instead of spending more money, having a poorer system and not delivering what was proposed? The market lifecycle and the presence of Open Source vendors dictate that quality has to be high to survive.

Segue Software's Ian MacLeod expressed his wish for extensive industry adoption of tool integration, noting that "Quality is all the elements of the ecosystem--requirements, development and test management, defect management, monitoring, and diagnostics across the deployment line and into operations."

In other words, the development process needs to be improved – more QA, whether it is with tools and/or services, would result in less expensive testing efforts, and less expense throughout the whole process.

Originally authored by Stelligent at testearly.com

Early Software Quality Absorbs Costs of Compliance

Compliance is a common theme heard in nearly every boardroom today. Determining whether business processes comply with the Health Insurance Portability and Accountability Act, Graham-Leach-Bliley, Sarbanes-Oxley or other regulatory demands imposes enormous cost.

For years, the impact of compliance on information technology has been projected. In early 2005, as the Sarbox Section 404 deadlines began to kick in, Qualcomm reported 67,000 man hours and over $7 million to achieve compliance.

Bernie Donnelly, vice-president of quality assurance and control at the Philadelphia Stock Exchange explains how execution of financial compliance is a function of information technology: "When [Sarbanes-Oxley] first came out, everybody was thinking about finances and the accuracy of year-end reports. But it starts to take on a life of its own. Because when you ask that one question-'Is this number accurate?' - then you have to ensure its accuracy. On the IT side, all these other things have to happen to answer that one question."

In the case of Sarbanes-Oxley, cost assessment metrics are emerging. In the InformationWeek article "Sarbox Isn't Just for the Big Guys" the accepted rule of thumb for Sarbanes-Oxley compliance is $1 million for every $1 billion in revenue.

Smaller companies face these costs, too. Often, as a supplier to a public company bound to offer transparency about dealings with vendors, smaller businesses find themselves in the ripple effect of regulatory compliance.

A PricewaterhouseCoopers survey of CEOs reports the bad news that most companies consider the benefits of compliance efforts unlikely to match these costs. Given the high adoption rate of smaller, private companies to regulatory compliance concerns and the known lack of ROI on these efforts, seasoned technology executives are looking to early software quality as an opportunity to achieve their corporate financial goals.

In the Stelligent whitepaper "The Business Case for Engineered Software Quality" the benefits of improved quality -- especially early in the development lifecycle -- are documented. Countering the explicit costs of regulatory compliance with the intrinsic benefits of improved quality yields a compelling financial argument for technology executives to contemplate.

The information technology component of compliance is ultimately about making sure the processes that collect, manipulate and maintain data all work as expected. There is great overlap between this mission and the mission of early software quality.

Early software quality has its own established business case. Regulatory compliance fails to meet ROI evaluations, but is required by government entities or strategic business pressures.

By creating a strategy that couples the two initiatives, savvy technology executives can leverage the huge savings of early quality techniques to absorb the costs of required compliance.

Originally authored by Burke Cox at testearly.com

Using a Cloud Infrastructure for Software Deployments

In the past couple of years, we've invested heavily into using cloud computing at Stelligent and for our customers. Since we create software delivery systems for our customers, it's been our goal to automate all of these processes and literally click a button to generate software that is ready to go to production. While we were able to automate many build, database assembly, deployment, testing, static analysis and other processes, the assumption was often that an instance of some sort (bare or virtual metal) had been setup. Even those that refer to the benefits of Continuous Deployment often make an assumption that the instance is available and configured. At Stelligent, we no longer make this assumption. We're automating the procurement and provisioning of the instance - including installing and configuring servers - using cloud computing.

Having spoken with several colleagues and at conferences about what we're doing, they thought it might be helpful for others to learn how we're using cloud computing resources on our internal projects and with some of our customers. While we've used various cloud providers, I will focus on Amazon Web Services (AWS) for the purpose of this blog entry.  

AWS - Elastic Compute Cloud (EC2)

EC2 provides on-demand virtual hardware instances using a pay-as-you-go model: you only pay for what you use. This makes it perfect for development and testing. We developed a web-based application that uses EC2 to provision a fully installed tech stack for Continuous Integration and target environments to deploy to our customers' non-production target environments. We are generalizing the functionality and releasing it as our CI as a Service Platform as a Service product in the coming weeks. So, using EC2, developers can use a web application to click a button and provision CI and target environments on demand. Developers are able to create clean target environments on demand when wanting to test a new feature in isolation against the project's technical stack. The application uses the Typica framework to access EC2's API.

AWS - Simple Storage Service (S3)

We're using S3 to store binary files in which we don't need to track multiple versions or for storing software distributions under development that can be recreated from source in other ways. S3 provides a great platform for storing transient files. Moreover, we're excited about utilizing AWS' recently announced S3 Reduce Redundancy Storage to lower our costs for truly transient files. We use tools such as Elasticfox for accessing files in S3. We're also beginning to use CloudFront for managing distributions for users who don't have access to our AWS account.

AWS - Elastic Block Storage (EBS)

EBS is used to attach data volumes that might need to persist after an EC2 instance is terminated. A typical use case goes something like this:

1. Test team uses CI as a Service to provision a target environment
2. Test team deploys/installs working application onto this provisioned target environment
3. Test team stores test data that want to use for future testing
4. Test team terminates the instance using CI as a Service (terminates the EC2 instance)
5. Test team provisions new instance using CI as a Service, installs software and reattaches EBS

Stelligent CI as a Service

Our CI as a Service product started to "scratch an itch" as we wanted to install and configure a complete target environment using a cloud infrastructure. We noticed we were doing basically the same thing each time: procuring the instance, selecting instance specs, selecting the operating system, configuring the OS, installing and configuring servers and tools such as Java, Ant, MySQL, Tomcat, (Hudson in some  cases), etc. So, we wrote a web application to perform the installation and configuration of this tech stack so that we could get a complete deployment environment at the click of a button. Development teams use this service on projects to setup (and terminate) these environments at the click of a button. 

In Summary...

This is meant to be an introduction to some of the cloud computing tools we are using on development projects for ourselves and our customers. It gets really interesting when considering process, usage and other factors when using a cloud infrastructure. Stay tuned for more...

Does Automation make us less communicative?

We spend a lot of time automating software delivery processes. For someone who spends a lot of time removing humans from repetitive, mundane and error-prone processes, I spend a lot of time talking with people. In particular, my time is spent in helping others understand that automation will make their jobs easier and that software/systems professionals should be focusing on novel activities, not doing the same thing over and over again and calling it work. However, inevitably, I've heard a similar question numerous times:

When you automate so many processes, do people spend less time communicating with each other?

The obvious implication is that automation leads to less communication. I cannot disagree more with this. In fact, I've found it leads to much more effective communication. Prior to designing and implementing software delivery automation systems for our customers, we typically find that organizations have instituted "walls" between teams (such as between development, QA and Systems/Operations) and no one is communicating with each other. They might be "talking", but they are not effectively communicating. Most of their conversations center around things like "but, I thought you were supposed to do that?" or "I forgot to copy the file to that location" or "will you provide me the authentication information for XYZ host?" If this is the kind of communication people envision in their theoretical "non-automated world", I'd rather talk about the latest dumb show on TV.

The reason we find that organizations communicate better after we automate their software delivery processes is because the automation becomes part of the overall software delivery system. For instance, when the developer checks a build/deployment script change into the version-control repository, it is used by QA and the Systems/Operations teams and vice-versa. No longer can team members get away with the "throw-it-over-the-wall" approach and abdicate their responsibility. Moreover, because it's all tied into CI and build management servers, they get instant notification when something does go wrong so that the fix is usually minor and made quickly. As I often like to say "Automation is for the People".

There's a big difference between automated and automatic

In our work, we interact with people from many different teams: developers, QA/testers, operations, etc.. In some organizations, operations may be further segmented into SCM, DBAs, Security, and so on. And, in other cases, some functions might be combined. But, if I had a nickel for every time I asked someone whether some process was automated and they incorrectly told me yes, well...you know the rest. This is why definitions matter sometimes. It's the difference between automated and automatic.

What they mean is:

• We wrote some scripts for that
• You have to log into each of the machines separately to run the processes
• Your environment/machine has to be configured just right to run the scripts
• You must manually execute each of the scripts separately
• A human must be present to run the scripts
• It's "automated"
  

What I mean is:

• The scripts are run headless (i.e. from a single command)
• The scripts are checked into the version-control repository
• The scripts have been integrated into the build and/or deployment scripts -- and all processes can be run from a single command
• That the automation is automatic

The difference between the approach they think is automated and what I think is automated is vast. But, it's not about using the right definitions; it's about having a common understanding of the goals you want to attain. The "manually automated" approach leads to significant bottlenecks (real scenario: 100 projects and 10 target environments per project means that your development is stopped every single time someone must manually perform these repetitive activities).

So, the next time someone tells you "(their) stuff is automated", you might want to ask a few questions because each manual task slows down the entire process which costs time, money and frustration.

Fire your best people...reward the lazy ones

Fire your “best” people…reward the “lazy” ones

In my experience, what most people consider to be their “best” people are often the root of most problems. It’s the difference between troubleshooters and troublepreventers.

Let me explain…

Bill is a troubleshooter (only). Many consider Bill to be the best employee in the company because he solves almost any problem that comes into the door. However, unbeknownst to many (especially management), Bill is the one causing most of the problems. Bill knows the purpose of every hard-coded lookup value and is the fastest to manually deploy the application to WebLogic using its web console. For instance, he knows that “14″ means a “transaction that is waiting for approval by second-line management”. However, instead of creating a human-readable constant, Bill hard codes the number “14″ in the source code. Therefore, Bill is the only one that can easily maintain these methods. In my personal experience, someone like Bill doesn’t do this for nefarious reasons, he’s just not thinking about how the next developer may need to maintain it. For every time a method is written, it’s read (and maintained) ten times (10:1). Therefore, people like Bill need to be writing source code for other humans, not the computer.

Let’s have a look at Peter…

Peter is a “lazy” developer (a troublepreventer). Because Peter is “lazy”, he hates doing the same thing twice, so he generalizes common functionality into a delegating class, creates interfaces for implementing classes to adhere to, or he writes build scripts that compile, integrate the database, runs tests and inspections, and deploys the software the same way every time. Peter finds it mind-numbing to repeat himself. Again, Peter is lazy so he doesn’t mind spending two hours automating something that normally takes 10 minutes to run manually. Why? Because, he very quickly reaps the benefits of this extra effort since every time the process is run he gets those 10 minutes back and even more importantly, he eliminates the possibility of human error (which, again, saves time and money).

The (”lazy”) troublepreventer thinks ahead. He extracts variable information into common properties files, seeks to reduce complex code, and automates repetitive, error-prone activities such as the build and deployment processes. He also ensures that others can very easily repeat what he has done. Anyone that has worked with me for even a day knows that I often sound like a broken record when I say “Is it in Subversion?” or “Have you updated the Wiki?” To me, if the knowledge is locked in your head, you are a less valuable, not more valuable, resource. Troublepreventers put their knowledge in the system, not just their heads, so that it runs the same way every time.

People like troubleshooters because they can solve a problem when a project is under pressure such as getting that emergency fix out the door immediately. Without question, you need troubleshooters on your project. However, many times the (exclusive) troubleshooters are the ones that cause the problem in the first place, be it a hard-coded value, duplication of code or a large complex method only they can understand.

Before you start thinking that I’m trying to gather together a group of slackers, I’m suggesting the complete opposite of this. I just want people to think about the total time involved, not just fixing the symptom. There are people that are both troublepreventers and troubleshooters. These are the people you want to keep and reward. However, on a given team, I’d opt for more troublepreventers than troubleshooters as they save everyone time, money and headaches.

Originally posted on testearly.com