Integrate Button: Continuous Integration, Deployment, Testing, Feedback

There's a big difference between automated and automatic

In our work, we interact with people from many different teams: developers, QA/testers, operations, etc.. In some organizations, operations may be further segmented into SCM, DBAs, Security, and so on. And, in other cases, some functions might be combined. But, if I had a nickel for every time I asked someone whether some process was automated and they incorrectly told me yes, well...you know the rest. This is why definitions matter sometimes. It's the difference between automated and automatic.

What they mean is:

• We wrote some scripts for that
• You have to log into each of the machines separately to run the processes
• Your environment/machine has to be configured just right to run the scripts
• You must manually execute each of the scripts separately
• A human must be present to run the scripts
• It's "automated"
  

What I mean is:

• The scripts are run headless (i.e. from a single command)
• The scripts are checked into the version-control repository
• The scripts have been integrated into the build and/or deployment scripts -- and all processes can be run from a single command
• That the automation is automatic

The difference between the approach they think is automated and what I think is automated is vast. But, it's not about using the right definitions; it's about having a common understanding of the goals you want to attain. The "manually automated" approach leads to significant bottlenecks (real scenario: 100 projects and 10 target environments per project means that your development is stopped every single time someone must manually perform these repetitive activities).

So, the next time someone tells you "(their) stuff is automated", you might want to ask a few questions because each manual task slows down the entire process which costs time, money and frustration.

The Myth of Build Management Servers

From time to time, I come across some who think that Build Management servers automate their delivery processes. There’s a big difference between executing automated scripts (what build management servers do) and writing the scripts that perform the automation (what experienced engineers do). For instance, I’ll hear people refer to the build management server as the tool that’s actually assembling the database or configuring the datasource that connects the application container to the database. The problem I have with this is that it’s not true (Or, if a particular process is "automated" by the server, your build architecture is not sustainable as I'll explain later). It's perpetuated by statements like:

"All three are big, expensive propositions that deliver considerable automation to build processes, as well as the early warning alerts implied by the term continuous in CI." (emphasis provided by me)

Often, it’s not the fault of the vendors selling their tools, but with ignorant managers or developers who vastly overstate or misunderstand the capabilities of the tool. The ironic part is that there’s no need to overstate the capabilities, because there’s great benefit in using a build management server.

Let me give you some examples of some build and deployment processes to illustrate my point that it takes experienced engineers to create effective automation systems.

• Database Assembly - Assembling a database and associated data involves extracting and writing DDL and DML SQL and that a build script of some sort executes these scripts in an ordered manner. A build management server executes the scripts that perform the database assembly. There are also frameworks such as Liquibase that help perform database upgrades, but you’ll need an experienced engineer to configure these tools for a project.
• Automated Tests - Depending on what’s being tested, writing automated tests requires an experienced developer, business analyst, or tester. These tests might be written in something like JUnit, NUnit, Selenium, etc.
• Dependency Management - Defining dependencies using tools such as Ivy or Maven. This requires experienced engineers defining versions, locations, transitive dependencies for each project. Some build management servers provide this capability, but using a build management server to do dependency management will lock you into using their tool just to build your software.
• Deployment Configuration - Automating deployments may involve many processes including configuring datasources (dynamically for each target environment), configuring LDAP and SMTP, configuring other authentication information, web container attributes (timeouts, connections, etc.), memory allocation, remote deployments, and the list goes on and on for any non-trivial project.
• Generating installers - Writing a GUI installer such as with a tool like IzPack or Install Shield. Experienced engineers need to configure and generalize scripts so that the install can work in any target environment (DEV and QA or different platforms such as Linux and Windows). A build management server may call a script that creates the GUI installer.

Other build and deployment processes…

• Code Quality Analysis - Configuring code quality analysis tools to assess code coverage, coding standards, dependency analysis, cyclomatic complexity, security analysis and coding standards may involve a developer to configure the tool and to define build thresholds based on project standards.
• Documentation - Writing scripts to generate documentation such as UML diagrams, ERDs, and developer documentation.

I’ve heard something similar to the following with some teams I’ve worked with:

We’ve already got automated deployments because we just spent tens of thousands on BigBuildManagementServer 2.0

Someone was sold a “bill of goods”! Again, it could’ve been because they didn’t understand what they were getting, not because the vendor misled them. And, they might be right if they’ve got a project where they can copy a file (e.g. WAR) over to a web container without changing anything else (even disregarding the problems you’ll experience trying to do this in a multi-target environment). However, I’ve yet to work on a trivial project like this.

There’s a fundamental principle that I suggest teams I work with should follow: the same scripts should be capable of running in any target environment. And, that means you shouldn’t need to rely upon a build management server to create working software on your local workstation. Moreover, you shouldn’t need to configure myriad environment variables or other manual configuration in each of these target environments - it should be part of the automated scripts that aren’t dependent on the build management server.

When I talk to teams about this in development organizations some will miss my point and incorrectly assume that I’m bashing build management servers. Moreover, as I mentioned before, there are tons of benefits in using a build management server; however, the items I mentioned above aren’t among them. Here are some of the benefits in using a build management server:

• Schedule build to run periodically or with every change
  
• Centralize all build and deployment execution across an organization
• Enforce a process of promotion and release to different target environments (DEV, QA, etc.)
• Manage deployments across multiple target environments and platforms
• Reduce errors attributed to local machine configuration
• Control access to people who are responsible for executing deployments
• Use tools that leverage multiple machines or other optimizations to improve build performance times (horizontal scalability)
• Provide a centralized web console for viewing reports

I’m sure a build management server vendor could provide many other beneficial features for their particular tool.

On projects, we often suggest and use several of the build management servers on the market as part of an overall software delivery automation system. But, I want to dispel the notion that a build management tool is writing your tests or writing the scripts to assemble the database or configure the application container or producing the documentation. It’s experienced engineers writing these automated scripts; these scripts should be capable of running with or without a build management server. The build management server executes these scripts in a controlled manner while optimizing resources and displaying the results of the build and deployment. While I mentioned that the vendors aren’t necessarily culpable, I will say that in reading some of the build management server vendor literature, it can be very easy to walk away thinking that their tool magically performs all of the deployment processes by clicking a button, without the need for anyone to write and customize scripts, and this is sad because it’s not even necessary, as their tools already have so much to offer without giving customers/prospects the impression that people can just install their tool and voila: instant software. The implication is almost like saying that because auto manufacturing plants use robots, there’s no need for people to put the cars together. Last I checked, we’re still a ways from achieving this in auto manufacturing and probably further from this in software development.

I realize there are automated build tools (separate from build management servers) that can be used to define builds using a GUI client. This can work well for non-complex build and deploy environments in which you can make numerous assumptions and treat servers as static assets. Often, the out-of-the-box ‘templates’ provided by these tools encourage hard-coded attribute values. That’s just too much assuming for me. Moreover, these tools limit the type of flexibility we often need when developing automated software delivery systems.

The other problem is getting locked into the vendor build management platform by configuring variables in the tool, hard-coding the datasource or using dependency management capabilities. I’m sure it’s good for a vendor’s business to be locked in to their tool, but probably not for your business. Again, there are many good reasons to use a build management server, but being unable to deploy the software unless you’re running the deployment from the build management server shouldn’t be one of them.

The encouraging thing is that, as an industry, we’re continuing to improve the efficiency of our automation. For instance, there are teams releasing software every 30 minutes, in some cases. It still requires experienced engineers to put together automated systems like these, but I’m optimistic that one day getting software into production will be as simple as pushing a button, without needing experienced engineers to write the automation. We’re just not there yet.

Email this entry