Integrate Button: Deliver Software at the Click of a Button

Continuous Inspection Article Published

The first article in IBM developerWork’s series “Automation for the people”, written by Stelligent’s CTO, Paul Duvall, examines how to use automated software “inspectors” using tools like CheckStyle, JavaNCSS, and CPD to enhance the development process and help improve quality.

Check out “Automation for the people: Continuous Inspection” and feel free to engage in the “Improve your Java Code Quality” discussion forum for feedback, questions, or to post your own comments!

Originally authored by Stelligent at testearly.com

Ambient Orb Ant task

Steve Matyas of 5AM Solutions and I developed an Ambient Orb Ant task that changes the color of an Ambient Orb based on a build event. There are many uses for an Orb, but the purpose of this task is to change the color, animation or comment of your Orb when a build passes or fails. If you're creative, you can configure your build script to gradually change the color of the Orb based on certain conditions (e.g. you are exceeding your code duplication percentage threshold). The Ambient Orb Ant task is hosted at Quality Labs.

So, rather than just a binary representation (such as X10), there are virtually limitless combinations of colors and animations with an Ambient Orb.

The task is setup as a listener that listens for build events (pass or fail). Consequently, when you define the task, you are registering the task to listen to build events. Based on the build status, the Orb color is changed. Because it checks the build status, you probably want to execute this task as one of the first tasks in your build script - in case your build fails early. For instance, you may choose to use your init task in Ant.

Steps to using the Ambient Orb Ant task 1) Download the Ambient Orb Ant task 2) Place the extracted jar in your Ant class path 3) Define an Ant target and configure as demonstrated below:

In this example, if the build passes, your Orb's color will change to green (12) or red (0) if it fails. Pretty cool, huh? In the example, you'll need to change the deviceId attribute. This attribute is the unique identifier located on the bottom of your Orb. Notice the use of the plus (+) sign to indicate spaces in the commentPass and commentFail attributes. Go here for additional documentation. A short movie demonstrates the installation, configuration and use of this task.

The project is open source and free to download and use. If you've got ideas for improving it, comment here or feel free to join the Quality Labs project.

Originally authored by Stelligent at testearly.com

Database Integration in your Build scripts

Imagine this scenario...

• Jayne (Developer): “I'm experiencing a lot of problems testing on v1.3.1.b1 of the database using build 418”
• Mike (DBA): “When using build 418, you should use v1.3.1.b2, but I also need to make a few changes to it first”
• Jayne: “I just spent three hours for no reason”
• Mike: “You should have asked me first”

Back in the early days of my career, I worked on quite a few projects that had a relatively efficient process for developing software. However, when it came to the database, we seemed to throw out all good judgment and retreat to the days without configuration management by using a "throw it over the wall" approach to managing the changes in the database. There was an understanding that the DBAs were doing “their thing” and we (the developers) were doing “our thing”. In those days, as developers, it was seldom that we could recreate the database on our machine using Data Definition Language (DDL) and Data Manipulation Language (DML) scripts because they either

1. Did not create the DDL/DML scripts or
2. Did not check them into the SCM repository. Instead, we would be at the whim of the DBA if we needed any kind of change to the database.

The scenario above is based on my own experiences (the names have been changed to protect the no so innocent).

Fortunately, some of us have awoken to what we could have been doing all along: integrating our DDL and DML into our build process to ensure we are working off the same schema and test data as our fellow developers and DBAs.

To effectively integrate your database definition and manipulation into the build process, you must establish the following: DDL scripts, DML scripts, an SCM tool, a local database, and build script(s). Each is described in more detail below.

DDL DDL is just a fancy acronym for SQL that creates your database schema. Typically, you will see code like this in a DDL script: CREATE TABLE beer(name VARCHAR(50), brewer VARCHAR(50), date_received DATE); DML DML is another one of them highfalutin acronyms for SQL that means inserting, updating, and removing (etc.) data in your database. You'll find this type of code in your DML:

INSERT INTO beer(name, brewer, date_received) VALUES ('Sam Adams Lager','Boston Beer Company','2005-12-09');

Just remember, we're talking about test data here; you're not going to be inserting millions of records into your local database. This is for another time and place.

SCM Tool You need to manage your DDL and DML just like you would your source code, so you'll need a source code management tool such as Subversion, ClearCase, Perforce or whatever you're using at your shop. You will check in the DDL and DML scripts that will recreate your database schema and insert all of the test data.

Local Database You need a local version of your database running so that your can run the DDL and DML scripts. If you are running this database on your developer workstation, you will probably want a lightweight version of your database so that you don't degrade your machine's performance. If you're using Oracle, look into Oracle Express or McKoi as lightweight options. If you're using SQL Server, then SQL Server Express is a viable option. If you're using MySQL, you're in luck, as you can use MySQL locally.

Build Scripts Now that you've got your DDL/DML scripts, a local database, and you are managing your database scripts in your SCM tool, you're ready for the final step: making your database definition/manipulation as a part of your build process. In makes no difference whether you are using Ant, NAnt, make or other build tool just as long as the script is run as part of a standard process of developing and building your software. An example in Ant and MySQL is listed below.

<target name="db:create-database" description="Create DB and Insert data">
<echo>Creating database...</echo>
<sql driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost:3306/brewery" userid="sam" password="@d@ms" classpathref="db.lib.path" delimiter=";">
<fileset file="${db.dir}/create-database.sql">

</fileset>
</fileset>

</sql>

</target> By making it a part of your build script, you can run it often as part of a continuous integration process. You can add more advanced features such as incorporating database migration into the process to ensure that changes can easily be applied to an existing database. On the projects I have used this process, it encourages a more collaborative effort between the developers and DBAs and allows the developers to make simple or more complex changes to the database (depending on their experience) that may be used by others on the team. Just as you do with your source code builds, when the build breaks, it must become a priority to fix the broken build. If you’re not integrating your database scripts into your build process, I’d encourage you to try it out and see how it works for you.

Integrating CC.NET with NAnt and Subversion on Windows

You may have heard of Continuous Integration (CI) for .NET, but perhaps thought it would take too long to install and configure the tools. By integrating the CC.NET, NAnt, and Subversion tools, you can quickly create a robust CI solution. In this entry, I walk you through the setup and configuration of these tools, step by step.

There are many ways to define CI, but just think of it as a process of integrating your software when a change is applied to your SCM repository. This may include compiling, inspecting, testing, deploying, and generating feedback.

Prerequisites You need to install and configure IIS and have access to a Subversion repository. If you'd like a Subversion project to use for this example, use http://www.qualitylabs.org/svn/ndbunit/. Note: You will need an account established at qualitylabs.org.

Step 1 - Download and install tools Subversion is the SCM tool used to manage your source code. Extract the Win32 version of the Subversion zip (svn-win32...) file to a designated directory NAnt is the build scripting tool to build your software. Extract the NAnt zip file to a designated directory CC.NET is the CI tool to run your build script whenever a change occurs in your SCM. Save and run the installation .exe file. CC.NET should be installed on a separate machine than the developer machines. However, for the purposes of this example, you will be installing CC.NET on the same machine.

Step 2 - Configure Subversion Create a Windows environment variable called SVN_HOME. The value should equal the location of the Subversion installation directory from step 1. Add %SVN_HOME%\bin; to the System PATH environment variable.

Open a command window and type svn. You should receive a message like: Type 'svn help' for usage.

Step 3 - Configure NAnt

Create a Windows environment variable called NANT_HOME. The value should equal the location of the NAnt installation directory from step 1.

Add %NANT_HOME%\bin; to the System PATH environment variable.

Open a command window and type nant. You should receive a message like: BUILD FAILED. Could not find a '*.build' file...

Step 4 - Create NAnt build file Create the following and save as default.build.

Open a command window again and type nant in the same directory where you placed the default.build file. After a short time, BUILD SUCCEEDED should be displayed.

Step 5 - Configure CC.NET Go to the directory where you installed CC.NET in Step 1. From here, go to the server subdirectory. In this directory there is a file called ccnet.config. Open this file and add the following:

  
  
  10000
  
    http://www.qualitylabs.org/svn/ndbunit/
    C:/dev/testearly/svn-win32-1.3.1/bin/svn.exe
    george.washington
    M@rtha
    
    
    true
  
  
    C:/dev/testearly/nant-0.85-rc3/bin/nant.exe
    C:/dev/testearly/
    default.build
    
      build
    
    300000
  
  
  
    .\website\log
  
  
  

Update the trunkUrl attribute to the location of your Subversion server and project. Open a web browser and type: http://localhost/ccnet/ The following should be displayed: Troubleshooting You may get tripped up with a few problems along the way. Here's a list of some of some of the issues along with some troubleshooting techniques:

PROBLEM: In attempting to access http://localhost/ccnet/, the CC.NET web dashboard page does not display

SOLUTION: Ensure that ASP.NET has been registered. It is usually located at C:\WINDOWS\Microsoft.NET\Framework\v1.1.4322 (or something similar) From here, run aspnet_regiis.exe /i. Also, ensure that the CruiseControl.NET service is installed and running by viewing the Services applet. Finally, ensure that you have installed and configured IIS (you may need to add a virtual directory for CC.NET if you install IIS after installing CC.NET.

PROBLEM: When you type nant and/or svn on the command line, you get a message like "'nant/svn' is not recognized as an internal or external command..."

SOLUTION: Ensure that the locations of the NANT_HOME and/or SVN_HOME environment variables reside on your computer. Also ensure that the % and ; symbols are used properly. Lastly, verify that the svn-win32 version has been installed on your machine, so that the svn.exe can be located.

Conclusion Of course, this is a very basic solution of integrating these tools. You'll want to expand the NAnt build script to provide meaningful activities such as running automated tests and inspections and deploying the software. If you need to use a different SCM, you can simply change the source control configuration block in ccnet.config. By integrating your software upon any material source code change, you can better ensure that you will have working software by running through consistent, repeatable steps that compile, test, inspect, and deploy your software (into a development/testing environment). Integrate early. Integrate often.

Originally authored by Stelligent at teatearly.com

How Much Building is Too Much?

Johanna Rothman, questions "How Much Building is Too Much?" in an article in StickyMinds. In it, she clearly seems to take the position that, for most projects, the more often you build your software, the better. I very much agree and I'd also like to address a few points a reader makes on this topic.

How often you build may be based on the needs and culture of your project. I wouldn't always suggest that a project immediately move from never building their software to Continuous Integration (CI) - building at every change to the SCM. A more incremental approach to CI may work for effectively in gaining acceptance in the group. One of the readers indicated that:

One of the biggest issues we had with frequent builds is that it encouraged poor development practices and poorer requirements definition.

I highlight this because I've heard this before as an excuse for not performing frequent builds. I'd suggest that ineffective programming practices, not building frequently, had everything to do with these problems. In this case with frequent builds, they were simply finding out sooner that there were problems with the software. This is actually a good thing; By using this feedback effectively, you can spend time on integrating quality practices into the group, such as writing automated tests, performing reviews or pairing, and/or running software inspection tools to highlight deficiencies.

Rothman also makes a good point that the testers serve a specific function and it is not feasible to have a tester test every build in an environment that performs frequent builds. There may be discussion amongst the development team on which builds the testers will focus their efforts.

Finally, remember that you should script every part of your build process so that you can execute it early and often. If you don't have a repeatable way to build or test your software, not much else matters.

Originally authored by Paul Duvall at testearly.com

Software Automation Continuum

I am often surprised/concerned to hear where many organizations are along their "software automation continuum". Developers spend most their time automating processes for users, yet don’t always see ways to automate their own development processes.

Sometimes developers/organizations are under the impression that they have automated an activity because they've written a few scripts to eliminate some steps in the development process. It's not about throwing a couple of scripts together and running them all the time. In order for automation to be effective for the entire team, you need to share your automation script(s) to the SCM repository, incorporate into the build system and finally, make it continuous (or at least scheduled). The figure below demonstrates the steps along the automation continuum.

1. Identify - the activity you seek to automate. It may be compile, test, inspection, deployment, database integration, and so on. An easy way to think of this is based on the major software disciplines/activities that most projects typically engage in - no matter which methodology they are using
2. Automate - the activity so that it is repeatable. This will involve writing some scripts
3. Share - using an SCM tool such as Subversion, so that others may use the scripts/programs you have written. If no one else can use your script, you are not leveraging the capabilities of automation.
4. Build – A process that you have automated that lives on its own is of minimal value. In many cases, you’d like one process to precede another. For example, you may want to remove certain files before you perform a compile. You can do this by incorporating your automation into a comprehensive build script. You may write this in a build script using a tool like NAnt.
5. Make it Continuous - so that humans do not need to run the automated process and so that risks are mitigated by ensuring all is well whenever a change is applied to the software. You may do this using a continuous integration tool such as CruiseControl.

Where is your organization along this automation continuum? Use the above heuristic to determine how automated your development activities are on your projects. Note, that you may be performing steps 2 and 4 together (for example, if you are using a build script). The more you can make these activities automatic, the more time you will have to deliver value to your users.

Originally authored by Paul Duvall at testearly.com

Using a Cloud Infrastructure for Software Deployments

In the past couple of years, we've invested heavily into using cloud computing at Stelligent and for our customers. Since we create software delivery systems for our customers, it's been our goal to automate all of these processes and literally click a button to generate software that is ready to go to production. While we were able to automate many build, database assembly, deployment, testing, static analysis and other processes, the assumption was often that an instance of some sort (bare or virtual metal) had been setup. Even those that refer to the benefits of Continuous Deployment often make an assumption that the instance is available and configured. At Stelligent, we no longer make this assumption. We're automating the procurement and provisioning of the instance - including installing and configuring servers - using cloud computing.

Having spoken with several colleagues and at conferences about what we're doing, they thought it might be helpful for others to learn how we're using cloud computing resources on our internal projects and with some of our customers. While we've used various cloud providers, I will focus on Amazon Web Services (AWS) for the purpose of this blog entry.  

AWS - Elastic Compute Cloud (EC2)

EC2 provides on-demand virtual hardware instances using a pay-as-you-go model: you only pay for what you use. This makes it perfect for development and testing. We developed a web-based application that uses EC2 to provision a fully installed tech stack for Continuous Integration and target environments to deploy to our customers' non-production target environments. We are generalizing the functionality and releasing it as our CI as a Service Platform as a Service product in the coming weeks. So, using EC2, developers can use a web application to click a button and provision CI and target environments on demand. Developers are able to create clean target environments on demand when wanting to test a new feature in isolation against the project's technical stack. The application uses the Typica framework to access EC2's API.

AWS - Simple Storage Service (S3)

We're using S3 to store binary files in which we don't need to track multiple versions or for storing software distributions under development that can be recreated from source in other ways. S3 provides a great platform for storing transient files. Moreover, we're excited about utilizing AWS' recently announced S3 Reduce Redundancy Storage to lower our costs for truly transient files. We use tools such as Elasticfox for accessing files in S3. We're also beginning to use CloudFront for managing distributions for users who don't have access to our AWS account.

AWS - Elastic Block Storage (EBS)

EBS is used to attach data volumes that might need to persist after an EC2 instance is terminated. A typical use case goes something like this:

1. Test team uses CI as a Service to provision a target environment
2. Test team deploys/installs working application onto this provisioned target environment
3. Test team stores test data that want to use for future testing
4. Test team terminates the instance using CI as a Service (terminates the EC2 instance)
5. Test team provisions new instance using CI as a Service, installs software and reattaches EBS

Stelligent CI as a Service

Our CI as a Service product started to "scratch an itch" as we wanted to install and configure a complete target environment using a cloud infrastructure. We noticed we were doing basically the same thing each time: procuring the instance, selecting instance specs, selecting the operating system, configuring the OS, installing and configuring servers and tools such as Java, Ant, MySQL, Tomcat, (Hudson in some  cases), etc. So, we wrote a web application to perform the installation and configuration of this tech stack so that we could get a complete deployment environment at the click of a button. Development teams use this service on projects to setup (and terminate) these environments at the click of a button. 

In Summary...

This is meant to be an introduction to some of the cloud computing tools we are using on development projects for ourselves and our customers. It gets really interesting when considering process, usage and other factors when using a cloud infrastructure. Stay tuned for more...

Continuous Integration as a Service Beta Now Available

Deploy working software at the click of a button

We are pleased to announce the release of our CI as a Service product from Stelligent. Using Continuous Integration as a Service, you get running with CI along with a complete development environment stack in minutes and manage it. Moreover, you have full unfettered access to the instance. You can start and stop fully configured virtual instances at any time.

The tool installs and configures the Hudson CI server, MySQL database, Subversion, Ant, Maven, Java and all other necessary tools to create and manage working software at the click of a button. It also runs a one-click build and deployment of working software from source code. At Stelligent, we use the CI as a Service to create working software from our SVN version-control repository - on demand.

See the video that demonstrates how to start using CI as a Service. The basic steps are:

1. Click the "Try it Out" button from the Stelligent website.
2. Enter your email address and password and click the Give me CI button.
3. Review the price information and click the Purchase button
4. You will receive an confirmation page. From this page, you can click the Download Key button to download your private key to access your virtual instance or monitor your CI instances.
5. In about 15 minutes, you will receive an email with links to your provisioned virtual instance and running Hudson Continuous Integration server

At this point, you have a fully-configured CI environment. Moreover, you have full access to this instance to configure as you please. No need to negotiate with your Systems/Operations team or wait for them to provision an environment. Build and deploy your software in a clean environment at a moment's notice. Sign up for the beta of CI as a Service and let us know your experiences with it.

CI as a Service is a product and accompanying service offering from Stelligent. Our CEO literally wrote the book on Continuous Integration. We are passionate about helping our customers automate the entire software delivery process to get software to users more quickly.

There's a big difference between automated and automatic

In our work, we interact with people from many different teams: developers, QA/testers, operations, etc.. In some organizations, operations may be further segmented into SCM, DBAs, Security, and so on. And, in other cases, some functions might be combined. But, if I had a nickel for every time I asked someone whether some process was automated and they incorrectly told me yes, well...you know the rest. This is why definitions matter sometimes. It's the difference between automated and automatic.

What they mean is:

• We wrote some scripts for that
• You have to log into each of the machines separately to run the processes
• Your environment/machine has to be configured just right to run the scripts
• You must manually execute each of the scripts separately
• A human must be present to run the scripts
• It's "automated"
  

What I mean is:

• The scripts are run headless (i.e. from a single command)
• The scripts are checked into the version-control repository
• The scripts have been integrated into the build and/or deployment scripts -- and all processes can be run from a single command
• That the automation is automatic

The difference between the approach they think is automated and what I think is automated is vast. But, it's not about using the right definitions; it's about having a common understanding of the goals you want to attain. The "manually automated" approach leads to significant bottlenecks (real scenario: 100 projects and 10 target environments per project means that your development is stopped every single time someone must manually perform these repetitive activities).

So, the next time someone tells you "(their) stuff is automated", you might want to ask a few questions because each manual task slows down the entire process which costs time, money and frustration.