DevOps in the Cloud

Screencast on Scripted Infrastructure with AWS CloudFormation and Chef

 

Continuous Delivery Tools List

I usually shy away from giving a list of tools that we use because people have their particular tool preferences and are sometimes indignant in considering others. However, I realize it's helpful for people to understand the tool landscape when it comes to Continuous Delivery in the Cloud just so they know where to start looking. After reading my Continuous Integration book, this is often the most common question I get from readers.

I want to say up front that I'm not advocating the use of any of these tools, just that we've used some of the tools or investigated when creating Continuous Delivery systems. I'm sure some of the tools that we use on a daily basis won't make it to this list.

The precise toolset a team may choose to use depends upon numerous factors including project, cost and customer constraints - to name a few. Therefore, I suggest that you focus more on the type of tool and determine which one meets their particular needs for their Continuous Delivery ecosystem. Just because I'm not mentioning a particular tool doesn't mean I'm not using it or that I don't think it's a good tool; these are meant to be illustrative. We tend to focus more on freely-available tools because people can download and use them quickly. There are good reasons to choose commercial tools. As implied before, you don't need to be using all of these tools to get significant benefit from Continuous Delivery. Start small and build it up. I've listed some of the tools in each category for the Java, .NET and Ruby platforms. Since, we lean heavily toward Cloud tools, you'll see that we opt for the SaaS-based tools, when applicable. Let me know if your preferred tool didn't make the list. Ok, there's my disclaimer. On with the list:

Application Containers - JBoss, Tomcat, IIS, Mongrel. NOTE: there are so many app containers, I'm not going to try to list all of them.
Build Tools - Ant, AntContrib, NAnt, MSBuild, Buildr, Gant, Gradle, make, Maven, Rake
Code Review - Crucible
Code Insight - Fisheye  
Continuous Integration - Bamboo, Jenkins, AntHill Pro, Go, TeamCity, TFS 2010
Cloud IaaS -  AWS EC2, AWS S3 ,  Windows Azure
Cloud PaaS - Google App Engine, AWS Elastic Beanstalk, Heroku
Database - Hibernate, MySQL, Liquibase, Oracle, PostgreSQL, SQL Server, SimpleDB, SQL Azure, Ant, MongoDB
Database Change Management - dbdeploy, Liquibase
Data Center Configuration Automation - Capistrano, Cobbler, BMC Bladelogic, CFEngine, IBM Tivoli Provisioning Manager, Puppet, Chef, Bcfg2, AWS Cloud Formation, Windows Azure AppFabric NOTE: There are many names and overlap for this tool "category".
Dependency Management - Ivy, Archiva, Nexus, Artifactory, Bundler
Deployment Automation - Java Secure Channel, ControlTier, Altiris, Capistrano, Fabric, Func
Information Sharing - Confluence, Google Apps
Installer - InstallShield, IzPack
Integrated Development Environment (IDE) - Eclipse, IDEA, Visual Studio
Issue Tracking - Greenhopper, JIRA
Multi-Type - rPath
Passwords - PassPack, PasswordSafe
Protected Configuration - ESCAPE, ConfigGen
Project Management - JIRA, Pivotal Tracker, SmartSheet
Provisioning - JEOS, BoxGrinder, CLIP, Eucalyptus, AppLogic
Reporting/Documentation - Doxygen, Grand, GraphViz, JavaDoc, NDoc, SchemaSpy, UmlGraph
Static Analysis - CheckStyle, Clover, Cobertura, FindBugs, FxCop, JavaNCSS, JDepend, PMD, Sonar, Simian
Systems Monitoring - CloudKick, Nagios, Zabbix, Zenoss
Testing - AntUnit, Cucumber, DbUnit, webrat, easyb, Fitnesse, JMeter, JUnit, NBehave, SoapUI, Selenium, RSpec, SauceLabs
Version-Control System - SVN/Subversion, git, Perforce

Elastic Operations - Scale Engineers in the Cloud

I’m happy to announce that we’re now offering Elastic Operations.  Elastic Operations is a managed service that eliminates all your hardware and replaces it with a reliable, scalable cloud supported by Operations Engineers. 

You get self-service provisioning, build, deployment, database administration, issue tracking, and system monitoring -- all managed by our expert engineers at one flat rate per month.

Your flat rate is based on the number of applications you’d like to manage with Elastic Operations.  And you can scale these experts up and down on a monthly basis, just like you do with Cloud Computing.

We offer various flat-rate plans for development operations, testing, and production. By utilizing 100% automation and the commodization of hardware via the cloud, we offer drastically reduced prices over traditional operations teams who manage data centers.

What applications would you like to manage better?  Sign up for Elastic Operations today, and your applications could be up and running in the cloud tomorrow. Check out the one-minute video on Elastic Operations here. 

 

To get more information on Elastic Operations or Stelligent, send an email to elasticops@stelligent.com

P.S. Use our Cloud ROI Calculator to learn how much you can save when moving to an Automated Operations Cloud.

One-minute Screencast on using Cloud ROI Calculator

A one-minute screencast on using the Cloud ROI Calculator to determine the costs between a Automated Cloud Operations team and a Traditional Operations team.

 

Continuous Delivery: How to deliver software to users at the click of a button

Continuous Delivery - part of the Martin Fowler Signature Series - was published last month and if delivering software to your users quickly and often is important to you, you must get this book. Continuous Delivery is what we provide our clients at Stelligent - delivering complete software systems to users at the push of a button. Jez Humble and David Farley go into great detail in describing everything you need to do to create a continuous delivery system. 

This is one of the most important software books published in years. From the beginning and throughout the book, the authors emphasize the importance in establishing one delivery team consisting of various experts throughout the software lifecycle - developers, DBAs, Systems/Operations, network specialists, testers and so on. The overarching pattern the authors describe is the Deployment Pipeline, which is basically a staged process consisting of all of the steps to go from bare/virtual metal to a working system whenever there is a change to source files. Of course, the only way this can be done is through copious amounts of automation. The other key point the authors make is that this automated delivery system - itself - is versioned with every change. Not just the custom source code, but also the operating system(s), tools, configuration and everything necessary to create a working software system. It's the first book I know to describe this concept in such great detail and not (really) provide any exceptions to the rule. In Continuous Integration (also in Fowler's series), I made a point of putting 'everything' in version control (referring to Berczuk and Appleton's Repository pattern), but Humble and Farley made it a core theme of their Deployment Pipeline and the overall book.

To sum up the book in a few bullets:

• The purpose of Continuous Delivery is to reduce the cycle time between an idea and usable software such that it's approaching zero
• Automate (almost) everything necessary to create usable software
• Version complete software systems (not just source code) for every change committed to version control system
• Employ a Deployment Pipeline in which the entire system is recreated whenever a change is committed to the version-control system and provide continuous feedback
• Identify one delivery team consisting of various delivery experts - build, deploy, provisioning, database, testing, etc. - a concept emphasized in the DevOps movement

The authors go into great detail in describing each of these themes. So, if you want the process of delivering software to any target environment - including production - to be a click of a button and something that can be accomplished as often as the business requires, get this book. When you employ the practices in this book, no longer will you need to artificially throttle changes delivered to users for months or even years because of the expense and risk required to deliver software.

As a note, I plan to extract and describe a combination of patterns from this book, the Continuous Integration book, and other patterns we use at Stelligent in upcoming blog posts.

Using a Cloud Infrastructure for Software Deployments

In the past couple of years, we've invested heavily into using cloud computing at Stelligent and for our customers. Since we create software delivery systems for our customers, it's been our goal to automate all of these processes and literally click a button to generate software that is ready to go to production. While we were able to automate many build, database assembly, deployment, testing, static analysis and other processes, the assumption was often that an instance of some sort (bare or virtual metal) had been setup. Even those that refer to the benefits of Continuous Deployment often make an assumption that the instance is available and configured. At Stelligent, we no longer make this assumption. We're automating the procurement and provisioning of the instance - including installing and configuring servers - using cloud computing.

Having spoken with several colleagues and at conferences about what we're doing, they thought it might be helpful for others to learn how we're using cloud computing resources on our internal projects and with some of our customers. While we've used various cloud providers, I will focus on Amazon Web Services (AWS) for the purpose of this blog entry.  

AWS - Elastic Compute Cloud (EC2)

EC2 provides on-demand virtual hardware instances using a pay-as-you-go model: you only pay for what you use. This makes it perfect for development and testing. We developed a web-based application that uses EC2 to provision a fully installed tech stack for Continuous Integration and target environments to deploy to our customers' non-production target environments. We are generalizing the functionality and releasing it as our CI as a Service Platform as a Service product in the coming weeks. So, using EC2, developers can use a web application to click a button and provision CI and target environments on demand. Developers are able to create clean target environments on demand when wanting to test a new feature in isolation against the project's technical stack. The application uses the Typica framework to access EC2's API.

AWS - Simple Storage Service (S3)

We're using S3 to store binary files in which we don't need to track multiple versions or for storing software distributions under development that can be recreated from source in other ways. S3 provides a great platform for storing transient files. Moreover, we're excited about utilizing AWS' recently announced S3 Reduce Redundancy Storage to lower our costs for truly transient files. We use tools such as Elasticfox for accessing files in S3. We're also beginning to use CloudFront for managing distributions for users who don't have access to our AWS account.

AWS - Elastic Block Storage (EBS)

EBS is used to attach data volumes that might need to persist after an EC2 instance is terminated. A typical use case goes something like this:

1. Test team uses CI as a Service to provision a target environment
2. Test team deploys/installs working application onto this provisioned target environment
3. Test team stores test data that want to use for future testing
4. Test team terminates the instance using CI as a Service (terminates the EC2 instance)
5. Test team provisions new instance using CI as a Service, installs software and reattaches EBS

Stelligent CI as a Service

Our CI as a Service product started to "scratch an itch" as we wanted to install and configure a complete target environment using a cloud infrastructure. We noticed we were doing basically the same thing each time: procuring the instance, selecting instance specs, selecting the operating system, configuring the OS, installing and configuring servers and tools such as Java, Ant, MySQL, Tomcat, (Hudson in some  cases), etc. So, we wrote a web application to perform the installation and configuration of this tech stack so that we could get a complete deployment environment at the click of a button. Development teams use this service on projects to setup (and terminate) these environments at the click of a button. 

In Summary...

This is meant to be an introduction to some of the cloud computing tools we are using on development projects for ourselves and our customers. It gets really interesting when considering process, usage and other factors when using a cloud infrastructure. Stay tuned for more...

Continuous Integration as a Service Beta Now Available

Deploy working software at the click of a button

We are pleased to announce the release of our CI as a Service product from Stelligent. Using Continuous Integration as a Service, you get running with CI along with a complete development environment stack in minutes and manage it. Moreover, you have full unfettered access to the instance. You can start and stop fully configured virtual instances at any time.

The tool installs and configures the Hudson CI server, MySQL database, Subversion, Ant, Maven, Java and all other necessary tools to create and manage working software at the click of a button. It also runs a one-click build and deployment of working software from source code. At Stelligent, we use the CI as a Service to create working software from our SVN version-control repository - on demand.

See the video that demonstrates how to start using CI as a Service. The basic steps are:

1. Click the "Try it Out" button from the Stelligent website.
2. Enter your email address and password and click the Give me CI button.
3. Review the price information and click the Purchase button
4. You will receive an confirmation page. From this page, you can click the Download Key button to download your private key to access your virtual instance or monitor your CI instances.
5. In about 15 minutes, you will receive an email with links to your provisioned virtual instance and running Hudson Continuous Integration server

At this point, you have a fully-configured CI environment. Moreover, you have full access to this instance to configure as you please. No need to negotiate with your Systems/Operations team or wait for them to provision an environment. Build and deploy your software in a clean environment at a moment's notice. Sign up for the beta of CI as a Service and let us know your experiences with it.

CI as a Service is a product and accompanying service offering from Stelligent. Our CEO literally wrote the book on Continuous Integration. We are passionate about helping our customers automate the entire software delivery process to get software to users more quickly.

There's a big difference between automated and automatic

In our work, we interact with people from many different teams: developers, QA/testers, operations, etc.. In some organizations, operations may be further segmented into SCM, DBAs, Security, and so on. And, in other cases, some functions might be combined. But, if I had a nickel for every time I asked someone whether some process was automated and they incorrectly told me yes, well...you know the rest. This is why definitions matter sometimes. It's the difference between automated and automatic.

What they mean is:

• We wrote some scripts for that
• You have to log into each of the machines separately to run the processes
• Your environment/machine has to be configured just right to run the scripts
• You must manually execute each of the scripts separately
• A human must be present to run the scripts
• It's "automated"
  

What I mean is:

• The scripts are run headless (i.e. from a single command)
• The scripts are checked into the version-control repository
• The scripts have been integrated into the build and/or deployment scripts -- and all processes can be run from a single command
• That the automation is automatic

The difference between the approach they think is automated and what I think is automated is vast. But, it's not about using the right definitions; it's about having a common understanding of the goals you want to attain. The "manually automated" approach leads to significant bottlenecks (real scenario: 100 projects and 10 target environments per project means that your development is stopped every single time someone must manually perform these repetitive activities).

So, the next time someone tells you "(their) stuff is automated", you might want to ask a few questions because each manual task slows down the entire process which costs time, money and frustration.