Using Bamboo for Deployments

As part of a project involving the complete redevelopment of a large UK based loyalty programme website, it was also necessary to assess the development processes that were being used and identify areas of improvement.

The status quo of the software lifecycle was that changes that were required on the server side were developed offshore and then the skeleton JSP files were provided to the onshore web team who would apply the styling (CSS) and front end validation (Javascript). These JSP files were managed and edited within the CMS which resulted in content editors having to know technologies such as HTML, CSS and JSP. Consequently, the web team were burdened with managing the content for the live website, usually at very short notice. Furthermore, the packaging of the server side, CMS and additional applications were bundled as one monolithic Java Enterprise Archive (EAR) distribution and deployed manually on application servers. Testing was also completely manual and although there were a number of test environments, each was configured slightly differently and not representative of the production environment. The entire deployment process was very labour intensive and consequently release into production (performed by a separate production support team) was a very exhausting and painful experience for those involved. Releases had to be scheduled every two weeks and performed in the early hours of the morning with a number of people on-call to provide support.

Instead of reusing any of the existing codebase, it was decided to start from the ground-up using the a new CMS as the platform to develop the new website that would serve multiple channels and also facilitate the marketing team to manage its content.

The business was also undergoing a transition from a traditional 'waterfall' method of development to using 'agile' processes. Behaviour Driven Development (BDD) was employed to capture and refine requirements.

There were also a number of technologies and tools adopted to deliver enhance productivity and deliver value. We decided to abandon the heavyweight enterprise application server as there was no need for the additional features that it provided, and use the more lightweight Tomcat servlet container. Jira [1] was used to capture requirements as stories, to raise issues and defects, and generally plan and track the project. Confluence [2] was adopted for documentation purposes and as a collaboration tool to discuss designs. Bamboo [3] was used as the continuous integration and build tool and all new source code was required to have unit tests. Functional and acceptance tests were developed using Cucumber [4] and Ruby [5] to create an automated regression testing suite.

Bamboo was configured to monitor every time code was committed to version control and trigger a build and test. With third-party plugins such as Sonar, the team was provided with visibility of test coverage and code quality. Using Bamboo to perform deployments into all environments also provided confidence in the deployment process so that there were no surprises during release into production. As the acceptance tests were also built and executed by Bamboo, it was possible to trigger them from Bamboo after a deployment using Bamboo's REST interface.

Bamboo deployment plans allow software artefacts created during the Bamboo build plans to be deployed into a given environment.

Bamboo Deployment Plans

The above screenshot shows how Bamboo can be used to deploy into different environments. Each environment needs to be configured with tasks that determine how the artefacts will be deployed.

Bamboo Deployment Tasks

The screenshot above shows the tasks used to deploy the website into the performance testing environment. The artefact is downloaded on to the Bamboo server and then a script is executed to perform variable substitution of environment specific configuration. The artefacts are then copied to the remote server using the SCP Task and finally Tomcat is stopped, the artefacts deployed and Tomcat restarted by executing a script on the remote server (SSH Task). These tasks are then repeated for the second server in the clustered environment. The tasks are replicated for each environment to ensure consistency in the deployment process but each environment also has some specific configuration such as the address of the database server and external web services. These can be specified as environment variables and then referenced from the deployment tasks such as $bamboo_catalina_base.

Bamboo Deployment Plan Variables

Using Bamboo for deployment made life so much easier for developers, testers and production support. Nevertheless, managing the Bamboo configuration was becoming cumbersome as with every change to the deployment process, the numerous deployment tasks and variables had to be updated for every environment. To streamline the deployment pipeline further, it was necessary to move the environment specific variables and tasks into property files and scripts that could be executed remotely. The deployment scripts were maintained in version control just like the website source code and packaged along with the application as a separate artefact by Bamboo. This had the benefit of decoupling the deployment process from Bamboo and reducing the number of tasks required to be configured in Bamboo for additional environments. So although Bamboo is still being used for deployments and executing the acceptance tests, it is much easier to manage and test changes to the deployment process.

Deployment Script

Environment Properties

The above screenshots show the deployment script that is now executed by Bamboo along with a sample properties file. The properties file captures the environment specific variables that were previously managed within Bamboo. Line 33 of the script shows how the properties file is read into the script. Lines 62 to 67 shows how an array is declared and the app_cluster from the properties file is read into the array. The array is then iterated over to configure the node that matches the ip address of the server. This allows additional nodes to be added to the cluster very easily by simply updating the properties file. Lines 86-100 perform the variable substitution on a number of configuration files by using the variable values specified within the properties file. Lines 107-118 then copy these configuration files to the relevant directories and restart Tomcat.

Bamboo now simply needs to invoke this script as shown below.

Invoke Deployment Script

The post-deployment smoke tests are invoked from Bamboo through its REST API.

curl --user $bamboo_smoke_test_user:$bamboo_smoke_test_password -X POST http://galbamboo:8085/rest/api/latest/queue/CMS-SMK?bamboo.variable.CMS_ENV=qa

References:

[1] https://www.atlassian.com/software/jira/
[2] https://www.atlassian.com/software/confluence/
[3] https://www.atlassian.com/software/bamboo/
[4] https://cucumber.io/
[5] https://www.ruby-lang.org/en/