Using Bamboo for Deployments

As part of a project involving the complete redevelopment of a large UK based loyalty programme website, it was also necessary to assess the development processes that were being used and identify areas of improvement.

The status quo of the software lifecycle was that changes that were required on the server side were developed offshore and then the skeleton JSP files were provided to the onshore web team who would apply the styling (CSS) and front end validation (Javascript). These JSP files were managed and edited within the CMS which resulted in content editors having to know technologies such as HTML, CSS and JSP. Consequently, the web team were burdened with managing the content for the live website, usually at very short notice. Furthermore, the packaging of the server side, CMS and additional applications were bundled as one monolithic Java Enterprise Archive (EAR) distribution and deployed manually on application servers. Testing was also completely manual and although there were a number of test environments, each was configured slightly differently and not representative of the production environment. The entire deployment process was very labour intensive and consequently release into production (performed by a separate production support team) was a very exhausting and painful experience for those involved. Releases had to be scheduled every two weeks and performed in the early hours of the morning with a number of people on-call to provide support.

Instead of reusing any of the existing codebase, it was decided to start from the ground-up using the a new CMS as the platform to develop the new website that would serve multiple channels and also facilitate the marketing team to manage its content.

The business was also undergoing a transition from a traditional 'waterfall' method of development to using 'agile' processes. Behaviour Driven Development (BDD) was employed to capture and refine requirements.

There were also a number of technologies and tools adopted to deliver enhance productivity and deliver value. We decided to abandon the heavyweight enterprise application server as there was no need for the additional features that it provided, and use the more lightweight Tomcat servlet container. Jira [1] was used to capture requirements as stories, to raise issues and defects, and generally plan and track the project. Confluence [2] was adopted for documentation purposes and as a collaboration tool to discuss designs. Bamboo [3] was used as the continuous integration and build tool and all new source code was required to have unit tests. Functional and acceptance tests were developed using Cucumber [4] and Ruby [5] to create an automated regression testing suite.

Bamboo was configured to monitor every time code was committed to version control and trigger a build and test. With third-party plugins such as Sonar, the team was provided with visibility of test coverage and code quality. Using Bamboo to perform deployments into all environments also provided confidence in the deployment process so that there were no surprises during release into production. As the acceptance tests were also built and executed by Bamboo, it was possible to trigger them from Bamboo after a deployment using Bamboo's REST interface.

Bamboo deployment plans allow software artefacts created during the Bamboo build plans to be deployed into a given environment.

Bamboo Deployment Plans

The above screenshot shows how Bamboo can be used to deploy into different environments. Each environment needs to be configured with tasks that determine how the artefacts will be deployed.

Bamboo Deployment Tasks

The screenshot above shows the tasks used to deploy the website into the performance testing environment. The artefact is downloaded on to the Bamboo server and then a script is executed to perform variable substitution of environment specific configuration. The artefacts are then copied to the remote server using the SCP Task and finally Tomcat is stopped, the artefacts deployed and Tomcat restarted by executing a script on the remote server (SSH Task). These tasks are then repeated for the second server in the clustered environment. The tasks are replicated for each environment to ensure consistency in the deployment process but each environment also has some specific configuration such as the address of the database server and external web services. These can be specified as environment variables and then referenced from the deployment tasks such as $bamboo_catalina_base.

Bamboo Deployment Plan Variables

Using Bamboo for deployment made life so much easier for developers, testers and production support. Nevertheless, managing the Bamboo configuration was becoming cumbersome as with every change to the deployment process, the numerous deployment tasks and variables had to be updated for every environment. To streamline the deployment pipeline further, it was necessary to move the environment specific variables and tasks into property files and scripts that could be executed remotely. The deployment scripts were maintained in version control just like the website source code and packaged along with the application as a separate artefact by Bamboo. This had the benefit of decoupling the deployment process from Bamboo and reducing the number of tasks required to be configured in Bamboo for additional environments. So although Bamboo is still being used for deployments and executing the acceptance tests, it is much easier to manage and test changes to the deployment process.

Deployment Script

Environment Properties

The above screenshots show the deployment script that is now executed by Bamboo along with a sample properties file. The properties file captures the environment specific variables that were previously managed within Bamboo. Line 33 of the script shows how the properties file is read into the script. Lines 62 to 67 shows how an array is declared and the app_cluster from the properties file is read into the array. The array is then iterated over to configure the node that matches the ip address of the server. This allows additional nodes to be added to the cluster very easily by simply updating the properties file. Lines 86-100 perform the variable substitution on a number of configuration files by using the variable values specified within the properties file. Lines 107-118 then copy these configuration files to the relevant directories and restart Tomcat.

Bamboo now simply needs to invoke this script as shown below.

Invoke Deployment Script

The post-deployment smoke tests are invoked from Bamboo through its REST API.

curl --user $bamboo_smoke_test_user:$bamboo_smoke_test_password -X POST http://galbamboo:8085/rest/api/latest/queue/CMS-SMK?bamboo.variable.CMS_ENV=qa

References:

[1] https://www.atlassian.com/software/jira/
[2] https://www.atlassian.com/software/confluence/
[3] https://www.atlassian.com/software/bamboo/
[4] https://cucumber.io/
[5] https://www.ruby-lang.org/en/

Migrating from SVN to Git

Our project team had been using SVN for the last 8 months as its source control management tool. All development was done on trunk and releases were created from a release branch. After 8 months, when the project (new responsive website on new multi-channel CMS platform) went live, we continued developing on trunk and any hot-fixes were made on the release branch and then merged into trunk. Merges were intentionally kept small and frequent and we avoided feature branches altogether.

Strategically the business wanted to adopt a number of new tools including Slack (team messaging, collaboration), Asana (Project and Task management) and Git, so I was tasked with migrating our project from SVN to Git. The repository would be hosted on github and thereby minimise maintenance as there was no server to manage to ensure there was sufficient disk space, or upgrading the scm tool to a later version.

This post describes the process.

The atlassian tutorial [1] provides 5 simple steps to help migrate users from SVN to Git.

1. Preparing the environment
2. Converting the SVN repository to a local Git repository
3. Synchronizing the local Git repository when the SVN repository changes
4. Sharing the Git repository with developers.
5. Migrating development effort from SVN to Git.

1. Preparing the Environment

After downloading and executing the migration script, it was found that the versions of Git and SVN that were available with the Linux distribution [2] were not of the required versions. It was therefore necessary to install the latest version of Git (2.5.2) from source [3].

Once again, after downloading the Git source, it was found that the necessary dependencies such as the gcc compiler and make tool were missing. These were installed using the yum Development Tools package.

The atlassian migration-script was still complaining about the SVN version so the latest version (1.8.14) was also installed by configuring an additional yum repository [4].

The final piece of the jigsaw was to install the correct subversion perl bindings for the updated SVN by executing yum install subversion-perl

Finally, the svn-migration-scripts verify command was able to successfully execute.

[root@devapp1 git-migration]# java -jar svn-migration-scripts.jar verify
svn-migration-scripts: using version 0.1.56bbc7f
Git: using version 2.5.2
Subversion: using version 1.8.14
git-svn: using version 2.5.2
[root@devapp1 git-migration]#

2. Converting the SVN repository to a local Git repository

The tutorial mentions running git svn clone --stdlayout --authors-file=authors.txt

 <svn-repo>/<project> <git-repo-name>

This fails with the following error if the SVN repository requires authentication.

Password for 'myuser': Can't locate Term/ReadKey.pm in @INC (@INC contains: /usr/lib/perl/site_perl/5.14
/usr/lib/perl5/site_perl/5.14/x86_64-cygwin-threads /usr/lib/perl5/vendor_perl
/5.14/x86_64-cygwin-threads /usr/lib/perl5/vendor_perl/5.14 /usr/lib/perl5/5.14/x86_64-cygwin-
threads /usr/lib/perl5/5.14 .) at /usr/lib/perl5/vendor_perl/5.14/Git.pm line 565.

Therefore it is necessary to first run the svn checkout command so that svn caches the credentials and then also install the Perl Term ReadKey package:

perl -MCPAN -e shell

After the shell initiation, install the ReadKey package:

install Term::ReadKey

It should now be possible to create a local Git repository from the SVN repository.

The cloning of the SVN repository creates the SVN branches and tags as remote branches. The clean-git script included in the svn-migration-scripts.jar kept on deleting the branches and tags that were being created so git branch only showed the master branch.

Executing "java -Dfile.encoding=utf-8 -jar ../svn-migration-scripts.jar clean-git --force --no-delete" allowed us to create the local git branches including all the obsolete and deleted ones.

Executing git branch now showed the local git branches, but git tag was still not returning any tags.

The following script was used to create the git tags changes the path to the tags.[5]

Create Git Tags

3. Synchronizing the local Git repository when SVN changes

The synchronizing of the local Git repository with SVN changes works as described in the documentation by simply fetching the latest changes, rebasing the Git repository and then doing a clean up.

git svn fetch

java -Dfile.encoding=utf-8 -jar ~/svn-migration-scripts.jar sync-rebase

java -Dfile.encoding=utf-8 -jar ~/svn-migration-scripts.jar clean-git --force

4. Sharing the Git repository

This step involves pushing the local Git repository to a public Git repository server and then allowing developers to pull and push commits to and from the public repository.

To facilitate pushing changes to a remote repository, Git uses a 'remote' reference as an alias for the remote repository URL. The remote reference can be anything one chooses but if the remote repository is going to serve as the official codebase for the project, it is conventionally referred to as 'origin',

git remote add origin https://\@bitbucket.org/\/\.git

Once the remote has been set up, the changes can be pushed to the remote repository.

git push -u origin --all

git push --tags

Instead of bitbucket, our project team used github to host our remote repository. The uploading of the git repository failed as there were some files with a history of being larger than 100MB in size which github doesn't allow. The file contained the binary data of images that were pruned once development matured (as the images were moved to be managed in the production environment and not within the codebase). However, to allow the uploading of the Git repository, it would be necessary to remove the history of the offending file. This is where the BFG utility came to the rescue [6].

To Be Continued....

References:

[1] https://www.atlassian.com/git/tutorials/migrating-overview
[2] Red Hat Enterprise Linux Server release 6.6 (Santiago)
[3] http://johnathanmarksmith.com/linux/git/programming/2013/05/15/how-to-install-git-182-on-fedora-centos-red-hat-and-scientific-linux/
[4] http://tecadmin.net/install-subversion-1-8-on-centos-rhel/
[5] http://blogs.atlassian.com/2012/01/moving-confluence-from-subversion-to-git/
[6] https://rtyley.github.io/bfg-repo-cleaner/

Hippo CMS with Spring Security

Hippo CMS already uses Spring for dependency injection as part of the HST framework used to develop websites. By default, it uses JAAS as its authentication mechanism and the CMS repository (via the HippoAuthenticationProvider) for authenticating login credentials and providing security roles for the authenticated user.

The existing website application already had in excess of 10 million users and there was no way that these users would be moved into the CMS repository. Fortunately Hippo can be configured with a custom AuthenticationProvider to use a database or LDAP service rather than the CMS repository for authenticating login credentials.

As part of the redevelopment of the website, the login UI was also redesigned. The default form based authentication uses the Hippo UI templates to render the login form. Again this was quite easy to replace with a custom login form template by following the online documentation [1].

At this point the development was looking good, the show and tell at the end of the first sprint demonstrated the new login story and the home page with integration to internal services for authentication and customer transactional data.

The next sprint required some changes to the login functionality based on insights gathered from user testing and UI/UX designers. Due to the business requirements, the login functionality was required to display a number of different messages depending on the cause of authentication failure. This may involve a simple error message, or asking the user that they need to activate their account or to contact the call centre if their account had been locked.

Using the default login mechanism proved to be difficult in propagating different error messages from the custom AuthenticationProvider back through the Hippo JAAS LoginModule and to the Hippo LoginServlet to render the login form. This was due to the Hippo LoginModule catching any exceptions from the AuthenticationProvider and throwing a new JAAS LoginException with a default error message thereby losing the error code and message from the original exception. There were also other issues with CSS when the login form was redisplayed after an authentication failure.

Furthermore, there was also a lurking requirement that the new mobile apps would be required to integrate with Hippo CMS for authentication and content. This would require developing a RESTful API to expose the content for the mobile apps to use. The mobile apps would not use http form based authentication for a RESTful API. Consequently the new website would need to provide a different authentication method depending on the access channel (web browser or mobile app).

The above issues and the complexity involved in customising the JAAS implementation were the main deciding factor to adopt Spring Security. Fortunately, we were not the first team that needed to do this and the Hippo community provides a project that details the integration of Spring Security with HST based website applications.[2]

The configuration is pretty simple and requires configuring the Spring Security filter within the web.xml file.

Spring Security Configuration in web.xml

This will now ensure all requests are passed through the Spring Security filter chain. The context files can be used to specify any security related Spring beans such as a custom authentication manager, authentication filters and success and failure handlers.

Thereafter, to integrate Spring Security within the HST based web application, it is simply a matter of adding the SpringSecurityValve into the HST request processing pipeline. As the HST pipeline employs the 'chain of responsibilty' pattern [3], by adding the SpringSecurityValve into the pipeline, HST will ensure the SpringSecurityValve has the opportunity to handle the request and establish a javax.security.auth.Subject if it finds a spring security Authentication instance.

The Hippo SpringSecurityValve allows securing the HST based website by configuring 'hst:sitemapitem' or 'hst:mount' nodes with security settings.

References
[1] http://www.onehippo.org/library/concepts/security/hst-2-authentication-and-authorization-support.html
[2] http://hst-springsec.forge.onehippo.org/
[3] http://en.wikipedia.org/wiki/Chain-of-responsibility_pattern