Rerun Jenkins Builds

Some consultants at our office needed to be able to roll back a Jenkins project that deployed to staging and production sites in case of an issue. The project had been setup to allow for a typical git checkout and then ended up using a php build tool, phing, to essentially rsync up to the servers.

Turns out ‘Parameterized Builds’ was able to resolve this. It took just a few steps:

  • Click ‘Configure’ on the project.
  • Check ‘This build is parameterized’.
  • Add a String Parameter with the name GIT_COMMIT and a default of ‘master’
  • Under Git > Branches to build, add the parameter as $GIT_COMMIT
  • Save

This allows you to specify the commit SHA for each release. If you need to roll back you just run it with the specified commit SHA. This can be configured to use git tags as well. Inspiration of this came from this post.

Timely Pull Request Responses

I contributed some pull requests this year to several projects including the classic Jenkins Chuck Norris Plugin. Some of them were accepted quickly, others had a bit of back and forth and a few sat for months. My pull request for the Chuck Norris Plugin attempted to fix an issue where the large Chuck images no longer showed up because Jenkins had changed its’ layout design. It took diving back into some Java and brushing up on Jelly templates to figure out what was going on with the bug. After some testing I submitted a pull request and hoped to be able to integrate the fix soon after.

That was December of 2014. The fix was finally merged in November of 2015. I don’t have any negative feelings about it. People get busy. It’s open source.

At some point people get bored with projects, move on, don’t have time, or they don’t work in the language anymore. I think there’s two reasonable ways to handle this:

  • Hand off maintainer duties. It’s the best option if you have someone interested in taking over the project.
  • Post your explanation for not updating the project on the README to let them know pull requests and issues probably won’t get any attention. Then someone who cares enough can fork it or at least it’s use will slowly dwindle down.

In the Chuck Norris case they got a new maintainer, and my little fix is finally merged in and released.

10,000 Tests and Counting

I played a “Yeah” sound effect in campfire a few weeks ago in celebration of checking in our 10,000th test. It was a milestone worth celebrating with both Creme Brûlée Bread Pudding and a chocolate chip cookie. Stepping back a few years I had to fight policy battles just to allot any development time to testing or even check it into CVS with the production code.

Some good things about 10,000 tests and counting:

  • We have pretty good confidence that we can catch breaking changes throughout the app. CI and a suite of much slower QA Acceptance tests add to that confidence.
  • We can run the entire suite of 10,000 RSpec examples in about 8 minutes on the newest Macbook Pros with 16GB of RAM and 4 cores plus hyper threading.
  • Finding old crufty areas of the codebase that aren’t’ tested is a rare surprise rather than a common experience.
  • Even our large “god” classes are generally well tested.
  • We’re constantly thinking about ways to increase the speed of the overall run to at least keep it under the 10 minute threshold rule of thumb. This tends to lead to good refactoring efforts to decouple slow tests from their slow dependencies.

Some not so good things:

  • Many of the ‘unit’ tests are really light integration tests since they depend on database objects, Rails ActiveRecord objects in our case.
  • Some of our ‘god’ classes have 3000+ lines of tests and take 2-3 minutes to run on their own.
  • We have to rely on methods like parallel tests to distribute our unit test running.
  • If it doesn’t look like a change will impact anything outside the new code we sometimes skip running a full spec and let the CI server catch issues.
  • Running individual specs that use ActiveRecord often take 5-8 seconds to spin up, which is painfully long for a fast TDD cycle.
  • Our full acceptance test suite still isn’t consistent enough and running on CI so we have even more of a dependency of trusting the indirect integration testing in our unit test suite.
  • We’d like to use things like guard or auto test, but we haven’t been able to make them work with such a large number of tests.

Even with all the cons of a really large test suite, I love that we have it and run it all day long.

One Year with Jasmine

It’s been about one year since we introduced Jasmine as our default for Javascript testing. Looking back it’s easy to declare it a success:

  • We test all our new javascript instead of just deciding it isn’t worth the effort.
  • Javascript is broken out into files instead of having the temptation of just leaving it inline in a view template.
  • Functions are broken down and refactored to be small and testable.
  • We’ve even been able to test some complex Closure javascript using Jasmine instead of JSUnit.
  • Running the full Jasmine suite is a part of every CI build.

If we’d only achieved a few of these things I’d consider it a big success. For a long time my default approach was to handle Javascript testing through functional Selenium based tests. While these are valuable tests, they certainly don’t help doing TDD with Javascript. Jasmine has finally allowed me to stay in a BDD/TDD workflow when switching between Ruby and Javascript.

Jenkins: My Personal Bodyguard

I’ve been running Jenkins as my CI solution for years now, but for the first 6 months at my new job I used it in an entirely new way. The typical CI pattern is you setup to run against checkins to your master branch run all the unit tests, and depending on how sophisticated you are integration tests, static analysis, or even performance tests.

In my current shop we had a standard CI build using CC.rb against checkins to our master branch. It ran about 5000+ specs sequentially and if everything’s green master is available for production pushes. That fulfills the standard case for CI just fine, but I quickly found that it a little light for my needs. The issue is we relied on feature branches for rolling out all of our stories and while those stories are being worked there’s no CI job at all. Another pain point is even with parallel specs and a nice 4-8 core processor the full test run takes 5-10 minutes so it doesn’t get run nearly as often as one might like.

I’d let Jenkins be my default choice for CI in Javaland, but I was really open to switching it out for a more Ruby style CI server. I figured the first place to look was our already running CC.rb implementation. I tried out CC.rb long ago and remembered liking it a lot better than the original cruise control, but with years of development on it I figured it had evolved. Turns out CC.rb is a pretty basic solution for CI and it hasn’t had much love over the years. It runs a build showing the console output and a very basic web console. Coming from Jenkins the lack of features and plugins was pretty shocking.

Given the Ruby community’s love for XP practices there must be a better option available. I looked at numerous CI servers for Ruby including CI Joe, Integrity, and Travis. All of them followed the lines of providing a very simple CI server much like CC.rb. Travis CI appeared to be a bit more with its use of a hosted open source solution, but it is completely tied to hosting your project on Github. None of these projects showed the depth of a solution like Jenkins.

So the solution was to run Jenkins as my own personal bodyguard. We were looking at swapping out the CC.rb implementation, but it wasn’t a priority right away. So I dived in and setup Jenkins to run a build on my laptop. Instead of just building master I added whatever git feature branch I was working on and hooked up an audible trigger to play James Brown “I feel good!” for successes and “Houston I think we have a problem” for failures. I expected to use the rake plugin to launch the build, but it turned out it was much easier with using things like rvm and bundler to simply execute a script. My final scripts for the build looked something like the following:

bash -l -c "rvm use 1.8.7-p330@acme"
bash -l -c "bundle install"
bash -l -c "rake db:migrate"
bash -l -c "rake parallel: prepare"
bash -l -c "rake parallel: spec"

So finishing up a bit of functionality, I’d commit and about 10 minutes later my laptop would exclaim “I feel good”, or occasionally “Houston I think we have a problem” to let me know I missed something. So I had my own personal bodyguard build and I could happily let it run in the background.

This year we replaced CC.rb with Jenkins and dropped feature branches so I’ve retired my laptop Jenkins, but I recommend the practice if you find yourself in a similar spot.