DevOps Zone is brought to you in partnership with:

C# ASP.NET developer who works with MVC but is sometimes known to dabble with WebForms. Passionate coder who cares about his craft. Has a real want to create clean, readable and maintainable code. Obsessed with CI and its effects on a good development team. Pushing the bounds of continuous delivery and studying the advantages it brings to developers. Paul is a DZone MVB and is not an employee of DZone and has posted 25 posts at DZone. You can read more from them at their website. View Full User Profile

Strategy to Make CI Builds Self Testing

09.01.2011
| 3435 views |
  • submit to reddit

This is post 5 in a series called ‘How to get started with CI’. The last post talked about how to implement CI to your first CI project. This post will extend that implementation by making sure that a build is ‘self testing.

A ‘self-testing’ build is a build that runs all of the required tests as part of it. It is important to stress the term tests here as there are different types of tests, unit, integration, regression, UI etc.). In general tests as described run in the following way:

  • unit tests – run in isolation, mock any dependencies and should be fast
  • integration / regression tests – require some sort of setup of data / environment and will be slower to run
  • UI / acceptance tests are slower to run and will require a replica of the application to test against. These will be a lot slower.
  •  
      It is important to understand that if a build that has triggered because of a VCS check in and doesn’t break the build then we cannot assume the integration of the code has worked.

We will not know that the integration has been successful until the code has been tested.

    The most efficient way of testing the code is with our test suite. If the integration fails with obvious errors, e.g. a compilation error, then that’s a huge clue that something has gone wrong.
    My usual strategy for CI builds and their tests runs is as follows:
  1. On VCS check in run all the unit tests

This will give us fast feedback about problems at a code level and will tell us if the code still behaves as expected. For this instant feedback to happen, builds (including test runs) must be fast. IF the runs are not fast then development could potentially be impacted by developers working with code that doesn’t build or hasn’t been tested. Always think of the 10 minute rule for the time a build starts till completion.

2. Every 2 hours run the integration / regression tests

These tests are very important to us but are a step below the priority of unit tests. They will be slower to run since we require data to be setup and torn down for the tests. They can require network and database interaction so we cannot hang around waiting for them to complete.

3. Nightly run the UI / Acceptance tests

These are again still import but the lowest of the priority order for a standard CI build. There will always be a round of testing by QAs or whoever before a release so there is time to fix any errors arising. From my nightly strategy, errors in these tests will only take at maximum a day to appear to a team. These tests are slower again and may require a full compilation and installation of an application to run against so running them overnight means there should be no real problem with time.

The steps listed about are the steps I use for a project is running a normal CI environment. These would change when an environment changes to include continuous delivery of the application. This will be discussed in a further post.

This is only a suggested approach from the experiences I have had with CI. If you run your builds in a different manner then I would love to hear about it – leave a comment below.

References
Published at DZone with permission of Paul Stack, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)