The concept of Software "Build Announcement"

Preface: 

I originally wrote this article in http://www.cmcrossroads.com/ during June 2009. But now, I found that this article is missing from the http://www.cmcrossroads.com/ and I try to trace it.

I also found that my original article was referred by  computer science department of Rochester Institute of Technology, NY.

Since the concept discussed in the article is still relevant, I thought to restore my article to my own blog (this one) with minor updates. I used Way Back Machine, to trace the original article source and I found it HERE.

I will also try to notify R.I.T CS department so that they may update the web link.

Lets continue to the article ...

The concept of Software "Build Announcement" 

This is a new era in Software Build and Release management. Continuous integration brings more frequent software builds. More frequent software builds means more frequent updates for testers, developers, and other stakeholders. This brings out new use cases / user stories:

1. As a Test/Validation engineer, I want to know, is the "Product Build output" valid to take forward for testing. 


2. As a Developer, I want to know, did the product build go through without any compilation issues? Was the Integration done correctly? 


3. As a stakeholder (Product owner, Program manager), I want to know, what is the status of product build?

Many build management tools (e.g. Jenkins) provide the methods for notifying task completion via e-mails and they also provide a dashboard that shows task status. But "All green" status does not always mean that the product build is "good" and can be promoted to the testing phase.

This article is about creating effective Software "Build Announcements". 

Let's understand the user stories / use cases,



What does SW validation / test engineer want to know?

Typically, a Software testing environment is a controlled (separate) environment from the production Software Build environment. So when the product needs to be tested, Tester / validation engineer will promote the product to the testing environment (which may be in a different subnet, with different firewall settings etc..) and the Software product needs to be installed in right environment with all its dependencies (older versions may need to uninstalled before installing latest build) and then, finally, it is ready for testing or validation.

Clearly the above steps involve efforts (which may raise questions), What if installer itself is broken? What if the one component failed to compile? What if the application is not starting at all?, Test engineer will find out anyway but then it is too late. The testing environment needs to be rolled back (Uninstall the latest build and install the previous build) in order to continue with the testing.

What does the Software developer want to know?

Software developers want to know that, the software build went through without compilation issues and components are integrated properly. Just parsing (e.g. looking at error patterns) compiler logs is not good enough because software products usually depend on many run time binaries and integration issues will be found only while using (testing) the software product.

Software developers depend upon the testing team to know the major integration issues or they themselves need to do the minimal testing (Sanity testing) to ensure that the software build is properly integrated.

What do all the stakeholders want to know?

Stakeholders (e.g. Product owners, program managers etc...) want to know the status of the product build in a rather "Boolean" way (Pass or Fail) because the build management tool's dashboard will not show the clear status. For example, compilation task may show green but the run time component integration may fail.

What about automated task completion notification?

Build management tools supports automated task notification mails. It just notifies us that a particular build / task has been completed. It is also for a particular task not for a complete product build.

The level of continuous integration differs from product to product. Less "weighty" products may build the whole product (including packaging) every time the source code changes. Heavy weight products (which can takes 6 to 14 hrs to build the complete product package) still depends on nightly builds for full build and it might build a few components for every source code change.

Either way, once the complete build is over, it is available for testing and then someone/something needs to ensure the quality of build.

So how do we satisfy all the above requirements? How do we make sure that everyone (Test Engineer, Developer and Stakeholders) gets the required information. 

Software Build Announcement

Here we need the process of "Software Build Announcement". This is the mail notification that contains the "Boolean" result of build status (Pass/Fail).

How do we assess that the build has passed/failed? - It should be based on the "Quality Gate", You can name it as a BAT (Build Acceptance Test) or Sanity Test quality gate.

I prefer the "Sanity test" because it verifies the overall sanity of product which involves some part of functionality testing.

Sanity Testing

"Sanity Test" quality gate should involve the mandatory test cases which will help to ensure the build's status.

For example, A software product can have the following test cases to ensure the product is ready to take forward to testing,

1.    Compiler logs - Should not have compiler issues
2.    Packaging logs - Should not have packaging issues
3.    Product installation - Product should install with out issues
4.    Product invocation - Product should come up without crash
5.    Product de installation - Product should un install with out issue

Hey, Wait!  Many test cases also involve testing functions, Should I test the product build before announce the build?

Yes!, Here comes the Joy of Automation. Automate your Sanity test cases and Build announcement.


Sanity Test Automation

In Nokia, (I was working in Nokia SW division in 2009, Now working in NXP Semiconductors, Automotive SW Division) We have automated both sanity testing and Build announcements. Sanity test frame work architecture looks like this,

As the architecture diagram explains, Once the product build is completed, the Build server invokes the Sanity test cases via the Sanity test framework and the build will be tested  for sanity.

If all the mandatory test cases are passed, the build will be shared in a production build share area and the build will be announced as a "successful" build via Email - else the build will be shared in a temp build share and it will be announced as "failure" build via Email.

A minimal sanity testing helps here to announce the builds.


Here is our success build announcement format,

Here is our success build announcement format,

Sub: Build Announcement: Sprint_<Sprint No>_<Product_Name> -- build <Build Number>

<Product> Sanity Test Report
 -----------------------------------
     1. Download_Build: PASS * 
     2. Unzip: PASS * 
     3. Test_compile_log: PASS *
             No.of Compilation Warnings: 42636
 
     4. Test_CMworkarea_log: FAIL
             No.of Explicit Conflicts: 2
 
     5. Test_Install: PASS * 
     6. Test_Invokation: PASS * 
     7. Test_Deinstall: PASS * 
     8. Cleanup_Build: PASS 

           Passed : 7
           Failed : 1
           Total  : 8

 Product used for this test :  <product>.zip

 Note: * indicates mandatory TestCase.

          Build PASSED Sanity Test.

 Build Location: \\<share_name>\<Build_Share_Dir>


 Build Report: \\<share_name>\<Build_Share_Dir>\Build_Report.html


 Build Log Location: \\<share_name>\<Build_Share_Dir>\Build_Log


 Sanity Test Log Location: \\<share_name>\<Build_Share_Dir>\<product>.log

 *** AUTO GENERATED MAIL - Pls do not Reply***
 
 SDK Sanity Test Automation

The above test cases help us to determine the success or failure build.

How Software "Build Announcement" is useful?

Since the Build announcement and sanity testing is automated, it will happen over the night and everyone gets the build announcement before step in to office.

For Test Enginner: Successful Build Announcement makes sure that latest build can be taken forward for testing. So they can take it into test machines without worrying about the roll back and efforts spend to move to latest builds will not go waste.

For Developer: The test cases include analyzing log files and invoking the product. If both are fine, we can make sure that Compilation & Integration is successful. Developer will get to know the Integration status without waiting for testing results.

For Stakeholder: Build announcement is a Boolean type. It says, build is Successful or Failed. No need to look in to the dashboard and interpret multiple build job's status. It also helps to collects the metrics, for example how many broken builds in this particular sprint (in scrum).


Conclusion

Build announcement with a sanity test quality gate tests the builds even before it is announced (released) for consumption. It will improve quality because it finds issues as  early as possible. It is not just analyzing the compiler log files but it actually tests the product and it will be helpful to find out integration issues.

More test cases can be added for improving the build success confidence level. Since the test cases are automated, it avoids the human errors and improves the quality.

Build Announcements helps to reduce the engineering effort and improve quality!