Difference between revisions of "Jenkins"

From Mpich
Jump to: navigation, search
(The MCS Jenkins CI Server and General Jenkins Overview)
Line 41: Line 41:
 
The short term goal is to augment the "old nightlies".  In the longer term it would be good to replace it with an all-Jenkins solution, provided that it remains stable and can provide us all important features that are currently offered by the "old nightlies".
 
The short term goal is to augment the "old nightlies".  In the longer term it would be good to replace it with an all-Jenkins solution, provided that it remains stable and can provide us all important features that are currently offered by the "old nightlies".
  
=== The MCS Jenkins CI Server and General Jenkins Overview ===
+
=== The MPICH Jenkins CI Server and General Jenkins Overview ===
  
Visit https://jenkins-ci.mcs.anl.gov/ to access the Jenkins server. '''This page is only accessible from within the ANL firewall!!!''' You should log in with your MCS username and password.
+
Visit https://jenkins.mpich.org/ to access the Jenkins server.  You should log in with your MCS username and password.
  
After logging in, on the home page you'll see a list of menu options on the left-hand side with an "executor status" table listed below that.  In the main central/right-hand panel you'll see a list of jobs which you are able to view.  I (goodell@) do not know how to filter this list automatically at this stage.  There is a concept of "views", but that doesn't seem to quite solve the problem in general.  Look at the list for jobs named "[https://jenkins-ci.mcs.anl.gov/job/mpich/ mpich]" or "mpich-SOMETHING".
+
After logging in, on the home page you'll see a list of menu options on the left-hand side with an "executor status" table listed below that.  In the main central/right-hand panel you'll see a list of jobs which you are able to view.  I (goodell@) do not know how to filter this list automatically at this stage.  There is a concept of "views", but that doesn't seem to quite solve the problem in general.  Look at the list for jobs named "[https://jenkins.mpich.org/job/mpich/ mpich]" or "mpich-SOMETHING".
  
 
; Helpful Jenkins Terminology
 
; Helpful Jenkins Terminology

Revision as of 16:58, 7 June 2013

MPICH has a relatively recent (as of early 2013) Jenkins continuous integration server setup at https://jenkins.mpich.org/. This page describes how we use this service in MPICH, how it works, and how we might use it in the future.

Executive Summary

If you don't have time to read the lovely prose below, at least internalize this info:

  • we now have a continuous integration server at https://jenkins.mpich.org/
  • this system runs automated build an test runs on regular intervals or whenever a commit is pushed to the revision control system
  • most of the jobs are setup to run automatically whenever code is pushed to http://git.mpich.org/mpich.git
  • the MPICH jobs are unsurprisingly named "mpich" or "mpich-SOMETHING"
  • build+test results are sent to builds@mpich.org. Click here to sign up for this list if you want to receive build status emails.
  • configuration is all done through the web interface (no more cron jobs!)

The Details

Goals

What are we even trying to accomplish by using Jenkins or any other continuous integration system?

  • reduce developer time and effort spent running tests by hand
  • reduce developer time spent fiddling with our existing automated testing systems
  • tighten the automated testing feedback loop from O(1 day) to O(1 hour) or better
  • improve accountability for "breaking the build" (low priority goal, given the current team)
  • improve software (MPICH) quality in several dimensions:
    • ensure correctness on multiple platforms (Linux, OS X, etc.)
    • ensure correctness with multiple compilers (GNU, Clang, Intel, PGI, etc.)
    • ensure correctness with multiple configure options and debugging levels
    • prevent performance regressions
  • track historical software quality information further back than just "last night"
  • reduce average build times, partly by tracking this information historically

Some of this is handled by the existing Nightly Tests infrastructure, though the "old nightlies" have a number of problems:

  • They are fragile. They are a cobbled together series of numerous shell scripts run as cron jobs by several different users on the team. It can be very confusing to track the entire flow of a test run.
  • They are not flexible. Adding a new test suite or configuration can be difficult.
  • They only run nightly.
  • They require the MCS NFS system (though so does our current Jenkins configuration).
  • They provide no way to suppress known test issues without completely disabling tests or platforms.
  • State from one build or day of testing is not always correctly cleaned up, leading to false positives and false negatives in some cases.

The short term goal is to augment the "old nightlies". In the longer term it would be good to replace it with an all-Jenkins solution, provided that it remains stable and can provide us all important features that are currently offered by the "old nightlies".

The MPICH Jenkins CI Server and General Jenkins Overview

Visit https://jenkins.mpich.org/ to access the Jenkins server. You should log in with your MCS username and password.

After logging in, on the home page you'll see a list of menu options on the left-hand side with an "executor status" table listed below that. In the main central/right-hand panel you'll see a list of jobs which you are able to view. I (goodell@) do not know how to filter this list automatically at this stage. There is a concept of "views", but that doesn't seem to quite solve the problem in general. Look at the list for jobs named "mpich" or "mpich-SOMETHING".

Helpful Jenkins Terminology
job (or sometimes "project") 
a logically related set of operations which should be executed in order to test a particular piece of software
build 
a particular execution of a job
workspace 
the working directory where a build executes
master (or sometimes "server") 
the Jenkins server which orchestrates builds, reports results, and manages configuration
slave (or "build executor") 
a host on which builds actually execute (that is, the job actions run on that host)
build status 
one of STABLE, UNSTABLE, or FAILED (colors are right for our server, STABLE is blue on stock Jenkins servers)

In order to control what happens in a build, you need to find your way to the "configure" panel for a given job. If you don't have the right permissions for the job, any link related to the job will probably yield an HTTP 404 for you.

MPICH Job Details

THE DETAILS BELOW ARE LIKELY TO DRIFT OVER TIME. PLEASE TREAT THE ACTUAL SERVER CONFIG AS AUTHORITATIVE. IF YOU ARE A GOOD PERSON, YOU'LL TRY TO KEEP THIS PAGE UP TO DATE (OR AT LEAST TRIM IT OF INACCURACIES)

All of the current jobs have the same basic pattern. They poll a git repository on a 5-minute interval looking for changes which have been pushed. Then they run the following commands in an zsh -xe shell environment, so the first command in the script which returns a non-zero exit code will cause the script (and the build) to fail.

#!/bin/zsh -xe
# uses zsh to have access to the $pipestatus array.  We want $pipestatus so that we can
# redirect command output, including stderr, to tee without breaking the "sh -e" functionality

check_status() {
   eval return $pipestatus[1]
}

cd $WORKSPACE
git config user.email "jenkins-ci@mpich.org"
git config user.name "Jenkins CI Server"

# we've had some trouble with old builds making newer builds appear to succeed even though the failed (or should have failed)
git clean -x -d -f 

./autogen.sh 2>&1 | tee autogen.log
check_status

# almost all of our performance tests fail in the shared MCS build environment, hence "--disable-perftest"
./configure --prefix=$WORKSPACE/_inst --enable-strict=c99 --enable-g --disable-perftest CC=clang CXX=clang++ 2>&1 | tee c.txt
check_status

make -j4 2>&1 | tee m.txt
check_status

make -j4 install 2>&1 | tee mi.txt
check_status

cat m.txt mi.txt | ./maint/clmake > filtered-make.txt

make testing

git status --porcelain | grep -v -E "autogen.log|c.txt|m.txt|mi.txt|filtered-make.txt|git-status.txt|_inst/" > git-status.txt || true

(All of the check_status stuff above makes sure that we can redirect the output of the various steps to files for later examination while still failing whenever the actual command fails. Otherwise the tee command's exit status will be used as the overall exit status for the pipeline, which will always be 0, indicating success.)

The MPICH test suite (run by make testing above) emits a test/mpi/summary.tap file in addition to the traditional test/mpi/summary.xml file. This TAP file is analyzed by the "Publish TAP Results" post-build action. This data is then displayed in two formats, a /tapTestReport/ page and a (much less useful) /tapResults/ page.

After the build completes, a build status email will be sent to builds@mpich.org. Go to the listinfo page to subscribe you want to receive these emails.

Jenkins also archives a specific set of files for each build and retains them a certain length of time (controlled by the job config). These files include c.txt and m.txt (from the MPICH README) as well as the test result files. Basically, anything that we might want to look at if someone reports a problem to us should be in this list.

Other Random Notes. The autogen.sh step currently relies on the autotools installed in Pavan's home directory (MPICH_AUTOTOOLS_DIR=/home/balaji/software/tools/install/i686/bin). The tests run on the build slaves as some MCS daemon user, though I can't remember the username right now.

"mpich"

https://jenkins-ci.mcs.anl.gov/job/mpich/

Our primary job right now. Does not really deviate from the stock template listed above.

In the future we will probably want to make this job stricter, since it's using a pretty weak level of compiler warnings and debug options. Many more problems are typically caught by stricter debug builds (--enable-g --enable-strict=c99).

"mpich-osx"

https://jenkins-ci.mcs.anl.gov/job/mpich-osx/

Same as "mpich", but runs test on the division's one lonely Mac server: frisbee.mcs.anl.gov. This machine is very underpowered, so many of the more strenuous tests in the MPICH test suite tend to fail on this platform.

Upgrading this machine to a recent vintage Mac Pro would be a big improvement. Alternatively, we should create a new --max-procs=NP configure option for the test suite or add some other way to prevent certain tests from being run.

"mpich-valgrind"

https://jenkins-ci.mcs.anl.gov/job/mpich-valgrind/

Similar to the basic template, but with two changes:

  • the configure line is: ./configure --prefix=$WORKSPACE/_inst --enable-g --enable-strict=c99 --disable-perftest
  • the MPICH test suite programs are run under valgrind, using export MPITEST_PROGRAM_WRAPPER='valgrind -q'

"mpich-review"

https://jenkins-ci.mcs.anl.gov/job/mpich-review/

Major differences:

  • the configure line is: ./configure --prefix=$WORKSPACE/_inst --enable-g --enable-strict=c99 --disable-perftest
  • the git repository polled for changes is any branch of git@git.mpich.org:review.git instead of "origin/master"

"mpich-multiconfig"

https://jenkins-ci.mcs.anl.gov/job/mpich-multiconfig/

Ignore this for now. It's an experimental setup to understand how Jenkins handles so-called "multiconfig" jobs.

Possible Future Uses of Jenkins in MPICH

  • run the other test suites as well (MPICH1, Intel, C++, LLNL I/O)
  • automated performance regression testing, including historical performance trend plotting
  • packaging our nightly snapshot tarballs
  • packaging our final release tarballs
  • write a script to filter TAP results for more sophisticated xfail criteria, possibly based on machine or test environment (e.g., exclude bcast2 failures due to MPIEXEC_TIMEOUT on shared machines)
  • gate pushes to origin on 100% clean tests
  • automated builds on platforms that are harder to integrate with the old nightlies (BG/Q, niagara machines, etc.)
  • multi-machine tests
  • build an extreme feedback device (google for more ideas) :)