NightlyBuilds

Nightly Builds

Nightly builds are performed on several different machines, with a variety of hardware, operating systems, configurations, and tests. At present, however, there are no nightly builds for ARM, Windows, or Common Larceny; those systems are built manually.

Collecting and reporting the results of those nightly builds became ad hoc (at best) when a couple of machines failed and our Trac site disappeared. Will is restructuring the process to create separate scripts for builds, tests, benchmarking, reporting, formatting, and publication. This will result in a process that does not depend on CDash (or similar software) but is capable of formatting results for publication via CDash.

Some of the machines that run nightly builds do not allow incoming SSH/SFTP, but can initiate outgoing SFTP connections. That motivates the following process:

• Machines run nightly builds, placing all log files in standard places.
• After running a nightly build and all appropriate tests/benchmarks, each machine uses SFTP to transfer its log files to its own particular subdirectory of a central directory on a fairly public file server.
• A formatting server uses SFTP to copy those directories to its local file system. (If the formatting server is the same as the file server, this step is unnecessary.)
• The formatting server scrapes the log files to create a summary of what happened, and converts those summaries into HTML and/or XML acceptable to CDash.
• The CDash format can be sent to a CDash server.
• A web server copies the log file directories and the formatted HTML into its own file system and generates a static HTML page that includes the HTML generated by the formatting server.

The web server needs to know about the formatting server, but does not need to know anything about the log file directory structure or nightly build machines. The formatting server needs to understand the log file directory structure, but does not have to communicate directly with any of the machines that perform nightly builds. The file server just holds the log file directories and accept incoming SFTP. The machines that perform nightly builds know about the file server, but do not need to know anything else.

We need scripts for

• building every configuration of Larceny we still want to support
  • IAssassin (IA32 and Sassy, for Linux, MacOS X, and Windows)
  • ARM (when we acquire a suitable machine)
  • PetitNasm (maybe)
  • Petit Larceny (for Linux x86 and MacOS X PowerPC)
  • Common Larceny (if we want to continue to support it)
• running every set of tests and benchmarks we want to run on a nightly basis
  • basic tests (test/Lib, test/Compiler)
  • Racket R6RS tests
  • R7RS tests (Chibi)
  • R7RS tests (Larceny)
  • CLisp number tests
  • trigtests
• copying build/test/benchmark log files to the file server
  • log files are:
    • git.log
    • build.log
    • buildR7RS.log
    • basic.log
    • racketR6RS.log
    • chibiR7RS.log
    • larcenyR7RS.log
    • clispNumbers.log
    • trigTest.log
    • srfi.log
    • benchmarks.log
  • configuration directories are:
    • linux86native (currently Ubuntu 12.04)
    • linux86petit (currently Ubuntu 12.04)
    • linux86native-regional
    • linux86petit-regional
    • linux86native-flat1
    • linux86petit-flat1
    • linux86native2 (current Ubuntu 14.04)
    • linux86petit2 (current Ubuntu 14.04)
    • macosx (currently 10.4.11)
    • macosx2 (currently 10.6.8, changing to 10.8.5)
    • macosx3 (for 10.10, when we have access to a desktop machine running Yosemite)
    • windowsXP
    • windows
    • commonLarceny
    • linuxARM
    • linuxARMpetit
• pulling that entire directory of log files from the file server
• scraping the log files
• formatting the summary information into HTML
• formatting the summary information into XML suitable for CDash
• pulling the log file directory and formatted summaries into the web server

Details:

• crontab -l lists the current user's crontab, and crontab -e edits a copy of the current user's crontab file and installs the copy as their crontab when editing is complete. (Also the only way to remove all lines from a crontab is crontab -r, according to the crontab man page.)
• Each nightly build uses git or svn to clone/checkout a copy of the source code.
  • If a machine does not accept the GitHub certificate, then the script will block. To fix that, you can do a manual clone/checkout on the machine in question, indicating permanent acceptance of the certificate.
• In the past, we have performed nightly builds on Windows and PowerPC/MacOSX machines.
  • The Windows builds were discontinued long ago because any infinite loop or other problem that occurred during the build could be resolved only by sitting in front of the physical machine.
  • The PowerPC builds have been discontinued because we don't have a PowerPC machine that's always turned on; PowerPC builds can still be performed manually.
• Each autobuild machine has a copy of a nightly-build script that drives the autobuild process each night. (Warning: some of the details below are no longer true.)
  • The usual way that it works is that each autobuild host has several symbolic links to that same script, but with specially chosen names for the link's name.
    • (Yes, this is a strange way to do things. But it does make the purpose of each link self-evident, once they are set up accordingly.)
  • The parameters for the build (whether it should use Larceny or MzScheme as the host, whether it is a Native or Petit build, etc) are encoded in the name of the link.
  • Then, each autobuild host has a single script (which Felix usually names checkout-svn-mail) that is run every night, which:
    1. runs each of the linked scripts
    2. uses sftp to send log files to the file server