-
Notifications
You must be signed in to change notification settings - Fork 38
CRAB Testing
This page describes CRABServer and CRABClient tests executed on each PR in corresponding GitHub repositories. The way in which CRABServer and CRABClient test are executed is identical, therefore the general approach how testing works will be explained without focusing on one specific repository. All needed files which are used in CRAB testing can be found in CMSDOCKS Gitlab repository.
All test are executed as Docker containers in DMWM Jenkins. CRABServer tests configuration can be found at https://cmssdt.cern.ch/dmwm-jenkins/job/DMWM-CRABServer-PR-test/configure while CRABClient at https://cmssdt.cern.ch/dmwm-jenkins/job/DMWM-CRABClient-PR-test/configure. However, you will need to request access rights to view/edit it (you can do that by writing an email to Shahzad). All CRAB builds are kept for 7 days, maximum 50 builds in total. After new PR is created, Jenkins generates a report that has 4 sections: 1) Pylint; 2) Python3 compatibility; 3) Default; 4) Unit tests. Each of this section has an indication whether that test passed or failed. Below all sections are described in more detailed.
Pylint configuration can be found in WMCore repository. So in case you want to run pylint test on your code before creating PR, use this file.
Things in the pylint report that are bold are causing PR tests to fail. Function which decides if test failed can be found here, while short summary would look like this:
- If pylint report has
warningsorerrors, then pylint test fails. However, there is a predefined list of warnings that do not cause pylint test to fail. This list is made of the following warnings:['0511', '0703', '0613']; - If pylint report score is below 8, then pylint test fails;
Pylint report in GitHub is divided into 3 sections:
- warnings and errors that must be fixed - these are warnings and errors in pylint report that are in bold and makes pylint test to fail;
- warnings - these are warnings and errors that that do not make pylint report to fail, i.e. following warnings:
['0511', '0703', '0613']; - comments to review - everything else what pylint report complains about are marked as comments.
As of now, unit tests are not executed for CRAB code, however, since same code to build the final report is used for WMCore and they do have unit tests executed, we have this section in the report as well. One can notice that this type of test always pass the test.