Common Statistical Methods

-

In this section, the most commonly used statistical methods from Combine will be covered, including specific instructions on how to obtain limits, significances, and likelihood scans. For all of these methods, the assumed parameter of interest (POI) is the overall signal strength r (i.e the default PhysicsModel). In general however, the first POI in the list of POIs (as defined by the PhysicsModel) will be taken instead of r. This may or may not make sense for any particular method, so care must be taken.

+

In this section, the most commonly used statistical methods from Combine will be covered, including specific instructions on how to obtain limits, significances, and likelihood scans. For all of these methods, the assumed parameter of interest (POI) is the overall signal strength r (i.e the default PhysicsModel). In general however, the first POI in the list of POIs (as defined by the PhysicsModel) will be taken instead of r. This may or may not make sense for any particular method, so care must be taken.

This section will assume that you are using the default physics model, unless otherwise specified.

Asymptotic Frequentist Limits

The AsymptoticLimits method can be used to quickly compute an estimate of the observed and expected limits, which is accurate when the event yields are not too small and the systematic uncertainties do not play a major role in the result. -The limit calculation relies on an asymptotic approximation of the distributions of the LHC test statistic, which is based on a profile likelihood ratio, under the signal and background hypotheses to compute two p-values p_{\mu}, p_{b} and therefore CL_s=p_{\mu}/(1-p_{b}) (see the FAQ section for a description). This means it is the asymptotic approximation for evaluating limits with frequentist toys using the LHC test statistic for limits.

+The limit calculation relies on an asymptotic approximation of the distributions of the LHC test statistic, which is based on a profile likelihood ratio, under the signal and background hypotheses to compute two p-values p_{\mu}, p_{b} and therefore CL_s=p_{\mu}/(1-p_{b}) (see the FAQ section for a description). This means it is the asymptotic approximation for evaluating limits with frequentist toys using the LHC test statistic for limits. In the definition below, the parameter \mu=r.

    -
  • The test statistic is defined using the ratio of likelihoods q_{r} = -2\ln[\mathcal{L}(\mathrm{data}|r,\hat{\theta}_{r})/\mathcal{L}(\mathrm{data}|r=\hat{r},\hat{\theta})] , in which the nuisance parameters are profiled separately for r=\hat{r} and r. The value of q_{r} is set to 0 when \hat{r}>r, giving a one-sided limit. Furthermore, the constraint r>0 is enforced in the fit. This means that if the unconstrained value of \hat{r} would be negative, the test statistic q_{r} is evaluated as -2\ln[\mathcal{L}(\mathrm{data}|r,\hat{\theta}_{r})/\mathcal{L}(\mathrm{data}|r=0,\hat{\theta}_{0})]
    • +
  • The test statistic is defined using the ratio of likelihoods q_{\mu} = -2\ln[\mathcal{L}(\mu,\hat{\hat{\nu}}(\mu))/\mathcal{L}(\hat{\mu},\hat{\nu})] , in which the nuisance parameters are profiled separately for \mu=\hat{\mu} and \mu. The value of q_{\mu} is set to 0 when \hat{\mu}>\mu, giving a one-sided limit. Furthermore, the constraint \mu>0 is enforced in the fit. This means that if the unconstrained value of \hat{\mu} would be negative, the test statistic q_{\mu} is evaluated as -2\ln[\mathcal{L}(\mu,\hat{\hat{\nu}}(\mu))/\mathcal{L}(0,\hat{\hat{\nu}}(0))]

This method is the default Combine method: if you call Combine without specifying -M, the AsymptoticLimits method will be run.

A realistic example of a datacard for a counting experiment can be found in the HiggsCombination package: data/tutorials/counting/realistic-counting-experiment.txt

@@ -334,7 +334,7 @@

Splitting points

combine -M AsymptoticLimits realistic-counting-experiment.txt --getLimitFromGrid limits.root

Asymptotic Significances

-

The significance of a result is calculated using a ratio of profiled likelihoods, one in which the signal strength is set to 0 and the other in which it is free to float. The evaluated quantity is -2\ln[\mathcal{L}(\textrm{data}|r=0,\hat{\theta}_{0})/\mathcal{L}(\textrm{data}|r=\hat{r},\hat{\theta})], in which the nuisance parameters are profiled separately for r=\hat{r} and r=0.

+

The significance of a result is calculated using a ratio of profiled likelihoods, one in which the signal strength is set to 0 and the other in which it is free to float. The evaluated quantity is -2\ln[\mathcal{L}(\mu=0,\hat{\hat{\nu}}(0))/\mathcal{L}(\hat{\mu},\hat{\nu})], in which the nuisance parameters are profiled separately for \mu=\hat{\mu} and \mu=0.

The distribution of this test statistic can be determined using Wilks' theorem provided the number of events is large enough (i.e in the Asymptotic limit). The significance (or p-value) can therefore be calculated very quickly. The Significance method can be used for this.

It is also possible to calculate the ratio of likelihoods between the freely floating signal strength to that of a fixed signal strength other than 0, by specifying it with the option --signalForSignificance=X.

@@ -505,29 +505,29 @@

Multidimensional bayesian cr

Computing Limits with toys

The HybridNew method is used to compute either the hybrid bayesian-frequentist limits, popularly known as "CLs of LEP or Tevatron type", or the fully frequentist limits, which are the current recommended method by the LHC Higgs Combination Group. Note that these methods can be resource intensive for complex models.

It is possible to define the criterion used for setting limits using --rule CLs (to use the CLs criterion) or --rule CLsplusb (to calculate the limit using p_{\mu}) and as always the confidence level desired using --cl=X.

-

The choice of test statistic can be made via the option --testStat. Different methodologies for the treatment of the nuisance parameters are available. While it is possible to mix different test statistics with different nuisance parameter treatments, we strongly do not recommend this. Instead one should follow one of the following three procedures,

+

The choice of test statistic can be made via the option --testStat. Different methodologies for the treatment of the nuisance parameters are available. While it is possible to mix different test statistics with different nuisance parameter treatments, we strongly do not recommend this. Instead one should follow one of the following three procedures. Note that the signal strength r here is given the more common notation \mu.

  • LEP-style: --testStat LEP --generateNuisances=1 --fitNuisances=0

      -
    • The test statistic is defined using the ratio of likelihoods q_{\mathrm{LEP}}=-2\ln[\mathcal{L}(\mathrm{data}|r=0)/\mathcal{L}(\mathrm{data}|r)].
      • +
    • The test statistic is defined using the ratio of likelihoods q_{\mathrm{LEP}}=-2\ln[\mathcal{L}(\mu=0)/\mathcal{L}(\mu)].
    • The nuisance parameters are fixed to their nominal values for the purpose of evaluating the likelihood, while for generating toys, the nuisance parameters are first randomized within their PDFs before generation of the toy.

  • TEV-style: --testStat TEV --generateNuisances=0 --generateExternalMeasurements=1 --fitNuisances=1

      -
    • The test statistic is defined using the ratio of likelihoods q_{\mathrm{TEV}}=-2\ln[\mathcal{L}(\mathrm{data}|r=0,\hat{\theta}_{0})/\mathcal{L}(\mathrm{data}|r,\hat{\theta}_{r})], in which the nuisance parameters are profiled separately for r=0 and r.
      • -
    • For the purposes of toy generation, the nuisance parameters are fixed to their post-fit values from the data (conditional on r), while the constraint terms are randomized for the evaluation of the likelihood.
      • +
    • The test statistic is defined using the ratio of likelihoods q_{\mathrm{TEV}}=-2\ln[\mathcal{L}(\mu=0,\hat{\hat{\mu}}(0))/\mathcal{L}(\mu,\hat{\hat{\nu}}(\mu))], in which the nuisance parameters are profiled separately for \mu=0 and \mu.
      • +
    • For the purposes of toy generation, the nuisance parameters are fixed to their post-fit values from the data (conditional on \mu), while the constraint terms are randomized for the evaluation of the likelihood.

  • LHC-style: --LHCmode LHC-limits , which is the shortcut for --testStat LHC --generateNuisances=0 --generateExternalMeasurements=1 --fitNuisances=1

      -
    • The test statistic is defined using the ratio of likelihoods q_{r} = -2\ln[\mathcal{L}(\mathrm{data}|r,\hat{\theta}_{r})/\mathcal{L}(\mathrm{data}|r=\hat{r},\hat{\theta})] , in which the nuisance parameters are profiled separately for r=\hat{r} and r.
      • -
    • The value of q_{r} set to 0 when \hat{r}>r giving a one-sided limit. Furthermore, the constraint r>0 is enforced in the fit. This means that if the unconstrained value of \hat{r} would be negative, the test statistic q_{r} is evaluated as -2\ln[\mathcal{L}(\mathrm{data}|r,\hat{\theta}_{r})/\mathcal{L}(\mathrm{data}|r=0,\hat{\theta}_{0})]
      • -
    • For the purposes of toy generation, the nuisance parameters are fixed to their post-fit values from the data (conditionally on the value of r), while the constraint terms are randomized in the evaluation of the likelihood.
      • +
    • The test statistic is defined using the ratio of likelihoods q_{\mu} = -2\ln[\mathcal{L}(\mu,\hat{\hat{\nu}}(\mu))/\mathcal{L}(\hat{\mu},\hat{\nu})] , in which the nuisance parameters are profiled separately for mu=\hat{\mu} and \mu.
      • +
    • The value of q_{\mu} set to 0 when \hat{\mu}>\mu giving a one-sided limit. Furthermore, the constraint \mu>0 is enforced in the fit. This means that if the unconstrained value of \hat{\mu} would be negative, the test statistic q_{\mu} is evaluated as -2\ln[\mathcal{L}(\mu,\hat{\hat{\nu}}(\mu))/\mathcal{L}(0,\hat{\hat{\nu}}(0))].
      • +
    • For the purposes of toy generation, the nuisance parameters are fixed to their post-fit values from the data (conditionally on the value of \mu), while the constraint terms are randomized in the evaluation of the likelihood.
@@ -777,8 +777,8 @@

Computing Significances with toys

  • LHC-style: --LHCmode LHC-significance , which is the shortcut for --testStat LHC --generateNuisances=0 --generateExternalMeasurements=1 --fitNuisances=1 --significance
      -
    • The test statistic is defined using the ratio of likelihoods q_{0} = -2\ln[\mathcal{L}(\textrm{data}|r=0,\hat{\theta}_{0})/\mathcal{L}(\textrm{data}|r=\hat{r},\hat{\theta})], in which the nuisance parameters are profiled separately for r=\hat{r} and r=0.
      • -
    • The value of the test statistic is set to 0 when \hat{r}<0
      • +
    • The test statistic is defined using the ratio of likelihoods q_{0} = -2\ln[\mathcal{L}(\mu=0,\hat{\hat{\nu}}(0))/\mathcal{L}(\hat{\mu},\hat{\nu})], in which the nuisance parameters are profiled separately for \mu=\hat{\mu} and \mu=0.
      • +
    • The value of the test statistic is set to 0 when \hat{\mu}<0
    • For the purposes of toy generation, the nuisance parameters are fixed to their post-fit values from the data assuming no signal, while the constraint terms are randomized for the evaluation of the likelihood.
    • @@ -1044,10 +1044,10 @@

    Using best fit snapshots

    Now we can load the best fit \hat{r},\hat{m}_{H} and fit for r freezing m_{H} and lumi_8TeV to their best-fit values,

    combine -m 123 -M MultiDimFit -d higgsCombineteststep1.MultiDimFit.mH123.root -w w --snapshotName "MultiDimFit" -n teststep2  --verbose 9 --freezeParameters MH,lumi_8TeV
    -

    Feldman Cousins

    +

    Feldman-Cousins

    The Feldman-Cousins (FC) procedure for computing confidence intervals for a generic model is,

      -
    • use the profile likelihood ratio as the test statistic, q(x) = - 2 \ln \mathcal{L}(\mathrm{data}|x,\hat{\theta}_{x})/\mathcal{L}(\mathrm{data}|\hat{x},\hat{\theta}) where x is a point in the (N-dimensional) parameter space, and \hat{x} is the point corresponding to the best fit. In this test statistic, the nuisance parameters are profiled, both in the numerator and denominator.
      • +
    • use the profile likelihood ratio as the test statistic, q(x) = - 2 \ln \mathcal{L}(x,\hat{\hat{\nu}}(x))/\mathcal{L}(\hat{x},\hat{\nu}) where x is a point in the (N-dimensional) parameter space, and \hat{x} is the point corresponding to the best fit. In this test statistic, the nuisance parameters are profiled, both in the numerator and denominator.
    • for each point x:
      • compute the observed test statistic q_{\mathrm{obs}}(x)
      • compute the expected distribution of q(x) under the hypothesis of x as the true value.
        • @@ -1069,9 +1069,9 @@

        Physical boundaries

        --setParameterRanges param1=param1_min,param1_max:param2=param2_min,param2_max ....

        The boundary is imposed by restricting the parameter range(s) to those set by the user, in the fits. Note that this is a trick! The actual fitted value, as one of an ensemble of outcomes, can fall outside of the allowed region, while the boundary should be imposed on the physical parameter. The effect of restricting the parameter value in the fit is such that the test statistic is modified as follows ;

        -

        q(x) = - 2 \ln \mathcal{L}(\mathrm{data}|x,\hat{\theta}_{x})/\mathcal{L}(\mathrm{data}|\hat{x},\hat{\theta}), if \hat{x} in contained in the bounded range

        +

        q(x) = - 2 \ln \mathcal{L}(x,\hat{\hat{\theta}}(x))/\mathcal{L}(\hat{x},\hat{\nu}), if \hat{x} in contained in the bounded range

        and,

        -

        q(x) = - 2 \ln \mathcal{L}(\mathrm{data}|x,\hat{\theta}_{x})/\mathcal{L}(\mathrm{data}|x_{B},\hat{\theta}_{B}), if \hat{x} is outside of the bounded range. Here x_{B} and \hat{\theta}_{B} are the values of x and \theta which maximise the likelihood excluding values outside of the bounded region for x - typically, x_{B} will be found at one of the boundaries which is imposed. For example, if the boundary x>0 is imposed, you will typically expect x_{B}=0, when \hat{x}\leq 0, and x_{B}=\hat{x} otherewise.

        +

        q(x) = - 2 \ln \mathcal{L}(x,\hat{\hat{\nu}}(x))/\mathcal{L}(x_{B},\hat{\hat{\nu}}(x_{B})), if \hat{x} is outside of the bounded range. Here x_{B} and \hat{\hat{\nu}}(x_{B}) are the values of x and \nu which maximise the likelihood excluding values outside of the bounded region for x - typically, x_{B} will be found at one of the boundaries which is imposed. For example, if the boundary x>0 is imposed, you will typically expect x_{B}=0, when \hat{x}\leq 0, and x_{B}=\hat{x} otherewise.

        This can sometimes be an issue as Minuit may not know if has successfully converged when the minimum lies outside of that range. If there is no upper/lower boundary, just set that value to something far from the region of interest.

        Info

        diff --git a/part3/nonstandard/index.html b/part3/nonstandard/index.html index f03faf7f89a..0b7d7710429 100644 --- a/part3/nonstandard/index.html +++ b/part3/nonstandard/index.html @@ -367,7 +367,7 @@

        Pre- and post-fit nuisance paramet

        The script has several options to toggle the thresholds used to decide whether a parameter has changed significantly, to get the printout of the absolute value of the nuisance parameters, and to get the output in another format for use on a webpage or in a note (the supported formats are html, latex, twiki). To print all of the parameters, use the option --all.

        By default, the changes in the nuisance parameter values and uncertainties are given relative to their initial (pre-fit) values (usually relative to initial values of 0 and 1 for most nuisance types).

        -

        The values in the output will be (\theta-\theta_{I})/\sigma_{I} if the nuisance has a pre-fit uncertainty, otherwise they will be \theta-\theta_{I} (for example, a flatParam has no pre-fit uncertainty).

        +

        The values in the output will be (\nu-\nu_{I})/\sigma_{I} if the nuisance has a pre-fit uncertainty, otherwise they will be \nu-\nu_{I} (for example, a flatParam has no pre-fit uncertainty).

        The reported uncertainty will be the ratio \sigma/\sigma_{I} - i.e the ratio of the post-fit to the pre-fit uncertainty. If there is no pre-fit uncertainty (as for flatParam nuisances), the post-fit uncertainty is shown.

        To print the pre-fit and post-fit values and (asymmetric) uncertainties, rather than the ratios, the option --abs can be used.

        @@ -380,20 +380,20 @@

        Pre- and post-fit nuisance paramet

        relDiffAsymErrs: This is the same as the default output of the tool, except that only constrained parameters (i.e. where the pre-fit uncertainty is defined) are reported. The uncertainty is also reported and calculated as \sigma/\sigma_{I}.

      • -

        unconstPullAsym: Report the pull as \frac{\theta-\theta_{I}}{\sigma}, where \theta_{I} and \sigma are the initial value and post-fit uncertainty of that nuisance parameter. The pull defined in this way will have no error bar, but all nuisance parameters will have a result in this case.

        +

        unconstPullAsym: Report the pull as \frac{\nu-\nu_{I}}{\sigma}, where \nu_{I} and \sigma are the initial value and post-fit uncertainty of that nuisance parameter. The pull defined in this way will have no error bar, but all nuisance parameters will have a result in this case.

      • -

        compatAsym: The pull is defined as \frac{\theta-\theta_{D}}{\sqrt{\sigma^{2}+\sigma_{D}^{2}}}, where \theta_{D} and \sigma_{D} are calculated as \sigma_{D} = (\frac{1}{\sigma^{2}} - \frac{1}{\sigma_{I}^{2}})^{-1} and \theta_{D} = \sigma_{D}(\theta - \frac{\theta_{I}}{\sigma_{I}^{2}}). In this expression \theta_{I} and \sigma_{I} are the initial value and uncertainty of that nuisance parameter. This can be thought of as a compatibility between the initial measurement (prior) and an imagined measurement where only the data (with no constraint on the nuisance parameter) is used to measure the nuisance parameter. There is no error bar associated with this value.

        +

        compatAsym: The pull is defined as \frac{\nu-\nu_{D}}{\sqrt{\sigma^{2}+\sigma_{D}^{2}}}, where \nu_{D} and \sigma_{D} are calculated as \sigma_{D} = (\frac{1}{\sigma^{2}} - \frac{1}{\sigma_{I}^{2}})^{-1} and \nu_{D} = \sigma_{D}(\nu - \frac{\nu_{I}}{\sigma_{I}^{2}}). In this expression \nu_{I} and \sigma_{I} are the initial value and uncertainty of that nuisance parameter. This can be thought of as a compatibility between the initial measurement (prior) and an imagined measurement where only the data (with no constraint on the nuisance parameter) is used to measure the nuisance parameter. There is no error bar associated with this value.

      • -

        diffPullAsym: The pull is defined as \frac{\theta-\theta_{I}}{\sqrt{\sigma_{I}^{2}-\sigma^{2}}}, where \theta_{I} and \sigma_{I} are the pre-fit value and uncertainty (from L. Demortier and L. Lyons). If the denominator is close to 0 or the post-fit uncertainty is larger than the pre-fit (usually due to some failure in the calculation), the pull is not defined and the result will be reported as 0 +/- 999.

        +

        diffPullAsym: The pull is defined as \frac{\nu-\nu_{I}}{\sqrt{\sigma_{I}^{2}-\sigma^{2}}}, where \nu_{I} and \sigma_{I} are the pre-fit value and uncertainty (from L. Demortier and L. Lyons). If the denominator is close to 0 or the post-fit uncertainty is larger than the pre-fit (usually due to some failure in the calculation), the pull is not defined and the result will be reported as 0 +/- 999.

      If using --pullDef, the results for all parameters for which the pull can be calculated will be shown (i.e --all will be set to true), not just those that have moved by some metric.

      This script has the option (-g outputfile.root) to produce plots of the fitted values of the nuisance parameters and their post-fit, asymmetric uncertainties. Instead, the pulls defined using one of the options above, can be plotted using the option --pullDef X. In addition this will produce a plot showing a comparison between the post-fit and pre-fit (symmetrized) uncertainties on the nuisance parameters.

      Info

      -

      In the above options, if an asymmetric uncertainty is associated with the nuisance parameter, then the choice of which uncertainty is used in the definition of the pull will depend on the sign of \theta-\theta_{I}.

      +

      In the above options, if an asymmetric uncertainty is associated with the nuisance parameter, then the choice of which uncertainty is used in the definition of the pull will depend on the sign of \nu-\nu_{I}.

      Normalizations

      For a certain class of models, like those made from datacards for shape-based analysis, the tool can also compute and save the best fit yields of all processes to the output ROOT file. If this feature is turned on with the option --saveNormalizations, the file will also contain three RooArgSet objects norm_prefit, norm_fit_s, and norm_fit_b. These each contain one RooConstVar for each channel xxx and process yyy with name xxx/yyy and value equal to the best fit yield. You can use RooRealVar::getVal and RooRealVar::getError to estimate both the post-fit (or pre-fit) values and uncertainties of these normalizations.

      @@ -528,7 +528,7 @@

      Nuisance parameter impacts

      A plot summarizing the nuisance parameter values and impacts can be made with plotImpacts.py, 

      plotImpacts.py -i impacts.json -o impacts
      -

      The first page of the output is shown below.

      +

      The first page of the output is shown below. Note that in these figures, the nuisance parameters are labelled as \theta instead of \nu.

      The direction of the +1σ and -1σ impacts (i.e. when the NP is moved to its +1σ or -1σ values) on the POI indicates whether the parameter is correlated or anti-correlated with it.

      For models with multiple POIs, the Combine option --redefineSignalPOIs X,Y,Z... should be specified in all three of the combineTool.py -M Impacts [...] steps above. The final step will produce the impacts.json file which will contain the impacts for all the specified POIs. In the plotImpacts.py script, a particular POI can be specified with --POI X.

      @@ -546,7 +546,7 @@

      Nuisance parameter impacts

      Info

      Since combineTool accepts the usual options for combine you can also generate the impacts on an Asimov or toy dataset.

    -

    The left panel in the summary plot shows the value of (\theta-\theta_{0})/\Delta_{\theta} where \theta and \theta_{0} are the post and pre-fit values of the nuisance parameter and \Delta_{\theta} is the pre-fit uncertainty. The asymmetric error bars show the post-fit uncertainty divided by the pre-fit uncertainty meaning that parameters with error bars smaller than \pm 1 are constrained in the fit. The pull will additionally be shown. As with the diffNuisances.py script, the option --pullDef can be used (to modify the definition of the pull that is shown).

    +

    The left panel in the summary plot shows the value of (\nu-\nu_{0})/\Delta_{\nua} where \nu and \nu_{0} are the post and pre-fit values of the nuisance parameter and \Delta_{\nu} is the pre-fit uncertainty. The asymmetric error bars show the post-fit uncertainty divided by the pre-fit uncertainty meaning that parameters with error bars smaller than \pm 1 are constrained in the fit. The pull will additionally be shown. As with the diffNuisances.py script, the option --pullDef can be used (to modify the definition of the pull that is shown).

    Breakdown of uncertainties

    Often you will want to report the breakdown of your total (systematic) uncertainty on a measured parameter due to one or more groups of nuisance parameters. For example, these groups could be theory uncertainties, trigger uncertainties, ... The prodecude to do this in Combine is to sequentially freeze groups of nuisance parameters and subtract (in quadrature) from the total uncertainty. Below are the steps to do so. We will use the data/tutorials/htt/125/htt_tt.txt datacard for this.

      diff --git a/search/search_index.json b/search/search_index.json index ddb42c0e1ae..4fb68ae2837 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"indexing":"full","lang":["en"],"min_search_length":3,"prebuild_index":false,"separator":"[\\s\\-]+"},"docs":[{"location":"","text":"Introduction These pages document the RooStats / RooFit - based software tool used for statistical analysis within the CMS experiment - Combine . Note that while this tool was originally developed in the Higgs PAG , its usage is now widespread within CMS. Combine provides a command-line interface to many different statistical techniques, available inside RooFit/RooStats, that are used widely inside CMS. The package exists on GitHub under https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit For more information about Git, GitHub and its usage in CMS, see http://cms-sw.github.io/cmssw/faq.html The code can be checked out from GitHub and compiled on top of a CMSSW release that includes a recent RooFit/RooStats Installation instructions Installation instructions and recommended versions can be found below. Since v9.0.0, the versioning follows the semantic versioning 2.0.0 standard . Earlier versions are not guaranteed to follow the standard. Within CMSSW (recommended for CMS users) The instructions below are for installation within a CMSSW environment. For end users that do not need to commit or do any development, the following recipes should be sufficient. To choose a release version, you can find the latest releases on github under https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit/releases Combine v9 - recommended version The nominal installation method is inside CMSSW. The current release targets the CMSSW 11_3_X series because this release has both python2 and python3 ROOT bindings, allowing a more gradual migration of user code to python3. Combine is fully python3-compatible and, with some adaptations, can also work in 12_X releases. CMSSW 11_3_X runs on slc7, which can be setup using apptainer ( see detailed instructions ): cmssw-el7 cmsrel CMSSW_11_3_4 cd CMSSW_11_3_4/src cmsenv git clone https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit.git HiggsAnalysis/CombinedLimit cd HiggsAnalysis/CombinedLimit Update to a recommended tag - currently the recommended tag is v9.1.0 : see release notes cd $CMSSW_BASE/src/HiggsAnalysis/CombinedLimit git fetch origin git checkout v9.1.0 scramv1 b clean; scramv1 b # always make a clean build Combine v8: CMSSW_10_2_X release series Setting up the environment (once): cmssw-el7 cmsrel CMSSW_10_2_13 cd CMSSW_10_2_13/src cmsenv git clone https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit.git HiggsAnalysis/CombinedLimit cd HiggsAnalysis/CombinedLimit Update to a recommended tag - currently the recommended tag is v8.2.0 : see release notes cd $CMSSW_BASE/src/HiggsAnalysis/CombinedLimit git fetch origin git checkout v8.2.0 scramv1 b clean; scramv1 b # always make a clean build SLC6/CC7 release CMSSW_8_1_X Setting up OS using apptainer ( see detailed instructions ): # For CC7: cmssw-el7 # For SLC6: cmssw-el6 cmsrel CMSSW_8_1_0 cd CMSSW_8_1_0/src cmsenv git clone https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit.git HiggsAnalysis/CombinedLimit cd HiggsAnalysis/CombinedLimit Update to a recommended tag - currently the recommended tag for CMSSW_8_1_X is v7.0.13 : cd $CMSSW_BASE/src/HiggsAnalysis/CombinedLimit git fetch origin git checkout v7.0.13 scramv1 b clean; scramv1 b # always make a clean build Oustide of CMSSW (recommended for non-CMS users) Pre-compiled versions of the tool are available as containers from the CMS cloud pages . These containers can be downloaded and run using Docker . If you have docker running you can pull and run the latest version using, docker run --name combine -it gitlab-registry.cern.ch/cms-cloud/combine-standalone:latest You will now have the compiled combine binary available as well as the complete package of tool. The container can be re-started using docker start -i combine . Standalone compilation The standalone version can be easily compiled using cvmfs as it relies on dependencies that are already installed at /cvmfs/cms.cern.ch/ . Access to /cvmfs/cms.cern.ch/ can be obtained from lxplus machines or via CernVM . See CernVM for further details on the latter. In case you do not want to use the cvmfs area, you will need to adapt the locations of the dependencies listed in both the Makefile and env_standalone.sh files. git clone https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit.git HiggsAnalysis/CombinedLimit cd HiggsAnalysis/CombinedLimit/ # git checkout . env_standalone.sh make -j 4 You will need to source env_standalone.sh each time you want to use the package, or add it to your login environment. Standalone compilation with LCG For compilation outside of CMSSW, for example to use ROOT versions not yet available in CMSSW, one can compile against LCG releases. The current default is to compile with LCG_102, which contains ROOT 6.26: git clone https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit.git HiggsAnalysis/CombinedLimit cd HiggsAnalysis/CombinedLimit source env_lcg.sh make LCG=1 -j 8 To change the LCG version, edit env_lcg.sh . The resulting binaries can be moved for use in a batch job if the following files are included in the job tarball: tar -zcf Combine_LCG_env.tar.gz build interface src/classes.h --exclude=obj Standalone compilation with conda This recipe will work both for linux and MacOS git clone https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit.git HiggsAnalysis/CombinedLimit cd HiggsAnalysis/CombinedLimit conda install --name base mamba # faster conda mamba env create -f conda_env.yml conda activate combine source set_conda_env_vars.sh # Need to reactivate conda deactivate conda activate combine make CONDA=1 -j 8 Using Combine from then on should only require sourcing the conda environment conda activate combine Note: on OS X, Combine can only accept workspaces, so run text2workspace.py first. This is due to an issue with child processes and LD_LIBRARY_PATH (see note in Makefile) Standalone compilation with CernVM combine , either standalone or not, can be compiled via CVMFS using access to /cvmfs/cms.cern.ch/ obtained using a virtual machine - CernVM . To use CernVM You should have access to CERN IT resources. If you are a CERN user you can use your account, otherwise you can request a lightweight account. If you have a CERN user account, we strongly suggest you simply run one of the other standalone installations, which are simpler and faster than using a VM. You should have a working VM on your local machine, compatible with CernVM, such as VirtualBox . All the required software can be downloaded here . At least 2GB of disk space should be reserved on the virtual machine for combine to work properly and the machine must be contextualized to add the `CMS`` group to CVMFS. A minimal working setup is described below. Download the CernVM-launcher for your operating system, following the instructions available [ here ] for your operating system (https://cernvm.readthedocs.io/en/stable/cpt-launch.html#installation Prepare a CMS context. You can use the CMS open data one already available on gitHub: wget https://raw.githubusercontent.com/cernvm/public-contexts/master/cms-opendata-2011.context) Launch the virtual machine cernvm-launch create --name combine --cpus 2 cms-opendata-2011.context In the VM, proceed with an installation of combine Installation through CernVM is maintained on a best-effort basis and these instructions may not be up to date. What has changed between tags? You can generate a diff of any two tags (eg for v9.1.0 and v9.0.0 ) by using the following url: https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit/compare/v9.0.0...v9.1.0 Replace the tag names in the url to any tags you would like to compare. For developers We use the Fork and Pull model for development: each user creates a copy of the repository on GitHub, commits their requests there, and then sends pull requests for the administrators to merge. Prerequisites Register on GitHub, as needed anyway for CMSSW development: http://cms-sw.github.io/cmssw/faq.html Register your SSH key on GitHub: https://help.github.com/articles/generating-ssh-keys Fork the repository to create your copy of it: https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit/fork (more documentation at https://help.github.com/articles/fork-a-repo ) You will now be able to browse your fork of the repository from https://github.com/your-github-user-name/HiggsAnalysis-CombinedLimit We strongly encourage you to contribute any developments you make back to the main repository. See contributing.md for details about contributing. CombineHarvester/CombineTools CombineTools is an additional tool for submitting Combine jobs to batch systems or crab, which was originally developed in the context of Higgs to tau tau analyses. Since the repository contains a certain amount of analysis-specific code, the following scripts can be used to clone it with a sparse checkout for just the core CombineHarvester/CombineTools subpackage, speeding up the checkout and compile times: git clone via ssh: bash <(curl -s https://raw.githubusercontent.com/cms-analysis/CombineHarvester/main/CombineTools/scripts/sparse-checkout-ssh.sh) git clone via https: bash <(curl -s https://raw.githubusercontent.com/cms-analysis/CombineHarvester/main/CombineTools/scripts/sparse-checkout-https.sh) make sure to run scram to compile the CombineTools package. See the CombineHarvester documentation pages for more details on using this tool and additional features available in the full package.","title":"Home"},{"location":"#introduction","text":"These pages document the RooStats / RooFit - based software tool used for statistical analysis within the CMS experiment - Combine . Note that while this tool was originally developed in the Higgs PAG , its usage is now widespread within CMS. Combine provides a command-line interface to many different statistical techniques, available inside RooFit/RooStats, that are used widely inside CMS. The package exists on GitHub under https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit For more information about Git, GitHub and its usage in CMS, see http://cms-sw.github.io/cmssw/faq.html The code can be checked out from GitHub and compiled on top of a CMSSW release that includes a recent RooFit/RooStats","title":"Introduction"},{"location":"#installation-instructions","text":"Installation instructions and recommended versions can be found below. Since v9.0.0, the versioning follows the semantic versioning 2.0.0 standard . Earlier versions are not guaranteed to follow the standard.","title":"Installation instructions"},{"location":"#within-cmssw-recommended-for-cms-users","text":"The instructions below are for installation within a CMSSW environment. For end users that do not need to commit or do any development, the following recipes should be sufficient. To choose a release version, you can find the latest releases on github under https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit/releases","title":"Within CMSSW (recommended for CMS users)"},{"location":"#combine-v9-recommended-version","text":"The nominal installation method is inside CMSSW. The current release targets the CMSSW 11_3_X series because this release has both python2 and python3 ROOT bindings, allowing a more gradual migration of user code to python3. Combine is fully python3-compatible and, with some adaptations, can also work in 12_X releases. CMSSW 11_3_X runs on slc7, which can be setup using apptainer ( see detailed instructions ): cmssw-el7 cmsrel CMSSW_11_3_4 cd CMSSW_11_3_4/src cmsenv git clone https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit.git HiggsAnalysis/CombinedLimit cd HiggsAnalysis/CombinedLimit Update to a recommended tag - currently the recommended tag is v9.1.0 : see release notes cd $CMSSW_BASE/src/HiggsAnalysis/CombinedLimit git fetch origin git checkout v9.1.0 scramv1 b clean; scramv1 b # always make a clean build","title":"Combine v9 - recommended version"},{"location":"#combine-v8-cmssw_10_2_x-release-series","text":"Setting up the environment (once): cmssw-el7 cmsrel CMSSW_10_2_13 cd CMSSW_10_2_13/src cmsenv git clone https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit.git HiggsAnalysis/CombinedLimit cd HiggsAnalysis/CombinedLimit Update to a recommended tag - currently the recommended tag is v8.2.0 : see release notes cd $CMSSW_BASE/src/HiggsAnalysis/CombinedLimit git fetch origin git checkout v8.2.0 scramv1 b clean; scramv1 b # always make a clean build","title":"Combine v8: CMSSW_10_2_X release series"},{"location":"#slc6cc7-release-cmssw_8_1_x","text":"Setting up OS using apptainer ( see detailed instructions ): # For CC7: cmssw-el7 # For SLC6: cmssw-el6 cmsrel CMSSW_8_1_0 cd CMSSW_8_1_0/src cmsenv git clone https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit.git HiggsAnalysis/CombinedLimit cd HiggsAnalysis/CombinedLimit Update to a recommended tag - currently the recommended tag for CMSSW_8_1_X is v7.0.13 : cd $CMSSW_BASE/src/HiggsAnalysis/CombinedLimit git fetch origin git checkout v7.0.13 scramv1 b clean; scramv1 b # always make a clean build","title":"SLC6/CC7 release CMSSW_8_1_X"},{"location":"#oustide-of-cmssw-recommended-for-non-cms-users","text":"Pre-compiled versions of the tool are available as containers from the CMS cloud pages . These containers can be downloaded and run using Docker . If you have docker running you can pull and run the latest version using, docker run --name combine -it gitlab-registry.cern.ch/cms-cloud/combine-standalone:latest You will now have the compiled combine binary available as well as the complete package of tool. The container can be re-started using docker start -i combine .","title":"Oustide of CMSSW (recommended for non-CMS users)"},{"location":"#standalone-compilation","text":"The standalone version can be easily compiled using cvmfs as it relies on dependencies that are already installed at /cvmfs/cms.cern.ch/ . Access to /cvmfs/cms.cern.ch/ can be obtained from lxplus machines or via CernVM . See CernVM for further details on the latter. In case you do not want to use the cvmfs area, you will need to adapt the locations of the dependencies listed in both the Makefile and env_standalone.sh files. git clone https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit.git HiggsAnalysis/CombinedLimit cd HiggsAnalysis/CombinedLimit/ # git checkout . env_standalone.sh make -j 4 You will need to source env_standalone.sh each time you want to use the package, or add it to your login environment.","title":"Standalone compilation"},{"location":"#standalone-compilation-with-lcg","text":"For compilation outside of CMSSW, for example to use ROOT versions not yet available in CMSSW, one can compile against LCG releases. The current default is to compile with LCG_102, which contains ROOT 6.26: git clone https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit.git HiggsAnalysis/CombinedLimit cd HiggsAnalysis/CombinedLimit source env_lcg.sh make LCG=1 -j 8 To change the LCG version, edit env_lcg.sh . The resulting binaries can be moved for use in a batch job if the following files are included in the job tarball: tar -zcf Combine_LCG_env.tar.gz build interface src/classes.h --exclude=obj","title":"Standalone compilation with LCG"},{"location":"#standalone-compilation-with-conda","text":"This recipe will work both for linux and MacOS git clone https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit.git HiggsAnalysis/CombinedLimit cd HiggsAnalysis/CombinedLimit conda install --name base mamba # faster conda mamba env create -f conda_env.yml conda activate combine source set_conda_env_vars.sh # Need to reactivate conda deactivate conda activate combine make CONDA=1 -j 8 Using Combine from then on should only require sourcing the conda environment conda activate combine Note: on OS X, Combine can only accept workspaces, so run text2workspace.py first. This is due to an issue with child processes and LD_LIBRARY_PATH (see note in Makefile)","title":"Standalone compilation with conda"},{"location":"#standalone-compilation-with-cernvm","text":"combine , either standalone or not, can be compiled via CVMFS using access to /cvmfs/cms.cern.ch/ obtained using a virtual machine - CernVM . To use CernVM You should have access to CERN IT resources. If you are a CERN user you can use your account, otherwise you can request a lightweight account. If you have a CERN user account, we strongly suggest you simply run one of the other standalone installations, which are simpler and faster than using a VM. You should have a working VM on your local machine, compatible with CernVM, such as VirtualBox . All the required software can be downloaded here . At least 2GB of disk space should be reserved on the virtual machine for combine to work properly and the machine must be contextualized to add the `CMS`` group to CVMFS. A minimal working setup is described below. Download the CernVM-launcher for your operating system, following the instructions available [ here ] for your operating system (https://cernvm.readthedocs.io/en/stable/cpt-launch.html#installation Prepare a CMS context. You can use the CMS open data one already available on gitHub: wget https://raw.githubusercontent.com/cernvm/public-contexts/master/cms-opendata-2011.context) Launch the virtual machine cernvm-launch create --name combine --cpus 2 cms-opendata-2011.context In the VM, proceed with an installation of combine Installation through CernVM is maintained on a best-effort basis and these instructions may not be up to date.","title":"Standalone compilation with CernVM"},{"location":"#what-has-changed-between-tags","text":"You can generate a diff of any two tags (eg for v9.1.0 and v9.0.0 ) by using the following url: https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit/compare/v9.0.0...v9.1.0 Replace the tag names in the url to any tags you would like to compare.","title":"What has changed between tags?"},{"location":"#for-developers","text":"We use the Fork and Pull model for development: each user creates a copy of the repository on GitHub, commits their requests there, and then sends pull requests for the administrators to merge. Prerequisites Register on GitHub, as needed anyway for CMSSW development: http://cms-sw.github.io/cmssw/faq.html Register your SSH key on GitHub: https://help.github.com/articles/generating-ssh-keys Fork the repository to create your copy of it: https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit/fork (more documentation at https://help.github.com/articles/fork-a-repo ) You will now be able to browse your fork of the repository from https://github.com/your-github-user-name/HiggsAnalysis-CombinedLimit We strongly encourage you to contribute any developments you make back to the main repository. See contributing.md for details about contributing.","title":"For developers"},{"location":"#combineharvestercombinetools","text":"CombineTools is an additional tool for submitting Combine jobs to batch systems or crab, which was originally developed in the context of Higgs to tau tau analyses. Since the repository contains a certain amount of analysis-specific code, the following scripts can be used to clone it with a sparse checkout for just the core CombineHarvester/CombineTools subpackage, speeding up the checkout and compile times: git clone via ssh: bash <(curl -s https://raw.githubusercontent.com/cms-analysis/CombineHarvester/main/CombineTools/scripts/sparse-checkout-ssh.sh) git clone via https: bash <(curl -s https://raw.githubusercontent.com/cms-analysis/CombineHarvester/main/CombineTools/scripts/sparse-checkout-https.sh) make sure to run scram to compile the CombineTools package. See the CombineHarvester documentation pages for more details on using this tool and additional features available in the full package.","title":"CombineHarvester/CombineTools"},{"location":"CernVM/","text":"Standalone use inside CernVM Standalone by adding the CMS group to the CVMFS Configuration. A minimal CernVM working context setup can be found in the CernVM Marketplace under Experimental/HiggsCombine or at https://cernvm-online.cern.ch/context/view/9ee5960ce4b143f5829e72bbbb26d382 . At least 2GB of disk space should be reserved on the virtual machine for Combine to work properly. Available machines for standalone combine The standalone version can be easily compiled via CVMFS as it relies on dependencies which are already installed at /cvmfs/cms.cern.ch/ . Access to /cvmfs/cms.cern.ch/ can be obtained from lxplus machines or via CernVM . The only requirement will be to add the CMS group to the CVMFS configuration as shown in the picture At least 2GB of disk space should be reserved on the virtual machine for combine to work properly. A minimal CernVM working context setup can be found in the CernVM Marketplace under Experimental/HiggsCombine . To use this predefined context, first locally launch the CernVM (eg you can use the .ova with VirtualBox, by downloading from here and launching the downloaded file. You can click on \"pair an instance of CernVM\" from the cernvm-online dashboard, which displays a PIN. In the VirtualBox terminal, pair the virtual machine with this PIN code (enter in the terminal using #PIN eg #123456 . After this, you will be asked again for username (use user ) and then a password (use hcomb ). In case you do not want to use the cvmfs area, you will need to adapt the location of the dependencies listed in both the Makefile and env_standalone.sh files.","title":"CernVM"},{"location":"CernVM/#standalone-use-inside-cernvm","text":"Standalone by adding the CMS group to the CVMFS Configuration. A minimal CernVM working context setup can be found in the CernVM Marketplace under Experimental/HiggsCombine or at https://cernvm-online.cern.ch/context/view/9ee5960ce4b143f5829e72bbbb26d382 . At least 2GB of disk space should be reserved on the virtual machine for Combine to work properly.","title":"Standalone use inside CernVM"},{"location":"CernVM/#available-machines-for-standalone-combine","text":"The standalone version can be easily compiled via CVMFS as it relies on dependencies which are already installed at /cvmfs/cms.cern.ch/ . Access to /cvmfs/cms.cern.ch/ can be obtained from lxplus machines or via CernVM . The only requirement will be to add the CMS group to the CVMFS configuration as shown in the picture At least 2GB of disk space should be reserved on the virtual machine for combine to work properly. A minimal CernVM working context setup can be found in the CernVM Marketplace under Experimental/HiggsCombine . To use this predefined context, first locally launch the CernVM (eg you can use the .ova with VirtualBox, by downloading from here and launching the downloaded file. You can click on \"pair an instance of CernVM\" from the cernvm-online dashboard, which displays a PIN. In the VirtualBox terminal, pair the virtual machine with this PIN code (enter in the terminal using #PIN eg #123456 . After this, you will be asked again for username (use user ) and then a password (use hcomb ). In case you do not want to use the cvmfs area, you will need to adapt the location of the dependencies listed in both the Makefile and env_standalone.sh files.","title":"Available machines for standalone combine"},{"location":"full-documentation/","text":"Full documentation Common options Choice of the prior for Bayesian methods For methods based on Bayesian inference (BayesianSimple, MarkovChainMC) you can specify a prior for the signal strength. This is controlled through option --prior , and the values that are currently allowed are: uniform : flat prior 1/sqrt(r) : prior proportional to 1/sqrt(r) , where r is the signal strength; for a counting experiment with a single channel and no systematics, this is Jeffrey's prior. Note that you might have to put the 1/sqrt(r) within quotes because for some shells the braces have a special meaning. If no prior is specified, a flat prior is assumed. Algorithm-specific options #ProfiLe ProfileLikelihood algorithm The ProfileLikelihood algorithm has only two options besides the common ones: the choice of the minimizer algorith (Minuit2 or Minuit), and the choice of the tolerance. If you see that the limit fails with one of the minimizer, try the other. Sometimes this problem happens due to numerical instabilities even if the model itself looks perfectly well behaved. If neither of the two succeeds, another possible way of circumventing the instability is sometimes to change the range of r by a small amount, or to change slightly one number in the model (e.g. in one simple counting experiment a test was failing when assuming \u0394B/B = 0.3, but was succeeding for \u0394B/B = 0.301 and \u0394B/B = 0.299, giving almost the same result) In case you experience numerical instabilities, e.g. failures or bogus results, you could be able to work around the problem by performing the minimization multiple times and requiring the results to be consistent. This can be done using the options below. maxTries : how many times the program should attempt to find a minimum at most (plausible range: 20-100) tries : how many times the program should succeed in computing a limit (plausible range: 5-10) maxOutlierFraction : maximum fraction of bad results within the tries ones (plausible: 0.15-0.30; default 0.25) maxOutliers : maximum number of bogus results before giving up for this point (plausible values: 2-5; default 3) preFit : don't try to profile the likelihood if Minuit already gave errors in finding the minimum. (suggested) BayesianSimple algorithm The BayesianSimple algorithm doesn't have parameters besides the ones specified above under the \"Common options\" section. #MarkoV MarkovChainMC algorithm This algorithm has many configurable parameters, and you're encouraged to experiment with those because the default configuration might not be the best for you (or might not even work for you at all) *Iterations, burn-in, tries* Three parameters control how the MCMC integration is performed: the number of tries (option --tries ): the algorithm will run multiple times with different ransom seeds and report as result the truncated mean and rms of the different results. The default value is 10, which should be ok for a quick computation, but for something more accurate you might want to increase this number even up to ~200. the number of iterations (option -i ) determines how many points are proposed to fill a single Markov Chain. The default value is 10k, and a plausible range is between 5k (for quick checks) and 20-30k for lengthy calculations. Usually beyond 30k you get a better tradeoff in time vs accuracy by increasing the number of chains (option tries ) the number of burn-in steps (option -b ) is the number of points that are removed from the beginning of the chain before using it to compute the limit. IThe default is 200. If your chain is very long, you might want to try increase this a bit (e.g. to some hundreds). Instead going below 50 is probably dangerous. Proposal The option --proposal controls the way new points are proposed to fill in the MC chain. uniform : pick points at random. This works well if you have very few nuisance parameters (or none at all), but normally fails if you have many. gaus : Use a product of independent gaussians one for each nuisance parameter; the sigma of the gaussian for each variable is 1/5 of the range of the variable (this can be controlled using the parameter propHelperWidthRangeDivisor ). This proposal appears to work well for a reasonable number of nuisances (up to ~15), provided that the range of the nuisance parameters is reasonable, like \u00b15\u03c3. It does not work without systematics. =ortho ( default ): (previously known as test ) This proposalis similar to the multi-gaussian proposal but at every step only a single coordinate of the point is varied, so that the acceptance of the chain is high even for a large number of nuisances (i.e. more than 20). fit : Run a fit and use the uncertainty matrix from HESSE to construct a proposal (or the one from MINOS if the option --runMinos is specified). This sometimes work fine, but sometimes gives biased results, so we don't recommend it in general. If you believe there's something going wrong, e.g. if your chain remains stuck after accepting only a few events, the option debugProposal can be used to have a printout of the first N proposed points to see what's going on (e.g. if you have some region of the phase space with probability zero, the gaus and fit proposal can get stuck there forever) #HybridNew HybridNew algorithm Type of limit By default, HybridNew computes hybrid Bayesian-frequentist limits. If one specifies the command line option --freqentist then it will instead compute the fully frequentist limits. Rule (option --rule ) The rule defines how the distribution of the test statistics is used to compute a limit. When using the CLs+b rule the limit to the value of the signal strength for which 95% of the pseudo-experiments give a result more signal-like than the current one, P(x < xobs|r*s+b) = 1 - CL . For the more conservative CLs rule, the limit is set by the condition P(x < xobs|r*s+b) /P(x < xobs|b) = 1 - CL . The default rule is CLs . Test statistics (option --testStat ) The test statistics is the measure of how signal-like or background-like is an observation. The following test statistics are provided: Simple Likelihood Ratio (option value LEP or SLR ): The LEP-like ratio of likelihoods ( L(x|r*s+b,\u03b8) / L(x|b, \u03b8) ) where numerator and denominator are evaluated for the reference values of the nuisance parameters \u03b8. Ratio of Profiled Likelihoods (option value TEV or ROPL ): The Tevatron-like ratio of profiled likelihoods, in which before computing each of the likelihoods is maximised as function of the nuisance parameters ( max\u03b8 L(x|r*s+b,\u03b8) / max\u03b8 L(x|b, \u03b8) ). Profile Likelihood modified for upper limits (option value LHC or MPL ): The LHC-like (or Atlas-like) profiled likelihood in which the maximization of the likelihood is done also in the signal strength ( max\u03b8 L(x|r*s+b,\u03b8) / maxr', \u03b8 L(x|r'*s+b,\u03b8) ), with the constraints 0 \u2264 r' \u2264 r where the upper bound is applied to force the method to always give an upper limit and not a two-sided interval. Profile Likelihood (not modified for upper limits) (option value PL ): The traditional profiled likelihood in which the maximization of the likelihood is done also in the signal strength ( max\u03b8L(x|r*s+b,\u03b8) / maxr', \u03b8L(x|x|r'*s+b,\u03b8) ), with just the physical constraints 0 \u2264 r' This test statistics can give two-sided limits, as it starts decreasing when the number of observed events is above the expectations from the signal+background hypothesis. The default value when computing hybrid Bayesian-frequentist limits is LEP . The default value when computing frequentist limits is LHC . Accuracy The search for the limit is performed using an adaptive algorithm, terminating when the estimate of the limit value is below some limit or when the precision cannot be futher improved with the specified options. The options controlling this behaviour are: rAbsAcc , rRelAcc : define the accuracy on the limit at which the search stops. The default values are 0.1 and 0.05 respectively, meaning that the search is stopped when \u0394r < 0.1 or \u0394r/r < 0.05. clsAcc : this determines the absolute accuracy up to which the CLs values are computed when searching for the limit. The default is 0.5%. Raising the accuracy above this value will increase significantly the time to run the algorithm, as you need N2 more toys to improve the accuracy by a factor N; you can consider enlarging this value if you're computing limits with a larger CL (e.g. 90% or 68%) Note that if you're using the CLs+b rule then this parameter will control the uncertainty on CLs+b , not on CLs . T or toysH : controls the minimum number of toys that are generated for each point. The default value of 500 should be ok when computing the limit with 90-95% CL. You can decrease this number if you're computing limits at 68% CL, or increase it if you're using 99% CL. Computing significances When computing the significances, there is no adaptive generation. You can control the number of toys with option T or toysH= and the option iterations (shortcut -i , default 1): the default of (1 iteration)\u00d7(500 toys) is not enough to probe a significances above ~2. We suggest that you uncrease the number of iterations instead of the number of toys, since the increase in time is linear with the iterations but non-linear with the toys. In order to compute the significance in multiple jobs, proceed as follows: Run N different jobs with the same inputs but different random seed (option -s ), specifying the additional option --saveHybridResult . Use hadd to merge the output root files in a single one. The program will complain with messages like Cannot merge object type, name: HybridCalculator _result which can be safely ignored. Compute the significance from the merged file running again but with options --readHybridResults= and --toysFile= Caveat: there is no check within the program that you're using consistent inputs for the different jobs. Simple hypotesis testing : Sometimes you just want to compute the p-values of the background-only hypotesis and of the signal plus background hypotesis for a fixed value of the signal strength. This can be done specifying the option singlePoint which will set the signal strength to that value and run the hypothesis test. It will generate toys until the required accuracy is met (see above for parameter clsAcc ). You can turn off adaptive generation setting clsAcc to zero, and then it will generate the toys once (or multiple times if you set option iterations to a value larger than 1). Just like for significance, you can run multiple times with different seeds and options --saveHybridResult , combine the output files with hadd and then compute the final result with --readHybridResult --toysFile=merged.root Performance issues The current hybrid code requires a lot of cpu resources. You can speed up the processing by using multiple cores (option fork , default value is 1). Note that even with fork set to 1, toy generation is done in a separate thread to avoid memory leaks. If you want to run in a single thread, e.g. to be able to read the debug output during generation, you should set the option to zero. If running multi-threaded on the cern batch cluster, you should declare it to the bsub command when submitting the jobs: e.g. for a job that uses four cores you should use bsub -n 4 -R \"'span[hosts=1]'\" ... #HybridNewGrid HybridNew algorithm usage for complex models or expected limits: grids If your model is complex, or you need to know the limit accurately, or you want expected limits, then running the computation in a single job might not be feasible. The alternative approach is to compute a grid of distributions of the test statistics for various values of the signal strength, a task that is easy to parallelize, and then use the that grid to compute the observed limit (and also the expected ones). This requires you to have some knowledge of where the limit should be, which you can gain e.g. from the ProfileLikelihood method Creating the grid: manual way The procedure to do this manually would be like the procedure for significances or simple hypothesis testing described previously: for each value r_i of the cross section, you write out one file with the distribution of the test statistics using combine card.txt -M HybridNew [--freq] [other options] -s seed_i --singlePoint r_i --saveToys --saveHybridResult and then you can merge all the output files for the different r_i with hadd . The [other options] should include --clsAcc 0 to switch off adaptive sampling, and you can tune the CPU time by working on the parameters -T and --iterations . It is important that you use different seed_i values for each point; if you don't care about exact reproducibility, you can just use --seed -1 and the code will take care of randomizing itself properly. Creating the grid: automated way, using CRAB Please note that the following is intended for use with crab2. For producing the grid with crab3, please see the instructions here Once you have a sense of the time needed for each toy, and of the range to consider, you can use the script makeGridUsingCrab.py to run the toys to create the grid in parallel either on LXBATCH or on regular T2s (or anything else that CRAB can digest). The procedure is very simple: makeGridUsingCrab.py card.txt minimum maximum -n points [other options] -o name This will create a crab cfg file name.cfg and a script name.sh , and possibly a binary workspace name.workspace.root . You can then just create and submit the jobs from that cfg file, and merge the output rootfiles with hadd=(note: =hadd will complain with messages like Cannot merge object type, name: HybridCalculator _result which can be safely ignored). The other options, that you can get executing the program with --help are: -T : same as in combine -r : use a random seed in each job (suggested) -I n : run only on 1/n of the points in each job (suggested if you want to have many points) -t , -j : choose the total number of toys and of jobs (can change later from the crab cfg file) --lsf , --queue ... : use lxbatch with the specific queue (can change later from the crab cfg file) Note that you can merge also the output of multiple crab submissions, if you have used random seeds. Using the grid for observed limits combine mydatcard.txt -M HybridNew [--freq] --grid=mygrid.root All other parameters controlling toys, accuracy and so on are meaningless in this context. Note that it might still take a while if you have many points and the test statistics is slow to evaluate. Add the option --saveGrid to save the value of the observed CLs at each grid point in the output tree. Using the grid for expected limits combine mydatcard.txt -M HybridNew [--freq] --grid=mygrid.root --expectedFromGrid 0.5 0.5 gives you the median. use 0.16/0.84 to get the endpoints of 68% interval, 0.025/0.975 to get the 95% one). Add the option --saveGrid to save the value of the expected quantile CLs at each grid point in the output tree. Plotting the test-statistics distributions The distribution of the test-statistic under the signal plus background and background only hypotheses can be plotted at each value of the grid using the following; python test/plotTestStatCLs.py --input mygrid.root --poi r --val all --mass MASS The output root file will contain a plot for each point found in the grid. FeldmanCousins The F-C method is used to compute an interval with the specified confidence level. If you run the model without special options, it will report the upper limit to the signal strength. If you want instead the lower end of the interval, just run it with option lowerLimit . The algorithm will search for a limit with an iterative procedure until the specified absolute or relative accuracy is met, as controlled by the parameters rAbsAcc , =rRelAcc . The default values are 0.1 and 0.05 respectively, meaning that the search is stopped when \u0394r < 0.1 or \u0394r/r < 0.05. The number of toys generated is adaptive. You can increase it by a factor using option toysFactor a value of 2 or 4 is suggested if you want to compute the limit with high accuracy. Running under CRAB The instructions below are for use with crab2 . For instructions on how to use the grid for toy based studies or complicated model scans under crab3 , follow the instructions given here . Running many toy MC for the limit calculation may be conveniently split among the different available GRID resources using CRAB. Examples of how to run on the GRID via CRAB are provided in the files: [[https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit/blob/master/test/combine_crab.sh][combine_crab.sh]] [[https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit/blob/master/test/combine_crab.cfg][combine_crab.cfg]] Preparing the ROOT workspace The first thing to do is to convert the datacards and possibly the shape model into a ROOT workspace. This model will be shipped to Worker Nodes for GRID processing. This is done via the utility script text2workspace.py . For instance: ../scripts/text2workspace.py ../data/benchmarks/simple-counting/counting-B0-Obs0-StatOnly.txt -b -o model.root Shell script for GRID Worker Nodes CRAB is designed mainly to provide automatic cmsRun job splitting providing the number of jobs one wants to run, and the number of 'events' in total one wants to process.The total number of toy MC we want to run. The maximum number of events is passed to the application to be executed via the variable $MaxEvents . In our case, we will use for convenience $MaxEvents as the number of toy MC to run per job. The script [[https://github.com/cms-analysis/HiggsAnalysis-CombinedLimit/blob/master/test/combine_crab.sh][combine_crab.sh]] runs the combiner code with the proper options, and prepares the output to be retrieved after the run completion on the Worker Nodes. It takes as argument the job indes ( $1 ), which we use as random seed. The main elements there are running the combiner and packing the output for final retrieval: echo \"job number: seed # i with n toys\" ./combine model.root -t n toys\" ./combine model.root -t n -sn -s i with n toys\" ./combine model.root -t