Scaling out MadGraph5 runs across Blue Waters

[matthewfeickert]

Now that Neubauer-Group/bluewaters-docker-images#14 is resolved and I've verified that MadGraph5 is able to produce simulation files on Blue Waters without crashing the next goal is to be able to scale up the production so that simulation can be produced in a distributed manner. This would mean figuring out a way to launch multiple Shifter jobs with the neubauergroup/bluewaters-mg5_amc:3.1.0 containers and then being able to write the output back to Blue Waters storage, collate it, and move the simulation off storage for studies.

Most of this work could be started in https://github.com/scailfin/MadGraph5-simulation-configs

[matthewfeickert]

@BenGalewsky Do you have any experience with parallelizing Shifter jobs on Blue Waters? I remember that in a group meeting you had mentioned that trying to use funcX here would be problematic on Blue Waters given storage. Can you elaborate a bit more on this (I'm probably missing something obvious).

[BenGalewsky]

@BenGalewsky Do you have any experience with parallelizing Shifter jobs on Blue Waters? I remember that in a group meeting you had mentioned that trying to use funcX here would be problematic on Blue Waters given storage. Can you elaborate a bit more on this (I'm probably missing something obvious).

It would not be a problem - all of the functions would have access to your scratch directory space, so they could just write to a local file as easy as you please. I could set up an endpoint. We could then use globus to transfer them over to the beast once the job is done.

One question is the size of the arguments to the funcX function. Is there a big object to be passed?

[matthewfeickert]

It would not be a problem - all of the functions would have access to your scratch directory space, so they could just write to a local file as easy as you please. I could set up an endpoint. We could then use globus to transfer them over to the beast once the job is done.

Sounds good.

One question is the size of the arguments to the funcX function. Is there a big object to be passed?

Uh, that I'm not really 100% sure on. I've been getting things working so far and haven't yet tried to figure out what needs to get built before in the image so that is is available for Python to call later. I hope that things should be small though.

[matthewfeickert]

@BenGalewsky @msneubauer After following up on some points with the MoMEMta developers who gave me some good information

then produce all the simulated events with the toolchain and then come back and have MoMEMta use that same generation process in combination with MoMEMta-MaGMEE to have MoMEMta know what was done?

That is correct. In MoMEMta it is basically assumed that you already have your events (simulated or data), and that you need to compute weights for them.

In your case it indeed comes down to generating the process in MG5 and writing out the C++ matrix element for MoMEMta on the one hand, and generating events using the MG5 toolchain (madevent, then Pythia or Delphes or whatever), with the same original generate command in MG5, on the other.

Note that in general there is no relationship between the two: you can compute weights under any process hypothesis, not necessarily the same that was used to generate your events (in fact, when you compute weights on data, you don't know how the events were produced!). Very often the generator used is also different, i.e. you can very well compute weights under the hypothesis of leading-order ttbar production (with a matrix element coming from MoMEMta-MaGMEE), whereas the events were simulated using powheg at NLO.

I first need to produce simulated detector events that pass through the MadGraph5 -> PYTHIA 8 -> DELPHES toolchain to be able to do the the weight calculations with MoMEMta. The first part of MadGraph5 -> PYTHIA 8 I can do now, which is good. Though I have a few questions about running for long periods of time on Blue Waters.

To do a local test, on Beast I ran the following MadGraph5 configurations (using just 1000 events as a test as PYTHIA will consume all CPU it has available)

$ cat drell-yan.mg5 
generate p p > l+ l-
output drell-yan_output
launch drell-yan_output
   shower=pythia8
   set nevents 1000

which then produces both the unweighted LHE files from MadGraph (unweighted_events.lhe.gz) and the HEPMC files from PYTHIA (tag_1_pythia8_events.hepmc.gz)

$ ls -l drell-yan_output/Events/run_01/
total 69776
-rw-r--r-- 1 feickert feickert     7271 Jun 17 14:09 parton_systematics.log
-rw-r--r-- 1 feickert feickert    25436 Jun 17 14:09 run_01_tag_1_banner.txt
-rwxr--r-- 1 feickert feickert      182 Jun 17 14:09 run_shower.sh
-rw-r--r-- 1 feickert feickert   774497 Jun 17 14:09 tag_1_djrs.dat
-rw-r--r-- 1 feickert feickert   774489 Jun 17 14:09 tag_1_pts.dat
-rw-r--r-- 1 feickert feickert     5602 Jun 17 14:09 tag_1_pythia8.cmd
-rw-r--r-- 1 feickert feickert 65839804 Jun 17 14:09 tag_1_pythia8_events.hepmc.gz
-rw-r--r-- 1 feickert feickert  3014281 Jun 17 14:09 tag_1_pythia8.log
-rw-r--r-- 1 feickert feickert   982678 Jun 17 14:09 unweighted_events.lhe.gz

(So this is all going one step past Neubauer-Group/bluewaters-docker-images#14 (comment))

The HEPMC files are going to be needed in by DELPHES in the particles -> detector events stage.

I now want to be able to scale this above step up on Blue Waters to maybe 10^6 events to start with. I also need to update the current neubauergroup/bluewaters-mg5_amc:latest image to take advantage of some changes.

@BenGalewsky I'm not sure what's the best approach for Blue Waters here. Can we figure out a way to send multiple Shifter jobs to the queue simultaneously to weakly parallelize this 10^6?

I'm currently working on getting run configurations setup on https://github.com/scailfin/MadGraph5-simulation-configs

---

cc @mihirkatare just to keep them informed (nothing you need to do, just wanted you to see some progress)