Convert from Rivet analysis to YAML format or vice versa

[GraemeWatt]

This issue is an extension of #5 ("Convert from YAML format to YODA"). A Rivet analysis consists of a .cc file (code, not relevant here), a .info file (metadata for the analysis), a .plot file (plot titles and axis labels), and a .yoda file with the actual data points. The last three files overlap with HEPData content and hence conversion between the two formats would be useful, i.e. both HEPData --> Rivet and Rivet --> HEPData. For example, I noticed yesterday that a particular Rivet analysis ATLAS_2014_I1315949 does not have a corresponding HepData record. Can we write a script to convert the relevant .info, .plot and .yoda files into the YAML format suitable for HEPData submission? Conversely, given a HEPData submission in the YAML format, can we export .info, .plot and .yoda (see #5) files suitable for inclusion in a Rivet analysis?

[eamonnmag]

Nice, that would be great to have.

[michal-szostak]

Is the .yoda file containing YODA data structures? There was a slight problem which I encountered while working on conversion from YAML to YODA - I had to use the same method as in the referenced code (https://gist.github.com/GraemeWatt/2d5e1514441e4775b718) - which is basically writing YODA file format by hand - which is still fine when writing it, but parsing it is a whole different story.

The main problem is that YODA pyhon bindings are not available by PyPI (or maybe I'm missing something? @GraemeWatt any corrections on you part?), which makes them hard to list as dependencies for the hepdata-converter library.

@eamonnmag - how should we proceed with this issue?

[michal-szostak]

After our discussion yesterday I checked, and there seems to be no package for yoda in any package manager (apt / pypi / etc) the only way to use it is source compilation (./configure && make && make install) - can someone confirm? @eamonnmag @GraemeWatt ?. I know that this is out of scope of hepdata-converter library but I would strongly advocate creating debian package for yoda, if we plan to integrate yoda output for hepdata. Without it continuous development with testing will be much more tedious.

Creating scripts for travis which download yoda then decompress it and compile it (+ downloading all dependencies) can be done, but the question is whether this should be the way to approach this issue

[eamonnmag]

Agreed, but I think this is something the YODA developers should be doing. Did you have any luck with them @GraemeWatt?

[michal-szostak]

The yoda dependencies has been resolved by using docker, the conversion to yoda is mostly complete (path fixes are on the way). Can I get the RIvet sample file in the meantime, in order to fix this issue? @GraemeWatt ?

[eamonnmag]

@lukasheinrich?

[lukasheinrich]

Do you need just any rivet sample output file? If so, I'm attaching one here. Let me know if you need mode. (remove the png, github has some stupid extension filtering :-/)

[lukasheinrich]

ok that didn't work. try: http://physics.nyu.edu/~lh1132/sample_rivet.yoda

[michal-szostak]

This should be good for a start if I need more I will let you know.

[GraemeWatt]

The original idea was to convert to/from Rivet input (experimental data) not Rivet output (theoretical predictions). Sample Rivet input files are linked from the first post above: others can be found by browsing in the same directory.

By the way, @lukasheinrich is now sharing my office and will work on conversion to/from Rivet, HistFactory, etc. for the next two weeks.

[lukasheinrich]

Yeah @GraemeWatt is correct, for example this would be a link directly to the raw input for the ATLAS_2011_I919017 rivet analysis

https://rivet.hepforge.org/hg/rivet/raw-file/tip/data/refdata/ATLAS_2011_I919017.yoda

[eamonnmag]

We can convert to the YODA files. I think this is enough, no? Converting from YODA to HEPData seems unnecessary.

[GraemeWatt]

Although the HEPData record should ideally be prepared before the Rivet analysis, in practice, within the experiments it seems that a Rivet analysis is often prepared at a much earlier stage. I'm reopening this issue for the moment, although it's not a top priority. I prefer that the experiments would alter their workflow to prepare the HEPData record at an earlier stage in the analysis. But if this is not practical, then we should try to make the preparation of the HEPData record easier.