multi-case web page

[bitterbark]

Do we have a dynamic way to generate code in the html files to reflect the number of cases that the MDTF might be called with?

It looks like the example_multicase.html is set up to display a plot with a static name

"href=model/example_model_plot.png>plot"

The POD I'm working on has one of these, but also individual plots:

"href=model/block_freq_anncycle.{{CASENAME}}.png>Model"
"href=obs/block_freq_anncycle.OBS.png>OBS"
"href=model/block_freq_anncycle.CAM5.png>CAM5 ensemble"

This worked fine for one CASENAME but obviously now we need something more dynamic that can write and replace in the html file something like:

"href=model/block_freq_anncycle.{{CASENAME_1}}.png>Model"
"href=model/block_freq_anncycle.{{CASENAME_2}}.png>Model"
...
"href=model/block_freq_anncycle.{{CASENAME_N}}.png>Model"

where N can vary with each invocation.

If there isn't something already here, I can try to write it. It would be helpful if anyone could tell me

• where the web page variable replacement happens in the mdtf code (function name is all I need)
• any recommendations of good ways to editing files with python. My instinct would be cat and sed which I assume we should avoid! :)

[wrongkindofdoctor]

@bitterbark Writing html in Python is done by embedding the desired html code in triple quotes. The contents are output using the basic file open and write methods. You can open and read existing html templates similarly. Example here

There is also a library called Beautiful Soup for reading and editing existing html files if you'd rather do find+replace-style editing on an existing file. Example here

[bitterbark]

That code was super helpful, @wrongkindofdoctor, thanks. I have run into an issue with the pod_env_vars replacement in the html file, though. Maybe I am missing something; if so please point it out. But if not, I would value your (and anyone else reading this) opinion on how to proceed. (Please note that I don't expect an answer tonight!)

First I'm going to summarize the parts of the process that are pertinent to writing the html page and saving the multi-run settings. Note that all code snippets are edited to show just what we need.

The replacement of {{variables}} in the html file happens in the function

  src/output_manager.py:
    def html_templating_dict(pod):
           template = config.global_env_vars.copy()
           template.update(pod.pod_env_vars)

The pod_env_vars dict is defined for the framework's use in

  src/environment_manager.py:
     class MultirunSubprocessRuntimePODWrapper(object):
      def setup_env_vars(self):
       skip_items = ['FIRSTYR', 'LASTYR', 'CASENAME'] 
       self.env_vars = {k: _envvar_format(v)
                         for k, v in self.pod.pod_env_vars.items() if k not in skip_items}

This works for my new multirun case writing its html file for the env vars that are used by the framework and the pod, except the skip_items, as I can see in the $pod.log file.

Information for the multiple cases, however, is written to the yaml file

 def write_case_env_file(self, case_list):
 for case_name, case in case_list.items():
        case_info[case_name] = {k: v
                               for k, v in case.env_vars.items()}
      for v in case.iter_vars_only(case, active=True):
         for kk, vv in v.env_vars.items():
            case_info[case_name][kk] = vv 

As far as I can tell, this information is not retained in the framework;
the pods access the yaml file for it. This is where I'm hoping I'm wrong!

If the variables are not retained, in order to get these variables into the html replacement, we need to do 2 things:

1. get the information for each case into the html replacement template
2. decide on a syntax to use in the html file itself so the right variables names are replaced (etg what to call variable in the {{variable}} text)

Options for 1)

• have the html template read the yaml file and store the information in a dict just like is done before the yaml write (seems nice
  and straightforward except the format is a nested dict that works fine to
  send to the yaml, but will it work in the html template replacement?)
• have a function near the yaml write that saves the information into a
  dict (adds to the burden that the framework carries around but means
  everything is done in one place)

2. In case it isn't clear, what I need here is something like

CASE1_CASENAME, CASE1_FIRSTYR,...
...
CASEN_CASENAME, CASE1_FIRSTYR,...

Or we could use CASE1:CASENAME if the variable replacement will work with
that.
Or we could be more general like GROUP1_CASE1_CASENAME, so groups
(ensembles or whatever) can end up together.

If I should proceed in this way, I am interested if you have an opinion as to how it is done, or other ideas for doing it easier.

[wrongkindofdoctor]

@bitterbark Good catch! I neglected to pass the CASENAME, FIRSTYR, LASTYR info to the html generator portion of the code in the multirun implementation because each case has unique values for them. You can see that the templates for the the multirun PODs in src/html only contain variable name placeholders for the environment variables that apply to all cases. I like your suggestion of simply modifying the html writer (currently handled by the output_manager module to read the multirun environment variable yaml file, and can add that to the redesigned framework.

[bitterbark]

I'm adding it now! Will send a PR when I'm done.

[wrongkindofdoctor]

Thanks, @bitterbark! I appreciate the dev assistance.