Create initial results visualization website

[wkiri]

Goal: visualize results of a given experiment

• Read in config file
• Give user option to specify output directory (has CSV files) if need to override out_dir (where the results are) in config file

Todo:

• Put relevant code to dora_viz/ directory
• Kiri: give Jake sample output for planetary rover
• Hannah: give Jake sample output for time series (maybe also visualize on a map)

[wkiri]

@jakehlee Please see https://github.com/nasaharvest/dora/tree/issue52-reorg-testdata/sample_data/planetary_rover/png (note it is in a branch due to issue #52 ) . I recommend running this sample case yourself to exercise DORA. If you do this on an MLIA machine, you can start the DORA virtual environment first and have all the packages installed:

$ source /proj/dora/dora-venv/bin/activate

Then you'll have some sample output to work with. Let me know if you run into any snags!

[jakehlee]

Minimum Viable Product pushed to branch issue61-vis-website. Instructions included to launch dev version, should be relatively simple as long as node is installed.

Workflow Walkthrough

Configuration Loading

This screen asks the user to locate the DORA config YAML filepath on the current system. This informs the app about:

1. Which methods' results to expect
2. Which dataloader was used
3. Where the data is
4. Where the results directory is

Configuration Parsing

It is possible for the config to have relative paths, or paths that were used on an external server. This step gives the user a chance to correct these steps.

1. It indicates that the image dataloader is supported. If not, the app cannot continue.
2. It indicates that it couldn't locate the data_to_score directory. The user can select a different folder from the current system.
3. It indicates that it couldn't locate the out_dir directory. The user can select a different folder from the current system.

Once all issues are resolved, the user can continue to the visualization.

Visualization

The current MVP implementation is a simple table with the following features:

1. Sortable columns
2. Pagination
3. Switching between methods output

TODOs

• Edge cases: current implementation assumes well-behaved user
  • Directory specified by user doesn't contain expected results per-method
  • Directory specified by user doesn't contain expected input data files
  • Configuration file specified by user doesn't contain expected fields
• Beautification: MVP includes no CSS/styling
• Aggregate method: aggregate results from all included methods into a single view
• Gallery view: show images in a grid instead of a table, with click-through information

[wkiri]

@jakehlee Great work! I love the video demo too :)

In our meeting today, perhaps we can discuss what would be useful to incorporate into the paper (as room permits).

Here are some initial feedback thoughts (not in priority order):

• I love that you can sort by various columns, but that wasn't clear to me until I saw you do it :) Maybe add an indicator (e.g., up/down arrows on the currently sorted column) to show that sorting is an option as well as the current sort direction?
• I suggest putting the algorithm selector at the top of the page (easy to miss buried in the paging controls). Maybe the title ("pca" in the example above) can just be a drop-down menu itself which would make the selection obvious.
• This particular example has really long filenames, which is awkward. On the beautification front, can you make the tiles prominent and relegate the filenames to hover text or something like that? Of course, it would be useful to have the text be selectable in case someone wants to copy/paste it somewhere. Or, maybe that's what "gallery view" will do.
• I suggest limiting scores to e.g. 2 decimal places. Or, we should do this upstream when the CSV is written out...?
• Styling: the columns are hard to see without any lines. (e.g., "Rank ID Filename" kind of looks like one thing) Can you add faint grey lines separating columns or other visual cue?
• When present, can you also load and display the comparison_plot_combined.png? Perhaps that would go with the "aggregate view". For the per-algorithm view, you could show the comparison_plot_<alg>.png that is within each algorithm's subdirectory (if present).
• The pagination is so fast that if I blink I miss it and don't realize I'm on another page of results. Maybe this is just me.

I'm looking forward to trying this out myself :)

(edited by Jake to change bullets into check boxes)

[wkiri]

@jakehlee I've merged my planetary rover results (actual experiment, not the test case you had before) so feel free to give a try running on this: https://github.com/nasaharvest/dora/tree/master/exp/planetary_rover/results

[jakehlee]

@wkiri Thanks for the UX notes! Those are all great points for when I do my UX pass. Thank you for the new results, I was able to download the dataset from /proj and run the app successfully. Let me implement some of these changes and post more screenshots/videos.

[jakehlee]

I've pushed some updates to address many of the points, and a demo video is available on Slack. I think I address most of the suggestions and cleaned everything up, although I'm a bit hesitant to add the comparison plots because I think those are only generated when labels are available, and I want to minimize rendering logic that only shows something if it exists. Maybe I'll find a good place to show it still though, so still under consideration.

[jakehlee]

Screenshots for the paper: