Metadata for hiding input/output or cells

[grst]

Hi @mwouts,

I am currently working on a pipeline that chains together jupyter/jupytext notebooks into a reproducible workflow. I created two 'render engines', one based on papermill and one on knitr.
Ideally, using jupytext, one could use them interchangeably on the same notebook.

I am currently struggling with the cell metadata to control the visibility of cell inputs/outputs. I have seen that you have been experimenting with both papermill and jupyterbook. I also observed that you implemented some conversion of cell metadata for Rmarkdown. Ideally, there would be common conventions between jupytext, jupyterbook, Rmarkdown etc.

First, my observations:

• nbconvert supports contolling cell visibility when creating a html report by using nbconvert.preprocessors.TagRemovePreprocessor
• Rmarkdown uses include=FALSE to hide an entire cell, results='hide' to hide output and echo=FALSE to hide input.
• jupyter book supports hide_input and remove_input to hide the input, remove_cell for an entir cell, and to my knowledge no option to hide output only.
• jupytext currently uses hide_output and hide_input when converting from Rmarkdown.

Now, my questions/suggestsions:

• Are you aware of more/different tags from other projects?
• Did you come up with current jupytext hide_output and hide_input, or are they supposed to be compatible with another project?
• Rmarkdown include=FALSE translates currently into hide_output. I believe it should translate into remove_cell or equivalent.
• Rmarkdown results='hide' is currently not converted. It should translate into hide_output.

Let me know what you think!

Best,
Gregor

[mwouts]

Hello @grst , thank you for this review, this is very helpful!

The current mapping of the R Markdown options to the Jupyter notebook ones dates from one year ago (at that time Jupyter Book was not yet available). The cell metadata inserted by Jupytext are intended to be compatible with the runtools extension for Jupyter Notebook:

jupytext/jupytext/cell_metadata.py

Lines 24 to 28 in 4c1f817

	# Map R Markdown's "echo" and "include" to "hide_input" and "hide_output", that are understood by the `runtools`
	# extension for Jupyter notebook, and by nbconvert (use the `hide_input_output.tpl` template).
	# See http://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/runtools/readme.html
	_BOOLEAN_OPTIONS_DICTIONARY = [('hide_input', 'echo', True),
	('hide_output', 'include', True)]

I agree with your point that I should have mapped hide_output to results='hide', I'll be happy to fix that. Also, I see that hide_input here (a boolean) is not the same as Jupyter Book's one (a tag). So really we need to go further, and try to agree among more projects on common cell metadata to hide input or outputs. I'm sure @choldgraf , the author of Jupyter Book, will be interested in this.

So let me summarize: we'd like to be able to

• hide inputs
• or outputs
• or both

Maybe I would also like to select only certain types of outputs (only text, or only HTML, etc, like the --NbConvertBase.display_data_priority option in jupyter nbconvert), and maybe I would like to have the option to encapsulate the output text either in an output cell (the default) or added to the main document (R Markdown's result='asis').

The different contexts in which we want to be able to do that are

• Jupyter Notebook (currently possible with the runtools extension) and Jupyter Lab (are you aware of an extension that can toggle cell visibility?)
• Jupyter Book (possible with tags, however just removing the outputs may not be an option yet)
• jupyter nbconvert (possible with either the runtools template, or as you mentionned with --TagRemovePreprocessor, which like Jupyter Book works with tags, cf. this example)
• R Markdown
• And maybe, in the future, Jupytext itself ? (Hide inputs, include outputs in Markdown format #220 )

I think we're seeing more tags here than boolean, so maybe we can choose the current tags used by Jupyter Book as the reference. Regarding the removal of outputs, I'd be curious to know if @choldgraf thinks there's a use case for that? Is anyone (other than me!) interested by the filtering of outputs (my use case is selecting png over js before sending Markdown over email)?

[choldgraf]

Thanks for looping me in - a few quick thoughts:

• Adding a tag for remove_output would be straightforward to do, I'm happy to have that feature in Jupyter Book

• Jupyter Lab (are you aware of an extension that can toggle cell visibility?)

  This works "out of the box" in JupyterLab - click the vertical blue bar next to any selected cell input/output and it'll collapse.

• I think we're seeing more tags here than boolean

  My reasoning for using tags in Jupyter Book is that there is / will be fairly straightforward UI for adding tags to cells, and it seemed like a simpler way to let people add metadata to cells without needing to write their own custom JSON.

• On the broader question of what syntax to support for publishing purposes, sometime soon we should put together a document that tries to determine some basic functionality we need, and syntax that should trigger that functionality. That way we all get on the same page.

[grst]

Jupyter Lab (are you aware of an extension that can toggle cell visibility?)

This works "out of the box" in JupyterLab - click the vertical blue bar next to any selected cell input/output and it'll collapse.

Ah, I wasn't aware of that. It adds the following metadata:
collapse input:

{
    "jupyter": {
        "source_hidden": true
    }
}

collapse output

{
    "jupyter": {
        "source_hidden": true
    }
}

collapssing both is simply a combination of the two

{
    "jupyter": {
        "source_hidden": true,
        "outputs_hidden": true
    },
    "collapsed": true
}

However, I'm not sure if when I just hide away a big output in jupyterlab, I also wouldn't want it to be included in a report that I create using papermill or nbconvert.

[grst]

On the broader question of what syntax to support for publishing purposes, sometime soon we should put together a document that tries to determine some basic functionality we need, and syntax that should trigger that functionality. That way we all get on the same page.

I'd really appreciate if a common standard could be established in the jupyter world!

[choldgraf]

However, I'm not sure if when I just hide away a big output in jupyterlab, I also wouldn't want it to be included in a report that I create using papermill or nbconvert.

Yeah I agree - that's why for Jupyter Book I just decided to make people explicitly state it with a tag.

I'd really appreciate if a common standard could be established in the jupyter world!

IMO we should try to pick a flavor of markdown to adopt, rather than create our own standard. Either RMarkdown or Pandoc Markdown seems reasonable to me. The trick is that I don't know of any python machinery to process these kinds of markdown, we'd need to use Pandoc (or R, I suppose)

[grst]

I don't know of any python machinery to process these kinds of markdown

I know at least two of them: https://github.com/jankatins/knitpy, https://github.com/pystitch/stitch
None of the two seems actively maintained, though. Both of them are mirroring the Rmarkdown syntax.

we'd need to use Pandoc

Are there any pandoc arguments to hide inputs/outputs? I wouldn't guess so, because in plain markdown it's not even clear what "output" belongs to which code block...

One the one hand, I'd opt to go for the Rmarkdown syntax, because it's pretty established. On the other hand I don't like it because it's neither intuitive nor consistent. (include=FALSE vs. echo=FALSE vs results='hide'). It took me ages to remember which does what. The tags you use in jupyter book are a lot better in that sense.

[choldgraf]

@grst very cool! I remember seeing these some time ago but had forgotten about them. I wonder if there are pieces in there that could be pulled out for the md->html generation

re: the RMarkdown syntax - I agree with you that the options for parameterization etc can be unintuitive. I'd probably opt to agree on the basic syntax structure (e.g. how parameters for code blocks are defined) rather than on the specific parameter key/values themselves. E.g., I like the idea of curly brackets for defining tags, something like

```python {tags=["tag1", "tag2"]}
a = 2

[grst]

I wonder if there are pieces in there that could be pulled out for the md->html generation

What for? Jupytext already does a great job converting (R)markdown to notebooks. And for notebooks to html there is nbconvert which is officially supported by the jupyter project.

FWIW, I stitched together jupytext, papermill and nbconvert in a quite straightforward python script to obtain that functionality.

I'd probably opt to agree on the basic syntax structure (e.g. how parameters for code blocks are defined) rather than on the specific parameter key/values themselves.

I would have rather said the opposite :D. How to format the parameters in markdown etc. is pretty much given by the formats supported by jupytext. Independent of the format, they end up as json-formatted cell metadata in the jupyter notebook format. It has to be clarified

• how they should be stored there, i.e. in tags or directly in the metadata and
• how they should be called.

Personally, I like the tags, because as @choldgraf mentioned, they are easy to add via UI and are supported by nbconvert.

[choldgraf]

What for?

Right now the flavor of Markdown that Jupyter supports is extremely vanilla - it doesn't support things like footnotes, citations, or other extensions of any kind. Some of that we can store in cell metadata, but some of it needs to be in-line with the text and we'll need to improve the markdown -> html renderer in nbconvert for that.

I agree that I think cell metadata is the way to go for storing most information and that jupytext can be the way to package that information with a notebook text file - I think that the region and endregion approach is a bit too clunky for my taste, and breaks with other flavors of markdown that are out there. That's something to discuss w/ the community though!

(and yes, I agree tags are good as well!)

[grst]

Good point! Actually, once jupytext supports storing outputs in markdown format (#220), then pandoc could be used to support all that features and generate reports in virtually any format.
But I believe that's a separte discussion!

[choldgraf]

@grst actually, pandoc already supports Jupyter Notebooks, you can do pandoc myntbk.ipynb -o mydocument.html and it'll work! There's a decent chance we'll start doing this with Jupyter Book actually, that'd solve a lot of our markdown + citations problems, the main things I'm concerned with is whether that'd still gracefully handle things like widgets etc

[mwouts]

Hi @grst , @choldgraf , that's a very promising discussion! I agree with everything that was said above. And I do prefer tags over additional cell metadata.

Regarding the tags names, maybe we could also consider names that could match nbconvert.preprocessors.TagRemovePreprocessor's argument names (i.e. "remove_cell", "remove_all_output", "remove_single_output", "remove_input"). Anyone nows how remove_single_output works? Does it filter outputs by mime type (what I am looking for) or by output index (I don't think I need that)?

@choldgraf , it's an interesting point that you're not fan of the "region" comments, and sure I'd like to use a more Markdown-ish approach. Pandoc's approach here could be of some inspiration, as they use nested divs (:::) to delimit regions. Yet, unlike the html comments, these markers where not recognized by the Markdown editors I do use (GitHub, VSCode, PyCharm).

Also, I would be interested to see an example notebook with notes. Would you like to share a link? Thanks!

[grst]

Anyone nows how remove_single_output works?

I checked the source code. Apparently outputs can have their own metadata and, therefore, tags. It checks cell.outputs[i].metadata.tags for the specified tags.

[choldgraf]

yeah - I'm still unsure of how exactly we can easily add tags to outputs, but I thought that was kinda cool that you can filter outputs by mimetype :-)

re: the "region" comments - I think they are totally fine to read, I just find them cumbersome to type. The reason I like the {} approach is that { is a very easy character to remember, and also is a common pattern in markdown (though I'm not sure if text editors recognize it, I know that at least kramdown, the jekyll github flavor, supports it). Maybe we could use that as a delimiter? (e.g. the existence of a {.someattribute} line effectively creates another cell below it? though then you'd need a way to "end" the cell region...)

[mwouts]

Hello @grst , @choldgraf , I think I'm good with the cell tags that we're discussing here, i.e. hide_input, remove_input, remove_output, remove_cell, and maybe also, later on, hide_output. My plans are to do the following:

• in the .ipynb representation, the tags have no specific effect: hiding input or output is actually done either by Jupyter Book, or by an extension to be developed.
• in the Jupytext Markdown representation, hide_input would have the effect to surround the code cell with a <details> section, so that the code is 'collapsed' when rendered by GitHub or VS code. What would be a good <summary>, i.e. the text that appears when the cell is collapsed? What do you think of something like 53 lines of code? And if the cell has a name or a title metadata, we could also use that one as a summary.
• similarly, remove_input would have the effect to move the code cell inside an HTML comment, so that it does not appear on GitHub or in VS Code preview.
• in the R Markdown representation, I'd map remove_input to echo=FALSE and remove_output to results='hide', and remove_cell to include=FALSE.

@grst , I see that R Markdown now has an option to hide (collapse) all the inputs in the HTML rendering of an Rmd document, see here. Do you know if this is something we can configure per cell? If yes, maybe I'd like to do the mapping of the hide_input tag to the corresponding option there.

Also, @choldgraf , when I compare to R Markdown I see that they give the option to change the default value for including inputs/outputs globally. Is this something that we would be interested to have? Should we have a default_cell_visibility entry in the notebook metadata with the list of the tags selected by default for all cells? But then, maybe we'd have to define as well include_cell, include_input and include_output to the list of supported tags to allow one to force the display of a given cell?

[grst]

@mwouts, I couldn't figure out a way of setting code_folding on a per-chunk level. There are some hacks described on stackoverflow.

Good initiative to get the runtools people onboard!

[choldgraf]

Hey all - a few thoughts to your questions:

• in the Jupytext Markdown representation, hide_input would have the effect to surround the code cell with a

  section, so that the code is 'collapsed' when rendered by GitHub or VS code. What would be a good

  , i.e. the text that appears when the cell is collapsed?

  • This seems reasonable, though I'd caution that (in my experience), the <details> tag has unexpected behavior with markdown inside. It always seems to give weird results where some things are rendered inside and some aren't...

• What do you think of something like 53 lines of code? And if the cell has a name or a title metadata, we could also use that one as a summary.

  • You mean "characters" of code? That seems reasonable (I'd probably cap it at the first line of code, maybe add a "Reveal code: " beforehand)

• Also, @choldgraf , when I compare to R Markdown I see that they give the option to change the default value for including inputs/outputs globally. Is this something that we would be interested to have?

  • Hmm - re: "notebook-level" hiding metadata, I wonder if a first-pass could be to improve the jupyterlab-celltags plugin so that you can add a tag to all highlighted cells, then adding a tag to each cell would just be ctrl-a -> click tags tab -> click the tag to add. The plugin already supports removing a tag from all cells that have it.

[psychemedia]

I'm not sure if this is the correct issue for this, but it riffs on the idea of "hiding" cells.

In particular, I'm trying out a workflow where I author a module in a notebook as an .ipynb file that mixes module code cells with testing code cells, tagged as active-ipynb in the notebook file.

I can pair the .ipynb file with a .py file that I load in as a module, with the active-ipynb code commented out in the .py file. But I properly want to hide the tagged code cells from the .py file, which is to say, I don't want the active-ipynb code cell content included in the .py file at all. (It doesn't matter that pairing/round tripping is broken; the .py file is never directly edited in my workflow, it is only ever generated from an .ipynb file.)

Is there a jupytext filter that lets me do this?

[mwouts]

Hello Tony, there's no jupytext filter for this. But maybe you could use nbconvert? I just gave it a try and it seems to be able to do what you are looking for.

Say we have this notebook:

Then we create a custom template for nbconvert with a file remove_inactive_cells.tpl that contains this code:

{%- extends 'python.tpl' -%}

{% block input_group -%}
{%- if "active-ipynb" in cell.metadata.tags -%}
{%- else -%}
{{ super() }}
{%- endif -%}
{% endblock input_group %}

and finally we call jupyter nbconvert:

jupyter nbconvert "Filtering active-ipynb cells.ipynb" --to script --template remove_inactive_cells.tpl --stdout

#!/usr/bin/env python
# coding: utf-8

# In[ ]:


"active cell"


# In[ ]:

[psychemedia]

@mwouts Great - thanks... that looks like it'll do the trick... I'll try to do a post on a workflow that includes this to try to explain why I think it might be useful!

PS post here: Rescuing Python Module Code From Cluttered Jupyter Notebooks

[mwouts]

I'll soon address this issue. Is everyone OK if I map the R Markdown options to the Jupyter Book ones by default? I also plan to offer an option for those who prefer to map them to the runtool ones.

[mwouts]

Hello @grst , I've updated the mapping between the R Markdown options and the runtools ones. Can you confirm that the mapping below is what you would have expected?

jupytext/tests/test_hide_remove_input_outputs_rmarkdown.py

Lines 8 to 27 in e519249

	@pytest.mark.parametrize('md,rmd', [('hide_input=true hide_output=true', 'include=FALSE'),
	('hide_output=true', "results='hide'"),
	('hide_output=true', 'results="hide"'),
	('hide_input=true', 'echo=FALSE')])
	def test_runtools_options_to_rmarkdown(md, rmd):
	"""Options set by the runtools extension are mapped to the corresponding R Markdown options
	https://jupyter-contrib-nbextensions.readthedocs.io/en/latest/nbextensions/runtools/readme.html"""
	md = '```python ' + md + """
	1 + 1
	```
	"""
	rmd = '```{python ' + rmd + """}
	1 + 1
	```
	"""
	nb_md = jupytext.reads(md, 'md')
	nb_rmd = jupytext.reads(rmd, 'Rmd')
	compare_notebooks(nb_rmd, nb_md)

Soon I will add the translation between the R Markdown options and the Jupyter Book tags (and that will become the default translation). Would the test below fit your specifications? Do you prefer to map the R Markdown options to remove_input/output, or to the hide_input/output ones?

@pytest.mark.parametrize('md,rmd', [('tags=["remove_cell"]', 'include=FALSE'),
                                    ('tags=["remove_output"]', "results='hide'"),
                                    ('tags=["remove_output"]', 'results="hide"'),
                                    ('tags=["remove_input"]', 'echo=FALSE')])
def test_jupyter_book_options_to_rmarkdown(md, rmd):
    """By default, Jupyter Book tags are mapped to R Markdown options, and vice versa #337"""
    md = '```python ' + md + """
1 + 1
```
"""

    rmd = '```{python ' + rmd + """}
1 + 1
```
"""

    nb_md = jupytext.reads(md, 'md')
    nb_rmd = jupytext.reads(rmd, 'Rmd')
    compare_notebooks(nb_rmd, nb_md)

[grst]

Sorry the delay. The test LGTM.

For the jupyter book tags, I think it doesn't matter too much.
Maybe remove would be the proper semantics, as afaik, the content is actually removed and not hidden, e.g. in a collapsable element.

[mwouts]

Thanks Gregor. Sure, I agree. And anyway if later on someone wants to map the Jupyter Book hide_* tags to their R Markdown counterpart, or reversely, we can still add another option to do so...

[mwouts]

Hello @grst, I have just released a new rc (pip install jupytext==1.3.0rc1). Would you like to give it a try? Thanks

[grst]

Actually, now I get a parsing error on a file that used to work fine:

> jupytext 02_analyze_data.Rmd --to ipynb
[jupytext] Reading 02_analyze_data.Rmd
Traceback (most recent call last):
  File "/home/sturm/anaconda3/envs/reporstrender_dev/bin/jupytext", line 10, in <module>
    sys.exit(jupytext())
  File "/home/sturm/anaconda3/envs/reporstrender_dev/lib/python3.7/site-packages/jupytext/cli.py", line 271, in jupytext
    exit_code += jupytext_single_file(nb_file, args, log)
  File "/home/sturm/anaconda3/envs/reporstrender_dev/lib/python3.7/site-packages/jupytext/cli.py", line 304, in jupytext_single_file
    notebook = read(nb_file, fmt=fmt)
  File "/home/sturm/anaconda3/envs/reporstrender_dev/lib/python3.7/site-packages/jupytext/jupytext.py", line 267, in read
    return read(stream, as_version=as_version, fmt=fmt, **kwargs)
  File "/home/sturm/anaconda3/envs/reporstrender_dev/lib/python3.7/site-packages/jupytext/jupytext.py", line 276, in read
    return reads(fp.read(), fmt, **kwargs)
  File "/home/sturm/anaconda3/envs/reporstrender_dev/lib/python3.7/site-packages/jupytext/jupytext.py", line 231, in reads
    notebook = reader.reads(text, **kwargs)
  File "/home/sturm/anaconda3/envs/reporstrender_dev/lib/python3.7/site-packages/jupytext/jupytext.py", line 79, in reads
    cell, pos = reader.read(lines)
  File "/home/sturm/anaconda3/envs/reporstrender_dev/lib/python3.7/site-packages/jupytext/cell_reader.py", line 123, in read
    pos_next_cell = self.find_cell_content(lines)
  File "/home/sturm/anaconda3/envs/reporstrender_dev/lib/python3.7/site-packages/jupytext/cell_reader.py", line 166, in find_cell_content
    cell_end_marker, next_cell_start, self.explicit_eoc = self.find_cell_end(lines)
  File "/home/sturm/anaconda3/envs/reporstrender_dev/lib/python3.7/site-packages/jupytext/cell_reader.py", line 350, in find_cell_end
    _, metadata = self.options_to_metadata(self.start_code_re.findall(line)[0])
  File "/home/sturm/anaconda3/envs/reporstrender_dev/lib/python3.7/site-packages/jupytext/cell_reader.py", line 403, in options_to_metadata
    return rmd_options_to_metadata(options, self.use_runtools)
  File "/home/sturm/anaconda3/envs/reporstrender_dev/lib/python3.7/site-packages/jupytext/cell_metadata.py", line 236, in rmd_options_to_metadata
    chunk_options = parse_rmd_options(others)
  File "/home/sturm/anaconda3/envs/reporstrender_dev/lib/python3.7/site-packages/jupytext/cell_metadata.py", line 212, in parse_rmd_options
    parsing_context.count_special_chars(char, prev_char)
  File "/home/sturm/anaconda3/envs/reporstrender_dev/lib/python3.7/site-packages/jupytext/cell_metadata.py", line 165, in count_special_chars
    'closing curly brackets'.format(self.line))
jupytext.cell_metadata.RMarkdownOptionParsingError: Option line "tags=c("parameters")}" has too many closing curly brackets

[psychemedia]

Just to try to provide some linkage between related issues across several projects, I notice that nbsphinx has had open issues and PRs on handling hidden cells for some time. For example, spatialaudio/nbsphinx#303

In terms of workflows and generalising publishing opportunities outside the production of documentation (which has its own set of concerns) I think it's useful if conventions for tagging cells are in place so that the same documents can be published via different workflows.

I don't think nbsphinx does support hiding cells (yet?) but if it does, it may be useful to have tests that check hidden cell publishing in eg an md->jupytext/nbsphinx->HTML pathway, though I don't know whether responsibility for that test would lay with nbsphinx or jupytext? Similarly, publishing via Jupytext?

[mwouts]

Thanks @grst for reporting this, I'll have a look.
Thank you @psychemedia for the link to the nbsphinx PR on hidden cells - I'll think about that!

[mwouts]

Gregor, indeed the backward compatibility switch (triggered by 'format_version: 1.1') for the Markdown format was also affecting the R Markdown format... I've fixed that with 17c9d34.

[grst]

Thanks for fixing! I confirm it works as expected now on the current master branch.

…

On Sat, 9 Nov 2019 at 11:23, Marc Wouts ***@***.***> wrote: Closed #337 <#337>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#337?email_source=notifications&email_token=ABVZRV4FQJ6M76YL3ZCKJODQS2FR3A5CNFSM4I2I4V5KYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOUYBSRVY#event-2785224919>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABVZRV7ZQ2CC2EFVIDLSQZLQS2FR3ANCNFSM4I2I4V5A> .