Merge pull request #33 from mwouts/v0.5.0

New version v0.5.0.
Fixes #30 #32 #29 #31 mwouts/nbsrc#5

HISTORY.rst

@@ -6,6 +6,26 @@ Release History
 dev
 +++
 
+0.5.0 (2018-08-21)
++++++++++++++++++++
+
+**Improvements**
+
+- Jupyter magic commands escaped when exported (#29)
+- 'endofcell' option for explicit (optional) end-of-cell marker (#31)
+- 'active' cell option now supported for .py and .R export (#30)
+- Raw cells now preserved when exported to .py or .R (#32)
+- Extensions can be prefixed, like `.nb.py`, (mwouts/nbsrc#5)
+- When a file with an extension not associated to 'ipynb' is opened and saved,
+no 'ipynb' file is created (mwouts/nbsrc#5)
+- Extensions can now be a sequence of groups. For instance,
+`nbrmd_formats="ipynb,nb.py;script.ipynb,py"` will create an `ipynb` file
+when a `nb.py` is opened (and conversely), and a `script.ipynb` file when a
+`py` file is opened (mwouts/nbsrc#5)
+- `nbsrc` script was moved to the `nbrmd` package. The `nbsrc` package now only
+contains the documentation (mwouts/nbsrc#3)
+
+
 0.4.6 (2018-07-26)
 +++++++++++++++++++
 

README.md

@@ -1,4 +1,4 @@
-# Jupyter notebooks from/to R markdown
+# Jupyter notebooks as R markdown, Python or R scripts
 
 [![Pypi](https://img.shields.io/pypi/v/nbrmd.svg)](https://pypi.python.org/pypi/nbrmd)
 [![Pypi](https://img.shields.io/pypi/l/nbrmd.svg)](https://pypi.python.org/pypi/nbrmd)
@@ -8,30 +8,27 @@
 [![pyversions](https://img.shields.io/pypi/pyversions/nbrmd.svg)](https://pypi.python.org/pypi/nbrmd)
 [![Binder](https://mybinder.org/badge.svg)](https://mybinder.org/v2/gh/mwouts/nbrmd/master?filepath=demo)
 
-This package is an implementation of the standard
-[R markdown](https://rmarkdown.rstudio.com/) notebook format for Jupyter.
-R markdown notebooks are source only notebooks, and they
-are great companion files for the standard `.ipynb` notebooks.
+This package offers a representation of Jupyter notebooks as Python scripts, R scripts, or [R markdown](https://rmarkdown.rstudio.com/) notebooks. 
 
-Use the `nbrmd` package if
-- you prefer to have simple text files under version control
-- if you want to use RStudio's advanced rendering of notebooks to PDF, HTML or [HTML slides](https://rmarkdown.rstudio.com/ioslides_presentation_format.html)
-- or, you have a collection of markdown or R markdown notebooks and you want to open them in Jupyter.
+These alternative representations allow to
+- edit notebooks in both Jupyter and your favorite IDE, and refactor them
+- extract executable scripts from your notebooks
+- get meaningfull diffs for notebooks under version control, and easily merge contributions
+- edit the same notebook in both Jupyter and Rstudio, and use RStudio's advanced rendering of notebooks to PDF, HTML or [HTML slides](https://rmarkdown.rstudio.com/ioslides_presentation_format.html).
 
-Only the source of your notebook is represented in R markdown.
-When a pair of `.Rmd`, `.ipynb` notebooks with identical names are opened
-in Jupyter, inputs are taken from the `.Rmd` file, and outputs, when they match the input,
-are taken from the `.ipynb` file. This allows you to edit the R markdown
-version in your favorite text editor, and reload the notebook in Jupyter with the
-convenience of preserving outputs when possible.
+Scripts and R markdown notebooks only store the source of the notebook, and work in pair with the original `.ipynb` file. Jupyter saves notebooks to both the classical `.ipynb` form, and to the text-only representation. When a text-only notebook is loaded in Jupyter, inputs are taken there, and outputs are taken from the `.ipynb` file, if present.
 
 ## Can I have a demo?
 
-Sure. Try our package on [binder](https://mybinder.org/v2/gh/mwouts/nbrmd/master?filepath=demo)
-and open our python-oriented R markdown notebook!
-As you will see there, the package also offers opening and saving
-notebooks as python or R scripts. Go to
-[nbsrc](https://github.com/mwouts/nbsrc) for a specific documentation on this.
+Sure. Try our package on [binder](https://mybinder.org/v2/gh/mwouts/nbrmd/master?filepath=demo). Notice that every `.py`, `.R` and `.Rmd` file now opens as a Jupyter notebook. I suggest you open the matplotlib demo there: `filled_step.py`, run it and save it, close notebook and reopen, to observe persistence of outputs. 
+
+The other examples demo how to *edit* the script and reload the notebook (preserving the kernel variables), and how to edit in Jupyter an interactive ioslide presentation.
+
+## How does the Python script look like?
+
+Python [notebook](https://mybinder.org/v2/gh/mwouts/nbrmd/master?filepath=tests/python_notebook_sample.py) in Jupyter  | Python [script](https://github.com/mwouts/nbrmd/blob/master/tests/python_notebook_sample.py)
+:--------------------------:|:-----------------------:
+![](https://raw.githubusercontent.com/mwouts/nbrmd/master/img/python_notebook.png)   | ![](https://raw.githubusercontent.com/mwouts/nbrmd/master/img/python_source.png)
 
 ## How does R markdown look like?
 
@@ -41,23 +38,25 @@ Rmd notebook in jupyter     | Rmd notebook as text
 
 ## Have you tested round-trip conversion?
 
-Round trip conversion is safe! And backed by hundreds of tests.
-- R markdown to Jupyter notebook, to R markdown again is identity. If you
+Round trip conversion is safe! A few hundreds of tests help to guarantee this.
+- Script to Jupyter notebook, to script again is identity. If you
 associate a Jupyter kernel to your notebook, that information will go to
-the yaml header of your notebook.
+a yaml header at the top of your script.
+- R markdown to Jupyter notebook, to R markdown again is identity. 
+- Jupyter to script, and Jupyter again preserves source and metadata.
 - Jupyter to R markdown, and Jupyter again preserves source and metadata.
-In some occasions (consecutive blank lines in
-markdown cells), cells may be splitted into smaller ones.
+In some occasions (consecutive blank lines in markdown cells), markdown cells may
+be splitted into smaller ones.
 
-## How do I activate R markdown notebooks in Jupyter?
+## How do I activate the companion scripts or R markdown notebooks in Jupyter?
 
 The `nbrmd` package offers a `ContentsManager` for Jupyter that recognizes
-`.Rmd` files as notebooks. To use it,
+`.py`, `.R` and `.Rmd` files as notebooks. To use it,
 - generate a jupyter config, if you don't have one yet, with `jupyter notebook --generate-config`
 - edit the config and include the below:
 ```python
-c.NotebookApp.contents_manager_class = 'nbrmd.RmdFileContentsManager'
-c.ContentsManager.default_nbrmd_formats = 'ipynb,Rmd'
+c.NotebookApp.contents_manager_class = "nbrmd.RmdFileContentsManager"
+c.ContentsManager.default_nbrmd_formats = "ipynb,py" # or "ipynb,nb.py" # or "ipynb,Rmd"
 ```
 
 Then, make sure you have the `nbrmd` package up-to-date, and re-start jupyter, i.e. run
@@ -66,21 +65,12 @@ pip install nbrmd --upgrade
 jupyter notebook
 ```
 
-With the above configuration, every Jupyter notebook will have a companion `.Rmd` notebook.
-And every `.Rmd` notebook will have a companion `.ipynb` notebook.
+With the above configuration, every Jupyter notebook will have a companion `.py` (`.nb.py`, or `.Rmd`) notebook. And every `.py` (`.nb.py`, or `.Rmd`) notebook will have a companion `.ipynb` notebook.
 
-If you prefer the `.ipynb` notebook not to be created by Jupyter when a `.Rmd`
-notebook is edited, set
-```
-c.ContentsManager.default_nbrmd_formats = ''
-```
-(as the default value is `ipynb`). Outputs for R markdown notebooks, however,
-will not be saved any more.
 
 ## Per notebook configuration
 
-If you prefer that the companion R markdown notebook be generated only for
- selected notebooks,
+If you prefer that the companion notebook be generated only for selected notebooks,
 remove the `c.ContentsManager.default_nbrmd_formats` line from Jupyter's
 configuration, and instead edit the notebook metadata as follows:
 ```
@@ -92,30 +82,48 @@ configuration, and instead edit the notebook metadata as follows:
   "language_info": {
     (...)
   },
-  "nbrmd_formats": "ipynb,Rmd"
+  "nbrmd_formats": "ipynb,py"
 }
 ```
 
-Accepted formats are: `ipynb`, `Rmd`, `py` and `R`.
+Accepted formats should have these extensions: `ipynb`, `Rmd`, `py` and `R`.
 
 In case you want both `py` and `Rmd`, please note that the
 order matters: the first non-`ipynb` extension
 is the one used as the reference source for notebook inputs when you open the `ipynb` file.
 
+Learn more on the possible values for `nbrmd_formats` [here](https://github.com/mwouts/nbsrc/issues/5#issuecomment-414093471).
+
 ## Command line conversion
 
-The package also provides a `nbrmd` script that converts Jupyter notebooks to R markdown notebooks, and vice-versa.
+The package provides two `nbrmd` and `nbsrc` scripts that convert Jupyter notebooks to R markdown notebooks and scripts, and vice-versa.
 
-Use it as:
+Use them as:
 ```bash
 nbrmd jupyter.ipynb         # this prints the Rmarkdown alternative
 nbrmd jupyter.ipynb -i      # this creates a jupyter.Rmd file
 nbrmd jupyter.Rmd   -i      # and this, a jupyter.ipynb file
 nbrmd jupyter.Rmd   -i -p   # update the jupyter.ipynb file and preserve outputs that correspond to unchanged inputs
+
+nbsrc jupyter.ipynb         # this prints the `.py` or `.R` alternative
+nbsrc jupyter.ipynb -i      # this creates a jupyter.py or jupyter.R file
+nbsrc jupyter.py    -i      # and this, a jupyter.ipynb file
+nbsrc jupyter.py    -i -p   # update the jupyter.ipynb file and preserve outputs that correspond to unchanged inputs
 ```
 
-Alternatively, the `nbrmd` package provides a `nbconvert` rmarkdown exporter that you can use with
+Alternatively, the `nbrmd` package provides a few `nbconvert` exporters:
 ```bash
 nbconvert jupyter.ipynb --to rmarkdown
+nbconvert jupyter.ipynb --to pynotebook
+nbconvert jupyter.ipynb --to rnotebook
 ```
 
+## Usefull cell metadata
+
+- Set `"active": "ipynb,py"` if you want that cell to be active only in the Jupyter notebook, and the Python script representation. Use `"active": "ipynb"` if you want that cell to be active only in Jupyter.
+- Code cells that contain two consecutive blank lines use an explicit end-of-cell marker `"endofcell"` in the script representation.
+- R markdown's cell options `echo` and `include` are mapped to the opposite of Jupyter cell metadata `hide_input` and `hide_output`.
+
+## Jupyter magics
+
+Jupyter magics are escaped in the script and R markdown representations so that scripts can actually be executed. Comment a magic with `#noescape` on the same line to avoid escaping. Non-standard magics can be escaped with `#escape`.

binder/requirements.txt

@@ -1,4 +1,4 @@
-nbrmd>=0.4.5
+nbrmd>=0.5.0
 plotly
 matplotlib
 pandas

demo/Sample notebook with python representation.py

@@ -33,7 +33,7 @@
 # 4. Now, modify the python file, save, and refresh the Jupyter notebook. Observe how inputs were updated, outputs preserved when possible, and kernel was left unchanged.
 # 5. Browse the github repo using Jupyter in binder, and open arbitrary python files as notebooks. Run some of them when applicable (test `filled_step.py` for instance).
 
-%matplotlib inline
+# %matplotlib inline
 
 
 import matplotlib.pyplot as plt

nbrmd/__init__.py

@@ -13,6 +13,8 @@
 
 try:
     from .rmarkdownexporter import RMarkdownExporter
+    from .srcexporter import PyNotebookExporter
+    from .srcexporter import RNotebookExporter
 except ImportError as err:
     RMarkdownExporter = str(err)
 

nbrmd/cell_metadata.py

@@ -48,7 +48,7 @@ def metadata_to_rmd_options(language, metadata):
     :param metadata:
     :return:
     """
-    options = language.lower()
+    options = (language or 'R').lower()
     metadata = copy(metadata)
     if 'name' in metadata:
         options += ' ' + metadata['name'] + ','
@@ -74,6 +74,8 @@ def metadata_to_rmd_options(language, metadata):
                     ', '.join(['"{}"'.format(str(v)) for v in opt_value])))
         else:
             options += ' {}={},'.format(opt_name, str(opt_value))
+    if not language:
+        options = options[2:]
     return options.strip(',').strip()
 
 
@@ -255,7 +257,7 @@ def json_options_to_metadata(options):
     :return:
     """
     try:
-        return json.loads(options)
+        return json.loads('{' + options + '}')
     except ValueError:
         return {}