response-to-reviewers.tex

% LaTeX rebuttal letter example.
%
% Copyright 2019 Friedemann Zenke, fzenke.net
%
% Based on examples by Dirk Eddelbuettel, Fran and others from
% https://tex.stackexchange.com/questions/2317/latex-style-or-macro-for-detailed-response-to-referee-report
%
% Licensed under cc by-sa 3.0 with attribution required.

\documentclass[11pt]{article}
\usepackage[utf8]{inputenc}
\usepackage{lipsum} % to generate some filler text
\usepackage{fullpage}
\usepackage{url}
\usepackage{xcolor}

% import Eq and Section references from the main manuscript where needed
% \usepackage{xr}
% \externaldocument{manuscript}

% package needed for optional arguments
\usepackage{xifthen}
% define counters for reviewers and their points
\newcounter{reviewer}
\setcounter{reviewer}{0}
\newcounter{point}[reviewer]
\setcounter{point}{0}

% Import this last to avoid mysterious errors.
\usepackage[implicit=false]{hyperref}

% This refines the format of how the reviewer/point reference will appear.
\renewcommand{\thepoint}{\thereviewer.\arabic{point}}

% command declarations for reviewer points and our responses
\newcommand{\reviewersection}{\stepcounter{reviewer} \bigskip \hrule
                  \section*{Reviewer \thereviewer}}

\newenvironment{point}
   {\refstepcounter{point} \bigskip \noindent {\textbf{Reviewer~Point~\thepoint} } ---\ }
   {\par }

\newcommand{\shortpoint}[1]{\refstepcounter{point}  \bigskip \noindent
    {\textbf{Reviewer~Point~\thepoint} } ---~#1\par }

\newenvironment{reply}
   {\medskip \noindent \begin{sf}\textbf{Reply}:\  }
   {\medskip \end{sf}}

\newcommand{\shortreply}[2][]{\medskip \noindent \begin{sf}\textbf{Reply}:\  #2
    \ifthenelse{\equal{#1}{}}{}{ \hfill \footnotesize (#1)}%
    \medskip \end{sf}}

\newcommand{\ggcomment}[1]{{\textcolor{yellow!60!red}{GG: #1}}}

\begin{document}

\section*{Response to the editor}
% General intro text goes here
Thank you for considering this manuscript for publication, for your
positive comments and in particular for the helpful and insightful reviews.

The changes that we have made mostly reflect an attempt to better
explain the purpose of Demes, its scope, and the rationale for
key design decisions. While the changes to the text
look fairly extensive (see the attached latex-diff), they are mostly
a function of moving text from the appendix to the main text, and
removing redundancy.

Specifically:

\begin{itemize}
\item We compressed the text and improved the presentation by removing
the Parsers appendix (there was little extra information that was not
already in the existing main text section).
\item Laurent made some good and fair points, but which were mainly
(we believe) reflecting our failure at communicating
the underlying rationale for design decisions effectively.
We have rearranged the text, bringing some content that was in the
appendix into the main text to try and address this up front
and avoid similar misunderstandings.
\end{itemize}

We have responded to all comments point-by-point, below.

\section*{Response to the reviewers}
% General intro text goes here
We thank the reviewers for their close reading of our manuscript and
insightful comments. In the following we address the points raised
in turn.

\reviewersection

\begin{point}
This manuscript presents the Demes data model and related libraries ecosystem.
I have been using it for some time and was happy to review the paper as I find
it really useful, intuitive and well designed (as are a lot of tools developed
by those authors and the whole community of the popsim consortium and tskit).
The main usage I have for it at the moment was to switch effortlessly between
msprime and SLiM using the same demographic model, loved that. But as described
in the paper, it has a lot of other qualities. Overall, this format is a really
useful addition to the field and which I hope will be largely adopted.
\end{point}
\begin{reply}
Thank you for the kind words, we are delighted that Demes has been useful
to you!
\end{reply}

\begin{point}
I don't have much to add to this manuscript other than a few comments that I
hope will help answer a few questions that other readers could have. My main
comment below is that an example of the YAML format should be included in the
main text.
\end{point}
\begin{reply}
We have addressed each of the comments in turn below.
\end{reply}

\begin{point}
I usually like when there is a link to the software directly in the abstract
 in those kind of papers, it avoids scrolling through it as links are
sometimes in different places in those type of manuscripts.
\end{point}
\begin{reply}
We agree that this is a useful practice.
In our case there are several software packages that are directly
and indirectly related to the Demes format as summarized in Table 1,
so we have instead added a link to the specification documentation in the abstract.
This documentation includes a tutorial introduction for users to learn
how to write a Demes YAML file, and a detailed specification that developers
may consult when implementing Demes support in their own sofware.
\end{reply}

\begin{point}
I suggest including the example of the YAML format in the main text.
This is too sad that the center piece of this paper has been relegated to the appendix :(
Show us how easy it is to understand the model from the YAML format!
Maybe FigA2 is a bit too long, but FigA1 A and B could work well.
\end{point}
\begin{reply}
Thank you for this excellent suggestion.
We have moved the example model in Figure A1.A (the HDM YAML) and
Figure A1.B (the model diagram) into the main text as Figures 1A and B,
while leaving the MDM form of the same model in the Appendix.
We have also added a section to the text that decribes the main features
of the model in relation to the YAML document.
\end{reply}

\begin{point}
A fair question that could be asked by any potential user could be: Are there
  versions to this data format? There is no version argument in the YAML file,
what if the format changes in the future? How would my model format would be
recognized by the parser? I get that it is designed with stability in mind and
it is done well, but we've seen formats change for other things through time
(e.g. in bioinformatics) to adapt to new usage.
\end{point}
\begin{reply}
We have had an issue open for this for some time (\url{https://github.com/popsim-consortium/demes-spec/issues/20}).
The current plan is to add a variable for this when the spec hits version 1.0, which will coincide
with this publication.
Files omitting a spec version will assume a default value of 1.0.
\end{reply}

\begin{point}
There might be the need to specify how the initialization time of the whole
  model is dealt with. This is kind of straightforward when thinking about
coalescent models for people used to it. But not for forward-time simulations.
Especially that in your examples there are no start\_time for the first
population. This might be hard for some users to wrap their head around.
\end{point}
\begin{reply}
Thank you for bringing this omission to our attention.
It's important that different forward simulators use the same strategy
for converting from times in the past (as used by Demes) into their
corresponding values when using a forwards-time convention.
The technique is to find the oldest non-infinite time in the model
and use this as the forwards-time reference point, suitably adjusted
for a burn-in. This same approach is used by DemesDraw for the model diagrams.
We have added a section to the specification detailing how this
conversion should be done
(\url{https://github.com/popsim-consortium/demes-spec/pull/159}),
and added a brief description to the ``Time units'' section in the manuscript's
appendix.
\end{reply}

\begin{point}
p2 you discuss the defaults of CLI and input parameter formats but not of
  APIs. It could be said that APIs require some learning (of both the interface
and the given programming language for users not used to it). Learning a new
API each time you change simulator is also time consuming.
\end{point}
\begin{reply}
We have updated the text to include this point.
\end{reply}

\begin{point}
p2 ``YAML format" you could specify that it is a data serialization language
\end{point}
\begin{reply}
We have updated the text to include this point.
\end{reply}

\begin{point}
p12 ``time units is in generations" you could precise it is the default value here, if it is?
\end{point}
\begin{reply}
There is no default value for the time units. This must be specified explicitly.
We mention that the time units field is mandatory in the new Example section
that describes the new Figure 1.
\end{reply}

\begin{point}
p12 ``such as years, accompanied by the generation time". Might be worth
  detailing that it is needed as simulators work with generations and not
years, so there needs to be some kind of conversion.
\end{point}
\begin{reply}
We have updated the text to include this point.
\end{reply}

\begin{point}
p12 ``function defined by start and end sizes" You should detail here what are
  the models of population expansion available and what is the default. Linear,
exponential, etc.?
\end{point}
\begin{reply}
We have updated the text to clarify that only the exponential population
size function is currently supported, but the specification may be updated
to include more.
\end{reply}

\begin{point}
p12 ``S are born via [...]" missing that S is a proportion
\end{point}
\begin{reply}
Fixed.
\end{reply}

\begin{point}
p13-14 ``Other implicit values [...]" until end of paragraph, this part with
  the example could be worth integrating into the main text as this is a major
thing to understand about the data model in my opinion.
\end{point}
\begin{reply}
We have rephrased this, and illustrated in the new ``Example'' section.
\end{reply}

\begin{point}
Section A3: you could also detail that having common parsers allows for
  consistent and informative error messages when it comes to missing values or
issues in formatting.
\end{point}
\begin{reply}
We have updated the Parsers main text section to make this point.
\end{reply}

\begin{point}
p15 what's the difference between ``genome annotations" and ``sequence annotations"?
\end{point}
\begin{reply}
Good point; we have deleted the ``sequence annotations''.
\end{reply}

\subsection*{Random thoughts I had while reading the paper, to discuss (or
not)}

\begin{point}
I wondered if such format couldn't be used as a way to increase (spoken)
  language inclusivity. I guess it would be fairly easy to have dictionaries
including multiple translations for each keyword of the format in the parser.
That way you could write your model in Japanese or Spanish and it could still
be read without issue.
\end{point}
\begin{reply}
The field names in a YAML file are variables, not words \textit{per se}.
Thus, allowing different inputs for the field names is like having code trying
to understand different names for the same piece of data.
While it would be useful to allow users to generate files with variable names
that are not English words, and in character sets other than Latin, those files would
require additional conversion into the variable/value format defined by \textit{demes}.
We feel that such conversion is beyond the scope of this project.
\end{reply}

\begin{point}
Do you think such plain text format also provide a better substrate for
versionning demographic models? Could be mentioned.
\end{point}
\begin{reply}
While plain-text formats are ideally suited for use with version control
systems such as git, almost all competing/custom formats are also
plain text and thus share this property. However, Demes models are more
clearly separated from code and CLI directives, which may make model
changes more obvious when reviewing change requests such as git pull requests.
\end{reply}

\reviewersection

\begin{point}
In their paper entitled ``Demes: a standard format for demographic models",
Gower et al. propose a new way to describe (historical) demographic models of
populations.

The aim of Demes is to provide a standardized way to describe demographic
models to be used by various programs performing genetic simulations, which
currently have different ways to specify the simulated demography. The
intention is to facilitate information exchange and avoid translation problems
when passing from say a simulation software to a parameter estimation program,
if both would use the same input file to describe a demographic model.

The Demes data model is implemented in a text file using the YAML serialization
language, which is often used for configuration files or bioinformatics
pipeline descriptions. YAML uses a convenient human-readable syntax, that is
relatively intuitive to read. Two formats are proposed: a Human data Model
(HDM) supposedly easy to read by human users, and a more elaborate Machine Data
Model (MDM) used for processing. A parser (implemented as Python or C
libraries) is provided to translate HDM into MDM. Overall, I find this
initiative highly welcome as it makes a lot of sense to have the same input
file that could be used by several programs. One could thus use a given
demography inferred from neutral markers (e.g. dadi) to specify a model using
selection implemented using a forward in time simulator (e.g. slim). This is
possible since the specification of demography is separated from the
specification of the genetic model (selection, mutation, recombination).

I have found the paper clearly written and I have just a few comments,
regarding the structure of the paper and the Demes implementation itself.
\end{point}
\begin{reply}
Dear Laurent,

Thank you for the kind words and the careful and insightful review.
We have found your comments extremely useful, helping us to see
parts of the manuscript where we had failed to sufficiently explain
key points and motivations. We have addressed each of your points
in turn below, and would be grateful for any further thoughts
you may have.

On a practical note, we have waited until Demes and the surrounding
software ecosystem is quite mature before submitting this paper
for publication. On the upside, this means that the features
we are describing are battle-hardened, and the abstractions that
we are using have worked well in practice for a range of
applications. On the downside, however, this means that our ability
to change the specification is rather limited, as we hope you
will appreciate.

\end{reply}

\begin{point}
I have two major concerns. The first one is about the two formats: HDM and MDM.
Contrary to what the author say, I have found the HDM more confusing to read
than the MDM, which is much more explicit (see minor comments below). My
impression is that naive users would more easily create mis-specified models in
HDM than MDM, as HDM is using a lot of implicit definitions (to avoid
redundancy). My relative extensive exposure to the feedback of users reporting
bugs for my own software tells me that users do not carefully check the
structure of their demographic models, and explicit definitions help them in
making it right. Related to that, it is unclear if the default format should
always be the HDM, which would each time need to be parsed into MDM or if
programs could use the MDM as the default input file format. Having both
possibilities would be great, but personally I would just remove the HDM
format, and add some of its nice features to the MDM (e.g. defaults)
\end{point}

\begin{reply}
This is a reasonable point, which we have addressed by
(a) making the rationale for the HDM clearer in the text; and (b) adding an
example IM model in HDM format, which we briefly explain. Hopefully
this will be enough for readers to understand the basic elements of the
model.

A key goal of the HDM is human \emph{understanding} --- we hope that
users can look at a Demes model (and perhaps the demesdraw rendering)
as illustrated in the new figure, and understand key features of it.
Perhaps people may even include models used in the text of their
papers (as we often see for \texttt{ms} command lines!).
Conciseness, and avoiding the repetition of ``obvious'' values,
is a key part of making this representation human-readable.

In terms of user-friendliness, we think that
the tools available for working with
Demes really are substantially better than the status quo.
We've spent considerable effort trying to anticipate ways in which users
could create erroneous demographic models. By design the Demes spec
makes many classes of model misspecification impossible to write,
it requires that errors are raised when models are ambiguous,
and it explicitly documents the required behavior for edge cases.
Of course, there will be some aspects of the HDM that will be
confusing for new users --- nothing is perfect.

The HDM was explicitly designed to be a less-verbose version of the MDM,
and our initial approach was to simply allow the omission of some fields
when there was an obvious default.
The result is that an MDM model is a valid HDM model (but not vice versa),
so users are free to be more explicit in any part of a model.
We have also pointed out that it is not necessary to be maximally
concise when describing models, and we encourage redundancy where
it increases legibility.

We provide the following resources to assist users when writing models.
\begin{itemize}
\item
The Demes tutorial
(\url{https://popsim-consortium.github.io/demes-spec-docs/main/tutorial.html})
is a gentle introduction to writing Demes models.
It covers 99\% of anticipated uses and assumes very little about the
reader's knowledge of population genetics.
\item
The C and Python parsers have command line interfaces that take HDM as
input and can output an MDM.
So if a user is in doubt about the interpretation of some aspect of the HDM,
then they can easily check how their model will be interpreted.
\item
Demes models can be visualised using DemesDraw
(\url{https://github.com/grahamgower/demesdraw}),
which provides a sketch of how demes are related and their relative sizes.
\end{itemize}

Finally, there is no requirement for simulation software to include
an HDM to MDM parser. As an example, the demes-slim
(\url{https://github.com/grahamgower/demes-slim/})
converter only accepts MDM models (and only in JSON format).
It thus relies on users converting HDM models using an
external parser, which is a small inconvenience for a large gain
in interoperability.
\end{reply}

\begin{point}
The second concern is about the use of a backward-in-time timescale (zero =
present) with a forward in time definition of epochs. It leads to strange
definitions, like a start time set to infinity (as in Fig. A1C), which will be
difficult to implement in forward in time simulators, as they always need to
define a burn-in time. I thus found quite confusing that a start\_time is always
larger than an end\_time. It would make more sense to use a backward approach
for both the time definition and the demographic definition of epochs (maybe
begin with the more recent epochs and finish with the more ancient ones).
\end{point}
\begin{reply}
This is an entirely reasonable point, and a question that
we agonised over at length. There is no correct answer, but
we believe that the current choice is the ``least-bad'' option.
All decisions about Demes are publicly recorded in github issues,
pull requests, and many experiments in both writing models and implementing
Demes-related software.
Two discussions relevant to the timescale are
\url{https://github.com/popsim-consortium/demes-python/issues/2}
and
\url{https://github.com/popsim-consortium/demes-python/issues/11}.

To summarise:
\begin{itemize}
\item
``Backwards time'' language is confusing when talking about migrations
(does one mean migrating individuals, or coalescent lineages moving
between demes?)
\item
Anchoring time in the past and having time values increase towards
the present would require that users manually translate from ``years ago''.
This is tedious and error prone.
\item
``Now'' is a natural time anchor.
\item
There is no perfect choice, but we must make a choice.
Supporting multiple timescales was considered, but this would be a
substantial undertaking without a clear benefit.
\end{itemize}

Demes uses forwards-time conventions in many places, such as for
``source'' and ``dest'' migration fields (migrants leave a ``source''
deme and have offspring in a ``dest'' deme),
epoch ordering (from the past towards the present),
and a deme must be listed in the input file before its
descendant demes.
This latter requirement means that older demes are listed first and that
demes are topologically sorted according to ancestry relationships.
Most non-trivial models of biological relevance are treelike with one
``root'' deme that stretches back in time towards infinity.
The ordering in Demes naturally aligns with how rooted trees are typically drawn:
top-down as in computer science (e.g.~Knuth)
or left-to-right as in phylogenetics
(e.g.~Felsenstein)

\end{reply}

\begin{point}
p. 12 Population dynamics. I am not sure that it is necessary to assume a
  specific population dynamics model (e.g. Wright-Fisher model) for a data
format. Similarly, it is not necessary to define how allele frequencies would
change in the population upon selection. Some simulators will implement
selection differently (viability selection, fecundity selection, and selection
could be implemented before or after migration) and, again, this does not
belong to a demography specifying data format. I understand that the authors
wanted to distinguish soft from hard selection, where population size might
depend on average fitness in the latter mode, but there ae so many ways to
implement selection that it should be left to the specific programs reading the
demography.
\end{point}
\begin{reply}
Excellent point, and we entirely agree. We have tried to clarify by simply
stating in the main text that dynamics are assumed to be Wright-Fisher,
with a more precise statement in the Appendix. We have also tried to clarify
the motivation for stating these assumptions in the main text.
\end{reply}

\begin{point}
- p. 12 selfing and cloning. These two parameters are making more sense for
  forward than backward simulators, and they thus do not seem to be fully
generic.
\end{point}
\begin{reply}
Indeed---this is a point that we had very long and detailed discussions on.
In the end, the decision to include them is pragmatic, since several of the
initial software implementations consider these parameters to be part of
the demography. We have updated the text in the paper to reflect this
choice, and stressed the pragmatic approach to defining the line between
genetics and demography.
\end{reply}

\begin{point}
p. 13: Defaults and inheritance. It is not entirely clear to me what happens
  when a default parameter is superseded in a given ancestral population. Is
the new value inherited in the daughter populations or is it the default value
that applies?
\end{point}
\begin{reply}
Default values are inherited hierarchically through the \emph{document},
rather than the graph of demes.

Please see the specification
(\url{https://popsim-consortium.github.io/demes-spec-docs/main/specification.html#defaults})
and tutorial
(\url{https://popsim-consortium.github.io/demes-spec-docs/main/tutorial.html#setting-defaults})
for the detailed documentation on this.
\end{reply}

\begin{point}
p. 14. Parsers. It is a good idea to provide HDM to MDM parsers, but I note
  that each program reading an MDM will also implement its own parser to read
it and translate it to program specific parameters. Again, it would make more
sense to me to have a single format and no parser. Rather some parser making
sure that the demographic model makes sense could be useful: i.e. checking that
population ancestors have all been defined, or that end\_times are compatible
with start\_time.
\end{point}
\begin{reply}
It's not necessarily true that a program that accepts the MDM will need to
write its own Demes (MDM) parser. For example, msprime uses the \texttt{demes}
Python module to accept demographic models in Demes format. In effect,
what msprime does is take the object model provided by the parser
which is identical to an MDM document. Msprime then processes this document,
converting it into its own internal Demography format. Once the conversion
into msprime's internal format is complete, \emph{this} object
model is validated using precisely the same code as if the model
had been produced by msprime's own demography APIs.
The only validation that msprime does on the Demes MDM document
is to check for parameters that it doesn't support (like selfing and cloning
rates).

Thus, from msprime's perpective, there is no difference between what we
have currently (the HDM converted to MDM via the parser) and what
you propose (the MDM validated by the parser).

If the HDM did not exist, we would still require parsers because validating
the full MDM is a lengthy and tedious task.
A key goal that we set was for the ecosystem to be ``batteries-included''
from the outset, which is why we have implemented several high-quality
parsers (Python, C, Rust, and Julia) which should
cover the vast majority of use-cases.
\end{reply}

\begin{point}
p. 16: migration section might become hard to read when there are more than a
  few migration rates to specify. A migration matrix representation would make
more sense, or a way to simplify notation, by perhaps specifying some template
for the parameters identifiers and only specify the values: Template: \{source,
dest, start\_time, end\_time, rate\} And then just entering values: \{ X, Y, 100,
0, 50, 0.0001\}
\end{point}
\begin{reply}
YAML supports a similar type of shorthand, and there are various
examples in the tutorial where we use it to increase legibility,
e.g.
\url{https://popsim-consortium.github.io/demes-spec-docs/main/tutorial.html#exponential-size-changes}

\end{reply}
\begin{point}
p. 17 : Whole HDM model is looking strange for a naive user, with a lot of
  implicit parameters probably set at default values. For instance
is start time infinity by default for ancestral population?
\end{point}
\begin{reply}
We have updated the main text to note that the goal of the paper is not to give the reader
a detailed understanding of the Demes model, but to argue for its necessity
and to explain the rationale for key design decisions.
However, to give the reader a very basic
understanding of how Demes works, we have added a new Example section that
gives a simple IM model, with a walk-through in the text.
Hopefully this, along with the extensive online materials
(in particular the tutorial), will be sufficient to address specific
points like this on the structure of the model.

\end{reply}
\begin{point}
AFR and EUR have no starting times, which suggests that they are created when
  their ancestral population ends. But what if AFR and EUR were branching off
ANC at different times?
\end{point}
\begin{reply}
See response to point 2.9.
\end{reply}

\begin{point}
migrations section looks strangely defined. Is symmetric migration
  implemented by default? Starting time is not specified for first epoch. Does
it mean the epoch starts with the creation of AFR and EUR? But again what if
AFR and EUR would have been created at different times?
\end{point}
\begin{reply}
See response to point 2.9.
\end{reply}

\begin{point}
It is good to have a standard way of specifying a given demographic scenario,
  but it would also be good to have a standard way to specify genetic data to
be simulated, and to link it to the demography.
\end{point}
\begin{reply}
We absolutely agree! However, as we stress in several places in the updated
text this is outside the scope of defining a demographic model, and this
is a vital simplification. We note in the updated ``Scope of the
Specification'' section that Demes could be part of a standard simulation
specification by embedding the demographic model within it.
\end{reply}

\begin{point}
Finally, I see the point of separating a model from the sampling, but in some
  cases it could be beneficial to mention them in the same file, for instance
if you would like to simulate data at different time points from the same
population. Making sample size part of the properties of a deme would make it
more efficient than specifying it from outside.
\end{point}
\begin{reply}
See the previous response for the general point; it is
more general and extensible to embed a demes model \emph{within} an
overall simulation description, than it is to include details required
for a simulation in the Demes model.
\end{reply}

\end{document}