ReferenceFormSet: Allow fixing invalid reference order.

If a project has an invalid order of references, we must provide a way
for the author or editor to review and fix the order.

The past behavior of the "project content" and "copyedit" pages was
that ReferenceFormSet would set the 'order' field for every form in
the formset.  The effect of that, however, was that 'order' would be
set for:
  (a) newly added references
  (b) existing references whose 'description' was changed
References that already existed, and whose 'description' field was
unchanged, would not have been saved (because their form.changed_data
was False.)

Since many existing projects may have scrambled lists of references
and can't be fixed automatically, we want broken projects to be
"flagged", and we want that flag to persist until someone is able to
review the project and check that it's fixed.  In particular, simply
opening the "project content" page and submitting it, without making
any changes, should not cause the flag to be cleared.

Therefore: if the project has an invalid reference order, then display
a warning message with a checkbox (at the bottom of the page, below
the reference list and above the submit button.)

By default, that checkbox isn't checked, and in that situation,
ReferenceFormSet should not alter any existing reference 'order'.
Newly-added references should have a correct 'order', but if existing
references were "invalid", they should remain "invalid".

If the checkbox is checked, then ReferenceFormSet should set all
reference 'order' to match the current form order (which is what the
old code was trying to do, and didn't work because it didn't set
changed_data.)

physionet-django/console/templates/console/copyedit_submission.html

@@ -54,6 +54,7 @@ <h3>Description</h3>
           {% include "project/content_inline_form_snippet.html" with form=description_form %}
           {% include "project/content_inline_form_snippet.html" with form=ethics_form %}
           {% include 'project/item_list.html' with item="reference" item_label=reference_formset.item_label formset=reference_formset form_name=reference_formset.form_name add_item_url=add_item_url remove_item_url=remove_item_url %}
+          {% include "project/confirm_reference_order_form.html" %}
           <h3>Access</h3>
           <div id="access">
             {% include "project/content_inline_form_snippet.html" with form=access_form %}

physionet-django/project/forms.py

@@ -732,11 +732,20 @@ class ReferenceFormSet(BaseGenericInlineFormSet):
     item_label = 'References'
     max_forms = 50
 
-    def __init__(self, *args, **kwargs):
-        super().__init__(*args, **kwargs)
+    def __init__(self, data=None, *args, **kwargs):
+        super().__init__(data, *args, **kwargs)
         self.max_forms = ReferenceFormSet.max_forms
         self.help_text = 'Numbered references specified in the metadata. Article citations must be in <a href=http://www.bibme.org/citation-guide/apa/ target=_blank>APA</a> format. Maximum of {}.'.format(self.max_forms)
 
+        # If user checked the "confirm reference order" box then they
+        # are confirming the order of references (as displayed in the
+        # form) is correct.  If user did not check that box, then do
+        # not touch the existing order.
+        if data and data.get('confirm_reference_order') == '1':
+            self.confirm_reference_order = True
+        else:
+            self.confirm_reference_order = False
+
     def clean(self):
         """
         - Check max forms due to POST refresh issue
@@ -760,9 +769,26 @@ def clean(self):
                 descriptions.append(description)
 
     def save(self, *args, **kwargs):
-        # change the value of order. set it as index of form
-        for form in self.forms:
-            form.instance.order = self.forms.index(form) + 1
+        if self.confirm_reference_order:
+            # If "confirm reference order" was checked, set the order
+            # of all references in the formset.
+            for form in self.forms:
+                form.instance.order = self.forms.index(form) + 1
+                form.changed_data = True
+        else:
+            # If "confirm reference order" was not checked, then set
+            # the order only for newly created references, leaving
+            # existing references alone.  New references should have
+            # "order" greater than any existing reference.
+            max_order = 0
+            for form in self.forms:
+                if form.instance.order is not None:
+                    max_order = max(max_order, form.instance.order)
+            for form in self.forms:
+                if form.instance.pk is None:
+                    form.instance.order = max_order + 1
+                    max_order += 1
+
         super().save(*args, **kwargs)
 
 

physionet-django/project/templates/project/confirm_reference_order_form.html

@@ -0,0 +1,28 @@
+{% comment %}
+Past bugs in the project editing forms can result in references having
+'order' set to None, or the order of 'order' not matching the order
+displayed in the content/copyedit page.  It is impractical to repair
+all existing projects automatically since that requires guessing the
+author's intent.
+
+Therefore, if a project's reference order is undefined or
+inconsistent, the message below should be displayed on the content
+page (when the project is author-editable) or copyedit page (when the
+project is copyeditable.)
+
+The "confirm_reference_order" checkbox will be handled by
+ReferenceFormSet (see project/forms.py).
+{% endcomment %}
+{% if not project.has_valid_reference_order %}
+<div class="alert alert-form alert-warning">
+  <div>
+    The References list may be incorrect due to a server error.
+    Please verify that the list shown above corresponds to the
+    correct numbered citations in the project text.
+  </div>
+  <label>
+    <input type="checkbox" name="confirm_reference_order" value="1">
+    Order of references is correct
+  </label>
+</div>
+{% endif %}

physionet-django/project/templates/project/project_content.html

@@ -17,7 +17,7 @@ <h2 class="form-signin-heading">2. Project Content</h2>
 <p>Please adhere to the standards specified in the helpstrings. Required fields are indicated by a <a style="color:red">*</a>.</p>
 <hr>
 
-<form action="{% url 'project_content' project.slug %}" onsubmit="return validateItems('reference-list', 'description', 'References')" method="post" class="no-pd">
+<form action="{% url 'project_content' project.slug %}" onsubmit="return validateItems('reference-list', 'description', 'References')" method="post">
   {% if not project.author_editable %}
     <div class="alert alert-form alert-warning alert-dismissible">
       <strong>The project cannot be edited right now.</strong>
@@ -32,6 +32,7 @@ <h2 class="form-signin-heading">2. Project Content</h2>
   {% include "project/content_inline_form_snippet.html" with form=description_form %}
   {% include 'project/item_list.html' with item="reference" item_label=reference_formset.item_label formset=reference_formset form_name=reference_formset.form_name add_item_url=add_item_url remove_item_url=remove_item_url %}
   {% if is_submitting and project.author_editable %}
+  {% include "project/confirm_reference_order_form.html" %}
   <hr>
   <button class="btn btn-primary btn-rsp btn-left" type="submit" name="edit_description">Save Description</button>
   {% endif %}