Consider design/implementation of consistent format for "networked $THING with attachments"

[eyelidlessness]

Background

There are a few similar concepts that fall into this category:

1. Form:

   • $THING: Form definition (ODK XForms XML)
   • attachments: anything specified to be referenced by jr: URL

   Currently, we handle this by parsing the form XML, identifying the resources, and calling a client-defined fetch-like function to retrieve the attachments.

2. Instance (outgoing for submission):

   • $THING: Instance XML, as serialized from form state
   • attachments: any file/upload data associated with a form question's value state, serialized as files separately from the instance XML (their respective nodes serialized with their respective file names)

   Currently, we handle this by building a (subclass/subtype of) FormData. The design of this web standard is closely analogous to multipart/form-data HTTP semantics.

3. Form and instance (incoming for edit):

   • $THING (1): As in item 1
   • attachments (1): As in item 1
   • $THING (2): As in item 2
   • attachments (2): As in item 2

   A serialized instance and its attachments does not contain sufficient information to initialize form state, we actually need the superset of items 1 & 2 to do that. We are currently designing an additive interface in Engine/client interface: editing serialized instance state #299, angling toward a solution conceptually similar but slightly different to the solutions for item 1 (and less similar to the solution for item 2, but likewise conceptually related notwithstanding network concerns).

I'm posting this now because the solution we're angling toward in #299 is probably right, but it smells really suspicious to have three similar-but-different solutions for the same general concept.

That smell is especially strong when we look at the prior art in item 2. Our solution there roughly corresponds to two HTTP transport designs, both having strengths in terms of reflecting HTTP semantics as a well worn path for expressing this kind of pattern.

Generalization: hypertext

I want to make this proposal as loose as possible, focused mainly on the underlying principles. In all three cases we have:

1. One or more document resources, each of which is incomplete or drastically limited without...
2. ... N ancillary resources, referenced by those documents.

We expect to load those ancillary resources (approximately) immediately, and (at least potentially) concurrently.

This is pretty much verbatim how one might describe "hypertext". I think it would be very compelling to define a meta-spec that wraps these bundle-y concepts up with consistent use of standard HTTP concepts.

Some pertinent concepts (non-exhaustive! not meant to be proscriptive!):

• As with outgoing instance submission, multipart/form-data might be conceptually pertinent. It is a suitable bidirectional payload format, capable of expressing the bare minimum requirements for every case considered above to function. However, multipart (alone) would tend to be in conflict with potential for concurrent loading.

• Another standard for referencing related resources in an HTTP payload is the Link header. This would facilitate both immediate and concurrent loading.

Supplementary to existing solutions

If we were to explore this concept, it would necessarily be supplementary to existing solutions. The Web Forms project isn't in a position to impose some new HTTP-based spec on integrators. For that matter, if we were to pursue something like this, we'd want to do it in collaboration with existing or interested integrators. But we'd need to continue supporting existing usage patterns in the meantime, and probably indefinitely.

The goal here isn't to replace existing concepts, but to think about how we could supplement them with a coherent design for each of these rather similar cases... and ideally without introducing yet another competing standard in the process.