Skip to content

APISerialization

Jump to bottom Edit New page
Andrew Byrd edited this page Feb 18, 2014 · 9 revisions

This page contains developer-oriented notes about how OTP REST API responses are converted to a textual format that is sent across the wire to the client.

Definitions

This Stack Overflow response clarifies the relationship: http://stackoverflow.com/a/17981041/778449 The JAX-RS implementation needs to receive and send objects (request and response bodies) over HTTP. When JAX-RS wants to return a response with the application/xml media type, it uses JAXB to figure out how to write objects out as XML.

Serialization to XML vs. JSON

JAXB entails adding annotations to classes indicating what fields should be serialized and how, such as @XmlRootElement and @XmlElement (actually, the original intent was to generate annotated class files from an XML schema). If we want to also provide other formats such as JSON, the JSON library could re-use the JAXB annotations to provide equivalent serialized responses for the same annotated objects. So the JAXB annotations (despite including the letters XML) can be applied to an XML or JSON serialization process.

However, there are ugly points in using JAXB to produce JSON. For example. the attributes vs. content distinction in XML does not naturally exist in JSON. When going through JAXB to produce JSON, HashMaps will also become strangely verbose. It is therefore often cleaner to simply use a library that converts objects directly to JSON. See: http://stackoverflow.com/questions/6001817/jackson-with-jaxb

While it is theoretically possible to use the same model classes and annotations to produce exactly equivalent responses in multiple formats (XML and JSON) in practice it is difficult to keep the two in sync, because special cases will be described and handled differently by the two libraries in use. Therefore it is likely that we will drop XML support in version 1.0 of the OTP API to avoid future maintenance concerns. Research has also shown that JSON is much less resource intensive on mobile devices.

Enabling JSON output from JAX-RS

It appears that Jersey does not automatically include a JSON MessageWriter, because if you remove the Maven dependency com.fasterxml.jackson.jaxrs:jackson-jaxrs-json-provider from the POM, then requests for Content-type:appication/json will fail with a:

javax.ws.rs.WebApplicationException: com.sun.jersey.api.MessageException: 
A message body writer for Java class org.opentripplanner.api.model.transit.RouteList, and Java type class org.opentripplanner.api.model.transit.RouteList, and MIME media type application/json was not found.

You need a JSON library that implements the extension points MessageBodyReader/Writer of JAX-RS, then declares its implementations to Java's ServiceLoader mechanism, allowing Jersey (and other software) to automatically detect and use it for JSON parsing/writing. OTP is using the Jackson library to supply JSON capabilities to Jersey.

Some wisdom gleaned from Stack Overflow and wikis on choosing and using a JSON library:

  • "Jackson and Gson are the most complete Java JSON packages regarding actual data binding support; many other packages only provide primitive Map/List (or equivalent tree model) binding." That is, Jackson and Gson allow marshalling and unmarshalling objects, other libraries may just parse the JSON into a tree of strings.
  • Jackson 1.x and 2.x live in different Java packages (1.x under org.codehaus.jackson, 2.x under com.fasterxml.jackson), be sure to import Jackson objects from matching versions.
  • Do not use Maven dependencies jersey-media-json (doesn't exist anymore in Jersey 2.x) and jersey-json (only for Jersey 1.x). Use either jersey-media-moxy (for JAXB) or jersey-media-json-jackson (for POJO). The Jersey User Guide has information on these modules in the JSON section at https://jersey.java.net/documentation/latest/media.html#json . Note that in OTP we are using yet another module from Jackson itself, I have no idea what the difference is with the jersey-media module.
  • "As of v1.18 of Jersey you do NOT need to write your own MessageBodyWriter and the return type of your @GET methods can be POJO-objects." That is, your Jersey resource methods can just return any old object, and Jackson will still be used to serialize it according to the usual rules.
  • "Jackson 1.7 added ability to register serializers and deserializes via Module interface. This is the recommended way to add custom serializers -- all serializers are considered "generic", in that they are used for subtypes unless more specific binding is found. The simplest way is to extend SimpleModule, add serializer(s), and register module with ObjectMapper."

POJO Mapping

We are using "POJO mapping" and "natural" serialization rather than JAXB. POJO mapping means just converting the public fields and getters of objects to JSON naturally, rather than requiring annotations.

We do need to somehow enable the "POJO mapping feature", which maps public fields and getters of objects to XML/JSON instead of requiring annotations, and "natural mapping" lest we automatically get "Badgerfish" JSON style which perfectly reproduces the semantics of XML but is bizarrely non-idiomatic in JSON.

POJO Serialization Rules

Jackson will simply include any public field or getter method when producing its JSON representation of an object.

Natural vs. Badgerfish Serialization

Custom JSON response serialization using Jackson and Jersey

The Jackson API provides multiple ways to do many things. Here are a few ways to specify a custom serialization method for a class:

The first three require annotations. In OTP we have a third-party class (AgencyAndId) that we want to serialize in a nonstandard way. There is always the possibility of wrapping the third-party class in a custom class we have control over, but imagine how many small modifications that will entail throughout the code. It would also be possible to add @JsonSerialize to every field of the third-party type, but some of those fields might be found in other third-party types (e.g. the One Bus Away model of GTFS which contains AgencyAndId fields we want to serialize in OTP Transit Index API responses).

The fourth option seems to be the best way to blanket-redefine how all instances of a third-party class are serialized. http://wiki.fasterxml.com/JacksonHowToCustomSerializers says that "Jackson 1.7 added ability to register serializers and deserializes via Module interface. This is the recommended way to add custom serializers."

Most of these methods require a separate serializer class, defined like so:

public class ThingSerializer extends JsonSerializer<Thing> {

    @Override
    public void serialize(Thing thing, JsonGenerator jgen,
        SerializerProvider provider) throws IOException, JsonProcessingException {
        jgen.writeStartObject();
        jgen.writeNumberField("x", thing.getNumber());
        jgen.writeEndObject();
    }

    @Override
    // Gets around type erasure, allowing 
    // module.addSerializer(new ThingSerializer()) to correctly associate this 
    // serializer with the proper type
    public Class<Thing> handledType() {
        return Thing.class;
    }

}

Once we have a custom serializer defined for AgencyAndId, we have to register it with Jackson. This involves making or getting an ObjectMapper, combining some serializers/deserializers into a Module, then registering that Module with the ObjectMapper, as described at http://wiki.fasterxml.com/JacksonHowToCustomSerializers. The ObjectMapper then constructs corresponding ObjectWriters which are used to output JSON. Unfortunately we are not calling ObjectMapper.writer().writeValue() manually. Instead JAX-RS (Jersey) is automatically writing out our result objects. Therefore need to somehow register our custom serializer Module with the specific ObjectMapper instance used by Jersey.

The only way I have found to do this is via a ContextResolver (thanks to this post http://jersey.576304.n2.nabble.com/Customizing-ObjectMapper-tp6234597p6234646.html). This ContextResolver is annotated with @Provider and detected by Jersey in package-scanning mode, in much the same way it detects REST resource classes or parameter classes. The purpose of a ContextResolver is to provide a specific ObjectMapper as a function of the type we are serializing. For example:

@Provider
@Produces(MediaType.APPLICATION_JSON)
public class OTPObjectMapperProvider implements ContextResolver<ObjectMapper> {

    private final ObjectMapper mapper;

    /**
     * Pre-instantiate a Jackson ObjectMapper that will be handed off to all incoming Jersey
     * requests, and used to construct the ObjectWriters that will produce JSON responses.
     */
    public OTPObjectMapperProvider() {
        // Create a module, i.e. a group of one or more Jackson extensions.
        // Our module includes a single class-serializer relationship.
        // Constructors are available for both unnamed, unversioned throwaway modules
        // and named, versioned reusable modules.
        SimpleModule module = new SimpleModule("OTP", Version.unknownVersion());
        module.addSerializer(AgencyAndId.class, new AgencyAndIdSerializer());
        mapper = new ObjectMapper();
        mapper.registerModule(module);
        mapper.setSerializationInclusion(Include.NON_NULL); // skip null fields
    }

    /** 
     * When serializing any kind of result, use the same ObjectMapper.
     * The "type" parameter will be the type of the object being serialized,  
     * so you could provide different ObjectMappers for different result types.
     */
    @Override
    public ObjectMapper getContext(Class<?> type) {
        return mapper;
    }

}

Tracing execution, we see that the Jersey IOC component provider instantiates the ContextResolver at startup, after the package scanning is finished and it logs "Initiating Jersey application". The specific ObjectMapper used to serialize a response is determined by calling getContext(Class type) while handling each incoming Jersey request. The "type" parameter contains the type of the response object being serialized, so you could provide differently configured ObjectMappers for different result types. The repeated calls to getContext are why we create the ObjectMapper once and return the pre-constructed instance in the getContext method.

Doc Links

Add a custom footer

The documentation on this wiki is outdated and should not be used

unless you are intentionally working with legacy versions of OpenTripPlanner. Please consult the current documentation at readthedocs

Clone this wiki locally