Behavior when parsing from stream with multiple JSON objects?

[dtgriscom-ta]

I'm interested in using this library to parse a TCP stream containing multiple LF-terminated JSON objects ("ndjson"). Scanning the issues database, it looks like this is possible, e.g.:

• Behavior of operator>> should more closely resemble that of built-in overloads. #367
• Parsing multiple JSON objects from a string or stream #438
• How to parse multi object json from file? #1470

However, the documentation doesn't mention this at all:

• https://json.nlohmann.me/api/basic_json/operator_gtgt/
• https://github.com/nlohmann/json#user-content-tofrom-streams-eg-files-string-streams

Does the streaming parser read a single JSON object/array/item and then stop reading, allowing a subsequent parse to work on a subsequent object/array/item?

[nlohmann]

Could this help? https://json.nlohmann.me/features/parsing/json_lines/

[dtgriscom-ta]

(Four-minute response time? Not bad....)

Reading a line into a string buffer and then parsing the buffer will clearly work, but if I could avoid that intermediate step I'd save memory and likely some CPU cycles (assuming the library doesn't itself read in the entire stream to EOF before starting to parse).

When parsing from a stream, when does the library stop? If it stops after a complete object/array/item, and then allows parsing a new object/array/item from the same stream, then Life Would Be Good. If instead it continues until it hits the end of the stream, then I'd have to do the string buffer trick. (And either way, your doc should probably elucidate.)

[nlohmann]

• The library does not make a copy of the data during parsing.
• When parsing from a stream, the library parses until a complete value has been processed. As described in https://json.nlohmann.me/features/parsing/json_lines/, code like

json j;
while (input >> j)
{
    std::cout << j << std::endl;
}

will not work, because the parser will try to parse one value after the last one.

However, you could try to catch that exception.

[dtgriscom-ta]

I'm guessing that the parser will throw the exception after it has begun reading the next value after the first one, which means I'd have to push that text back onto the stream before the next time I try to parse. Right?

It looks like parse, accept and operator>> do blocking reads until they hit an error or an EOF (right? If so, would be good to add to the doc.)

If I use the parser_callback_t hook, I can detect when the end of the top-level value is received, and could even grab the value of the top-level JSON object. but the bool "discard this item" return value can't tell the parser "stop right there"; it'll keep parsing into the next object, triggering that exception and requiring me to push back again.

Since the parser can unambiguously detect the end of the top-level value before it starts reading the next value (right?), could there be a new option to have the parser read one value and then return, leaving subsequent data on the stream ready to be read? (I doubt you want to add yet another optional parameter to the parse() function, but I don't know how else you'd do this...)

[dtgriscom-ta]

Looking at the PR, I'm not clear: does the change have the stream parser stop parsing when a complete JSON object/array/item has been consumed?

[imwhocodes]

The parser already stop reading when the object/value end if std::istream is used as input
The PR that I linked is addressing a corner case where with a custom iterable it was consuming one char more than the one it should

Can you explain your problem? (I'm still working on the PR so maybe it could be addressed)

[dtgriscom-ta]

I would like to be able to feed a stream of JSON objects/arrays/items, and have the parser stop reading from the input stream when a single complete object has been read. Then, you would be able to invoke the parser a second time on the same stream to read the next object. Now, the parser continues past the end of the first object and into the second, triggering an exception, which makes handling an ongoing stream difficult.

[imwhocodes]

Are you sure?

Check this example , as you can see every iteration of the loop consume one single json object,
the exception is the fact that the parser try to read an empty buffer

[edit]:
extended example

[dtgriscom-ta]

I'm in no way sure what it actually does, but your example does seem to work, although the author above says that it won't work. Again, it would be lovely if the doc was clear one way or another.