Support for Streaming/Incremental JSON Parsing #219
Description
Currently, yyjson does not support a streaming or incremental parsing mode. This limits its use in network-oriented scenarios such as TCP or HTTP ingestion, where:
- Multiple complete JSON messages may be received in a single buffer
- The last message might be incomplete due to chunked delivery
- Applications need to parse valid messages while buffering the rest
In Fluent Bit, for example, we use a tokenizer like jsmn that allows us to track the last byte consumed during parsing. This makes it possible to process the valid portion of the buffer and leave the remainder for the next read cycle.
Desired Behavior
We would like yyjson to support a mode that allows:
- Parsing of one or more complete JSON documents from a buffer
- Detection and handling of incomplete trailing content
- Reporting of how many bytes were successfully consumed
- Graceful partial parsing without requiring the entire buffer to be valid
Suggested Features
- A new return value (e.g., "partial success") indicating that a valid JSON document was parsed, but trailing data remains
- A simple way to retrieve the number of bytes consumed from the input buffer
- An optional parsing flag to enable multiple-document or stream-aware parsing
- An optional decoder state object that maintains parsing position and allows repeated extraction of JSON values from a buffer
- A tolerant mode that accepts trailing or incomplete data and stops at the end of the last valid JSON element
These features would make yyjson viable for high-performance streaming environments and enable it to replace low-level tokenizers used in Fluent Bit.
Describe alternatives you've considered
keep as is now, or extend jsmn by using our SIMD helper, however we think would be better to move to yyjson.
Optional idea
Introduce a simple state...
yyjson_decoder_state state;
yyjson_decoder_init(&state, json, len);
while ((doc = yyjson_decoder_next(&state)) != NULL) {
....
}