Won't emit a document until the next document is available

[edward@…]

parse_document_end won't emit an event until it has seen the next token after the document end. This makes it difficult to use PyYAML to parse live streams of documents in a timely way, because the next document may not actually be available for some undetermined amount of time.

Using PyYAML-3.04.

[xi]

Does the stream contain the document end marks: "...\n"? These marks are necessary for parsing live streams.

[edward@…]

Program demonstrates problem

[edward@…]

Yes, the stream contains "...\n".

I just attached a script that demonstrates the issue. Run it once with "--server" and once without to test over loopback.

You'll see that the receiving side is always one message behind the sending side.

I think the problem is on parser.py line 189. This loop won't exit until the next non-DocumentEndToken? is seen. If no token is available, it blocks. This only seems necessary if multiple DocumentEndTokens? will occur in the stream -- is that even legal?

You'll see in the script that I'm wrapping a socket object to get a stream with "read" and "write". I have also tried the built-in "socket.makefile", but this performs even worse, because the read doesn't return any data until the whole buffer is full (or EOF).

[xi]

Fixed in [249] and [250].

Thanks for the bug report. Your analysis of the problem is correct. Multiple document end indicators are, indeed, allowed by the yaml specification, ironically, to make emitting a stream of yaml documents easier.

[edward@…]

Thanks for the quick resolution.