Ticket #51 (closed defect: fixed)

Opened 3 years ago

Last modified 3 years ago

Won't emit a document until the next document is available

Reported by: edward@sweetbytes.net Assigned to: xi
Priority: normal Component: pyyaml
Severity: normal Keywords:
Cc:

Description

parse_document_end won't emit an event until it has seen the next token after the document end. This makes it difficult to use PyYAML to parse live streams of documents in a timely way, because the next document may not actually be available for some undetermined amount of time.

Using PyYAML-3.04.

Attachments

test_socket.py (1.0 kB) - added by edward@sweetbytes.net on 04/16/07 12:18:54.
Program demonstrates problem

Change History

04/16/07 02:32:34 changed by xi

Does the stream contain the document end marks: "...\n"? These marks are necessary for parsing live streams.

04/16/07 12:18:54 changed by edward@sweetbytes.net

  • attachment test_socket.py added.

Program demonstrates problem

04/16/07 12:28:19 changed by edward@sweetbytes.net

Yes, the stream contains "...\n".

I just attached a script that demonstrates the issue. Run it once with "--server" and once without to test over loopback.

You'll see that the receiving side is always one message behind the sending side.

I think the problem is on parser.py line 189. This loop won't exit until the next non-DocumentEndToken? is seen. If no token is available, it blocks. This only seems necessary if multiple DocumentEndTokens? will occur in the stream -- is that even legal?

You'll see in the script that I'm wrapping a socket object to get a stream with "read" and "write". I have also tried the built-in "socket.makefile", but this performs even worse, because the read doesn't return any data until the whole buffer is full (or EOF).

04/17/07 13:44:29 changed by xi

  • status changed from new to closed.
  • resolution set to fixed.

Fixed in [249] and [250].

Thanks for the bug report. Your analysis of the problem is correct. Multiple document end indicators are, indeed, allowed by the yaml specification, ironically, to make emitting a stream of yaml documents easier.

04/17/07 14:02:41 changed by edward@sweetbytes.net

Thanks for the quick resolution.


Add/Change #51 (Won't emit a document until the next document is available)




Change Properties
Action