Modify

Ticket #51 (closed defect: fixed)

Opened 7 years ago

Last modified 7 years ago

Won't emit a document until the next document is available

Reported by: edward@… Owned by: xi
Priority: normal Component: pyyaml
Severity: normal Keywords:
Cc:

Description

parse_document_end won't emit an event until it has seen the next token after the document end. This makes it difficult to use PyYAML to parse live streams of documents in a timely way, because the next document may not actually be available for some undetermined amount of time.

Using PyYAML-3.04.

Attachments

test_socket.py Download (999 bytes) - added by edward@… 7 years ago.
Program demonstrates problem

Change History

comment:1 Changed 7 years ago by xi

Does the stream contain the document end marks: "...\n"? These marks are necessary for parsing live streams.

Changed 7 years ago by edward@…

Program demonstrates problem

comment:2 Changed 7 years ago by edward@…

Yes, the stream contains "...\n".

I just attached a script that demonstrates the issue. Run it once with "--server" and once without to test over loopback.

You'll see that the receiving side is always one message behind the sending side.

I think the problem is on parser.py line 189. This loop won't exit until the next non-DocumentEndToken? is seen. If no token is available, it blocks. This only seems necessary if multiple DocumentEndTokens? will occur in the stream -- is that even legal?

You'll see in the script that I'm wrapping a socket object to get a stream with "read" and "write". I have also tried the built-in "socket.makefile", but this performs even worse, because the read doesn't return any data until the whole buffer is full (or EOF).

comment:3 Changed 7 years ago by xi

  • Status changed from new to closed
  • Resolution set to fixed

Fixed in [249] and [250].

Thanks for the bug report. Your analysis of the problem is correct. Multiple document end indicators are, indeed, allowed by the yaml specification, ironically, to make emitting a stream of yaml documents easier.

comment:4 Changed 7 years ago by edward@…

Thanks for the quick resolution.

View

Add a comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
The resolution will be deleted. Next status will be 'reopened'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.