[ts-gen] redirecting shim output to python script

[Mike Thornton]

Hi Bill,

Just a quick note to confirm what Paul observes.  When I comment (# =
comment in python FYI) out all references to the poll command, the python
script is able to read the entire shim header info (what is printed to
stdout before any commands are sent).  However, when a poll command is used,
the script is able to read only part of the header -- the poll command
continually reports an empty stdout pipe partway through the header.

Thanks.
Mike



if __name__ == '__main__':
    shimProcess = subprocess.Popen('/usr/local/shim/shim --risk cout file
save', bufsize=1, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
stderr=subprocess.STDOUT)
    #shimPoll = select.epoll()
    #shimPoll.register(shimProcess.stdout)
    while True:
        #a = shimPoll.poll(0)
        #print "poll = %s" % a
        #while (len(a) == 0):
        #    time.sleep(0.05)
        #    a = shimPoll.poll(0)
        #    print "poll = %s" % a
        line = shimProcess.stdout.readline().strip()
        print line


On Thu, May 14, 2009 at 11:59 AM, Bill Pippin <pippin at owlriver.net> wrote:

> I'd like to thank Paul for the cogent report and his excellent
> diagnostic testing.  I'll be continuing to look into this, and
> might have more to say later, but for now, first, what I believe
> to be a work around for Mike's benefit; second, some comments on
> what Paul found, and third, some plans for changes to the shim's
> initialization setup to help with this issue.
>
> Thanks as well, Mike, for your thorough diagnostic posting; I would
> have been able to diagnose this without Paul's message, although his
> text handed me the solution on a plate.
>
> So, about what Mike is seeing:
>
> > I am still having a bit of difficultly reading data from a pipe ...
> >     ... my python script ... does not fully read all of the
> > initial shim account header information.  After reading a total of
> > about 8kB of data, the script enters a state where it continually
> > polls an empty pipe --- the rest of the ... info is not read.
>
> True; it almost certainly has not been sent yet.
>
> > ... the script is not locking up since it ends up continually
> > polling an empty pipe ...
>
> True; it's fair to say the script is livelocked; the shim is waiting
> for more messages to fill its buffers, and waiting as well for the
> commands that might trigger those messages.
>
> One safety measure that I believe is worth using for this problem is
> to always submit a market data subscription for something, anything,
> to ensure that messages are arriving on a regular basis.  Then, when
> they don't show up, you can begin looking for problems, e.g., with
> the IB tws network connection to the upstream servers.
>
> This hides but does not completely solve the problem here, however;
> as Paul's report makes clear, log text is being buffered in
> unacceptably large blocks.
>
> Mike, please consider using the log file in place of the stdout for
> now.  That is, start the shim with the file option, open the log
> file, default name log/ShimText, or whatever you've renamed it via
> your config file options, and read the log lines from that file.
>
> Yes, this is clumsy, and yes, it's marginally less efficient, and
> yes, it sacrifices --- for now --- the elegance of Unix pipes.  It
> should, however, work, which is the main virtue of this approach.
>
> Now, about Paul's report:
>
> > ...  The log/ShimText file contains all the expected output,
> > irrespective of what the pipe does.
>
> True.  The shim sets line buffering for the log file, in part so
> that the tail script, bin/tail.window, will work properly.  This
> is how I observe the shim in real time, in particular for testing;
> I run scripts, and glance at the tail'd output in a konsole window
> to see what's happening.
>
> > ...  Uncommenting the shim.puts line:
>
> >       # shim.puts "wait 10;\nquit;\n" # ...
>
> > causes the shim to quit after 10 seconds, at which point the rest
> > of the output then appears.
>
> True.  The stdout is flushed when a process, in this case the shim,
> terminates, and similarly, for all of the test scripts in the exs
> directory, delays in output are hidden by process termination.
>
> > ...  I can resolve the problem ... by adding a fflush(stdout) [in]
> > src/dump.c:
>
> >    Sig(print_cout) {    f.fputs(stdout,      b); fflush(stdout); }
>
> Clearly line buffering is not set for stdout when users include the
> cout option on the command line.  Looking at the logic, I see that
> I set line buffering for those files that are explicitly opened,
> but not for the stdout.  I vaguely remember deciding not to for
> reasons probably having to do with bulk data collection when using
> the cout option ...
>
> > ... [whether] the shim source code is correct as it is [?] ...
>
> ... nevertheless, block buffering makes the cout option useless for
> downstream apps using risk mode.  So, the shim needs to provide some
> form of user control over the choice of bufffering policy, about
> which more in the next section.
>
> >                                                  ... just that it
> > seems to resolve it for these python/ruby scripts, and so may give
> > us a clue.
>
> Indeed.  Paul, your diagnostics provide more than a clue, they
> pinpoint the issue exactly, and not so incidently suggest a work
> around.  Thank you.
>
> Now, about changes to the shim to deal with the problem Mike is
> seeing.  My plan for now, once given the cout option, is:
>
>    * for both risk and data modes, to make line buffering for
>      stdout the default; and
>
>    * for data mode only, to add an option, "bulk", to let the
>      operating system choose the buffer policy, and so maintain
>      the current behavior.
>
> Comments about this change are welcome, although I would need to
> be given *strong* reasons to justify any plan that fails to make
> line buffering the default in risk mode; Mike has uncovered an
> important issue on this point.  Also, other semantics might be
> added to the bulk option in the future, although this is uncertain.
>
> As for when the change will go out, I'm working in other parts of
> the code right now, and so don't plan to make a release today.  I
> do, however, expect to get one out soon, perhaps tomorrow, or else
> likely early next week.
>
> Mike, if you can't solve the problem you're seeing by using the
> log file, please feel free to repost, and again, thanks Paul.
>
> Bill
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.trading-shim.org/pipermail/ts-general/attachments/20090518/7e9e18ef/attachment.html