[ts-gen] redirecting shim output to python script

[Paul C]

Mike,
I've seem a similar problem before and have just tried your script. 
Some observations that may help somebody resolve the problem:

1. I get the same result when I run your python script (albeit I had to
remove the shimPoll lines before python would run - I don't know
python). In my case the output gets a bit further always
(consistantly) stopping at

   1280|36323|  15179183|3| 6| 2|NetLiquidation                  | 26500.15|USD | **** |

2. The following ruby script produces the *same* behaviour; again
stopping at the same point after the NetLiquidation line.

       path = "/home/paul/bin"
       prog = "shim --data file cout save"
       exec = "#{path}/#{prog}"
       shim = IO.popen(exec, "w+");
       #shim.puts "wait 10;\nquit;\n" # note that this line is commented out                                   
       shim.close_write   
       while line = shim.gets do
         puts line                 
       end 

Note1: Uncommenting the shim.puts line causes the shim to quit after 10
seconds, at which point the rest of the output then appears.

Note2: The log/ShimText file contains all the expected output,
irrespective of what the pipe does.

3. I can resolve the problem (for both python and ruby) by adding a
fflush(stdout) to src/dump.c:

   Sig(print_cout) {    f.fputs(stdout,      b); fflush(stdout); }

Please understand that I've only done this for diagnostic reasons. I'm
not suggesting that the fflush is the best solution to the problem (probably 
the shim source code is correct as it is), just that it seems to 
resolve it for these python/ruby scripts, and so may give us a clue. 

Regards

Paul C


On Wed, May 13, 2009 at 11:08:49PM -0700, Mike Thornton wrote:
> Hi folks,
> 
> I am still having a bit of difficultly reading data from a pipe when a test
> python script connects to the shim.  The script basically polls the pipe for
> data and reads it when available.  What I observe is that my python script
> (code below) does not fully read all of the initial shim account header
> information.  After reading a total of about 8kB of data, the script enters a
> state where it continually polls an empty pipe -- the rest of the usual header
> info is not read.  What I suspect is happening is that there is a buffer
> overflow with the pipe.  I understand that linux 2.6 kernels (what I have) have
> 64kB allocated to pipes with a maximum of 4kB for each atomic access.  The last
> few lines printed out are shown below.  Is it possible that the shim is writing
> more than 4kB to the pipe at once, thereby causing the script to lose the tail
> of the header?  The header prints out just fine if I run the shim command line
> without a script wrapper, and the script is not locking up since it ends up
> continually polling an empty pipe...
> 
> 
> Thanks.
> Mike
> 
> 
> 
> -------------------------- last several shim lines printed by
> script-------------------------
> poll = [(5, 1)]
> 3510|44731|   4169714|3| 6| 2|GrossPositionValue-S            |     0.00|USD |
> DU11111|
> poll = [(5, 1)]
> 3510|44731|   4169733|3| 6| 2|InitMarginReq                   |     0.83|USD |
> DU11111|
> poll = [(5, 1)]
> 3510|44731|   4169751|3| 6| 2|InitMarginReq-C                 |     0.00|USD |
> DU11111|
> poll = [(5, 1)]
> 3510|44731|   4169775|3| 6| 2|InitMarginReq-S                 |     0.83|USD |
> DU11111|
> poll = [(5, 1)]
> 3510|44731|   4169791|3| 6| 2|Leverage-S                      |     0.00|    |
> DU11111|
> poll = []
> poll = []
> poll = []
> poll = []
> poll = []
> poll = []
> 
> ------------------------ Python script ------------------------------------
> import subprocess
> import select
> import time
> 
> if __name__ == '__main__':
>     shimProcess = subprocess.Popen('/usr/local/shim/shim --risk cout file
> save', bufsize=1, shell=True, stdin=subprocess.PIPE, stdout=subprocess.PIPE,
> stderr=subprocess.STDOUT)
>     shimPoll = select.epoll()
>     shimPoll.register(shimProcess.stdout)
>     while True:
>         a = shimPoll.poll(0)
>         print "poll = %s" % a
>         while (len(a) == 0):
>             time.sleep(0.05)
>             a = shimPoll.poll(0)
>             print "poll = %s" % a
>         line = shimProcess.stdout.readline().strip()
>         print line
> -------------------------------------------------------------