[ts-gen] Downstream architecture (3rd)

[Bill Pippin]

Today's release includes a new directory, rsc, for ruby sample client,
with example code for popen3 and select.  The code is preliminary and
leaves much room for improvement, yet includes the minimum architectural
features of a call to Open3.popen3 to open the connection to a shim
process, and use of IO.select to read both the shim stdout and stderr.

Users may want to use the ruby class definitions in the rsc directory
as the starting point for downstream script development.

Since these sample client sources are also used by the exs/test.rb
regression script, they are a starting point for me in improving
test coverage, in particular to avoid problems such as the one noted
earlier this week by Ken, where execution report requests had silently
stoped working.

In an earlier post back in Aug,

http://www.trading-shim.org/pipermail/ts-general/2009-August/000570.html

I wrote:

> ... users should already be using popen3() or its equivalent
> in their favorite scripting language, and select() ditto ...

> It would be nice to provide example code via a ruby sample
> client, and that's just one of many things ... [not] completed
> yet.

Users attempting to use Open3.popen3 as suggested above have probably
had problems with livelock, and may well have fallen back on using
psuedo ttys.  [Note that Pty.spawn(...) is nearly a drop in replacment
for Open3.popen3(...), with the difference that the ptty merges stdout
and stderr, so that the client must disambiguate them.]

The current release of the shim changes the buffering for the shim
stdin to unbuffered, which is the last step needed to overcome the
problems with livelock.  Previous changes included the choice of
line buffering for the shim's stdout, with the cout option; and
advice to the list to eliminate script buffering to the pipes, e.g.,
via the assignments in the popen3 block below:

>   Open3.popen3(text.cmd_line) do | cmd_send, msg_recv, err_text |

>     cmd_send.sync =
>     msg_recv.sync =
>     err_text.sync = true

>     test.prog(cmds, read, proc, cmd_send, msg_recv, err_text)
>   end

The livelock would occur where commands to the shim were accumulating
in a stdlib buffer, the shim did not know of them and was loop waiting
using select for commands, and the client was loop waiting for events
that would not occur until the commands were processed.

The text of the new ruby sample client follows my sig, and, as noted
earlier, is also included in the rsc directory in the today's release.

Also, the problem with history queries for ECBOT:YM is new, known, and
I'll be getting to that next.  It seems to be data (symbol) dependent.

Thanks,

Bill 

::::::::::::::
test.rb
::::::::::::::
#!/usr/bin/ruby
 
#  author: Bill Pippin, <pippin at trading-shim.com>, msgs may gate to the list
#  copyright (c) 2008-2009 Trading-shim.com, LLC  Columbus, OH
#  GPL version 3 or later, see COPYING for details

=begin

  Given the command line defined by CmdArgText; the predefined commands,
  by CannedCmds; the select/read loop, by SelectRead; and the processing
  filter, by ProcFilter: for the class Popen3Shim, exec the command as a
  child using popen3, and use the SampleTest instance to send cmds to it,
  read what it writes back, and dump results.  See the ProcFilter class
  for further details of processing.

=end

require "rsc/CmdArgText"
require "rsc/CannedCmds"
require "rsc/SelectRead"
require "rsc/ProcFilter"
require "rsc/SampleTest"
require "rsc/Popen3Shim"

text = CmdArgText.new
cmds = CannedCmds.new
read = SelectRead.new
proc = ProcFilter.new
test = SampleTest.new
       Popen3Shim.new.prog(text, cmds, read, proc, test)
exit
::::::::::::::
SampleTest.rb
::::::::::::::
#  author: Bill Pippin, <pippin at trading-shim.com>, msgs may gate to the list
#  copyright (c)      2009 Trading-shim.com, LLC  Columbus, OH
#  GPL version 3 or later, see COPYING for details

=begin

  Given both: objects to send commands, perform processing, and read
  results; and files for output, input, and stderr for the related IO;
  then glue the object processing together via a simple test harness.

=end

class SampleTest

  def prog(cmds, read, proc, cmd_send, msg_recv, err_text)

       cmds.test(cmd_send)
    if read.loop(msg_recv, err_text, proc)
       cmds.exit(cmd_send)
       proc.dump
    end

  end
end

::::::::::::::
CmdArgText.rb
::::::::::::::
#  author: Bill Pippin, <pippin at trading-shim.com>, msgs may gate to the list
#  copyright (c)      2009 Trading-shim.com, LLC  Columbus, OH
#  GPL version 3 or later, see COPYING for details

=begin

  Define a reasonable command line with which to start a data mode shim,
  given the goal of regression testing; feel free to modify as needed.

=end

class CmdArgText

  def cmd_line
    "./shim --data cout file save join diff"
  end
end

::::::::::::::
Popen3Shim.rb
::::::::::::::
#  author: Bill Pippin, <pippin at trading-shim.com>, msgs may gate to the list
#  copyright (c)      2009 Trading-shim.com, LLC  Columbus, OH
#  GPL version 3 or later, see COPYING for details

=begin

  Open the command line Text using popen3, and apply the prog method of
  the Test object to each of the object parameters Cmds, Read, and Proc
  as well as the resulting pipes. 

  Note that pseudo ttys can be used as nearly a drop in replacement for
  popen3.  I.e. Pty.spawn(text.cmd_line) do | cmd_send, msg_recv, pid |
  There's the drawback that the process filter must distinguish tws api
  events from shim writes to the stderr, but given the log format, this
  is not difficult.

=end

require 'open3'

class Popen3Shim

  def prog(text, cmds, read, proc, test)

    Open3.popen3(text.cmd_line) do | cmd_send, msg_recv, err_text |

      cmd_send.sync =
      msg_recv.sync =
      err_text.sync = true

      test.prog(cmds, read, proc, cmd_send, msg_recv, err_text)
    end
  end
end

::::::::::::::
CannedCmds.rb
::::::::::::::
#  author: Bill Pippin, <pippin at trading-shim.com>, msgs may gate to the list
#  copyright (c)      2009 Trading-shim.com, LLC  Columbus, OH
#  GPL version 3 or later, see COPYING for details

=begin

  Define as text various commands in the trading-shim input language, and
  provide operations to print those commands as part of regression tests.

  Currently this class includes only non-transactional tests, those that do
  not perform any trades.  For now, see the test script exs/risk.rb for an
  example of simple trades.

  Although this script is meant in part to illustrate correct syntax for
  the shim commands included herein, there are a number of cases where the
  alternatives in the plain shim scripts --- executable text files with the
  shim itself as the shebang interpreter, found in the directory ../exs ---
  are more flexible and upto date.

  In particular, the comments in the tick, info, and bars scripts may be of
  interest to new users.

  test_cmds(shim_cmd)

      Once contructed, and given an IO object connected to the unbuffered
      stdin of a trading-shim process, send session level, market data,
      marked depth, and history commands to the shim.  This top-level cmd
      will trigger all the other methods of this class, and so is the only
      method the client need use.

      By the way, the contracts are meant to be for Apple and the current
      front month of ECBOT:YM .

=end

class CannedCmds

  def test shim
      data_test(shim)
      tick_test(shim)
      book_test(shim)
      past_test(shim)
      tick_test(shim)
  end

  def data_test(shim); shim.write data_text(); end
  def tick_test(shim); shim.write tick_text(); end
  def book_test(shim); shim.write book_text(); end
  def past_test(shim); shim.write past_text(); end
  def exit     (shim); 
                     STDERR.print "exit \n"
                       shim.write "exit;\n";   end

  def data_text()
    return <<-"EOT"
set loglevel Detail;
select next;
select exec AAPL 9:00:00;
select news all;
cancel news;
select info ibc:  266093 at SMART new;
select info ibc:56578477 at ECBOT all;

    EOT
  end

  def tick_text()
    return <<-"EOT"
select tick ibc:  266093 at SMART 1;               wait  2;
cancel tick ibc:  266093 at SMART;                 wait  0;

select tick ibc:56578477 at ECBOT 1;               wait  2;
cancel tick ibc:56578477 at ECBOT;                 wait  0;

    EOT
  end

  def book_text()
    return <<-"EOT"
select book ibc:56578477 at ECBOT 5;               wait  4;
cancel book ibc:56578477 at ECBOT;                 wait  0;

    EOT
  end

  def past_text()
    return <<-"EOT"
select past ibc:56578477 at ECBOT h1  11  1d now;  wait  6;

    EOT
  end
end

::::::::::::::
SelectRead.rb
::::::::::::::
#  author: Bill Pippin, <pippin at trading-shim.com>, msgs may gate to the list
#  copyright (c)      2009 Trading-shim.com, LLC  Columbus, OH
#  GPL version 3 or later, see COPYING for details

=begin

  Given two inputs, the first for a shim format event log, and the second
  from the shim stderr, loop to multiplex reads for each via select, send
  each log input line to proc.filter, each error text line to proc.stderr,
  and terminate if:

      eof or errors occur with either the event or error input files; or
      twelve or more one-second timeouts occur during the call to select.

=end

class SelectRead

  def loop(msg_recv, err_text, proc)

    limit = 12
    count = 0
    child = true
    input = [msg_recv, err_text]

    while count < limit && child
      result = IO.select(input, nil, input, 1)
      if result != nil
        if result[2] == []
          result[0].each do |f|
            case f
              when msg_recv then child &= proc.filter msg_recv.gets
              when err_text then child &= proc.stderr err_text.gets
            end
          end
        else STDERR.print "IO error in select loop\n"; child = false end
      else count += 1 end
    end
    return child
  end
end

::::::::::::::
ProcFilter.rb
::::::::::::::
#  author: Bill Pippin, <pippin at trading-shim.com>, msgs may gate to the list
#  copyright (c)      2009 Trading-shim.com, LLC  Columbus, OH
#  GPL version 3 or later, see COPYING for details

=begin

  Given shim format event lines or stderr text, count events for the former,
  and echo the latter; and otherwise, if the input line is nil, again echo
  it, and otherwise do nothing.  The dump method lists event totals.

  For debugging and other purposes there is also a commented-out statement
  to list log format events.  That statement will, given the split on bars,
  slice the array to drop the first three fields, reconstitute via join,
  and slice the resulting string to fit the terminal display.

  The event counting is crude, and will in the future be modified to use
  database information about the various event types.

=end

class ProcFilter

  attr_reader :cmd_totals, :req_totals, :msg_totals
  def initialize
    @cmd_totals = []
    @req_totals = []
    @msg_totals = []
    (0..100).each do |i|
      @cmd_totals[i] = 0
      @req_totals[i] = 0
      @msg_totals[i] = 0
    end
  end

  def filter line
    if line != nil
      fields = line.chop.split('|')
      length = fields.size

      if length >= 5 then
        src = fields[3].to_i
        tag = fields[4].to_i
        ver = fields[5].to_i
        if src == 3
#         STDERR.print fields[3,length].join('|')[0..78], "|\n"
        end
        case src
          when 1 then cmd_totals[tag] += 1
          when 2 then req_totals[tag] += 1
          when 3 then msg_totals[tag] += 1
        end
      end
    end
    return line
  end

  def stderr line
    STDERR.print line
    return line
  end

  def dump
    1.upto(20) { |i| printf("%4u", cmd_totals[i]) }
    1.upto(20) { |i| printf("%4u", req_totals[i]) }
    1.upto(20) { |i| printf("%4u", msg_totals[i]) }
  end

end