[ts-gen] cross session orders, client ids

pippin at owlriver.net pippin at owlriver.net
Thu Dec 4 14:10:30 EST 2008


You're coming at the problem of identifying unique orders from a number
of directions, so let me restate some material to start with before I
address your questions directly.

An order is uniquely identified by a value from either of two key spaces:

    * from the standpoint of the IB tws, by an a clientid/ordertag pair,
      at least until it decides to start renumbering order ids;

    * from the standpoint of downstream clients, by the order item
      variable name, at least until the journal is cleared by truncation
      or database recreation.  The item var name is what I believe you
      are referring to in your post as the "date/time string", and in
      any case the attribute "var" of the CreateEvent journal table,
      with a declaration of:

        var varchar( 96) not null, unique key(var),

The shim takes responsibility for mapping between the two key spaces,
to the extent possible: order-related commands use the order item
variable name as an identifier; requests to the tws have an ordertag,
with an implicit clientid ordinarily determined by the initial
connection handshake; and related messages provide the full
clientid/ordertag pair.

Note that: there is no way for the shim to change the clientid
for a particular socket connection to the tws, since it is determined
by the handshake; the tws will only allow one active connection per
clientid at any one time; and the tws ordinarily uses the connection
clientid as the order clientid.  In addition, we've designed each shim
process to use just one connection, with the expectation that you run
multiple shims as needed, up to the tws-imposed limit of eight socket
connections per tws at any one time.

In your first question, you mention:

> I get the following error from time to time:

    $ exs/kill.rb

              The trading shim has connected to the database server
              and loaded 51649 products;

    Problem: 422 the client id  8 has not yet been released

    Problem: 523 the IB tws server handshake failed

Yes.  The kill.rb script runs with the "once" option, to ensure that
both connections to the tws run with a predictable clientid, here the
starting value of 8.  [Note that data-mode shims are confined to the
clientid space of 1-7.]  Without the once option, the second shim would
loop making connection attempts, for the case above presumably
succeeding with clientid 9, in which case, cross-session order
management would fail, since the connections would be working with
disjoint upstream key spaces --- that's why the "once" option. 

As the conclusion to that question, you ask how to prevent the
intermittent failure for the second connect attempt:

> restarting tws helps. Any other way?

There is currently a one second wait in the kill.rb script between the
termination of the first connection and the opening of the second, and
based on your report I'm increasing it to two seconds.  Please keep in
mind that kill.rb is a regression test script, and not meant to be 
production code.

You are welcome to increase that delay, and that would probably reduce
the frequency of the error you see above.  You are also free to wrap
the second connection attempt in a loop until it succeeds, which begs
the question of how many tries to make before giving up.  Ultimately,
you either give up the assurance of termination, or accept the
possibility of errors.  For our regression script, I prefer the latter.

You might use ps and netstat to check the process and socket listings,
and you are also free, as you've already discovered, to fix the problem
by a stop/restart of the tws.

Until the IB tws releases a clientid from one connection, it
can not be used by another.  As noted above, this is a restriction
of the tws api, and outside the control of the shim.  In my
experience, the IB tws ordinarily releases clientids within a couple
of, or at most three, seconds, and if it is bogged down to the point
where it doesn't, you probably have more serious problems than this
to worry about.

Now, about your second question, about cross session order management:

> Having the example kill.rb running, is it also possible to access
> orders from a different script, which are marked in OrderStatus
> table as 'Submitted'. In a first try I picked the date/time string
> from that table to do the steps as in kill.rb, it didn't quite work
> out however. Is there a more direct way to refer to such an order?

By design, the *only* way to refer to orders in downstream commands
to the shim is by the order item variable name, which is what I
believe you are referring to when you mention a "date/time string";
and, given our provision of cross-session support, you should be
able to freely use such order references in order-related shim
commands, as long as the shim succeeds in mapping them to clientid-
ordertag pairs, and the clientid found matches that for the current

Whether the resulting request to the IB tws works as you intend would
then depend on the state of the order according to the IB tws, and your
network connection to the IB servers, as well as theirs, to the various
markets.  In brief, many factors, all more or less outside the control
of the shim.

The order item var name, or key, doesn't have to be a date/time
string.  In the risk.rb regression test script, I use the following
code fragment to generate two unique keys:

    sequence = "-aa-"
    identity =  sequence  + Time.new.strftime("%y%m%d-%H-%M-%S")
    take_pos = 'take-pos' + identity
    exit_pos = 'exit-pos' + identity

    src1 = "bind key(#{take_pos}) to oid(1);\n"
    src2 = "bind key(#{exit_pos}) to oid(2);\n"

In the above code, the sequence string "-aa-" might vary on a
per-position basis, the identity variable is constant across both
keys of the two quoted order commands, and the take-pos and exit-pos
literals are simply a means to give distinct, mnemonic variable
names to the entry and exit command keys.  More flexible code would
also include a counter variable, and increment it each time a new
variable name was generated.

Once given such a counter, and code to ensure that it did not overlap
from one session to the next, downstream scripts would be free to
replace the approach to key generation above with something more
concise.  You may use whatever keys you like as long as they are
ASCII, accepted by the shim's input scanner, and fit the database
attribute constraints of varchar(96), not null, and unique key.

The shim lexical scanner type for order item variables is that of
"sym", for symbols, and reviewing the ScanCtor::simple_commands()
procedure in lib/dfsa.c, you'll see that, for 7bit ASCII, such symbols
are strings matching the regular expression [a-zA-Z_][-+a-zA-Z_0-9]*
that is, ordinary programming language variable names with in addition
the characters for plus and minus included once given the initial
alphabetic character.

Note that the scanner also treats those bytes with the high-order bit
set as alphabetic characters, and so long as the database allows the
insert, there is, to the best of my knowledge, nothing stopping you
from using valid multibyte utf8.  Such use is not freely supported:
you are welcome to report your experiences to the list, but if you
have problems with locale or database setup, we may not find it feasible
to duplicate your setup or problem.

Now, preliminaries done, the essence of your question about concurrent
order management:

> ... is it also possible to access orders from a different script ...

You can of course access orders from a different script, though
each script must have the *same* clientid, else the tws key spaces
are disjoint.  This means that a risk-mode script that creates an
order *must* drop its connection to the IB tws before another
risk-mode script that would connect for modification or cancellation
of that order, and that the second script must wait until the IB tws
releases the clientid for reuse.

This is admittedly very inconvenient, but is, for the most part,
a constraint imposed by the tws api.  Recall from above that I say
"the tws ordinarily uses the connection clientid as the order
clientid".  There is one way that concurrent order management might
be made, if not feasible, then at a minimum possible, although the
shim does not yet support it.

An api client that connects as clientid 0 can, according to my
understanding of the tws api, change the clientid context in which
the IB tws interprets its order requests by submitting an account
data request for that clientid, and learn the current ordertag base
value in that key space by submitting a next id request.

There are problems here with managing the ordertag counter space;
a clientid(0) risk-mode shim would have to maintain multiple mappings
for the various risk mode clientids, allow for holes in the 
ordertag counter space for each clientid, and handle concurrent
changes to that space triggered by other risk-mode clients.

There is also an unavoidable race condition if two shims attempt to
modify the same order, so we have not been very interested in
designing for such functionality.  A clientid(0) shim would be
useful for monitoring orders, however, so we probably will work on
this at some point, just not right away.

If you have some application in mind that involves concurrent order
processing of shared orders by distinct processes, please feel free
to describe it to the list.  I suspect that, given such a use case,
it would make sense to merge the heavyweight processes to threads in
a single ruby script with a single shim subprocess, where each ruby
thread shared access to the shim via locks on a containing class for
the shim stdin.

The original purpose for cross-session order management was to allow
modifies and cancels by a follow-on process, so that you could work
with orders from one session to the next.  Concurrent order processing
for shared orders is a harder problem, and one that I'm not sure
many users will want to work with.  My experience has been that,
although I can avoid race conditions for the shim to the extent that
the tws api allows, the resulting design pushes them off onto the
downstream clients, who have a hard time dealing with the resulting

The problem here is the clientid context, and the stateful nature of
the related ordertag counter space, both of which are part of the tws
api.  The mysql server already gives us database concurrency, so that
multiple shim processes can be inserting to the journal, and learning
about previous updates by concurrent processes as the inserts are read
back in, but this doesn't help with the IB tws api.  The api seems to
expect that each connection control the state of its own orders, with
perhaps a clientid(0) process in addition that monitors but does not
modify the orders of other clients.



More information about the ts-general mailing list