[ts-gen] Examples, and mysql, and ssl

[Bill Pippin]

John,

With respect to performance:

> Say I have about 100 tickers and getting quotes, volumes, bid/trade
> signals real time via IB tws (which is what shim is designed to do).
> Question is can shim + mysql handle this?  Plus analysis (query on
> the mysql server) etc.

> It's more of an open-ended question than anything specific.

The short answer is that performance will be fine on modern
hardware as long as you use common sense in the way you design
your downstream.  After all, outside of downstream analytics,
the api itself is just data transfer, essentially network IO, and
linux on modern cpus does a great job of handling Internet IO.

In more detail, you've raised the issues of capacity performance for:
the IB tws; the shim; and the downstream, there with analytics and
possibly ongoing sql queries.  I'll consider each in turn, before
considering what I consider to be the key issue, that of latency.

First, the IB tws:

You do realize that IB has defined the tws api to limit demands on
both the tws and the upstream servers?  In particular, there can be,
as you mention, at most 100 subscriptions or other market lines;
no more than three market depth subscriptions; and no more than fifty
requests from the downstream per second, an average of one per 20
milliseconds, an eternity in computer terms.  In addition, their
tick datastream is known to be summarized when need be, to ensure
that it doesn't fall behind during busy trading; depending on the
symbol and market conditions, tick streams max out at either three or
ten new last prices per second, for a per-subscription tick cycle time
of either 100 or 300 milliseconds.  And as for history queries,
capacity limits there are of an entirely different order of magnitude;
the steady-state rate for query submission is one every *ten seconds*.

So, within these limits, and by my observations on modern hardware,
the IB tws seems to do fine.  The performance of the upstream servers
presumably depends on the amount of money that IB throws at the issue;
they seem to have a good reputation in this area, though you should
judge for yourself.

Now, for the shim, and how you can judge performance:

The shim does one thing, provide convenient, efficient access
to the IB tws api, and it does that one thing very well.  In
particular, it's fast.

The shim timestamps every event --- command, request, message, and
internal log comment --- to microsecond precision, and these timestamps
are included in the log format.  You can easily see the time between
events by using the bin/diff.awk script that's included with the
shim; just create a log file by running exs/test.rb, and filter the
log file, probably log/ShimText, through that script, e.g.:

    bin/diff.awk < log/ShimText | more

You'll see that the shim spends almost all its time waiting on the
outside world, that is either waiting for 20 milliseconds, the queued
command time slot interval, to pass, or for the OS to wake it because
input has arrived.  Once that happens, the individual messages are read
in microseconds.  For the high volume messages, market data, market
depth, and history detail, these figures are in the single digits, and
often low single digit microseconds.  The waits for IO, however, can
easily be half, one, or two milliseconds.  A long time, admittedly, but
this is idle time, so you can be sure your system has plenty of spare
cycles for your downstream aps.

Nevertheless, there are optimizations I'd like to work on for the shim,
in particular with the memory allocation subsystem, and that is on our
roadmap.

You mentioned sql queries.  The shim reads in its data at startup,
and after this only writes data, currently history query answer and
order journal records as needed; by way of example, the regression
test history query answer sql insert takes about ten milliseconds.
These writes are always at the end of a time slot, time that the
shim would probably otherwise be using to wait on IO.

A projected feature may allow for sql reads after startup, to obtain
options data, but these will be in response to user commands, and
meant to be batched together at startup, as an initialization step.

If other processes were to hammer the mysql server, this could
conceivably impact the time to return from sending sql writes, and
we keep an eye on the logs to see if this becomes a problem.  It
shouldn't be, since the mysql server is threaded, and in any case
this is one area we'll optimize if need be.

The potential performance impact of a sql database has been raised
many times on the yahoo list, and so I'd like to emphasize that
the shim's use of a database is not a performance issue.  To recap,
the database is used for three roles, and for those roles, the
performance impact is unimportant, nonexistent, or trivial
compared to larger issues.

First, the symbols data obtained at startup allows the shim to have
a simple command language that maps to the elaborate request formats
defined by the IB tws api.  This, together with the conversion from
text to binary and back, is what makes the tws api accessible. The
simplicity of short textual commands is a key justification for the
shim, since otherwise, if the command language were as complex as the
request format, why use an api access program at all?  Here the time
cost, on the order of single digit seconds and therefore quite
obvious, is paid entirely at startup and so has no performance impact
after, so that this time cost is unimportant.

Second, the other key role for the database, order journalling, is
essentially an after-the-fact time cost; order records are written
*after* the request goes out, or a related message has been relayed
to the downstream, and so add no time at all to the key statistic of
order submission latency, or for that matter the submission of other
requests, since those are at least 20 milliseconds apart.  So, the
performance impact here is nonexistent.

Third, for the last use of the database, to archive the answers to
history queries, keep in mind that these are relatively rare,
arriving around ten seconds apart, so that they are of little use
in high frequency trading; they are, as the name suggests, historical.

Instead, consider submitting history queries via a separate shim
process, one dedicated to history collection.  After all, leaving
aside the time to insert the data in a single sql statement, consider
the time cost for the IB tws in relaying the answer for a large
history query, one with a thousand or more detail lines.  This has to
act like a pig in a python for other messages, so do you really want
such traffic interspersed with actual live orders?  I think not.
Here, the time cost, though real, is unimportant in the context of
the real issue, the segregation of traffic --- consider using your
paper account for history, and dedicating your live account to
orders and urgent tick data.

Now, about the downstream, and in particular, the possibility of
making sql queries.  The downstream shouldn't be making sql queries
in the midst of trading; there is no good reason to do so.  All of
the data from the event stream is faithfully passed by the shim to
whatever downstream channels you've selected at startup.  If you are
concerned about performance, you should glue your downstream app to
the shim by using the cout option to send all log events to the
standard output, and let the OS pipe them into your app.

Instead of using sql queries to track journalling and the like, which
design is not just inefficient but also flawed, due to the possible
race conditions, read whatever you need from the database at startup,
and use the event stream from the shim for the rest.  Modern scripting
languages such as perl/python/ruby make it easy to dispatch events by
the event codes, and split those events into attributes at the vertical
bars.

Beyond this, it's up to the user to choose efficient algorithms and
code.  There are lots of spare cycles just going to waste for a centos
or debian system running X11, java, the IB tws, and the shim, so
you'll have to try hard to load the cpu.  I keep an ongoing eye on
cpu load using xosview, an excellent performance meter for X11, which
you might like to try while running the shim and IB tws.  You'll see
that the cpu is just twiddling its metamoriphacal thumbs.

The industry seems to see a significant role for the R language for
the analytics, though in any case, numerical algorithms can chew
through cycles fast, so this is really up to you.  Such downstream
processing is outside the control of the shim, and would be an issue
for any trading system, so about which no more.

Now I'd like to leave aside your question about capacity --- "keeping
up" with 100 tickers, etc --- which I hope I've convinced you is not
an issue on modern hardware.

Instead, I've come to view the key statistic as latency, and not
just on the local machine, though even there the IB tws can sometimes
take its own sweet time answering a request.  IB seems to have worked
hard on their order processing, however; for an open order, say a limit
order with a "far out" limit price, so that it's already staged upstream
at or beyond IB's servers, and just hanging around, a modify command to
the shim to change the limit price seems to give impressively fast
executions.  There is an earlier post about this in the archives, and
you should check for yourself if you want to know the expected
execution time in milliseconds, since it will of course vary depending
on your net access.

If latency is the key issue, as I believe it to be, the two points to
look at here are the lag for market ticks, and the round-trip time for
pre-staged orders.  IB's tick summarization seems to avoid congestion
during busy markets, and appears to be reasonably robust, though in the
absence of originating-source timestamps for their ticks, how can we
be sure?  In any case, there's not much you can do about the data other
than choose another data provider.  The round trip modify/execute order
time is something you can measure using the shim, and if it matters to
you, you should do so.  Here, if you are focused on the fastest
possible response time, you'll see that the key insight is to stage
orders upstream, and modify them in response to changing conditions.

Beyond this, you could always consider server co-location, and a data
feed from DTN NxCore.

Thanks,

Bill