[ts-gen] Keeping TWS/shim up for an entire week?

[Ken Feng]

Hi Bill,

I apologize for being somewhat muddy in my last posting.  It was an
attempt to "ping" the level of robustness/limits of the entire
TWS/shim system - it was definitely >NOT< a complaint.

For example, without testing, I am pretty certain that if I wanted to
use TWS/shim to trade US stocks once a day, I would have few problems.
 I could shut down TWS/shim after the market close and open a little
before the market open.

On the other extreme, the forex markets are open 24/5, and I know from
some precursory testing that latencies of 600ms+ are to be expected in
the system when querying for 5 second bars.  Given these limitations,
trading in time-scales of smaller than 5-10 seconds would be playing
with fire and even less than 1 minute would be foolish if the ISP is
not robust.

So we know the "useable region" to be somewhere between these two
extremes.  I am trying to determine where the boundaries are.

> By the way, you should know that one IB employee with detailed
> knowledge of IB tws internals has given [on the Yahoo list] what I
> read as friendly but very direct warnings about the unsafe nature of
> the IB tws over multi-day timeframes.

I was not aware of the Yahoo TWSAPI boards - thanks for pointing that
out.  It looks like there is a lot of activity on that board.  What
particular problems have you heard about not restarting TWS daily?
Are there known memory leaks or crashing issues in TWS if left over 24
hours?  If so, that would definitely explain IB"s intent on shutting
down the TWS once a day.

BTW, I get around this by using the Jemmy module in Java to start up
TWS.  Jemmy uses introspection to detect and grab handles into GUI
elements.  Once I am able to grab a handle into the panel and widgets
which set the log-off time, I am able to schedule a Timer which sets
the now log-off time to be one hour before Now.  I schedule it once an
hour.  The code is about a couple of pages long and not 100% robust as
there may be timeout issues involved, or attempting to reset the
log-off time when TWS has lost its connection.  But if I reset the
widget once an hour, TWS seems to stay up.  Whether or not TWS itself
is behaving correctly after a period of 24 hours is currently
unknown/untested.

Back to the issue of robustness.  I define it has knowing where the
caca is so I can avoid stepping in it.  If one of these limits is a
24hour window, I still have to pick when to shut-down and when to
start up for forex, as the market itself is running non-stop.

> As for the shim, you can use ping to make sure it's alive, select next
> to make sure the IB tws is alive (both of these steps are mentioned
> in the manual), and then drop and reselect your symbol of interest to
> see if the upstream is providing data.  You should know that a recent
> post to the Yahoo list suggested that the kind of data quality problems
> you are seeing are sometimes a sign of communication problems with the
> exchange, and that the occurrence of such was a key reason why the
> writer did not try to run his system unattended.

Thank you for this hint - I will look into the manual and search the
Yahoo boards.

> Make sure to drop the old subscription first, and then, yes absolutely,
> if you lose data flow for any reason, resubscribing makes perfect sense,
> and, as you imply, waiting until network connectivity is restored is
> probably your best bet:

You anticipated my problem - I didn't drop the old subscription first
and ended up with duplicate bars!  I will try out your recommendation.

>>                    ...  In trying to keep TWS/shim up, it seems
>> to me that I have to code for all sorts of states - I guess this
>> is a characteristic of real-time systems.
>
> Indeed, and some of the behavior you'll see, such as the evidently
> bad data you mention later, is to my way of thinking a good reason to
> restart your IB tws at one day intervals as recommended by them, in
> which case you'll be restarting the shim as well.  Note that there is
> no particular reason you can't keep your downstream running, just that
> you almost certainly should restart the IB tws, and that in that case
> you will need to restart the shim also.

Here's a question - is it possible to restart TWS/Shim once every 24
or 12 hours without any loss of incoming ticks?  If so, how would you
go about it?

In reality, such a stringent requirement may not be necessary as it
seems that during certain periods of day (eg very regularly at 3:55UTC
for forex), TWS seems to hiccup - I may use that opportunity to reset
the system.

I will definitely post my findings, and I will go to the Yahoo boards
now to see what others have found out.

Thx.

- Ken