[ts-gen] Obtaining the close for IND:NYSE:INDU

[Bill Pippin]

I'm reposting the following in case it is of interest.

A poster on the yahoo tws api list asked about obtaining information
with respect to the Dow Industrials, that is the index NYSE:INDU:

> I'm having trouble getting the Dow Industrial Index Net Change using
> the API (using INDU, IND, NYSE, NYSE). I've tried getting the daily
> history using the previous day's close as well as using the Tick
> Request but can't get an accurate quote that comes close to setting it
> up in TWS. ...

History queries typically include a close, which for bar
intervals of one day should be what you want.

Presumably you know this, and it's not clear to me whether
you are having problems collecting data, or whether you are
unhappy with what the api provides.  For the latter case, you
probably would need to pay for some data subscription service
besides the api.

If by "accurate" you refer to the official exchange close, which
may be fixed only hours after the close of trading, and this
happens to differ from what IB has collected and stored on their
servers from the datastream they collected throughout the day, I
don't see any alternative.

As for the api, I have no problem subscribing to market data or
in making history queries for this symbol, which is what I focus
on in the following.

Given an installation of the trading-shim, it's the work of
moments to collect 1-day bars with a command script such as:

    #!./shim -f

    # Query for IND:NYSE:INDU 1-day history bars.
    # In the history query, 29163 is the contract
    # record id for the symbol of interest, while
    # the label d1 is the bar size, and the value
    # 1w is the duration.

    select past 29163 d1   1  2w Ymd_T(20081001  9:00:00); wait 10;
    exit;

The contract record unique id number 29163 was
found via the query of a symbol-database listing:

    sql$ grep INDU syms.txt | grep NYSE
    29163   29154   1935181 IND     NYSE    INDU    USD     US

Using either symbol descriptions or the ib contract id of 1935181
would be more convenient, and that feature is part of our planned
enhancements.  We can name the script above as indu, dump it in
the examples directory exs, and run it with the following command.

    exs/indu --data file save

The resulting commands, requests, and messages are serialized in
text form to a log file.

Note that in the log listing below, due to license prohibitions on
the redistribution of data obtained via the api, I've changed the
price figures to 10000.0 even.  Of the other values, the first column
is a timestamp difference, in microseconds from the creation time for
the previous event.  Due to command queueing the creation times are
not in serial order, so on occasion a difference is negative.  The
second column uses 1, 2, or 3 to indicate whether the event is a
command, request, or message, respectively.  Since the history detail
lines are long, and wrap, I've trimmed the has_gaps value (false)
and the appended symbol info, IND:NYSE:INDU, from the history query
detail lines.

  0.000080|1| 0| 0|# Query for IND:NYSE:INDU 1-day history bars.|
  0.000085|1| 0| 0|# In the history query, 29163 is the contract|
  0.000084|1| 0| 0|# record id for the symbol of interest, while|
  0.000097|1| 0| 0|# the label d1 is the bar size, and the value|
  0.000077|1| 0| 0|# 1w is the duration.|
  0.000065|1| 0| 0||
  0.000167|1|12| 0|select past 29163 d1   1  2w (20081001  9:00:00); |
  0.000027|1| 2| 0|wait 10;|
  0.000081|2|20| 3|20|3|1|INDU|IND||0.00||1|NYSE||USD||20081001  09:00:00|11|2 w|0|TRADES|1|
  0.148712|3|17| 3|         1|10|
 -0.000268|3| 1| 1|20080917|10000.0|10000.0|10000.0|10000.0|0|0| ...
  0.000027|3| 1| 1|20080918|10000.0|10000.0|10000.0|10000.0|0|0| ...
  0.000026|3| 1| 1|20080919|10000.0|10000.0|10000.0|10000.0|0|0| ...
  0.000038|3| 1| 1|20080922|10000.0|10000.0|10000.0|10000.0|0|0| ...
  0.000026|3| 1| 1|20080923|10000.0|10000.0|10000.0|10000.0|0|0| ...
  0.000026|3| 1| 1|20080924|10000.0|10000.0|10000.0|10000.0|0|0| ...
  0.000025|3| 1| 1|20080925|10000.0|10000.0|10000.0|10000.0|0|0| ...
  0.000026|3| 1| 1|20080926|10000.0|10000.0|10000.0|10000.0|0|0| ...
  0.000037|3| 1| 1|20080929|10000.0|10000.0|10000.0|10000.0|0|0| ...
  0.000026|3| 1| 1|20080930|10000.0|10000.0|10000.0|10000.0|0|0| ...
  0.029402|4|110| 0|# |4|110|0|event: history insert|(29154, 11, 20080917 -- 20080930)|
 -0.177822|1| 1| 0|exit;|

If the close values the api gives you are not what the NYSE
reports --- and the close fixed by any particular exchange may
have only a tenuous connection with the actual trades that end
the day --- then I'm not sure what more you can expect from the
api.   I suspect this is a case of take it or leave it.

Thanks,

Bill