[ts-gen] Getting historical data outside regular hours

Bill Pippin pippin at owlriver.net
Thu Sep 18 10:58:27 EDT 2008


With respect to:

> I've been finaly able to get some historical data from my
> favorite european futures contracts,  but I'm not able to
> get historical outside the "regular hours".

What is the value of the history filter, and in particular the
rth_only attribute of the HistoryTag?  In the history query
from exs/test.rb copied below, the number 8 is a PastFilter
record index:

    select past 141  8 now;         wait 10;

[The initial values of this table are populated via insert
statements in sql/rest.sql; feel free to add to this and
recreate the database tables via sql/create.sql, so that the
additional values are permanently available.]  

Looking at the table definitions for PastFilter and HistoryTag
in sql/subs.sql, we see that beyond providing the period,
PastFilter has a foreign key to HistoryTag, and that there is
an rth_only attribute in HistoryTag:

create table PastFilter
    uid         int unsigned not null auto_increment primary key,
    tid         int unsigned not null references  HistoryTag(uid),
    period      int unsigned          references     BarSize(uid),

create table HistoryTag
    uid         int unsigned not null auto_increment primary key,
    rth_only    bool                            not null,
    format      enum('ymdt', 'epoch')           not null,
    what        enum('TRADES', 'MIDPOINT',
                     'BID', 'ASK', 'BID/ASK')   not null

Although you may expect some changes to the database here in the near
future, since it's likely that the next release will collapse these
two tables to one, the use of the history tag table as a means to define
the more stable and obscure parts of the subscription history filter
is likely to remain in one form or another.  This applies in particular
to the rth_only, date format, and what-to-show attributes.

PastFilter and HistoryTag were first split to distinguish variable
and stable attribute values, and the most important of these, the
duration, has since been added to the query itself as a parameter.
Changes to replace references to PastFilter by direct references to
HistoryTag have been made in some parts of the code, but not others.

As for the next release, one to clean up these table definitions may
occur very soon; it depends on what I find as I dig further.



More information about the ts-general mailing list