[ts-gen] Getting historical data outside regular hours
pippin at owlriver.net
Thu Sep 18 16:46:40 EDT 2008
This is a follow-on post to my earier reply to Sam, about
There I mentioned, with respect to the PastFilter and HistoryTag tables
that are used to define history query filters, the possiblity of:
> ... some changes to the database ... in the near future ...
The changes to the database have now occurred, bump the version by
one, and are otherwise minor. Of more interest is the parallel change
to the format of the history query command.
History query commands now include the query duration, a time period
with catenated unit. This parameter used to be provided as part of
the PastFilter record, and in fact the database version change is due
to the deletion of the related attributes from that table.
As a consequence, the past command syntax of, e.g.:
select past 142 8 now;
has changed to:
select past 142 8 2w now;
See the script exs/past as well for another example.
As a result of lifting the duration argument from the database to
the command, the PastFilter table attributes are now limited to
two categories, those shim-specific features that are not directly
visible to the IB tws, the support for periodic queries and history
query processing scripts; and the foreign key for HistoryTag, which
latter table has those history filter inputs not provided directly
by command argument values (still bar-size, rth-only, time-format,
The non-primary-key attributes of HistoryTag are finite domains, so
that the records inserted into that table at load time can be
enumerated via a cross-product select, and values are related to
record indices in a predictable pattern.
The initial records of PastFilter have the identity mapping to the
resulting 220 records, so that although the revisions to the database
have changed PastFilter table numbering, and you will need to revise
the PastFilter table index argument to existing history query
commands in scripts as a result, the changes are also predictable;
for low record numbers, e.g., 6 and 8 as used in the previous test
scripts, the PastFilter index changes to 116 and 118.
The NEWS file describes other changes. In particular, the front month
for YM has rolled, and the arbitrary test symbol STK:AIG has been
replaced with STK:AMAT. In addition to these changes which are visible
in the test scripts, there are some hidden changes.
The scanner has been enhanced to allow any reasonable floating point
format input for those command and message tokens that have decimal type.
Perhaps more to the point, the command and message languages recognize
exactly the same numeric formats. Changes to the scanner have tightened
tokenization type checking, and there is a new transient exception
added as well. If you observe trace output to the stderr of the form:
For type target TYPE_NAME and token text: <TEXT>
Problem: 401 unexpected token text: syntax error
please report the type name and related text string to the list.
I'd like to thank Sam for questions about the history query facility,
which suggested the need for the improvement to the history command
syntax, and an earlier bug report by Nils, which led me to improve
the input scanner.
More information about the ts-general