[ts-gen] version update/database migration
R P Herrold
herrold at owlriver.com
Sat Aug 23 22:15:45 EDT 2008
On Sat, 23 Aug 2008, sam wrote:
> Does the database structure is supposed to be changed again ?
> It's a bit off-topic, but I'll like to have a Perl module
> for Geniustrader, for accessing to the shim database.
... more below ...
> I'm extracting text files from the Shim database for beeing
> after that backtested by Geniustrader, it's not the best
We are closing in on that part of the database (HistoryBar)
and its foriegn key dependencies not changing; I think you
will find that it has had only two other forms in the last two
and a half years. The addition of the IB ConID value when we
did an effort to populate out the Products within the last six
months occasioned the last change, and before that the
Product, Exchange pairing.
I posted to the GT mailing list, back when adding intra-day
was being discussed a couple of years ago, proposing
essentially the same table form as we have today, with those
two changes excepted. As Bill mentioned, we consciously did
some denormalization when those columns were added, in the
interest of making migrating retrieved data forward as changes
happen elsewhere in the database.
We have to have some freedom for coding purposes to be able to
make schema changes, while we are in development mode (getting
Journalling just right, in preparation for cross-session Order
management has been tricky in the last couple months.) We are
closing in on a formal cutover from development mode to a
first stable API and database production release, and know
what _we_ want to have in (from a 'feature completeness' point
of view) at that point, and the order in which we will get
there. We are willing to consider more, but the cycle is
late. Speak up in the list, please. We hold a planning
meeting every Monday (and a progress review meeting each
The recent surfacing of end user feedback and reports has been
most welcome, and we hope that our fast turnaround on
addressing issues raised has been helpful. Please continue!
As to externally raised issues, I think the only
non-documentation item left outstanding is Nils' cross-session
order management question. We have some others as well, being
./lib and memory management matters, rolling in 'BIND', and
help and documentation. I am away from my office notes, of
course, and may be forgetting something.
The GT mailing list has had some strange back and forth in
recent months. The squabbling about Date::Time is hard for me
to understand. R, ta-lib, quantlib, and qtstalker are other
projects we watch, package, follow CVS (SVN) on, and so forth.
Bill noted, as have I that Debian seems to be the Free
Software distribution family of choice used by most of the
external users; and, interestingly (as it has changed as the
trading-shim project had been going), a shift to 64 bit from
32 bit (even though for theoretical reasons, 64 bit may not be
as efficient as 32). I went through the work with xen virtual
instances I have been doing earlier this week, and I'll be
cutting Bill over to a 64 bit base installation (and distcc
farm) next week. For those new to the list, I have also run
earlier versions of the shim through the IBM 'chiphopper'
certification process on Power 5 (PPC64), and s390
architectures, which pointed up some changes needed in ./lib/
As to a perl module for access, we have looked at adding
'swig' bindings which would also give us Ruby, Python, and
many more. Feel free to front run us on this, and ask
questions on the list or privately. We might be a bit slower
on response than as to bugfixing, as some design work is
needed there. We would ask that all contributions to the
project be a 'under the GPLv3 or later' License, and for code
which we adopt into the tarball, an assignment of rights
executed and sent to us; alternatively we may read the code,
and re-implement as needed.
-- Russ herrold
More information about the ts-general