[ts-gen] version update/database migration

[R P Herrold]

On Sat, 23 Aug 2008, sam wrote:

> Does the database structure is supposed to be changed again ?

> It's a bit off-topic, but I'll like to have a Perl module 
> for Geniustrader, for accessing to the shim database.

... more below ...

> Today 
> I'm extracting text files from the Shim database for beeing 
> after that backtested by Geniustrader, it's not the best 
> solution...

We are closing in on that part of the database (HistoryBar) 
and its foriegn key dependencies not changing; I think you 
will find that it has had only two other forms in the last two 
and a half years.  The addition of the IB ConID value when we 
did an effort to populate out the Products within the last six 
months occasioned the last change, and before that the 
Product, Exchange pairing.

I posted to the GT mailing list, back when adding intra-day 
was being discussed a couple of years ago, proposing 
essentially the same table form as we have today, with those 
two changes excepted.  As Bill mentioned, we consciously did 
some denormalization when those columns were added, in the 
interest of making migrating retrieved data forward as changes 
happen elsewhere in the database.

We have to have some freedom for coding purposes to be able to 
make schema changes, while we are in development mode (getting 
Journalling just right, in preparation for cross-session Order 
management has been tricky in the last couple months.)  We are 
closing in on a formal cutover from development mode to a 
first stable API and database production release, and know 
what _we_ want to have in (from a 'feature completeness' point 
of view) at that point, and the order in which we will get 
there.  We are willing to consider more, but the cycle is 
late.  Speak up in the list, please.  We hold a planning 
meeting every Monday (and a progress review meeting each 
Friday).

The recent surfacing of end user feedback and reports has been 
most welcome, and we hope that our fast turnaround on 
addressing issues raised has been helpful.  Please continue!

As to externally raised issues, I think the only 
non-documentation item left outstanding is Nils' cross-session 
order management question.  We have some others as well, being 
./lib and memory management matters, rolling in 'BIND', and 
help and documentation.  I am away from my office notes, of 
course, and may be forgetting something.

The GT mailing list has had some strange back and forth in 
recent months.  The squabbling about Date::Time is hard for me 
to understand.  R, ta-lib, quantlib, and qtstalker are other 
projects we watch, package, follow CVS (SVN) on, and so forth.

Bill noted, as have I that Debian seems to be the Free 
Software distribution family of choice used by most of the 
external users; and, interestingly (as it has changed as the 
trading-shim project had been going), a shift to 64 bit from 
32 bit (even though for theoretical reasons, 64 bit may not be 
as efficient as 32).  I went through the work with xen virtual 
instances I have been doing earlier this week, and I'll be 
cutting Bill over to a 64 bit base installation (and distcc 
farm) next week.  For those new to the list, I have also run 
earlier versions of the shim through the IBM 'chiphopper' 
certification process on Power 5 (PPC64), and s390 
architectures, which pointed up some changes needed in ./lib/

As to a perl module for access, we have looked at adding 
'swig' bindings which would also give us Ruby, Python, and 
many more.  Feel free to front run us on this, and ask 
questions on the list or privately.  We might be a bit slower 
on response than as to bugfixing, as some design work is 
needed there.  We would ask that all contributions to the 
project be a 'under the GPLv3 or later' License, and for code 
which we adopt into the tarball, an assignment of rights 
executed and sent to us; alternatively we may read the code, 
and re-implement as needed.

-- Russ herrold