[ts-gen] How symbols and contracts are defined during database creation

[R P Herrold]

On Fri, 29 Aug 2008, sam wrote:

> Thanks for the answer.
>
> The db structure is more clear for me now, and I know now 
> what was wrong for me, when I was trying to get historical 
> data for some of my european stocks.

   ;)

> First, one stock name change its symbol/name these last 
> months, but keep the same ibc (it's the stock named now GSZ 
> instead of GAZ, ibid 35052259). So sym/A0Stk05.sql might be 
> changed or re-generated for this stock.

darn ;(  We hoped IB would assign a new ibid in such cases 
[there are reasons to do so and reasons not to do so, I know].

This to some degree is something we need to design for in 
holding lot specification markers on orders (under some tax 
lawd, one can specify that one is selling from part of a 
holding which one bought, say, 18 months ago, rather than say 
1 month ago, and get differing tax treatment -- long term 
'investor' vs. short term 'trader' treatments.)

Thank you for this report.

> Second, I still have a strange issue with a french stock, 
> TFI, ibid 29612096. I'm unable to get any data for this 
> stock, it's maybe an IB issue, because I've found a strange 
> thing, when I get a chart in TWS for this stock, the symbol 
> name is FSE instead of TFI...

Researching TFI here comes up in the TWS GUI with:
 	a US SPDR stock, which invests in muni bonds
 		(probably a closed end?) mutual fund traded on
 		the AMEX (closed end mutuals look and
 		trade like stocks) in USD
 	'Transforce' a stock, on the Toronto SE, one assumes
 		in CAD
 	'TFI', a media company Listing trading on Exchange SBF
 		in EUR, in Region: Europe *AND* having Fut and
 		Opt derivatives
 	A stock I do not have rights enabled to trade:
 		BEST STK/Top TFI "TREVI FINANZIARIA SPA", in
 		EUR, in Region: Europe
 	Another stock I do not have rights enabled to trade:
 		BM STK/Top TFI "TABLEROS DE FIBRA", in EUR, in
 		Region: Europe

... my head hurts  ;)


> Third point, I tried to add some E-Mini contracts, which 
> don't seems to be here in sym/*.sql. I thought I was just 
> need to add the correct entries in, for example, 
> sym/A0Fut00.sql, and in Expiry.sql. But I don't get the 
> newly added contracts with a get_id.sql, maybe i'll need to 
> change the where clauses in this script ?

We have usually just populated the YM for the next few Front 
Month and back Expirations in the emini's for the tarballs we 
ship, as it took up space and caused load times to be longer, 
and presented a maintenance load in the past.  As we have 
made some changes 'under the hood' and have more planned for 
database loads, we will revist this in planning Monday.

Bill may well have a short term workaround on this, but this 
is the first time anyone has asked for them in the mailing 
list community. ;)

> Unfortunately, I was forced to stop my tests, I have a 
> strange issue, with the lastest tarball, a fresh  database, 
> when I try to get historical data (here example with AIG):
> ------------------------
> select past 6 5 Ymd_T(20080801 23:00:00)
>
> Problem: 550 ice: req wire-level format typecheck error
> Exiting
> ------------------------
> I've restarted few times my TWS, but still got this error for.

I have brought your email to Bill's attention.  We are quite 
interested, and he will probably have more to offer.

> About Warrants, I'm not so big fan of these products, but I 
> can help if needed.

We want and NEED people _testing_ a feature, before expending 
the effort to support it.  When we do not test, we may miss a 
function loss in a refactoring.  As the trading shim homepage 
notes (and has noted from day one):

 	... [in part] we are exploring software engineering
 	issues related to ... test-driven development.

> It's a good idea to include th .ps file in the tarball, I 
> had some issues for generate the file with Graphviz on my 
> Debian based distribution.

It is not unusual for recent Debian 'testing' to not be able 
to rebuild the documentation -- they seem to have lost a 
required package.  It _is_ unusual to get a Graphviz 
complaint.  I'll test it ... more later.  As to the LaTeX 
rebuild of the manual, I get this error on a ** new ** Debian 
'testing' installation [from the netiso earlier this week]:

Package latex2html is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
E: Package latex2html has no installation candidate

------------------------------------

which builds the HTML version of the documentation.

'latex2html' is needed by a Makefile target:

online: manual          ;       latex2html -debug $(L2H_CA) manual

and in the TeX as well [it provides 'html.sty'].  We have a 
documentation re-write on the horizon, and rather than do a 
poor fix now, let's wait to see if Debian has the package 
return, and not do doco rebuilds on Debian.

As to Graphviz, Bill mentioned earlier in the week:

> The postscript files above were produced using the graphviz 
> dot program via the script bin/fkey.rb .  If you lack the 
> tools to produce the graphs, and one or more is missing from 
> your tarball, I've just pushed another release that has all 
> three of the above postscript files in the sql directory

hmmm ... But there ** is ** something wrong as to the 
[re-]generation of 'foreign.ps' -- we will look into that.

herrold at dhcp-232:~/shim/shim-builder/shim-080828/sql$ find -name '*.ps'
./syms.ps
./foreign.ps
./xact.ps
./subs.ps
herrold at dhcp-232:~/shim/shim-builder/shim-080828/sql$ rm -rf \
 	./syms.ps ./foreign.ps ./xact.ps ./subs.ps
herrold at dhcp-232:~/shim/shim-builder/shim-080828/sql$ find -name '*.ps'
herrold at dhcp-232:~/shim/shim-builder/shim-080828/sql$ bin/fkey.rb
herrold at dhcp-232:~/shim/shim-builder/shim-080828/sql$ find -name '*.ps'
./syms.ps
./xact.ps
./subs.ps
herrold at dhcp-232:~/shim/shim-builder/shim-080828/sql$


This is our Debian 'testing' environment:

herrold at dhcp-232:~/shim/shim-builder/shim-080828/sql$ sudo apt-get install lsb
   ... snip much ...
herrold at dhcp-232:~/shim/shim-builder/shim-080828/sql$ lsb_release  -a
LSB Version:    core-2.0-ia32:core-2.0-noarch:core-3.0-ia32:
 	core-3.0-noarch:core-3.1-ia32:core-3.1-noarch:core-3.2-ia32:
 	core-3.2-noarch:cxx-3.0-ia32:cxx-3.0-noarch:cxx-3.1-ia32:
 	cxx-3.1-noarch:cxx-3.2-ia32:cxx-3.2-noarch:desktop-3.1-ia32:
 	desktop-3.1-noarch:desktop-3.2-ia32:desktop-3.2-noarch:
 	graphics-2.0-ia32:graphics-2.0-noarch:graphics-3.0-ia32:
 	graphics-3.0-noarch:graphics-3.1-ia32:graphics-3.1-noarch:
 	graphics-3.2-ia32:graphics-3.2-noarch:qt4-3.1-ia32:qt4-3.1-noarch
Distributor ID: Debian
Description:    Debian GNU/Linux testing (lenny)
Release:        testing
Codename:       lenny
herrold at dhcp-232:~/shim/shim-builder/shim-080828/sql$

I have pushed a freshly updated copy of my Debian environment 
setup and building script to:

 	ftp://ftp.owlriver.com/pub/local/COLUG/shim-debian.sh

You should get identical results after running that 
environment setup.  If not, I am VERY interested in a report.

-- Russ herrold