[ts-gen] Fix for negative realtime bar "quantities" [Was: FOREX Issues ...]

[Bill Pippin]

Ken,

About the 402 warnings for unexpected negative numbers for realtime
bars:

> When running, for BID/ASK:
> select bars CASH:IDEALPRO:USD:JPY 4;
> I get the following 402 WARNINGS.
 
> Problem: 402 unexpected minus sign, input clamped to zero
> For token text: [      -1] and type target [quantity]
> predicted token type:  pos, scanner saw:  neg
> Problem: 402 unexpected minus sign, input clamped to zero
> For token text: [      -1] and type target [count%6lu]
> predicted token type:  pos, scanner saw:  neg

Thanks much for this report.  I've gone ahead and changed the quantity
and count attributes for the real time bars message type, index number
50, from unsigned ints --- natural numbers, Nat --- to integers; in
tags.c, from:
 
      Qm50(*new(perm) tag:: Nat(e, t.pos   , "quantity" ,   "%6lu")),
      Nm50(*new(perm) tag:: Nat(e, t.pos   , "count"    ,   "%6lu")),
to:
      Qm50(*new(perm) tag:: Int(e, t.neg   , "quantity" ,   "%6ld")),
      Nm50(*new(perm) tag:: Int(e, t.neg   , "count"    ,   "%6ld")),

The order of the related attribute declarations in syms.h also had to
be changed accordingly.

> Trade bars doen't seem to work:
> select bars CASH:IDEALPRO:USD:JPY 1;

Yes, the log shows the request triggered by the above command as
leading to an IB tws api error, with index 420, "invalid
realtime query":
 
3| 4| 2|15016059| 420|Invalid Real-time Query|

This is a limitation of the datafeed; just as you pointed out
above that currency datafeeds don't necessarily include volumes,
so also they do not always include actual trades; I've seen
comments to this effect on the Yahoo twsapi list.

In any case, the shim works fine for realtime bar trades for
stocks and the like --- see exs/bars for examples --- and you
can look at the various logs to see that the realtime bar request
for the above query is formatted properly.  The shim is working
fine in this case.

> ... I am not surprised that for FX, they have set the quantity field
> to be -1.  For Forex bars, most brokers don't aggregate volumes.  So
> this really shouldn't be an error/warning, in the case of Forex.

Please feel free to let me know about this kind of problem if you
observe 402 messages for other foreign exchange quantities, e.g.,
for market data, market depth, and history.  The problem is easy
to fix, and I'd much prefer to make these changes on a case by
case basis as instances of the problem are observed in the wild.

There are many places in which the careful distinction between 
unsigned and signed integral types makes the source code for the
shim internally more robust, but unfortunately this comes at the
cost of the shim being picky about its input.  I'd prefer not to
make wholesale changes of attributes such as quantity to be
possibly negative, and instead rely on reports such as yours to
fix the special cases where this occurs.

Also, it's arguably preferable to elide the warning, since the
shim's response, to treat the input as zero, may well be preferable
to using -1.  This is a point about which reasonable people might
disagree, and I hesitate to filter the tick data from IB, but I'm
definitely interested in the views of users such as yourself who
are actually working with currency data.

Thanks,

Bill