[ts-gen] segmentation faults

[Bill Pippin]

Nils [and Jose],

I've pushed a new release; please try it out and let us know how
it works.  The db version has not changed, so if you have an
up-to-date database, it's just download and recompile.

Thanks for the comprehensive trace info.  Clearly I asked the wrong
questions; apropos would have been:

    What is your memory architecture?

Fortunately the gdb banner you included indicated the problem:

> This GDB was configured as "x86_64-linux-gnu"...

I should have mentioned that the shim's memory allocator was not
yet working for 64bit architectures, and that was indeed the problem.
I discussed feature priorities during a recent post this month,
including refactoring of the memory allocator which is to be
done for a number of reasons, not least being 64bit support:

    http://www.trading-shim.org/pipermail/ts-general/2008-August/000231.html

That post was in response to Jose Casado, who also is trying to
use the shim on a 64bit desktop, so although this issue already
had a reasonably high priority, between the two of you your
reports raised it that much higher.

Therefor, I went through the allocator code cleaning up the
use of manifest constants, mostly by improving the implementation
and client use of MemConstants, an existing class that had been
somewhat incomplete.  For my immediately accessible 64bit platform,
amd64, this new version of the shim works fine for the regression
tests, so I've pushed it out to the web site so it's available to
interested users.

This is not yet the full-fledged refactoring of the memory
allocator that is on the priority list; it's simply all the code
cleanup I can see to do without hitting a slippery slope.

So, I hope it works for now; please let the list know whether it
works for you, since there has not yet been much 64bit testing
with this newest version; and, for now, for me it's back to
cash symbols, session-related order journalling and the like.

I do expect to take another look at memory allocation
after I finish the problems I'm working on now.

The rest of this post is an analysis of the trace you posted,
work I did prior to the code cleanup of MemConstants; this
post is really two-in-one, and the graft occurs right here.

Editing the trace you supplied as follows:

> Program received signal SIGSEGV, Segmentation fault.
>      in Cursor::Cursor(this, a=ref, b=ref, c=0x0)      at src/envs.c:241
> 241              eRror(*new (a.temp)    nat(0))
> Current language:  auto; currently c++
> (gdb) 
> (gdb) where
> #14               main(argc, argv=ptr)                  at src/main.c: 13
> #13 Shim::        shim(argc, argv=ptr)                  at src/shim.c: 16
> #12 SingleTree :: root( ref, a=ref)                     at src/once.c: 34
> #11       ::Singletons(this, c=ref)                     at src/once.c: 21
> #10               Init(this, c=ref, perm=ref, a=ref)    at src/once.c: 46
> # 9         Shim::Shim(this, p=ref, l=ref, args=ref)    at src/once.c: 56
> # 8 Shim::        init(this, e=ref, s=ref, f=ref, r=ref)at src/init.c: 39
> # 7 LateData::LateData(this, e=ref, m=ref, i=ref)       at src/init.c: 54
> # 6 Main::        tabs(this, e=ref, c=ref)              at src/init.c: 77
> # 5 InitE::tables_init(this)                            at src/init.c:118
> # 4 Tables::      init(this, i=ref, r=ref)              at src/init.c:143
> # 3 Tables::      read(this, e=ref, tabs=ref)           at src/fill.c: 21
> # 2 Persistent::  read(this, q=ref)                     at src/fill.c: 46
> # 1 Mysql::      query(this, e=ref, v=ref, q=ptr)       at src/fill.c:105
> # 0     Cursor::Cursor(this, a=ref, b=ref, c=0x0)       at src/envs.c:241

We can see that the shim is performing table fill, just as the
partial banner suggests; that it is reading records into a 
database table image vector, that the mysql connection has
accepted and performed a query, and, given the Cursor ctor
call, that the shim is about to construct a record from one
of the rows in the query result.  The actual crash is in the
ctor, during a memory allocation, which just goes to emphasize
that the memory allocator was not working for 64bit.

Looking at the ctor call where the segfault occurs, the arg
values appear valid (the null pointer as the initial value
for the rule variable is perfectly legitimate here, and even
if not, could only lead to problems later, once dereferenced),
and the actual fault itself is during the last call to new,
due presumably to a bad return value from the temporary memory
pool a.temp .

> iof::Cursor
>    ::Cursor(Atom_e a, bool_ref b, Rule_0 c)
>    : reader(a.reader),
>        lOop(b),
>        atom(a),
>        rule(*new (a.temp) Rule_0(c)),
>       sTate(*new (a.temp)    nat(start())),
>       index(*new (a.temp)    nat(0)),
>       words(*new (a.temp)    nat(0)),
>       eRror(*new (a.temp)    nat(0))
> {}

The other trace shares a common call graph prefix for the
first 14 nodes, up to the MySql::query() call, and I believe
the underlying problem there was the same, that is a bug
in the memory allocator for 64bit.

As always, if you have questions or problems, please post to the
list, and again, thanks very much for the bug reports.

Thanks,

Bill