In the allocation tracker, not only did I use a dog-slow linear search,
I forgot to stop on the first empty spot, and actually used the last
empty spot... Add the missing break, which made leak detection 10% faster.

A better implementation would be preferable, but this is a low hanging
fruit

Various improvements to "osv leak show":

1. Somewhat faster performance (but still slow).

2. Better report of progress (percent done). Previously, much of the`
   work of fetching the backtraces from the guest was actually delayed
   until sort time, so was wrongly attributed to the sort phase. Now
   the fetching phase takes most of the time, and percent of its
   progress is shown.

3. Due to popular request: sort leak records by size:

   Instead of outputting immediately each leak record (summary of all
   living allocations from a particular call chain), we now save them
   in memory, making it very easy to sort these records by any interesting
   criterion.
   In this patch, I sort them in decreasing order of total bytes - i.e.,
   the first record one sees is the one responsible for most allocated
   bytes.

   The additional sort takes only a fraction of a second, and makes the output
   of "osv leak show" much more useful.

As Avi suggested, add an option (turned on by default) to remember only
the most recent function calls - instead of the most high-level function
calls like I did until now - in an allocation's stack trace.

In our project, where we often don't care about the top-level
functions (various Java stuff), it is more useful.

Detached threads delete themselves, so the auto-join creates an infinite loop.

Avoid by checking whether this is a detached thread when destroying it.

The detached thread reaper deletes zombies, but our pthread implementation
also deletes dead pthreads (using the container object).

Fix by making the base thread use the set_cleanup() method to set up a
deleter, which is then overridden by pthreads.

Improved detached threads handling and pthread_mutex_t.

the FPU on a context switch caused by a function call (as opposed
to a preemption during interrupt), in practice this makes the
"sunflow" benchmark from SpecJVM fail, producing wrong results.

This patch saves the FPU on any context switch and makes "sunflow"
work correctly, at the price of slower context switches and an
unsolved puzzle on why the heck this is needed in the first place :(

We had a klugey pmutex class used to allow zero initialization of
pthread_mutex_t.  Now that the mutex class supports it natively we
can drop it.

pthread_mutex_t has a 32-bit field, __kind, at offset 16.  Non-standard
static initializers set this field to a nonzero value, which can corrupt
fields in our implementation.

Rearrange field layout so we have a hole in that position.  To keep the
structure size small enough so that condvar will still fit in
pthread_condvar_t, we need to change the size of the _depth field to
16 bits.

Use the generic one instead; the cleanup function allows destroying
the pthread object.

Instead of a subclass, make it a thread attribute.  This simplifies usage
and also allows detaching a thread after creation.

Detached threads are auto collected, so give users a chance to execute some
cleanup function before dying.

for the very deep calls in Java. Increase it.

In the future, I should have an option to save only the deepest calls,
not the calls nearest the root.

read_random() is used indirectly by the TCP stack to randomize a local port number,
before this patch it was identical in each execution, and that caused NAT problems.

also add a FIXME note to implement real random one day.

the linux/musl definition is 2 bytes long instead of 1 byte as in
freebsd, this patch is fixing this issue which caused ifconfig to fail

Attempting to put an end to the linux<->freebsd confusion.

"osv leak show" can take a very long time until you start seeing output.
This patch shows the progress of the first phase (getting data from the
debugged program), and then says the second phase (sorting) starts.
By the way, it turns out the sorting phase is slower (complexity-wise,
this is obvious, but I didn't really expect it in this case).

The previous implementation of backtrace() required frame pointers.
This meant it could only be used in the "debug" build, but worse,
it also got confused by libstdc++ (which was built without frame pointers),
leading to incorrect stack traces, and more rarely, crashes.

This changes backtrace() to use libunwind instead, which works even
without frame pointers. To satisfy the link dependencies, libgcc_eh.a
needs to be linked *after* libunwind.a. Because we also need it linked
*before* for other reasons, we end up with libgcc_eh.a twice on the
linker's command line. The horror...

mmu::allocate(), implementing mmap(), used to first evacuate the
region (marking it free), then allocate a tiny vma object (a start,end
pair), and finally populate the region.

But it turns out that the allocation, if it calls backtrace() for the first
time, ends up calling mmap() too :-) These two running mmap()s aren't
protected by the mutex (it's the same thread), and the second mmap could
take the region just freed by the first mmap - before returning to the
first mmap who would reuse this region.

We solve this bug by allocating the vma object before evacuating the
region, so the other mmap picks different memory.

Before this fix, "--leak tests/tst-mmap.so" crashes with assertion
failure. With this fix, it succeeds.

This patch fixes two deadlocks which previously existed in the allocation
tracking code in core/mempool.cc. These deadlock have become more visible
in the new backtrace() implementation which uses libunwind and more often
allocates memory during its operation (especially from dl_iterate_phdr()).

1. alloc_page() wrongly held the free_page_ranges lock for a bit too
   long - including when calling tracker_remember(). tracker_remember()
   then takes the tracker mutex. Unfortunately, the opposite lock order
   also occurs: Consider a tracker_remember() (e.g., from malloc)
   needing to allocate memory and through the memory pool, end up calling
   alloc_page().

   This is a classic deadlock situation. The solution is for alloc_page()
   to drop free_page_ranges_lock before calling tracker_remember().

2. Another deadlock occured between the tracker lock and a pool::_lock -

   thread A: malloc calls remember() taking the TRACKER LOCK, and then
   calling malloc() (often in dl_iterate_phdr()), which calls
   memory::pool::alloc and takes the POOL LOCK.

   thread B: malloc calls memory::pool::alloc which takes the POOL LOCK
   and then if the pool is empty, calls alloc_page() which is also
   tracked so it takes the TRACKER LOCK.

   Here the solution is not to track page allocations and deallocations
   from within the memory pool implementation. We add
   untracked_alloc_page() and untracked_free_page() and use those in
   the pool class. This not only solves the deadlock, it also provides
   better leak detection because pages held by the allocator are now
   no longer considered "leaks" (just the individual small objects
   themselves).

   The fact alloc_page() now calls untracked_alloc_page()
   also made solving problem 1 above more natural (the free_pages
   lock is held during untracked_alloc_page()).

When the allocation tracking code itself does allocations, we do not
track these allocations - and we notice this by the "depth" of the
alloc_tracker mutex. This avoids some messy infinite recursion, and also
improves performance because alloc_tracker does do a bunch of allocations
(most of them not apparent from the code), and tracking them too would
significantly slow it down, with no benefit because we're not debugging
the allocation tracker itself.

But while the existing code ignored nested alloc_tracker::remember(),
we forgot to also ignore nested alloc_tracker::forget()! This meant
that for each allocation inside alloc_tracker, we never tracked it,
but wasted our time trying to delete it from the list of living
allocations. This oversight caused a huge slowdown of alloc_tracker(),
which this patch fixes. alloc_tracker() is now just very slow, not
very very very slow ;-)

libunwind, which the next patches will use to implement a more reliable
backtrace(), needs the msync() function. It doesn't need it to actually
sync anything - just to recognize valid frame addresses (stacks are
always mmap()ed).

Note this implementation does the checking, but is missing the "sync" part
of msync ;-) It doesn't matter because:

1. libunwind doesn't need (or want) this syncing, and neither does anything
   else in the Java stack (until now, msync() was never used).

2. We don't (yet?) have write-back of mmap'ed memory anyway, so there's
   no sense in doing any writing in msync either. We'll need to work on
   a full read-write implementation of file-backed mmap() later.

As Nadav found out, we don't have to call defineClass and register each visible
Java class within the js scope, this makes it much simpler to integrate with Java
from the cli.

Currently, if we're requested to preempt while preemption is disabled, we
just ignore the request and wait until the next interrupt (which is not even
guaranteed to arrive) hits us.  This isn't the best arrangement.

Fix by latching the preemption request into a 'need_reschedule' thread local
variable, and examining it during preempt_enable().

Use includes from musl and external/, avoiding the host development
environment.

Not provided by the musl headers, and also unneeded.