Not wired up yet due to the problems of creating a zfs fs on the host.

This patch solves issue #11, where join() hangs were seen in tst-mutex, with
threads remaining in "terminating" state.

The problem was that complete() assumed that _cpu->terminating_thread was
nullptr, so it could overwrite it with the current thread. This is usually
true - when we switch to any thread, if terminating_thread!=0 it is
handled. The problem is that when we switch to a *new* thread (in
sched::init(), call switch_to_first()) this code does not run, and if
this thread quickly terminates, _cpu->terminating_thread gets overwritten
instead of being handled.

The simplest workaround, in this patch is simply to handle (i.e., call
unref()) the previous _cpu->terminating_thread before overwriting it.

The downside with this approach is that the termination of a thread
may be delayed by the run time of the first time-slice of a new thread.

But we anyway plan to eventually replace this termination mechanism
(see issue #10), so I think this solution is fine.

Starting the CLI and using the run command by specifying it as a run.py argument
didn't work due to a cast problem (run expected NativeArray).

previousely this didn't work:

$ sudo ./scripts/run.py -n -e "java.so -jar /java/cli.jar run tools/netserver-osv -D -4 -f -N" -c2 -m1G

This reverts commit ccaad30c.

VIRTIO_RING_F_EVENT_IDX and VIRTIO_RING_F_INDIRECT_DESC are
defined in virtio.hh

No need to assign the condvar initializer, this is C++ after all and objects
are initialized by default anyway.

When a callout is deleted, it is properly deleted from the set of
callouts, but if it was the next-in-line to run, it was also saved
in a local variable while waiting for its timer to expire, and could
be run despite being deleted.

The Shrew test HTTP server (see bug 7) exposed this bug - every once
in a while (usually very quickly) when a socket was deleted had a
crash when a callout referring to the deleted socket was run.

Thanks to Guy for finding and fixing this bug.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>

Without <endian.h> we won't pick up any byte order defines.  In addition
Solaris also uses single underscore prefixed versions of the defines instead
of the more common non- or double underscore prefixed ones.

Our code had a serious bug in thread completion: When a thread complete()s,
i.e., finishes its work, it wake()s the thread doing join() on it,  and
that joiner thread in turn deletes the completed thread and its stack.
On rare occasions, the wake() was very slow but the joiner thread was very
quick in deleting the thread - leading to a crash on return (retq)
from wake() because the stack on which it was running has been deleted.

This patch includes a simple, but effective, fix for this bug:

We add a new per-cpu field, cpu::terminating_thread. complete() no longer
calls unref() itself - as the thread unref()ing itself caused the bug.
Instead, complete() just sets terminating_thread to the current thread.
After the scheduler on this CPU switches to the next thread, we
call unref() on the thread specified in terminating_thread. We know
this is safe because this thread is no longer running.

This fix seems simple and effective (the crashes that were apparent
in tst-wake and the sunflow benchmark seem to be gone, as far as I
can tell). Its biggest downside is an extra "if" on every context switch.
It is possible to devise different solutions, without the cost of the
extra if, but these solutions are more complicated and require a lot
more code changes. I'll add a bug-tracker entry documenting them.

same as we can tell the host to disable interrupts via the _avail ring,
the host can tell us to supress notification via the _used ring.
every notificaion, or kick consumes about 10ns as it is implemented as
writing to an io port, which travels to usespace qemu in the host.

this simple patch, increase netperf's throughput by 600%, from a
300mbps to 1800mbps.

Indirect is good for very large SG list but isn't required
in case there is enough place on the ring or the SG list is tiny.
For the time being there is barely use of it so I set it off
by default

The feature allows the hypervisor to batch several packets together
as one large SG list. Once such header is received, the guest rx
routine interates over the list and assembles a mega mbuf.

The patch also simplifies the rx path by using a single buffer for
the virtio data and its header. This shrinks the sg list from size of
two into a single one.

The issue is that at the moment I haven't seen packets w/ mbuf > 1
being received. Linux guest does receives such packets here and there.
It may be due to the use of offload features that enalrge the packet size

this rwlock gives precedence to writers, it relies on a mutex and 2 condvars
for it's implementation.

it also supports taking the lock recursively for both readers and writers.

this implementation is not fully tested but yet the TCP stack uses it
extensively, so far without any seen races (tested TCPDownload and netperf).

1. it is much cleaner that the header files perform extern "C" themselves,
   so they can be included both from C and C++ code.

2. when doing extern "C" from a C++ file then __cplusplus is also defined,
   and compilation can break in some situations.

3. as a bonus, this patch increase compilation time.

netport.h defines a log() macro, which is an unfortunate choice of name
because log is also a pretty-well-known mathematical function, and this

So rename this macro bsd_log(), and change the dozen files which used
log() to use bsd_log().

Java's os::available() requires the FIONREAD on fds which do not
implement seek. So we need to support this ioctl for the console.

Use the new wake_with() in lock-free mutex

Before commit 1b53ec56,
condvar_wake_all() had a crash which could be seen tst-pipe.so (which
apparently tests some condvar code paths that weren't tested by
tst-condvar.so).

The fix in this commit was for condvar_wait() to regain the condvar
internal lock after the wait, even if not needed (it's only really
needed in case of timeout). This masked the bug (see details below)
but also deteriorated performance: the woken up thread will now often
goes back to sleep to wait for the lock which is still held by
condvar_wake().

This patch reverts that commit, i.e., condvar_wait() does not retake
the lock when woken. Instead, we fix the real bug: The bug was in
condvar_wake_all() which did:

        sched::thread *t = wr->t;
        wr->t = nullptr;
        t->wake();
        wr = wr->newer;

but after the wake(), wr is no longer valid (the waiter, being woken,
would quickly exit the condvar_wait() function which held wr on the stack).

However, by not taking the lock after the wait we also have another
potential for bug - in rare cases merely doing wr->t = nullptr can
cause the thread t to start running, and if it not only stops waiting
but also exits - the call to t->wake() will refer to an invalid thread
and may crash. So we need to use the new wake_with() thread method
introduced in a previous patch.