Mention all package that need to be installed on a bare bones Fedora system,
make sure service zfs-fuse start is done using sudo.

Now that we have a reliable number of CPUs indicator the implementation
is trivial.

Fix various issues with the debug allocator.

map_file() takes the vm lock, then calls read() to pre-fault the data.
However read() may cause allocations, which then require the vm lock as well.

Fix by faulting in the data after dropping the lock.

The standard allocator returns page-aligned addresses for large allocations.
Some osv code incorrectly relies on this.

While we should fix the incorrect code, for now, adjust the debug allocator
to return aligned addresses.

The debug allocator now uses the following layout:

  [header page][guard page][user data][pattern tail][guard page]

Required by the virtio spec.

Virtio and other hardware needs physically contiguous memory, beyond one page.
It also requires page-aligned memory.
Add an explicit API for contiguous and aligned memory allocation.

While our default allocator returns physically contiguous memory, the debug
allocator does not, causing virtio devices to fail.

virtio_blk pre-allocates requests into a cache to avoid re-allocation
(possibly an unneeded optimization with the current allocator).  However,
it doesn't take into account that requests can be completed out-of-order,
and simply reuses requests in a cyclic order. Noted by Avi although
I had it made using a peak into the index ring but its too complex
solution. There is no performance degradation w/ smp due to the good
allocator we have today.

In commit 7ecbf29f I added to the
poll_install() stage of poll() a check of the current state of the file -
to avoid the sleep if the file became ready before we managed to "install"
its poll request.

However, I wrongly believed it was necessary to put this check inside
the FD_LOCK together with the request installation. In fact, it doesn't
need to be in the same lock - all we need is for the check to happen
*after* the installation. The call to fo_poll() doesn't need to be in
the same FD_LOCK or even in an FD_LOCK at all.

Moreover, as it turns out, it must NOT be in an FD_LOCK() because this
results in a deadlock when polling sockets, caused by two different
code paths taking locks in opposite order:

1. Before this fix, poll() took FD_LOCK and called fo_poll() which
   called sopoll_generic() which took a SOCKBUF_LOCK

2. In the wake path, SOCKBUF_LOCK was taken, then so_wake_poll()
   is called which calls poll_wake() which takes FD_LOCK.

virtio_driver::wait_for_queue() would often hang in a memcached and
mc_benchmark workload, waiting forever for received packets although
these *do* arrive.

As part of the virtio protocol, we need to set the host notification
flag (we call this, somewhat confusingly, queue->enable_interrupts())
and then check if there's anything in the queue, and if not, wait
for the interrupt.

This order is important: If we check the queue and only then set the
notification flag, and data came in between those, the check will be
empty and an interrupt never sent - and we can wait indefinitely for
data that has already arrived.

We did this in the right order, but the host code, running on a
different CPU, might see memory accesses in a different order!
We need a memory fence to ensure that the same order is also seen
on other processors.

This patch adds a memory fence to the end of the enable_interrupts()
function itself, so we can continue to use it as before in
wait_for_queue(). Note that we do *not* add a memory fence to
disable_interrupts() - because no current use (and no expected use)
cares about the ordering of disable_interrupts() vs other memory
accesses.

This patch fixes two bugs in so_wake_poll(), which caused us missing
some poll wakeups, resulting in poll()s that never wake up. This can be
seen as a hang in the following simple loop exercising memcached:
         (while :; do; echo "stats" | nc 192.168.122.100 11211; done)

The two fixes are:

1. If so_wake_poll() decides *not* to call poll_wake() - because it sees
   zero data on this packet - it mustn't reset the SB_SEL flag on the
   socket, or we will ignore the next event even when it does have data.

2. To see if the socket is readable, we need to call soreadable(), not
   soreadabledata() - the former adds the connection close event to the
   readability. See sopoll_generic(), which also sets a readability
   event in that case.

I'm returning Dor's original virtio_driver::wait_for_queue().

The rewrite just masked, with its slightly different timing and redundant
second check before waiting, the real bug which a missing memory barrier
(see separate patch fixing that).

Dor's original code has the good feature that after waking up from a
sleep - when presumably we already have something in the queue - we
check the queue before pessimisticly enabling the host notifications.
So let's use Dor's original code.

It is usually very useful, together with PAGE_SIZE

No plans in sight to page out anonymous memory, so for now, stub this.
We may have to revisit this once we support proper mmap semantics, specially
with ranged mlock/munlock

From dlopen man page:
"If filename  is  NULL,  then the returned handle is for the main program."

What I do in this patch is exactly this

Since will now have the xen interface files for BSD anyway, let's
use their more readable definitions instead of hardcoded numbers and duplicated
strings.

This was badly adapted from Avi's 5 argument hypercall. It is not in use for now,
so let's just delete it.

verbatim header files import, plus the code in build.mk to find them

In my memcached tests (with mc_benchmark as the driver), I saw
virtio_driver::wait_for_queue appears to have some bug or race condition -
in some cases it hangs on waiting for the rx queue - and simply never
returns.

I can't say I understand what the bug in this code is, however.
Instead, I just wrote it from scratch in a different way, which I think
is much clearer - and this code no longer exhibits this bug.

I can't put my finger on why my new version is more correct than
the old one - or even just difference... Dor, maybe you can find a
difference? But it definitely behaves differently.

virtio-vring and it's users (net/blk) were changed so no request
header will be allocated on run time except for init. In order to
do that, I have to change get_buf and break it into multiple parts:

        // Get the top item from the used ring
        void* get_buf_elem(u32 *len);
        // Let the host know we consumed the used entry
        // We separate that from get_buf_elem so no one
        // will re-cycle the request header location until
        // we're finished with it in the upper layer
        void get_buf_finalize();
        // GC the used items that were already read to be emptied
        // within the ring. Should be called by add_buf
        // It was separated from the get_buf flow to allow parallelism of the two
        void get_buf_gc();

As a result, it was simple to get rid of the shared lock that protected
_avail_head variable before. Today only the thread that calls add_buf
updates this variable (add_buf calls get_buf_gc internally).

There are two new locks instead:
  - virtio-net tx_gc lock - very rarely it can be accessed
    by the tx_gc thread or normally by the tx xmit thread
  - virtio-blk make_requests - there are parallel requests

Trivial: Move code above, preparation for preventing past path allocations for the virtio request data

wakeup() and wakeup_one() dropped the lock before waking the thread(s).
But if a thread timed out at the same time we are planning to wake it,
and if the thread imediately exited, our attempt to wake() it could crash.

Therefore wakeup() needs to continue holding the lock until after the
wake(). When msleep() times out, before it returns it reacquires the
lock, ensuring it cannot return before a concurrent wakeup() does
its wakes.

Note that in the ordinary wakeup() case (not racing with a timeout),
we are *not* negatively affected by holding the lock while wake()ing
because the waking msleep() does not try to reacquire the lock in the
non-timeout case.

Additionally, we use wake_with() instead of separate set and wake
instructions. Without wake_with() we have the (exceedingly rare)
possibility that merely setting _awake=true causes msleep() to
return, and its caller then decides to exit the thread, the
wake() right after the _awake=true would crash.

Aside from mutex, BSD always implements sx locks. They are a form of rwlock,
and according to BSD's sx.h, the differences from their own rwlock are an
implementation detail. Let's just use them as rwlocks for now.

The declarations are uglier than I wanted. But this file ends up being included
from c and cc code, and rwlock.h calls condvar.h inside - which is full of
sync_stub.h - and as a result condvar.h itself - are usually included in extern
"C" blocks. Because we are not expected to use sxlocks from C, it should be
fine.