Without this we get errors reading/writing from larger imageṡ.  Our current
10g usr.img actually is enough to trigger this, but our lack of error handling
papered over it so far.

bio queue is the name used by BSD. Since it is just a name difference,
I would better change our code, since there are few users (only ramdisk),
than to patch all code I am importing from BSD that uses it.

Based on FreeBSD virtio code
Provides a x7 boost for rx netperf

Fix memcached networking issue that was caused by unsecure, parallel device invocation. I wasn't aware that the a device lock has to be held on the tx callback. Will look into it deeper tomorrow. The patch solves the issue

This way it's possible to wake a thread while holding the lock
that protects the thread pointer of going away. The lock itself
won't be held by the waker and thus the wakee will be able to
use it immedietly w/o ctx. Suggested by Nadav.

Make the early allocator available earlier to support the dynamic
per-cpu allocator.

No code change.

Instead of cancelling block requests due to no space on the ring
that lead to corruption of the upper layer, block until there is
space.

Required by the virtio spec.

virtio_blk pre-allocates requests into a cache to avoid re-allocation
(possibly an unneeded optimization with the current allocator).  However,
it doesn't take into account that requests can be completed out-of-order,
and simply reuses requests in a cyclic order. Noted by Avi although
I had it made using a peak into the index ring but its too complex
solution. There is no performance degradation w/ smp due to the good
allocator we have today.

virtio_driver::wait_for_queue() would often hang in a memcached and
mc_benchmark workload, waiting forever for received packets although
these *do* arrive.

As part of the virtio protocol, we need to set the host notification
flag (we call this, somewhat confusingly, queue->enable_interrupts())
and then check if there's anything in the queue, and if not, wait
for the interrupt.

This order is important: If we check the queue and only then set the
notification flag, and data came in between those, the check will be
empty and an interrupt never sent - and we can wait indefinitely for
data that has already arrived.

We did this in the right order, but the host code, running on a
different CPU, might see memory accesses in a different order!
We need a memory fence to ensure that the same order is also seen
on other processors.

This patch adds a memory fence to the end of the enable_interrupts()
function itself, so we can continue to use it as before in
wait_for_queue(). Note that we do *not* add a memory fence to
disable_interrupts() - because no current use (and no expected use)
cares about the ordering of disable_interrupts() vs other memory
accesses.

I'm returning Dor's original virtio_driver::wait_for_queue().

The rewrite just masked, with its slightly different timing and redundant
second check before waiting, the real bug which a missing memory barrier
(see separate patch fixing that).

Dor's original code has the good feature that after waking up from a
sleep - when presumably we already have something in the queue - we
check the queue before pessimisticly enabling the host notifications.
So let's use Dor's original code.

In my memcached tests (with mc_benchmark as the driver), I saw
virtio_driver::wait_for_queue appears to have some bug or race condition -
in some cases it hangs on waiting for the rx queue - and simply never
returns.

I can't say I understand what the bug in this code is, however.
Instead, I just wrote it from scratch in a different way, which I think
is much clearer - and this code no longer exhibits this bug.

I can't put my finger on why my new version is more correct than
the old one - or even just difference... Dor, maybe you can find a
difference? But it definitely behaves differently.

virtio-vring and it's users (net/blk) were changed so no request
header will be allocated on run time except for init. In order to
do that, I have to change get_buf and break it into multiple parts:

        // Get the top item from the used ring
        void* get_buf_elem(u32 *len);
        // Let the host know we consumed the used entry
        // We separate that from get_buf_elem so no one
        // will re-cycle the request header location until
        // we're finished with it in the upper layer
        void get_buf_finalize();
        // GC the used items that were already read to be emptied
        // within the ring. Should be called by add_buf
        // It was separated from the get_buf flow to allow parallelism of the two
        void get_buf_gc();

As a result, it was simple to get rid of the shared lock that protected
_avail_head variable before. Today only the thread that calls add_buf
updates this variable (add_buf calls get_buf_gc internally).

There are two new locks instead:
  - virtio-net tx_gc lock - very rarely it can be accessed
    by the tx_gc thread or normally by the tx xmit thread
  - virtio-blk make_requests - there are parallel requests

Trivial: Move code above, preparation for preventing past path allocations for the virtio request data

returns the current time in nanoseconds.

Use a single instance per queue vector of sglist data.
Before this patch sglist was implemented as a std::list
which caused it to allocate heap memory and travel through pointers.
Now we use a single vector per queue to temoprary keep the
buffer data between the upper virtio layer and the lower one.

When the guest reads the used pointer it must make sure that
all the other relevant writes to the descriptors by the
host are up to date.
The same goes for the other direction when the guest
updates the ring in get_buf - the write to used_event should
make all the descriptor changes visible to the host

Instead of enabling interrupts for tx by the host when we have
a single used pkt in the ring, wait until we have 1/2 ring.
This improves the amount of tx irqs from one per pkt to practically
zero (note that we actively call tx_gc if there is not place on the
ring when doing tx). There was a 40% performance boost on the
netperf rx test

Convert avail._idx to a std::atomic variable As a result, get rid of the compiler barrier calls since we use std:atomic load/store instead

Easier to compute deltas.  Limits range to 292 years.

The optimization improves performance by letting each side
of the ring know what was the last index read by the remote
party. In case were the other side has older indexes, waiting
for processing we won't trigger another update (kick or irq,
depending on the direction).
The optimization reduces irq injection by a 7%.

Along the way, collapse vring::need_event to the code and use
std::atomic(s) to access any variable which is shared w/ the host

Unlike KVM, we won't use percpu variables because Xen already lays down
statically the shared info structure, that includes the vcpu info pointer
for each cpu.

We could in theory use percpu variables to store pointers to the current cpu
vcpu info, but I ended up giving up this route.  Since our pcpu implementation
have the overhead of computing addresses anyway, we may as well pay the price
and compute it directly from the xen shared info.

One of the things that comes with it, is that we can compute precise timings
using xenclock very early. Since we don't have *that* much to do early, it is
unclear if KVM needs to be improved in this regard (probably not), so this
becomes just a slight bonus.

The designer of kvmclock wrote it in a way so as to be ABI-compatible
with xen's pvclock. We can reuse the same structures, then, so let's do it.

Because we are now pre-computing wall clock, we only need preemption disabled during system
time calculation.

This reverts commit dae99589.