This way it's possible to wake a thread while holding the lock
that protects the thread pointer of going away. The lock itself
won't be held by the waker and thus the wakee will be able to
use it immedietly w/o ctx. Suggested by Nadav.

No code change.

Instead of cancelling block requests due to no space on the ring
that lead to corruption of the upper layer, block until there is
space.

virtio_blk pre-allocates requests into a cache to avoid re-allocation
(possibly an unneeded optimization with the current allocator).  However,
it doesn't take into account that requests can be completed out-of-order,
and simply reuses requests in a cyclic order. Noted by Avi although
I had it made using a peak into the index ring but its too complex
solution. There is no performance degradation w/ smp due to the good
allocator we have today.

virtio-vring and it's users (net/blk) were changed so no request
header will be allocated on run time except for init. In order to
do that, I have to change get_buf and break it into multiple parts:

        // Get the top item from the used ring
        void* get_buf_elem(u32 *len);
        // Let the host know we consumed the used entry
        // We separate that from get_buf_elem so no one
        // will re-cycle the request header location until
        // we're finished with it in the upper layer
        void get_buf_finalize();
        // GC the used items that were already read to be emptied
        // within the ring. Should be called by add_buf
        // It was separated from the get_buf flow to allow parallelism of the two
        void get_buf_gc();

As a result, it was simple to get rid of the shared lock that protected
_avail_head variable before. Today only the thread that calls add_buf
updates this variable (add_buf calls get_buf_gc internally).

There are two new locks instead:
  - virtio-net tx_gc lock - very rarely it can be accessed
    by the tx_gc thread or normally by the tx xmit thread
  - virtio-blk make_requests - there are parallel requests

Trivial: Move code above, preparation for preventing past path allocations for the virtio request data

Use a single instance per queue vector of sglist data.
Before this patch sglist was implemented as a std::list
which caused it to allocate heap memory and travel through pointers.
Now we use a single vector per queue to temoprary keep the
buffer data between the upper virtio layer and the lower one.

we have to disable virio interrupts before msix EOI so disabling
must be done in the ISR handler context. This patch adds an std::function
isr to the bindings.

references to the rx and tx queues are saved as well (_rx_queue and _tx_queue),
so they can be used in the ISR context.

this patch reduces virtio net rx interrupts by a factor of 450.

We cannot relay on the overlay protocol to provide this length and
we must receive it from the hypervisor.

This allows taking BIO_ERROR into the block layer core and remove unsafe
bio_flags manipulations in the virtio-blk driver.

Move osv/bio.h into the .cc implementation and along with it move the virtio_blk_req structure that can be hidden from the header.

virtio_device is now a simple set of accessors around pci::device.  Since
the only users of virtio_device are virtio_driver and its subclasses, we
can simply fold it into virtio_driver, eliminating the middleman.

Resolves two FIXMEs where we leaked virtio_device objects created during
probing.

While a great feature, it's not supported in C++.  G++ does support it as an
extension, but eclipse's static code analyzer does not, which causes a lot
of false errors to be reported.

pci::pci_device is redundant, use pci::device.

The pci layer knows about virtio devices and creates them, even though
virtio is a higher layer.

Fix by making virtio_device contain a reference to the pci device, instead
of inheriting from it.  The functionality could probably be moved to
pci_driver.

Instead of requiring the user to construct an msix entry -> thread binding
map, they can now pass it directly:

  _msi.easy_register({ { 0, isr0 },  { 1, isr1 } });

Since the interrupt manager is now local to a device, we can initialize
the device pointer once and remember it.  This simplifies the API.

There are actually no interactions between msix registrations of different
drivers, so it doesn't help to use a global instance.  We can simplify the
interface by making it local, and also remove locking considerations.

These are subsumed into the driver constructor/destrutor.

Driver enumeration requires instantiating a driver even for devices which
are not present; and if a device is present multiple times, we need to
pre-create multiple driver instances, which is awkward.  Further, driver
life-cycle is complicated, with separation between instantiation and binding
to a device.

Switch (back) to the traditional device-driven model, where we iterate over
all devices, try to find a matching driver factory, and when found, call it
to instantiate the driver and bind to the device.

driver itself. It removes the hard coded binding to queue 0
and will enable flexibility for the upcoming virtio-net registration.

and call biodone when appropriate

Now it's possible to stress virtio through the external tests so
we do not need this code

debug output and do some order

Discover that bio->bio_offset is zero while the data starts in
the middle of the page so I adjusted the code accordingly.
Will need to sort it out later on.

The logic is called as callback invoked by virtio interrupts.

In addition, polish the test code. Before all of the write request
had filled the ring and left no space for the read requests. Now I rotate
between the two.