New interface allows to send a single doorbell to the host per
buffers bulk.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

(u16)_avail_added_since_kick wrap-around should be avoided since we may loose
an _avail_event update and thus miss the point where the HV should have
been woken (kicked).

We choose half a u16 range as a threshold since kick() is usually called for a
bulk of buffers and not for every separate buffer. Number of buffers in such a bulk is
limited by a size of a virtio ring. The virtio ring size is unlikely to have 32K
entries in the nearest future (it has 256 entries now) thus kicking() at least
every 32K buffers will ensure that we prevent the mentioned above wrap-around
as a result of such a bulking.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Using malloc() fails with the debug allocator, and can fail for the normal
allocator as well since it does not guarantee physical memory.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It can reduce the duplicated code

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Useful for tracing virtio queue utilization issues.

To get plottable data:
 1. Save traces to gdb.txt
 2. grep virtio_add_buf gdb.txt | grep "queue=0" \
     | awk '{print $3 " " $6}' | sed -r 's/avail=(.*)/\1/'

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Clean up virtio-vring.hh and move add_buf_wait() out of line. Benefits:

  - Plays well with tracepoints

  - Less recompilation when code is changed

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

With indirect descriptor, we can queue more buffers in the queue.
Indirect descriptor helps block device by making the large request does
not consume the entire ring and making the queue depth deeper. Indirect
descriptor does not help net device because it makes the queue longer so
it adds latency. The tests show that indirect descriptor makes blk
faster and there is no real measurable degradation on net. Also the
indirect will turn on only when we are short of descriptors.

This patch only enables indirect descriptor for vblk and vscsi. vnet is
not enabled.

1) vblk
Before: 340MB/s
After:  350MB/s

2) vscsi
Before: 320MB/s
After:  410MB/s

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Replace divides by variables and by the hard constants with masks and
divides by easy constants.

Improves netperf by about 1.6%.

Noted by Vlad.

Reviewed-by: Dor Laor <dor@cloudius-systems.com>
Reviewed-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

It is useful to test if we can do gc on the used ring.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

When the _avail_count is less than 1/3 of the ring, we start using
indirect descriptor.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Dor Laor <dor@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

There is no reason we should do the scale.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Dor Laor <dor@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

This patch fixes:

- The order of _used_ring_host_head and _used->_idx, the latter is more
  advanced than the former.
- The unwanted promotion to "int"

Pekka wrote:

   However, on the right-hand side, the expression type in master
   evaluates to "int" because of that innocent-looking constant "2" and
   lack of parenthesis after the cast.  That will also force the
   left-hand side to promote to "int".

   And no, I really don't claim to follow integer promotion rules so I
   used typeid().name() verify what the compiler is doing:

   [penberg@localhost tmp]$ cat types.cpp
   #include <typeinfo>
   #include <stdint.h>
   #include <cstdio>

   using namespace std;

   int main()
   {
       unsigned int _num = 1;

       printf("int                = %s\n", typeid(int).name());
       printf("uint16_t           = %s\n", typeid(uint16_t).name());
       printf("(uint16_t)_num/2)  = %s\n", typeid((uint16_t)_num/2).name());
       printf("(uint16_t)(_num/2) = %s\n", typeid((uint16_t)(_num/2)).name());
   }
   [penberg@localhost tmp]$ g++ -std=c++11 -Wall types.cpp
   [penberg@localhost tmp]$ ./a.out
   int                = i
   uint16_t           = t
   (uint16_t)_num/2)  = i
   (uint16_t)(_num/2) = t

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add Cloudius copyright and license statement to drivers/*.

A couple of header files were based on Linux's BSD-licensed header files
(e.g., include/uapi/linux/virtio_net.h) so they included the BSD license,
but not any copyright statement, so we can just replace that by our own
statement of the BSD license.

Required by the virtio spec.

virtio_driver::wait_for_queue() would often hang in a memcached and
mc_benchmark workload, waiting forever for received packets although
these *do* arrive.

As part of the virtio protocol, we need to set the host notification
flag (we call this, somewhat confusingly, queue->enable_interrupts())
and then check if there's anything in the queue, and if not, wait
for the interrupt.

This order is important: If we check the queue and only then set the
notification flag, and data came in between those, the check will be
empty and an interrupt never sent - and we can wait indefinitely for
data that has already arrived.

We did this in the right order, but the host code, running on a
different CPU, might see memory accesses in a different order!
We need a memory fence to ensure that the same order is also seen
on other processors.

This patch adds a memory fence to the end of the enable_interrupts()
function itself, so we can continue to use it as before in
wait_for_queue(). Note that we do *not* add a memory fence to
disable_interrupts() - because no current use (and no expected use)
cares about the ordering of disable_interrupts() vs other memory
accesses.

virtio-vring and it's users (net/blk) were changed so no request
header will be allocated on run time except for init. In order to
do that, I have to change get_buf and break it into multiple parts:

        // Get the top item from the used ring
        void* get_buf_elem(u32 *len);
        // Let the host know we consumed the used entry
        // We separate that from get_buf_elem so no one
        // will re-cycle the request header location until
        // we're finished with it in the upper layer
        void get_buf_finalize();
        // GC the used items that were already read to be emptied
        // within the ring. Should be called by add_buf
        // It was separated from the get_buf flow to allow parallelism of the two
        void get_buf_gc();

As a result, it was simple to get rid of the shared lock that protected
_avail_head variable before. Today only the thread that calls add_buf
updates this variable (add_buf calls get_buf_gc internally).

There are two new locks instead:
  - virtio-net tx_gc lock - very rarely it can be accessed
    by the tx_gc thread or normally by the tx xmit thread
  - virtio-blk make_requests - there are parallel requests

Use a single instance per queue vector of sglist data.
Before this patch sglist was implemented as a std::list
which caused it to allocate heap memory and travel through pointers.
Now we use a single vector per queue to temoprary keep the
buffer data between the upper virtio layer and the lower one.

When the guest reads the used pointer it must make sure that
all the other relevant writes to the descriptors by the
host are up to date.
The same goes for the other direction when the guest
updates the ring in get_buf - the write to used_event should
make all the descriptor changes visible to the host

Instead of enabling interrupts for tx by the host when we have
a single used pkt in the ring, wait until we have 1/2 ring.
This improves the amount of tx irqs from one per pkt to practically
zero (note that we actively call tx_gc if there is not place on the
ring when doing tx). There was a 40% performance boost on the
netperf rx test

Convert avail._idx to a std::atomic variable As a result, get rid of the compiler barrier calls since we use std:atomic load/store instead

The optimization improves performance by letting each side
of the ring know what was the last index read by the remote
party. In case were the other side has older indexes, waiting
for processing we won't trigger another update (kick or irq,
depending on the direction).
The optimization reduces irq injection by a 7%.

Along the way, collapse vring::need_event to the code and use
std::atomic(s) to access any variable which is shared w/ the host

same as we can tell the host to disable interrupts via the _avail ring,
the host can tell us to supress notification via the _used ring.
every notificaion, or kick consumes about 10ns as it is implemented as
writing to an io port, which travels to usespace qemu in the host.

this simple patch, increase netperf's throughput by 600%, from a
300mbps to 1800mbps.

Indirect is good for very large SG list but isn't required
in case there is enough place on the ring or the SG list is tiny.
For the time being there is barely use of it so I set it off
by default

The feature allows the hypervisor to batch several packets together
as one large SG list. Once such header is received, the guest rx
routine interates over the list and assembles a mega mbuf.

The patch also simplifies the rx path by using a single buffer for
the virtio data and its header. This shrinks the sg list from size of
two into a single one.

The issue is that at the moment I haven't seen packets w/ mbuf > 1
being received. Linux guest does receives such packets here and there.
It may be due to the use of offload features that enalrge the packet size