Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

vec_sz in net::txq::try_xmit_one_locked(net_req*) should be initialized to 1 and not to 0
since there is always net_hdr element and vec_sz is incremented only for fragments not taking
into an account the header element.

This miscalculation caused vqueue->add_buf() return false while vqueue->avail_ring_has_room(vec_sz)
was returning true when the amount of elements in the avail ring was equal to vec_sz
since the actual size of the _sg_vec was "vec_sz + 1".

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The lock has been pushed down to a helper class and cannot be accessed here,
so remove the assert.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This involves the introduction of an xmit iterator and a worker thread.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The current tracepoint coverage does not handle all situations
well. In particular:

 * it does not cover link layer devices other than virtio-net. This
   change fixes that by tracing in more abstract layers.

 * it records incoming packets at enqueue time, whereas sometimes it's
   better to trace at handling time. This can be very useful when
   correlating TCP state changes with incoming packets. New tracepoint
   was introduced for that: net_packet_handling.

 * it does not record protocol of the buffer. For non-ethernet
   protocols we should set appropriate protocol type when
   reconstructing ethernet frame when dumping to PCAP.

We now have the following tracepoints:

 * net_packet_in - for incoming packets, enqueued or handled directly.

 * net_packet_out - for outgoing packets hitting link layer (not
   loopback).

 * net_packet_handling - for packets which have been queued and are
   now being handled.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

There was no way to sniff packets going through OSv's loopback
interface. I faced a need to debug in-guest TCP traffic. Packets are
logged using tracing infrastructure. Packet data is serialized as
sample data up to a limit, which is currently hardcoded to 128 bytes.

To enable capturing of packets just enable tracepoints named:
  - net_packet_loopback
  - net_packet_eth

Raw data can be seen in `trace list` output. Better presentation
methods will be added in the following patches.

This may also become useful when debugging network problems in the
cloud, as we have no ability to run tcpdump on the host there.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

When we discard a bad buffer, we don't advance to the next one, so we'll free
it again on the next pass, resulting in a double-free.

Tested-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Curently virtio uses MCLBYTES (2KB) buffers for virtio.  This hits a weakness
in our UMA and memory allocation implementations:

- UMA appends a 4-byte reference count to the allocation, resulting in an
  2052 byte allocation request
- Our allocator only services requests up to 2048 bytes from the mempool
  allocator; larger requests are serviced by the large allocator, which is
  significately slower, and is also serialized by a mutex.

Switch to an EXT_EXTREF mbuf with the buffer and refcount allocated
separately.  This also allows us to switch to 4K allocations serviced by
the page allocator (memory::alloc_page()), which is our fastest allocator,
and halves the lengths of the mbuf chains on streaming workloads.

Increases iperf stream (host->guest) results from ~12Gbps to ~34Gbps on my
desktop.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Zero req->mhdr.num_buffers in Tx path when MRG_RXBUF is negotiated as virtio spec demands.

Reviewed-by Zhi Yong Wu <zwu.kernel@gmail.com>
Reviewed-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

don't copy a virtio virtio_net_hdr to a local buffer in receiver().
This prevents CPU's optimizations.
Cast the mbuf's beginning to the virtio_net_hdr and use the relevant
fields where needed allowing a proper load prediction (as a result of
branch prediction).

Reviewed-by: Dor Laor <dor@cloudius-systems.com>
Reviewed-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

msi interrupt is not always implemented. Fallback to gsi interrupt when
msi interrupt is not available.

Signed-off-by: Asias He <asias@cloudius-systems.com>

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Remove unused #include of <drivers/clock.hh>.
Except the clock drivers and <osv/clock.hh>, no source file now now
include this header. Rather, <osv/clock.hh> should be used. Code including
<sched.hh> will also get <osv/clock.hh> automatically.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Several source files include <drivers/clockevent.hh>, though this is a
very low-level feature which they don't actually use.

sched.cc does use <drivers/clockevent.hh>, but already gets it through
sched.hh, so also doesn't need to include it explicitly.

This patch removes the unnecessary includes.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Indicate to the virtual hardware that our stack supports UDP fragmentation
offload.

This improves performance by a factor of about 3.3 (from ~20Gbps to ~66Gbps)
running the netperf UDP STREAM test.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

We reported the size of the last buffer in the packet, rather than the size
of the complete packet.

Fix to report the total size.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

VIRTIO_RING_F_INDIRECT_DESC belongs to the base features bits. No need
to specify it in net driver.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Reviewed-by: Dor Laor <dor@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

if_getinfo is shorter and more consistent all the other if_ functions
name, e.g., if_init, if_ioctl.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Rename it to txq_stats and rxq_stats which are more consistent with
struct txq and struct rxq.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

We are under virtio namespace, no need to repeat the virtio prefix.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

We are under the virtio namespace, it makes no sense to repeat the
virtio prefix again in the virito_net driver.

Change the naming from virtio::virtio_net to virtio::net.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It's better not to indent after the namespace specifier.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Added ifnet->if_get_if_info() method to collect the
internally gathered statistics into the standard if_data struct.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Reviewed-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Reviewed-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Introducing rxq and txq structs representing the Rx and
Tx queues data correspondingly.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Reviewed-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>