We are heavily using this function to grab the address of the current thread.
That means a function call will be issued every time that is done, where a
simple mov instruction would do.

For objects outside the main ELF, we don't want that to be inlined, since that
would mean the resolution would have to go through an expensive __tls_get_addr.
So what we do is that we don't present the symbol as inline for them, and make
sure the symbol is always generated.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It is sometimes useful to programatically learn if something will end up
in the main ELF or in a shared library. This is the function usually
performed by _KERNEL. However, using that proved quite difficult when
Asias was trying to conditionally compile some zfs tools.  This is
because a lot of our headers expect _KERNEL to be defined and they are
also included in the tools build. Too messy, unfortunately.

Because of that, I am defining a new constant that will be available for
every object that will end up in the main ELF, but won't be defined
otherwise.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Remove the unnecessary default vnc :1 argument during make.
It also does not allow make to proceed if another vm is already using :1

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Jaspal Singh Dhillon <jaspal.iiith@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

$ ./scripts/run.py -e 'zpool.so list'
OSv v0.08-81-gc3422a7
eth0: 192.168.122.15
NAME   SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
osv   9.94G   148M  9.79G     1%  1.00x  ONLINE  -

$ ./scripts/run.py -e 'zfs.so list'
OSv v0.08-81-gc3422a7
eth0: 192.168.122.15
NAME      USED  AVAIL  REFER  MOUNTPOINT
osv       148M  9.64G    32K  /
osv/zfs   148M  9.64G   148M  /zfs

Enable it by compiling the missing functions.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: KANATSU Minoru <icc.pot.tyew272@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

add symbol for porting CRuby
this function is depend on execv()

Signed-off-by: KANATSU Minoru <icc.pot.tyew272@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

add symbol for porting CRuby

Signed-off-by: KANATSU Minoru <icc.pot.tyew272@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The version built on muninn does not have the leading
'tracepoint_base::log_backtrace(trace_record*, unsigned char*&)'
frame.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

They are important in some cases so we should not hide them. Lambda
functions are reported like that.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Our "Makefile" runs build.mk with "make -r", to disable all the default
rules. This flag is saved by make in the MAKEFLAGS variable, which is
passed on to child processes. Unfortunately, it is also passed to the
makefiles of individual modules (apps/*) when we build specific modules.
These makefiles are completely stand-alone, and are normally written to
assume that the default rules (e.g., .c->.o) do exist. They will fail when
run with MAKEFLAGS=r.

We recently noticed both redis-memonly and memcached to exhibit this problem
and committed workarounds (see 914bef6f75d9f3e7f268aecdb00a023b86439b85 and
e2f7ba1b1d80bb5e89d7fca71b4994c7de15ed4e in apps.git).

However, the right fix is for our build.mk to acknowledge that the modules'
makefiles are stand-alone, and must not inherit our prefered flags like
"-r". So this patch clears the MAKEFLAGS variable when calling module.py
(module.py will later invoke "make module" in the module's directory).

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

RX descriptor length is 14 bit field, we overflow it
by allocating 16K buffers and device treats RX descriptor
buffer length as zero.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Packet discard logic on RX path is incorrect because
it doesn't take into account that incoming packet may be spread
over a few rings.

At the same time discard logic is called in case of newbuf() failure
only and our newbuf() never fails since we have no memory allocation
errors, so discard logic may be safely dropped and this is an
excellent fix for the original problem.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Function vmxnet3::rxq_eof() may exit in the middle
of multi-descriptor RX packet processing. In this case
it is expected to resume processing of the same packet
on next invocation.

Fixed by making corresponding pointers vmxnet3_rxqueue class
members instead of stack variables.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

2nd ring of RX queue was filled with descriptors
and immediately cleared. This lead to inability of driver
to receive packets loger than 16K.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This involves the introduction of an xmit iterator and a worker thread.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

  This class is a heart of a per-CPU Tx framework.
  Except for a constructor it has two public methods:
   - xmit(buff): push the packet descriptor downstream either to the HW or into
     the per-CPU queue if there is a contention.

   - poll_until(cond): this is a main function of a worker thread that will
     consume packet descriptors from the per-CPU queue(s) and send them to
     the output iterator (which is responsible to ensure their successful
     sending to the HW channel).

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

  This class will represent a single per-CPU Tx queue.

  These queues will be subject to the merging by the nway_merger class in
  order to address the reordering issue. Therefore this class will
  implement the following methods/classes:
   - push(val)
   - empty()
   - front(), which will return the iterator that implements:
       - operator *() to access the underlying value
   - erase(it), which would pop the front element.

  If the producer fails to push a new element into the queue (the queue is
  full) then it may start "waiting for the queue": request to be woken when the
  queue is not full anymore (when the consumer frees some some entries from the
  queue):

   - push_new_waiter() method.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This class allows efficiently merge n sorted containers.
It allows both a single-call merging with a merge() method and
the iterator-like semantincs with a pop() method.
In both cases the merged stream/next element are streamed to
the output iterator.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

New interface allows to send a single doorbell to the host per
buffers bulk.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pawel Dziepak <pdziepak@quarnos.org>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch implements lockfree_queue (which is used as incoming_wakeup_queue)
so that it doesn't need exchange or compare_exchange operations.

The idea is to use a linked list but interleave actual objects stored in the
queue with helper object (lockless_queue_helper) which are just pointer to the
next element. Each object in the queue owns the helper that precedes it (and
they are dequeued together) while the last helper, which does not precede any
object is owned by the queue itself.

When a new object is enqueued it gains ownership of the last helper in the
queue in exchange of the helper it owned before which now becomes the new
tail of the list.

Unlike the original implementation this version of lockfree_queue really
requires that there is no more than one concurrent producer and no more than
one concurrent consumer.

The results oftests/misc-ctxs on my test machine are as follows (the values
are medians of five runs):

before:
colocated: 332 ns
apart: 590 ns

after:
colocated: 313 ns
apart: 558 ns

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pawel Dziepak <pdziepak@quarnos.org>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

and do it early (before the loop around init_array)

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

fix build for Aarch64 broken by commit 66317807 and b3fa77d3.

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Instead of searching for next data offset in a file before getting ARC
buffer this patch adds dmu_buf_hold_array_sparse() function which is variant
of dmu_buf_hold_array() but it returns an error if requested offset
belongs to a hole. This reduces amount of locks taken and avoids file
syncing that dmu_offset_next() does.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This introduces a simple timer-based sampling profiler which is
reusing our tracing infrastructure to collect samples.

To enable sampler from run.py run it like this:

 $ scripts/run.py ... --sampler [frequency]

Where 'frequency' is an optional parameter for overriding sampling
frequency. The default is 1000 (ticks per second). The bigger the
frequency the bigger sampling overhead is. Too low values will hurt
profile accuracy.

Ad-hoc sampler enabling is planned. The code already takes that into
account.

To see the profile you need to extract the trace:

 $ trace extract

And then show it like this:

 $ trace prof

All 'prof' options can be applied, for example you can group by CPU:

 $ trace prof -g cpu

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

First couple of frames always come from trace logging functions. I
think we should not show them because they do not add any information.

There was already a list of function names which should be filtered
out. This is not appropriate in this case because tracing goes through
a function named 'log' which is a pretty generic name and could as
well be an application symbol, so we cannot just filter out all
functions named 'log. This patch adds support for knocking off a
leading sequence of frames form the backtrace which is more accurate
and seems good enough.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Sampler will need to set and later restore value of this option.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>