It's a good practice to propagate failures so that make does not
succeed if something goes wrong.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

To make it work self._closed needs to be initialized early because
even though exception is thrown from __init__(), __exit__() still will
be called and that one expects self._closed to be defined.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Found during code inspection.

Qemu process is started with stdout=PIPE but the pipe will not be read
until _process.communicate() is called and it is called when nbd_file
is closed. So if the pipe buffer fills up we may have a deadlock. Qemu
will be blocked on stdout.write() and imgedit.py will be waiting for
it to open a socket.

Solution here is to redirect to /dev/null as this has the same
effect as current code.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This is not a documented behavior but glibc's dlopen behaves this way.

Spotted when debugging assertion failure when java is started with a
debugger, we try to read elf header from "/" on dlopen("", ...):

Assertion failed: data.uio_resid == 0 (osv/fs/fs.cc: read: 27)

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Useful if you want to know who created that large pile of threads.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

gcc -O3 complains about uninitialized use of 'nims'. In fact,
inm_get_source() can return an error without setting nims which will
be later read from RB_FOREACH_REVERSE_FROM macro. It looks like 'nims'
is intended to hold the last value for which inm_get_source() returned
success. The uninitialized access would happen if this function never
succeded. I am not sure if this is possible in practice, but let's
initialize nims to NULL, which will cause no iteration in
RB_FOREACH_REVERSE_FROM macro.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Gcc comaplins about attempt to read 'size' via dereferencing a pointer
in xdr_u_int() in case xdrs->x_op == XDR_ENCODE.  However, in this
case the size will be set from the switch case inside xdr_string()
before xdr_u_int() is invoked. I think it's spurious because the
code clearly assumes that xdrs->x_op cannot change between these two
execution points.

Let's initialize 'size' to 0 to make gcc happy.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently map_file() do three passes over vma memory in a worst case.
First it maps memory with write permission while zeroing it, then it
reads a file into memory and, if vma is read only, it does one more
pass to fix memory permissions. Fix it by providing new specialization
of fill_page class which builds iovec of all allocated memory and
reads from a file using the iovec at the end of populate stage.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Commit 2ebfd915 ("build.mk: Generate a manifest of tests automatically")
separated native tests from Java tests but unfortunately broke the build
because it forgot to add dependency to the new java_tests variable.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

In issue #145 I reported a crash during boot in start_early_threads().
I wasn't actually able to replicate this bug on master, but it happens
quite frequently (e.g., on virtually every "make check" run) with some
patches of mine that seem unrelated to this bug.

The problem is that start_early_threads() (added in 63216e85)
iterates on the threads in the thread list, and uses
t->remote_thread_local_var() for each thread. This can only work if
the thread has its TLS initialized, but unfortunately in thread's
constructor we first added the new thread to the list, and only later
called setup_tcb() (which allocates and initializes the TLS). If we're
unlucky, start_early_threads() can find a thread on the list which still
doesn't have its TLS allocated, so remote_thread_local_var() will crash.

The simple fix is to switch the order of the construction: First
set up the new thread's TLS, and only then add it to the list of
threads.

Fixes #145.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Generate a manifest of tests automatically from the makefile
based on the value of $(tests) and $(java_tests) variables.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently we just call poweroff() which does not umount
file systems which leads to data loss on next boot.

This happens for example if you start the CLI and type exit. On next
boot you will get:

 TX_CREATE
 TX_CREATE
 Solaris: WARNING: ZFS replay transaction error 95, dataset osv/zfs,
 seq 0x1, txtype 2

Reported-by: Pekka Enberg <penberg@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

In order to reuse the logic it needs to be extracted.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

When compiling with -O3 gcc detects that 'th' may be uninitialized.
This is true, it happens when prio_find_thread is given
which == PRIO_USER or which == PRIO_PGRP.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

MultiJarLoader reads the command line parameters from a file
and uses split to make a list of params out of them before
calling parseArgs.

Currently the regular expression is \\s which means that
consecutive space will be split it to an empty string.
It should be \\s+ so consecutive spaces will be ignored.

Found by code inspection.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>

Instead of the old C-style file-operation function types and fo_*()
functions, since recently we have methods of the "file" class. All our
filesystem code is now C++, and can use these methods directly.

So this patch drops the old types and functions, and uses the class methods
instead.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

The code has bitrotted, and it doesn't support wait morphing.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Instead of free standing functions, use member functions, which are easier
to work with.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Prevent memset() overwriting an initialized mutex.  Instead initialize
members individually.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

the _ prefix helps to distinguish between members and non-members; helps with
the next patch.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Make the code more maintainable by removing the #ifdefs; it doesn't make
sense to disable wait morphing.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Drop the socket uma zone and replace it with calls to new and delete.  This
allows the constructor and destructor to be called, so we can add C++
objects to the socket structure.

Take care to use the default-initializing form of the constructor since
the socket code requires zero initialization.

We lose the ability to limit the socket count, so we'll have to re-add it in
the future if we want it.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Each implementation of "struct file" needs to implement 8 different file
operations. Most special file implementations, such as pipe, socketpair,
epoll and timerfd, don't support many of these operations. We had in
unsupported.h functions that can be reused for the unsupported operation,
but this resulted in a lot of ugly boiler-plate code.

Instead, this patch switches to a cleaner, more C++-like, method:
It defines a new "file" subclass, called "special_file", which implements
all file operations except close(), with a default implementation identical
to the old unsupported.h implementations.

The files of pipe(), socketpair(), timerfd() and epoll_create() now inherit
from special_file, and only override the file operations they really want
to implement.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Right now most of mmap related functions have the same bug related to vma
locking: they validate mapping under vma lock then release the lock
and do actual vma operation, but since between validation and operation
the mapping can go away it is incorrect. This patch fixes this by doing
validation and operation under the same lock.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Currently offset calculation is incorrect. Fix it by tracking base
address of a region and calculating an offset by subtracting base
address from current mapping address.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

map_range() is a entry point into a page mapper, so make it
impossible to instantiate map_level class directly by making all of its
function and constructor private and declaring map_range() as a friend.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

One for initial call, another for recursion.

This gets rid of default parameters, the need for std::integral_constant
and makes code much more readable.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Bring back haproxy.py that I deleted by accident in commit 3e48ad65
("modules: support "include"ing a module list").

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Instead of listing modules from the apps/ repository in OSv's config.json,
this patch adds support for an "include" specification in config.json,
which we use to include apps/modules.json.

This allows adding modules to the apps/ repository without needing to
patch also the main OSv repository.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Example output looks as follows:

(gdb) osv zfs
:: ZFS TUNABLES ::
    zil_replay_disable=0
    zfs_nocacheflush=0
    zfs_prefetch_disable=0
:: ARC SIZES ::
    Actual ARC Size: 16742872
    Target size of ARC: 16777216
    Min Target size of ARC: 16777216
    Max Target size of ARC: 16777216
        Most Recently Used (MRU) size: 1222656 (7.29%)
        Most Frequently Used (MFU) size: 15554560 (92.71%)
:: ARC EFFICIENCY ::
Total ARC accesses: 47477
    ARC hits: 44668 (94.08%)
        ARC MRU hits: 18737 (41.95%)
            Ghost Hits: 183
        ARC MFU hits: 25733 (57.61%)
            Ghost Hits: 1084
    ARC misses: 2809 (5.92%)
Total Hash elements: 341
    Max Hash elements: 449
    Hash collisions: 98
    Hash chains: 13

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Useful for tracing virtio queue utilization issues.

To get plottable data:
 1. Save traces to gdb.txt
 2. grep virtio_add_buf gdb.txt | grep "queue=0" \
     | awk '{print $3 " " $6}' | sed -r 's/avail=(.*)/\1/'

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Clean up virtio-vring.hh and move add_buf_wait() out of line. Benefits:

  - Plays well with tracepoints

  - Less recompilation when code is changed

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Most likely introduced in eb48b150

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

See apps/haproxy/README for howto run haproxy.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This was the cause of poor ZFS performance in misc-fs-stress test.

Before:

 Wrote 168.129 MB in 10.12 s = 16.610 Mb/s
 Wrote 194.688 MB in 10.00 s = 19.469 Mb/s
 Wrote 183.004 MB in 10.06 s = 18.186 Mb/s
 Wrote 167.754 MB in 10.28 s = 16.315 Mb/s

After:

 Wrote 636.227 MB in 10.00 s = 63.623 Mb/s
 Wrote 666.979 MB in 10.00 s = 66.696 Mb/s
 Wrote 613.512 MB in 10.00 s = 61.350 Mb/s
 Wrote 573.502 MB in 10.00 s = 57.346 Mb/s
 Wrote 668.607 MB in 10.00 s = 66.857 Mb/s
 Wrote 630.920 MB in 10.00 s = 63.087 Mb/s

It turned out that the limiting factor was the ARC cache. A check
inside arc_tempreserve_space() was forcing txg to be synced too often
(once every 400ms). The arc_c variable was only 16M (arc_c_min) which
allowed to write only 8M per transaction. It turns out that arc_c
depends on kmem_size() which is based on physmem which was never
initialized.

I would hold with commiting this yet because of several reasons,
which I want to put under your consideration.

While this improves write throughput it makes the boot time after make
much longer, on my disk the boot time is increased from 1.5s to 10s.
This is because zfs verifies the last 3 txgs upon mount. This patch
increases txg size, which results in more data to check in the next
boot. I'm working on solving this right now.

Something worth noting is that while larger transactions sync less
often incresing throughput they also sync longer increasing worst case
latency. In my test the pauses get as high as 3 seconds with 1G of
guest memory.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The current initialization is to something like 5 bit words, which truncates
everything to control characters on VMware.  QEMU somehow ignores the word
size.

Fix to initialize the serial port to 8N1, yielding useful output.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

mprotect(PROT_WRITE) on a file opened as read only should fail,
but current mprotect() implementation is missing the check. The patch
implements it.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>