Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Now that total is long casting to float may cause precision loss.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This is useful to notify us that something goes wrong.

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

bytes_written can overflow while testing on a fast device.

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch fixes two places where tst-mmap.cc doesn't munmap() everything
it mmap()ed. This only makes a difference if doing leak detection on
tst-mmap.cc to see if our page-table handling code leaked memory.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

We use sched::thread::attr to pass parameters to sched::thread creation,
i.e., create a thread with non-default stack parameters, pinned to a
particular CPU, or a detached thread.

Previously we had constructors taking many combinations of stack size
(integer), pinned cpu (cpu*) and detached (boolean), and doing "the
right thing". However, this makes the code hard to read (what does
attr(4096) specify?) and the constructors hard to expand with new
parameters.

Replace the attr() constructors with the so-called "named parameter"
idiom: attr now only has a null constructor attr(), and one modifies
it with calls to pin(cpu*), detach(), or stack(size).

For example,
    attr()                                  // default attributes
    attr().pin(sched::cpus[0])              // pin to cpu 0
    attr().stack(4096).pin(sched::cpus[0])  // pin and non-default stack
    and so on.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

I saw this Abort:

35.159 Mb/s
50.230 Mb/s
46.648 Mb/s
68.850 Mb/s
Wrote 613.418 MB in 10.00 s
Aborted

The backtrace says:

  (gdb) bt
  #0  0x000000000035bb82 in halt_no_interrupts () at
  /home/asias/src/cloudius-systems/osv/arch/x64/processor.hh:241
  #1  osv::halt () at
  /home/asias/src/cloudius-systems/osv/core/power.cc:28
  #2  0x0000000000218142 in abort (msg=msg@entry=0x55197f "Aborted\n") at
  /home/asias/src/cloudius-systems/osv/runtime.cc:89
  #3  0x000000000021816e in abort () at
  /home/asias/src/cloudius-systems/osv/runtime.cc:79
  #4  0x000000000039eaa2 in osv::generate_signal (siginfo=...,
  ef=0xffffc0003eb56008) at
  /home/asias/src/cloudius-systems/osv/libc/signal.cc:58
  #5  0x000000000039eb0c in osv::handle_segmentation_fault
  (addr=<optimized out>, ef=<optimized out>) at
  /home/asias/src/cloudius-systems/osv/libc/signal.cc:73
  #6  0x000000000030b45c in mmu::vm_sigsegv
  (addr=addr@entry=17592186060800, ef=ef@entry=0xffffc0003eb56008) at
  /home/asias/src/cloudius-systems/osv/core/mmu.cc:763
  #7  0x000000000030b54b in mmu::vm_fault (addr=<optimized out>,
  addr@entry=17592186061840, ef=ef@entry=0xffffc0003eb56008)
      at /home/asias/src/cloudius-systems/osv/core/mmu.cc:773
  #8  0x000000000032bff5 in page_fault (ef=0xffffc0003eb56008) at
  /home/asias/src/cloudius-systems/osv/arch/x64/mmu.cc:35
  #9  <signal handler called>
  #10 0x0000100000004410 in ?? ()
  #11 0x000000000031e5fd in virtio::blk::req_done
  (this=0xffffc0003eddb800) at
  /home/asias/src/cloudius-systems/osv/drivers/virtio-blk.

Wait until all the bio are done to fix this use after free.

This patch also make the test to measure completed writes instead of
submitted writes.

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This test includes a large number of tests for the timerfd_*() functions
and many of their bizarre use-cases and corner cases.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Printing asynchronously is not affected by write() stalls and hence the period
for partial amount can be more or less constant.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Fix some compilation errors that would arrise when NDEBUG is defined:
 - tests/misc-tcp.cc: missing #include<iostream> that would come from
   include/boost/assert.hpp: line 81 when NDEBUG is not defined.
 - bsd/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_dataset.c:
   when NDEBUG is not defined assert() is defined to __assert_fail,
   which has "noreturn" attribute, which satisfies the compiler.
   When NDEBUG is defined and assert() completly compiles out, the
   compiler start complaining about the missing "return" statment.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Rename blacklisted tests, from tst-wake.cc et al. to misc-wake.cc.

The different name will cause these tests not to be automatically run
by "make check" - without needing the separate blacklist in test.py
(which this patch deletes).
After this patch, testrunner.so will also only run tests called tst-*,
so will not run the misc-* tests.

The misc-* tests can still be run manually, e.g.,
  run.py -e tests/misc-mutex.so

In addition to the previously blacklisted tests, this patch "blacklists"
(renames) a few additional tests which fail quickly, but test.py didn't
know because they didn't use the word "fail". An example is tst-schedule.so,
which existed immediately when not run on 1 vcpu. So this patch also
renames it to misc-schedule.so, so "make check" or testrunner.so won't
run this test.

Note that after this patch, testrunner.so is a new way to run all tests,
but it isn't working well yet because it still exposes new bugs that do not
exist in the separate tests (depending on your view point, this might be
considered a feature, not a bug, in testrunner.so...).

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Using hard-coded path names is problematic because other test cases may
use the same path names and forget to clean up after them.

Make tst-fs-link.so more robust by using mktemp() to generate unique
path names.

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The last part of the standard thread tests created 4 threads and calls the
detach of one from the body of the other. They live in the same block to
guarantee that they will all be destroyed more or less at the same time (we
expect). Avi, however, demonstrated that a mistake prevents that from being
the actual case:

    t1 starts to run
    t2 starts to run
    t3 starts to run
    t4 starts to run
    t4 is detached
    t4 is destroyed (ok)
    t3 is destroyed. wasn't detached or join, to terminate
    t1, t2, t3 are detached, but too late

This introduces a simple wait mechanism to avoid having the threads
terminated after the block is gone.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

        sched::thread *t2 = nullptr;
        sched::thread *t1 = new sched::thread([&]{
            // wait for the t2 object to exist (not necessarily run)
            sched::thread::wait_until([&] { return t2 != nullptr; });
            if (quick) {
                return;
            }
            sched::thread::sleep_until(nanotime() + 10_ms);
        }, sched::thread::attr(sched::cpus[0]));

        t2 = new sched::thread([&]{
            t1->wake();
        }, sched::thread::attr(sched::cpus[1]));

        t1->start();
        t2->start();
        delete t1
        delete t2;

t1 may start, complete, and be destroyed before t2 gets a chance to run.  In
this case the call to t1->wake() will access deallocated memory.

Fix by making sure t1 is only destroyed after t2 completes.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

This patch fixes the error codes in four error cases:

1. unlink() of a directory used to return EPERM (as in Posix), and now
   returns EISDIR (as in Linux).

2. rmdir() of a non-empty directory used to return EEXIST (as in Posix)
   and now returns ENOTEMPTY (as in Linux).

3. rmdir() of a regular file (non-directory) used to return EBADF
   and now returns ENOTDIR (as in Linux).

4. readdir() of a regular file (non-directory) used to return EBADF
   and now returns ENOTDIR (as in Linux).

This patch also adds a test, tst-remove.cc, for the various unlink() and
rmdir() success and failure modes.

Fixes #123.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Our implementation of operate() will try to fill as much as the address space
as possible with huge pages. If that fails, we should be able to fill the range
with small pages instead of failing. This test should make sure that in such
scenarios, the resulting mapping looks sane.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This reverts commit bdd99c7b.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

tst-pipe.cc read a buffer after freeing it, which could have theoretically
caused segfaults (it didn't in practice, but better fix this oversight).

Also, it forgot to return a return code, so it doesn't play nicely in
a test framework like testrunner.so. I'm surprised the C++ compiler wasn't
bothered by an int main() not returning an int.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The remove() function is part of the ISO C 1989 standard, and used, for
example, to implement Java's File.delete(). It's supposed to remove a
file, regardless of whether unlink() or rmdir() is needed to remove it.

Our implementation (from Musl's) assumed that unlink() on a directory fails
with EISDIR, and only on that case it tried rmdir(). However, returning
EISDIR on unlink() is a Linux extension, which (deliberately) goes against
the Posix standard - which specified EPERM should be returned in that case.
Our ZFS implementation of unlink, following Solaris and FreeBSD (and not
Linux), returns EPERM in that case.

This meant that remove() used to fail deleting empty directories, and
Java code (like the SpecJVM2008 "derby" benchmark) using it to recursively
delete a directory, left behind undeleted empty directories.

So this patch fixes remove() to try rmdir() if unlink() returned either
the Linux-specific EISDIR, or the Posix-standard EPERM. It also adds
to the readdir test another test which verifies that remove() can delete
all files in a directory - both regular files and empty directories.

Fixes #112.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Before this patch, tst-readdir.cc assert()ed on every test, meaning that
a failure will cause a crash. Change this to use a report() function,
which counts failures instead of immediately crashing on the first one.

This patch doesn't change anything in what this test actually tests for.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

That test goes together with thread detach, but I am also calling joins
to make sure we're not breaking them. It is unfortunate that this is quite
non-deterministic and we can't really surely test for failure. But on the
flip side, it did help me catch a couple of bugs in my implementation. So
it will eventually explode somewhere if a bug appears.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Since we are using ZFS now, we don't need workarounds
for tmpfs bugs.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Revive testrunner.so (run with run.py -e testrunner.so), which runs all tests
in the tests/ directory in the image.

Changed it to C++, and for simplicity use the osv::run() API.

The next steps on the TODO list should be:
1. Change the manifest so that non-tests are NOT copied to the tests/ on the
   image.
2. Add an option to run tests in parallel, for maximum testing speed. Will not
   be trivial (will need to redirect the tests' output so they don't mix, and
   also figure out the load so we can run as many tests in parallel as we need
   to fill up the CPUs).

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

A ';' at the end of a parameter mark the end of a program's arguments list.

The goal of this patch is to be able to split mkfs.so in to parts mkfs.so and
cpiod.so.

The patch uses a full spirit parser to escape "" and split commands around ';'.

Signed-off-by: Benoit Canet <benoit@irqsave.net>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The main purpose is to exercise the new parameter checks added to
mmap to handle wrong combination of parameters.
Let's intentionally force errors and catch them with the respective errno codes.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

storage/runtime arguments for tracepoint can be inferred from
assign() function signature instead of specified explicitly
by storage_args/runtime_args. This makes boilerplate code smaller.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add a simple test case for "/dev/random" which just reads from the file
and prints out the random bytes.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add missing join() in tst-loadbalance, to avoid rare crashes during the
test.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch adds tst-scheduler.cc, containing a few tests for the fairness
of scheduling of several threads on one CPU (for scheduling issues involving
load-balancing across multiple CPUs, check out the existing tst-loadbalance).

The test is written in standard C++11, so it can be compiled and
run on both Linux and OSv, to compare their scheduler behaviors.
It is actually more a benchmark then a test (it doesn't "succeed" or "fail").

The test begins with several tests of the long-term fairness of the
schduler when threads of different or identical priorities are run for
10 seconds, and we look at how much work each thread got done in those
10 seconds. This test only works on OSv (which supports float priorities).

The second part of the test again tests long-term fairness of the scheduler
when all threads have the default priority (so this test is standard C++11):
We run a loop which takes (when run alone) 10 seconds, on 2 or 3
threads in parallel. We expect to see that all 2 or 3 threads
finish at (more-or-less) exactly the same time - after 20 or 30
seconds. Both OSv and Linux pass this test with flying colors.

The third part of the test runs two different threads concurrently:
 1. One thread wants to use all available CPU to loop for 10 seconds.
 2. The second thread wants to loop in an amount that takes N
    milliseconds, and then sleep for N milliseconds, and so on,
    until completing the same number of loop iterations that (when run
    alone) takes 10 seconds.

The "fair" behavior of the this test is that both threads get equal
CPU time and finish together: Thread 2 runs for N milliseconds, then
while it is sleeping for N more, Thread 1 gets to run.
This measure this for N=1 through 32ms. In OSv's new scheduler, indeed both
threads get an almost fair share (with N=32ms, one thread finishes in 19
seconds, the second in 21.4 seconds; we don't expect total fairness because
of the runtime decay).

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Allows longer tests to be run.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>