Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add additional error patterns to test.py:
1) For tests that are missing. Without it test.py was reporting the
   missing tests as PASSED.
2) For tests that fail with a critical errors like missing symbols.
   Without it the "make check" was hanging silently leaving u dazed
   and confused.
3) For tests that return non-zero status:
       a) Those that use exit().
       b) Those that user return xx.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

aio=native makes QEMU to use Linux native implementation of aio instead
of emulated thread pool which improves tst-bdev-write.so performance:

  blk, aio=threads
  Wrote 691.910 MB in 10.03 s
  Wrote 590.020 MB in 10.00 s
  Wrote 605.578 MB in 10.01 s
  Wrote 662.828 MB in 10.01 s
  Wrote 624.762 MB in 10.01 s

  blk, aio=native
  Wrote 789.566 MB in 10.00 s
  Wrote 744.691 MB in 10.00 s
  Wrote 537.125 MB in 10.00 s
  Wrote 732.230 MB in 10.00 s
  Wrote 683.383 MB in 10.00 s

  scsi, aio=threads
  Wrote 200.863 MB in 10.02 s
  Wrote 193.758 MB in 10.02 s
  Wrote 193.680 MB in 10.03 s
  Wrote 195.211 MB in 10.03 s
  Wrote 190.762 MB in 10.02 s

  scsi, aio=native
  Wrote 414.344 MB in 10.05 s
  Wrote 483.148 MB in 10.00 s
  Wrote 537.477 MB in 10.01 s
  Wrote 477.727 MB in 10.01 s
  Wrote 462.805 MB in 10.01 s

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

vma layout changed in commit bbec1a18.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Use --scsi or -S to boot from virtio-scsi instead of virtio-blk device.

This options is default to false.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This reverts commit 97626d36. As pointed
out Tomek, "-u" flag is already passed in mkzfs.py.

Since there's no user data on the image, we can safely cache it, improving
build performance.  Helpful duing 'git bisect'.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Commit 649654af ("Fix race between join() and thread completion") fixed
the problem so bring back the test case.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch adds sorting as an option (-s). Examples:

 Not sorted:

   gdb$ osv trace duration

 Sorted, narrowed down to one function:

   gdb$ osv trace duration -s vfs_pwritev

Not sorting allows us to start printing traces right away. There's
also no need to keep them in memory, which makes the command a bit
faster.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Make blacklisted tests visible during test.py run to attract attention
to them and hopefully get them fixed:

  TEST tst-strerror_r.so         OK  (1.083 s)
  TEST tst-threadcomplete.so     SKIPPED
  TEST tst-tracepoint.so         OK  (1.120 s)

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The test hangs at times and while there's a fix brewing, it's making
'make check' less useful.

Lets add it back when it works all the time.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Previously, the variables image_file, cmd_args and opt_path were used
globally along the run.py script, I did not consider this convenient and
thus, opted to convert these global variables, into one variable that is
passed as a parameter.

This variables was renamed to options, with the mindset that, if
desired, this variable could come from a configuration file, thus making
the passing of command line arguments optional. So, if this
functionality for a configuration file... Or perhaps default values, be
added to the script, there would not be much need for refactoring (or
renaming) as there would with these global variables.

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Juan Antonio Osorio Robles <jaosorior@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The '-nographic' option is deprecated and no longer imposes
the behavior which we want since this commit:

qemu.git 02c4bdf1d2ca8c02a9bae16398f260b5c08d08bf

We should use 'signal' option of the chardev instead, which
works with current qemu master as well as on 1.4.0

This patch renames '-g' option to '-s' as the former no longer
has adequate name.

Reported-by: Juan Antonio Osorio Robles <jaosorior@gmail.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

After commit dc40b49e, the place of
the _cpu and _status fields of a thread has moved (into a structure
pointed by the _detached_thread field). So fix loader.py to use
these new locations, so "osv info threads" will work again.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Rename blacklisted tests, from tst-wake.cc et al. to misc-wake.cc.

The different name will cause these tests not to be automatically run
by "make check" - without needing the separate blacklist in test.py
(which this patch deletes).
After this patch, testrunner.so will also only run tests called tst-*,
so will not run the misc-* tests.

The misc-* tests can still be run manually, e.g.,
  run.py -e tests/misc-mutex.so

In addition to the previously blacklisted tests, this patch "blacklists"
(renames) a few additional tests which fail quickly, but test.py didn't
know because they didn't use the word "fail". An example is tst-schedule.so,
which existed immediately when not run on 1 vcpu. So this patch also
renames it to misc-schedule.so, so "make check" or testrunner.so won't
run this test.

Note that after this patch, testrunner.so is a new way to run all tests,
but it isn't working well yet because it still exposes new bugs that do not
exist in the separate tests (depending on your view point, this might be
considered a feature, not a bug, in testrunner.so...).

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add a '--repeat' option to test.py that repeats the test suite until a
test fails.  This is useful for detecting test cases that fail some of
the time.

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Make the test runner output look pretty and show test duration to make
it visible which tests take the longest time to run.  The output looks
as follows now:

    TEST tst-af-local.so           OK  (3.288 s)
    TEST tst-bdev-write.so         OK  (1.058 s)
    TEST tst-bsd-evh.so            OK  (1.071 s)
    TEST tst-bsd-kthread.so        OK  (1.234 s)
    TEST tst-bsd-taskqueue.so      OK  (1.062 s)
    TEST tst-bsd-tcp1.so           OK  (2.114 s)
    TEST tst-commands.so           OK  (1.141 s)
    TEST tst-condvar.so            OK  (1.776 s)
    TEST tst-dns-resolver.so       OK  (2.560 s)
    TEST tst-epoll.so              OK  (1.952 s)
    TEST tst-except.so             OK  (1.146 s)
    TEST tst-fpu.so                OK  (2.630 s)
    TEST tst-fs-link.so            OK  (1.051 s)
    TEST tst-fs-stress.so          OK  (1.027 s)
    TEST tst-fsx.so                OK  (1.067 s)
    TEST tst-hub.so                OK  (6.256 s)
    TEST tst-huge.so               OK  (2.199 s)
    TEST tst-kill.so               OK  (4.147 s)
    TEST tst-libc-locking.so       OK  (2.110 s)
    TEST tst-loadbalance.so        OK  (1.070 s)
    TEST tst-mmap-file.so          OK  (1.080 s)
    TEST tst-mmap.so               OK  (1.087 s)
    TEST tst-pipe.so               OK  (7.306 s)
    TEST tst-preempt.so            OK  (1.119 s)
    TEST tst-pthread.so            OK  (1.100 s)
    TEST tst-queue-mpsc.so         OK  (3.748 s)
    TEST tst-ramdisk.so            OK  (1.078 s)
    TEST tst-readdir.so            OK  (1.094 s)
    TEST tst-remove.so             OK  (1.030 s)
    TEST tst-rename.so             OK  (1.157 s)
    TEST tst-resolve.so            OK  (1.095 s)
    TEST tst-scheduler.so          OK  (1.087 s)
    TEST tst-sleep.so              OK  (3.083 s)
    TEST tst-solaris-taskq.so      OK  (1.061 s)
    TEST tst-stat.so               OK  (1.106 s)
    TEST tst-strerror_r.so         OK  (1.102 s)
    TEST tst-tcp-sendonly.so       OK  (2.014 s)
    TEST tst-tcp.so                OK  (1.080 s)
    TEST tst-threadcomplete.so     OK  (2.770 s)
    TEST tst-tracepoint.so         OK  (1.109 s)
    TEST tst-truncate.so           OK  (1.083 s)
    TEST tst-utimes.so             OK  (1.079 s)
    TEST tst-vblk.so               OK  (1.310 s)
    TEST tst-vfs.so                OK  (1.118 s)
    TEST tst-yield.so              OK  (1.992 s)
    TEST tst-zfs-mount.so          OK  (1.087 s)
  OK (58 tests run, 82.944 s)

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Switch the whitelist to a blacklist to increase testing coverage.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Make test.py silent by default and only print out OSv log on error or if
the '--verbose' command line option is passed.

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

GDB python API does not handle inlined functions as nicely as regular
'backtrace' command does. Part of the frame attributes point to
the inlined function (variables, symtab) and part point to the caller.

For example frame.function() returns the nearest non-inlined function.

This breaks code which prints thread joining information.
The code thinks it's in "sched::thread::join()" when actually it's in
sched::schedule() context which does not have 'this' variable.

This solution skips inlined functions when considering print
candidates.  The printed information would be confusing anyway: file
and line number would be of the inlined function but printed function
name would belong to the caller. Finally we will reach the non-inlined
caller and print the call site properly.

Fixes #124.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

When all resolved frames are blacklisted we try to print the oldest
resolved frame.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Our implementation of operate() will try to fill as much as the address space
as possible with huge pages. If that fails, we should be able to fill the range
with small pages instead of failing. This test should make sure that in such
scenarios, the resulting mapping looks sane.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

That test goes together with thread detach, but I am also calling joins
to make sure we're not breaking them. It is unfortunate that this is quite
non-deterministic and we can't really surely test for failure. But on the
flip side, it did help me catch a couple of bugs in my implementation. So
it will eventually explode somewhere if a bug appears.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

My thread patch broke info threads. My bad: Nadav noticed in his review
that it would, but I ended up forgetting about when I reworked it. In
any case, with the following fix it works again.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

1. Use multithreaded build when possible
2. Suppress output on bucket removal

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

1. It turned out that Amazon changes standard AMIs from time to time,
   AMI ID changes as well in this case. As a result Windows AMI we
   used as a template for OSv releases became anavailable.
   The script will use earlier OSv release AMI instead to avoid
   this class of problems.

2. Invalid strings compare syntax fixed.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently, clicking on the "monitor" link of OSv web console, you get a
nasty crash.  Switch to 2 GB to make things work out of the box.

Fixes #63.

Cc: Tzach Livyatan <tzach@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This is a temporary measure before the module definition is moved to
the mgmt repo.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Manifests should contain paths in the same file system in which the
image is built. Currently manifests assumed that the module will be
downloaded/copied into build/*/module/ and that the CWD is inside
build/*. This is not always the case now that we have "direct-dir"
module type, which is is not copied.

Example use:

    /usr/x:${MODULE_DIR}/x

This patch adds MODULE_DIR variable, which is replaced with the path
to the module in local file system. Depending on the module type this
can be either direct path or a path inside build/*/module/.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

    == Description of the problem ==

Currently modules can only specify files which need to be
copied. There is a need for a module to also declare ways it can be
run, so that we can automatically prepare a runnable image. It
should be easy to switch between run configurations.

Currently it is enough for image configuration to happen at the
time of image make process. In future this should be allowed
on the already built image.

We also need to be able to combine multiple modules in one
image. For example in addition to the main application one might want
to start various services like CLI, management API server, etc.

Java apps should be able to specify JVM arguments, which must be
passed to the JVM upon its creation, as well as regular run-java
arguments (classpath, main classes, main arguments, etc.)

    == Solution ==

This is not intended to be a permanent solution. The aim is to solve
immediate need to have a fully modularized build in a scalable way.

Every module has a new kind of file in its root directory which
holds its configuration. The file is named 'module.py' and is
a python script which uses osv's api for declaring run configurations.

Using python as config language has several advantages:
 - more expresiveness, unlike json it allows for expression reuse
 - it's easier to extend the config language
 - we don't need as much parsing, gluing, error checking, error
   reporting code because we have it already

There are currently two kinds of applications which can be declared:

   run(cmdline)  <- basic .so application
   run_java(jvm_args=[], classpath=[], args=[])  <- java applications

Run configurations can be declared as simple module attributes
which can be referenced from the image configuration file.

Image configuration

There is a new configuration file kind, which defines which modules
and which run configurations should be included in the image. Files
are located using path: ${OSV_BASE}/images/$(image-name).py

Syntax:

  require(module) <-- declares that module should be included in the image
  		      and returns an object which allows to access module's
		      attributes.

  run = []  <-- list of run configurations

Example:

  _mgmt = require('mgmt')
  run = [ _mgmt.shell ]

To use a particular image configuration run make like this:

  make image=fancy-tomcat

The default configuration is named 'default'.

This patch extracts mgmt into a module, which is embedded under
${OSV_BASE}/modules/mgmt

The purpose of ${OSV_BASE}/config.json has been changed. It does not
list modules which should be included anymore, image config file does
that. It's a module look-up configuration which tells the build where
to look for modules.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This will allow the users to populate /usr with simples binaries like haproxy
and their config at runtime.
It's intented to be combined with the loader ability to run multiple binaries
in a sequential way.
For example -e '/tools/cpiod.so; /usr/haproxy.so -f /usr/haproxy.conf'

Signed-off-by: Benoit Canet <benoit@irqsave.net>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This pave the way for the creation of bare images that the user can populate
using cpiod when he need it.

Signed-off-by: Benoit Canet <benoit@irqsave.net>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This is a temporary fix. I really feel we need failure detection based
on exit code.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add error scanning to test.py based on known patterns in stdout and
stderr and stop test execution on error to make broken test cases stand
out.

Example output looks as follows:

  PASS: link returns ENOENT if source path does not exists
  PASS: create a file
  PASS: close the file
  PASS: create a hard link
  PASS: link returns EEXIST if destination path exists
  PASS: read content from another link
  PASS: stat the file
  PASS: hard link count is increased
  PASS: stat the hard link
  PASS: stat device IDs match
  PASS: stat inode numbers match
  PASS: remove the hard link
  PASS: stat the file
  FAIL: hard link count is decreased
  PASS: remove the file
  SUMMARY: 15 tests, 1 failures
  Powering off.

  Test '/tests/tst-fs-link.so' FAILED
  make: *** [check] Error 1

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This adds the virtio-rng driver to OSv.  The implementation is simple:

  - Start a thread that keeps 64 byte of entropy cached in internal
    buffer.  Entropy is gathered from the host with virtio-rng.

  - Create device nodes for "/dev/random" and "/dev/urandom" that both
    use the same virtio_rng_read() hook.

  - Use the entropy buffer for virtio_rng_read().  If we exhaust the
    buffer, wake up the thread and wait for more entropy to appear.

We eventually should move device node creation to separate
drivers/random.c that multiplexes between different hardware RNG
implementations.  However, as we only support virtio-rng, I'm leaving
that to whomever implements support for the next RNG.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Shell command line should be passed as one argument. Additional
arguments are passed to the shell itself.

Reported-by: Nadav

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

tput is more portable than raw colors specification

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

There is a bug in AWS engine making copied public AMIs invisible.
This patch implements a work-around for this problem as described at
https://forums.aws.amazon.com/thread.jspa?messageID=454676&#454676



Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch changes module.py so that it generates both manifests
at once. We do not need to have it split. Doing it in one step
makes we also need to resolve modules once.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>