That test goes together with thread detach, but I am also calling joins
to make sure we're not breaking them. It is unfortunate that this is quite
non-deterministic and we can't really surely test for failure. But on the
flip side, it did help me catch a couple of bugs in my implementation. So
it will eventually explode somewhere if a bug appears.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

My thread patch broke info threads. My bad: Nadav noticed in his review
that it would, but I ended up forgetting about when I reworked it. In
any case, with the following fix it works again.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

1. Use multithreaded build when possible
2. Suppress output on bucket removal

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

1. It turned out that Amazon changes standard AMIs from time to time,
   AMI ID changes as well in this case. As a result Windows AMI we
   used as a template for OSv releases became anavailable.
   The script will use earlier OSv release AMI instead to avoid
   this class of problems.

2. Invalid strings compare syntax fixed.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently, clicking on the "monitor" link of OSv web console, you get a
nasty crash.  Switch to 2 GB to make things work out of the box.

Fixes #63.

Cc: Tzach Livyatan <tzach@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This is a temporary measure before the module definition is moved to
the mgmt repo.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Manifests should contain paths in the same file system in which the
image is built. Currently manifests assumed that the module will be
downloaded/copied into build/*/module/ and that the CWD is inside
build/*. This is not always the case now that we have "direct-dir"
module type, which is is not copied.

Example use:

    /usr/x:${MODULE_DIR}/x

This patch adds MODULE_DIR variable, which is replaced with the path
to the module in local file system. Depending on the module type this
can be either direct path or a path inside build/*/module/.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

    == Description of the problem ==

Currently modules can only specify files which need to be
copied. There is a need for a module to also declare ways it can be
run, so that we can automatically prepare a runnable image. It
should be easy to switch between run configurations.

Currently it is enough for image configuration to happen at the
time of image make process. In future this should be allowed
on the already built image.

We also need to be able to combine multiple modules in one
image. For example in addition to the main application one might want
to start various services like CLI, management API server, etc.

Java apps should be able to specify JVM arguments, which must be
passed to the JVM upon its creation, as well as regular run-java
arguments (classpath, main classes, main arguments, etc.)

    == Solution ==

This is not intended to be a permanent solution. The aim is to solve
immediate need to have a fully modularized build in a scalable way.

Every module has a new kind of file in its root directory which
holds its configuration. The file is named 'module.py' and is
a python script which uses osv's api for declaring run configurations.

Using python as config language has several advantages:
 - more expresiveness, unlike json it allows for expression reuse
 - it's easier to extend the config language
 - we don't need as much parsing, gluing, error checking, error
   reporting code because we have it already

There are currently two kinds of applications which can be declared:

   run(cmdline)  <- basic .so application
   run_java(jvm_args=[], classpath=[], args=[])  <- java applications

Run configurations can be declared as simple module attributes
which can be referenced from the image configuration file.

Image configuration

There is a new configuration file kind, which defines which modules
and which run configurations should be included in the image. Files
are located using path: ${OSV_BASE}/images/$(image-name).py

Syntax:

  require(module) <-- declares that module should be included in the image
  		      and returns an object which allows to access module's
		      attributes.

  run = []  <-- list of run configurations

Example:

  _mgmt = require('mgmt')
  run = [ _mgmt.shell ]

To use a particular image configuration run make like this:

  make image=fancy-tomcat

The default configuration is named 'default'.

This patch extracts mgmt into a module, which is embedded under
${OSV_BASE}/modules/mgmt

The purpose of ${OSV_BASE}/config.json has been changed. It does not
list modules which should be included anymore, image config file does
that. It's a module look-up configuration which tells the build where
to look for modules.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This will allow the users to populate /usr with simples binaries like haproxy
and their config at runtime.
It's intented to be combined with the loader ability to run multiple binaries
in a sequential way.
For example -e '/tools/cpiod.so; /usr/haproxy.so -f /usr/haproxy.conf'

Signed-off-by: Benoit Canet <benoit@irqsave.net>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This pave the way for the creation of bare images that the user can populate
using cpiod when he need it.

Signed-off-by: Benoit Canet <benoit@irqsave.net>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This is a temporary fix. I really feel we need failure detection based
on exit code.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add error scanning to test.py based on known patterns in stdout and
stderr and stop test execution on error to make broken test cases stand
out.

Example output looks as follows:

  PASS: link returns ENOENT if source path does not exists
  PASS: create a file
  PASS: close the file
  PASS: create a hard link
  PASS: link returns EEXIST if destination path exists
  PASS: read content from another link
  PASS: stat the file
  PASS: hard link count is increased
  PASS: stat the hard link
  PASS: stat device IDs match
  PASS: stat inode numbers match
  PASS: remove the hard link
  PASS: stat the file
  FAIL: hard link count is decreased
  PASS: remove the file
  SUMMARY: 15 tests, 1 failures
  Powering off.

  Test '/tests/tst-fs-link.so' FAILED
  make: *** [check] Error 1

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This adds the virtio-rng driver to OSv.  The implementation is simple:

  - Start a thread that keeps 64 byte of entropy cached in internal
    buffer.  Entropy is gathered from the host with virtio-rng.

  - Create device nodes for "/dev/random" and "/dev/urandom" that both
    use the same virtio_rng_read() hook.

  - Use the entropy buffer for virtio_rng_read().  If we exhaust the
    buffer, wake up the thread and wait for more entropy to appear.

We eventually should move device node creation to separate
drivers/random.c that multiplexes between different hardware RNG
implementations.  However, as we only support virtio-rng, I'm leaving
that to whomever implements support for the next RNG.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Shell command line should be passed as one argument. Additional
arguments are passed to the shell itself.

Reported-by: Nadav

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

tput is more portable than raw colors specification

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

There is a bug in AWS engine making copied public AMIs invisible.
This patch implements a work-around for this problem as described at
https://forums.aws.amazon.com/thread.jspa?messageID=454676&#454676



Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch changes module.py so that it generates both manifests
at once. We do not need to have it split. Doing it in one step
makes we also need to resolve modules once.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This is a refactoring before further modifications. Should
not have any semantic changes.

Items:
 - file.readlines() is not necessary, file is line-iterable
 - use "cwd" named parameter in subprocess.call() instead of explicit "cd"
 - auto close using scope manager: "with xxx as yyy:"
 - os.path.exists() instead of os.access(mmod_path, os.F_OK) == False
 - extract paths containing ".." to variables

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Dumps virtio drivers and the state of their queues.

Sample output:

(gdb) osv info virtio
virtio::virtio_net at 0xffffc0003ff0ec00
  queue 0 at 0xffffc0003ff2ba00
    avail g=0xbe09 h=0xbe09 (0)
    used   h=0xbe09 g=0xbd11 (248)
    used notifications: enabled
  queue 1 at 0xffffc0003ff2bc00
    avail g=0xc951 h=0xc951 (0)
    used   h=0xc951 g=0xc8fd (84)
    used notifications: enabled
  queue 2 at 0xffffc0003ff2bd00
    avail g=0x0 h=0x0 (0)
    used   h=0x0 g=0x0 (0)
    used notifications: enabled
virtio::virtio_blk at 0xffffc0003fefd400
  queue 0 at 0xffffc0003fee5100
    avail g=0x15f h=0x15f (0)
    used   h=0x15f g=0x15f (0)
    used notifications: enabled

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Fix the printout of runtime by "osv info threads" and "osv runqueue"
debugger commands, to fit the new scheduler (different variable name,
and it is float, not integer).

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This feature will be used to release images
with preinstalled applications.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Duration analysis is based on trace pairs which follow the convention
in which function entry generates trace named X and ends with either
trace X_ret or X_err. Traces which do not have an accompanying return
tracepoint are ignored.

New commands:

  osv trace summary

      Prints execution time statistics for traces

  osv trace duration {function}

      Prints timed traces sorted by duration in descending order.
      Optionally narrowed down to a specified function

gdb$ osv trace summary
Execution times [ms]:
name          count      min      50%      90%      99%    99.9%      max    total
vfs_pwritev       3    0.682    1.042    1.078    1.078    1.078    1.078    2.801
vfs_pwrite       32    0.006    1.986    3.313    6.816    6.816    6.816   53.007

gdb$ osv trace duration
0xffffc000671f0010  1    1385318632.103374   6.816 vfs_pwrite
0xffffc0003bbef010  0    1385318637.929424   3.923 vfs_pwrite

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The iteration logic was duplicated in two places. The patches yet to
come would add yet another place, so let's refactor first.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch adds the unsafe-cache option to run.py and changes mkzfs.py
to always call run.py with this option enabled.
Thus, we're doing this change just for the build run (Suggested by Nadav Har'El).
The main goal is to boost the time it takes to complete the entire process.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Some older version of qemu-nbd cuases error exit with nbd_client.py.
(Look at: https://groups.google.com/d/msg/osv-dev/EW5BtNFNfzs/I33BeFXg2f0J

)
This is because nbd_client.py is sending FLUSH command unconditionally, but it's extended feature, nbd client should check nbd server has the capability to accept FLUSH.
nbd server sends capability flags on negotiation stage, it sends HAS_FLAGS(0x1) and SEND_FLUSH(0x4) when server supports FLUSH.

This patch adds these capability check, and skips to send FLUSH if server doesn't support it.

Signed-off-by: Takuya ASADA <syuu@dokukino.com>
Reviewed-by: Benoît Canet <benoit.canet@irqsave.net>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Show running test case name.  Makes debugging test failures less
painful...

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Use the new pivot_root() functionality to switch to ZFS root filesystem
once OSv is up and running.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

I committed the file accidentally.  It's a temporary file that shouldn't
be there.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The idea of the patch is basically described in prevoius post:

https://groups.google.com/d/msg/osv-dev/RL2S3AL9TNE/l4XZJo3-lI0J

Whis this patch, you will be able to install OSv apps into disk image on
"make all" stage.

These apps does not require to exist in OSv repository, you can install
apps which is on any git repository or svn repository, or on local
directory.

You'll need to write a config file to add apps, format of the file is
JSON.

Here's a sample of the file:
{
   "modules":[
      {
	 "name":"osv-mruby",
         "type":"git",
         "path":"https://github.com/syuu1228/osv-mruby.git",
         "branch":"master"
      }
   ]
}

If you add "module" on config file, make all calls script/module.py.

This scripts perform "git clone" to fetch repository to $(out)/module,
and invoke "make module" on each module.

"make module" should outputs bootfs.manifest/usr.manifest on module
directory, the script merge bootfs.manifest.skel/usr.manifest.skel and
module local manifests to single file
$(out)/bootfs.manifest/$(out)/usr.manifest.

Here's app Makefile example:

  https://github.com/syuu1228/osv-mruby/blob/master/Makefile



It have "module" target, and the target builds all binaries and
generates *.manifest.

Signed-off-by: Takuya ASADA <syuu@dokukino.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It simply tells QEMU not to start OSv at startup.

To continue the execution, it's possible to use either the QEMU monitor
or a GDB remote connection.

Signed-off-by: Raphael S. Carvalho <raphael.scarv@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

EC2 does not support QCOW2. Force build to create RAW image.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Xen requires explicit image format specification.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>