This patch changes module.py so that it generates both manifests
at once. We do not need to have it split. Doing it in one step
makes we also need to resolve modules once.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

There is a race between "usr.manifest" and "bootfs.manifest" rules
which both call module.py. The script does complex stuff wrt module
preparation like fetching module files, calling make, etc. and
should not be run concurrently.

This change fixes the problem by moving the calls into one rule.

This is not the end of the story, more refactoring will follow.  The
module.py script should be split into parts, one that fetches modules
and one that generates manifests. This way the dependencies could be
made more fine grained and jobs paralellized.

This fixes issue #100.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch adds tst-scheduler.cc, containing a few tests for the fairness
of scheduling of several threads on one CPU (for scheduling issues involving
load-balancing across multiple CPUs, check out the existing tst-loadbalance).

The test is written in standard C++11, so it can be compiled and
run on both Linux and OSv, to compare their scheduler behaviors.
It is actually more a benchmark then a test (it doesn't "succeed" or "fail").

The test begins with several tests of the long-term fairness of the
schduler when threads of different or identical priorities are run for
10 seconds, and we look at how much work each thread got done in those
10 seconds. This test only works on OSv (which supports float priorities).

The second part of the test again tests long-term fairness of the scheduler
when all threads have the default priority (so this test is standard C++11):
We run a loop which takes (when run alone) 10 seconds, on 2 or 3
threads in parallel. We expect to see that all 2 or 3 threads
finish at (more-or-less) exactly the same time - after 20 or 30
seconds. Both OSv and Linux pass this test with flying colors.

The third part of the test runs two different threads concurrently:
 1. One thread wants to use all available CPU to loop for 10 seconds.
 2. The second thread wants to loop in an amount that takes N
    milliseconds, and then sleep for N milliseconds, and so on,
    until completing the same number of loop iterations that (when run
    alone) takes 10 seconds.

The "fair" behavior of the this test is that both threads get equal
CPU time and finish together: Thread 2 runs for N milliseconds, then
while it is sleeping for N more, Thread 1 gets to run.
This measure this for N=1 through 32ms. In OSv's new scheduler, indeed both
threads get an almost fair share (with N=32ms, one thread finishes in 19
seconds, the second in 21.4 seconds; we don't expect total fairness because
of the runtime decay).

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Some functions (strerror_r()) are defined differently based on the source
dialect.  We need to provide both dialects since we have mixed source.

Add a source-dialect macro (defaulting to _GNU_SOURCE) and override it
as appropriate.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

v2: Let's convert everything to std::chrono::timepoint (Avi Kivity)
v3: Use the to_timeptr approach suggested by Nadav Har'El

This test checks the functionality of the utimes support.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Start up shell and management web in parallel to make boot faster.  Note
that we also switch to latest mgmt.git which decouples JRuby and CRaSH
startup.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This adds a simple mmap microbenchmark that can be run on both OSv and
Linux.  The benchmark mmaps memory for various sizes and touches the
mmap'd memory in 4K increments to fault in memory.  The benchmark also
repeats the same tests using MAP_POPULATE for reference.

OSv page faults are slightly slower than Linux on first iteration but
faster on subsequent iterations after host operating system has faulted
in memory for the guest.

I've included full numbers on 2-core Sandy Bridge i7 for a OSv guest,
Linux guest, and Linux host below:

  OSv guest
  ---------

  Iteration 1

       time (seconds)
   MiB demand populate
     1 0.004  0.000
     2 0.000  0.000
     4 0.000  0.000
     8 0.001  0.000
    16 0.003  0.000
    32 0.007  0.000
    64 0.013  0.000
   128 0.024  0.000
   256 0.052  0.001
   512 0.229  0.002
  1024 0.587  0.005

  Iteration 2

       time (seconds)
   MiB demand populate
     1 0.001  0.000
     2 0.000  0.000
     4 0.000  0.000
     8 0.001  0.000
    16 0.002  0.000
    32 0.004  0.000
    64 0.010  0.000
   128 0.019  0.001
   256 0.036  0.001
   512 0.069  0.002
  1024 0.137  0.005

  Iteration 3

       time (seconds)
   MiB demand populate
     1 0.001  0.000
     2 0.000  0.000
     4 0.000  0.000
     8 0.001  0.000
    16 0.002  0.000
    32 0.005  0.000
    64 0.010  0.000
   128 0.020  0.000
   256 0.039  0.001
   512 0.087  0.002
  1024 0.138  0.005

  Iteration 4

       time (seconds)
   MiB demand populate
     1 0.001  0.000
     2 0.000  0.000
     4 0.000  0.000
     8 0.001  0.000
    16 0.002  0.000
    32 0.004  0.000
    64 0.012  0.000
   128 0.025  0.001
   256 0.040  0.001
   512 0.082  0.002
  1024 0.138  0.005

  Iteration 5

       time (seconds)
   MiB demand populate
     1 0.001  0.000
     2 0.000  0.000
     4 0.000  0.000
     8 0.001  0.000
    16 0.002  0.000
    32 0.004  0.000
    64 0.012  0.000
   128 0.028  0.001
   256 0.040  0.001
   512 0.082  0.002
  1024 0.166  0.005

  Linux guest
  -----------

  Iteration 1

       time (seconds)
   MiB demand populate
     1 0.001  0.000
     2 0.001  0.000
     4 0.002  0.000
     8 0.003  0.000
    16 0.005  0.000
    32 0.008  0.000
    64 0.015  0.000
   128 0.151  0.001
   256 0.090  0.001
   512 0.266  0.003
  1024 0.401  0.006

  Iteration 2

       time (seconds)
   MiB demand populate
     1 0.000  0.000
     2 0.000  0.000
     4 0.001  0.000
     8 0.001  0.000
    16 0.002  0.000
    32 0.005  0.000
    64 0.009  0.000
   128 0.019  0.001
   256 0.037  0.001
   512 0.072  0.003
  1024 0.144  0.006

  Iteration 3

       time (seconds)
   MiB demand populate
     1 0.000  0.000
     2 0.001  0.000
     4 0.001  0.000
     8 0.001  0.000
    16 0.002  0.000
    32 0.005  0.000
    64 0.010  0.000
   128 0.019  0.001
   256 0.037  0.001
   512 0.072  0.003
  1024 0.143  0.006

  Iteration 4

       time (seconds)
   MiB demand populate
     1 0.000  0.000
     2 0.001  0.000
     4 0.001  0.000
     8 0.001  0.000
    16 0.003  0.000
    32 0.005  0.000
    64 0.010  0.000
   128 0.020  0.001
   256 0.038  0.001
   512 0.073  0.003
  1024 0.143  0.006

  Iteration 5

       time (seconds)
   MiB demand populate
     1 0.000  0.000
     2 0.001  0.000
     4 0.001  0.000
     8 0.001  0.000
    16 0.003  0.000
    32 0.005  0.000
    64 0.010  0.000
   128 0.020  0.001
   256 0.037  0.001
   512 0.072  0.003
  1024 0.144  0.006

  Linux host
  ----------

  Iteration 1

       time (seconds)
   MiB demand populate
     1 0.000  0.000
     2 0.001  0.000
     4 0.001  0.000
     8 0.001  0.000
    16 0.002  0.000
    32 0.005  0.000
    64 0.009  0.000
   128 0.019  0.001
   256 0.035  0.001
   512 0.152  0.003
  1024 0.286  0.011

  Iteration 2

       time (seconds)
   MiB demand populate
     1 0.000  0.000
     2 0.000  0.000
     4 0.001  0.000
     8 0.001  0.000
    16 0.002  0.000
    32 0.004  0.000
    64 0.010  0.000
   128 0.018  0.001
   256 0.035  0.001
   512 0.192  0.003
  1024 0.334  0.011

  Iteration 3

       time (seconds)
   MiB demand populate
     1 0.000  0.000
     2 0.000  0.000
     4 0.001  0.000
     8 0.001  0.000
    16 0.002  0.000
    32 0.004  0.000
    64 0.010  0.000
   128 0.018  0.001
   256 0.035  0.001
   512 0.194  0.003
  1024 0.329  0.011

  Iteration 4

       time (seconds)
   MiB demand populate
     1 0.000  0.000
     2 0.000  0.000
     4 0.001  0.000
     8 0.001  0.000
    16 0.002  0.000
    32 0.004  0.000
    64 0.010  0.000
   128 0.018  0.001
   256 0.036  0.001
   512 0.138  0.003
  1024 0.341  0.011

  Iteration 5

       time (seconds)
   MiB demand populate
     1 0.000  0.000
     2 0.000  0.000
     4 0.001  0.000
     8 0.001  0.000
    16 0.002  0.000
    32 0.004  0.000
    64 0.010  0.000
   128 0.018  0.001
   256 0.035  0.001
   512 0.135  0.002
  1024 0.324  0.011

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The test creates and destroys threads, each of which creates a random number
of connections, each transferring a random number of bytes to an echo server.

This is used to stress the tcp/ip stack.

The test is portable, and builds on the host with the command

  g++ -O2  -g3 -pthread -std=gnu++11 -lboost_program_options -lboost_system tests/tst-tcp.cc

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Commit 0dcf1f8f ("OSv module support") didn't add a dependency to
bootfs.manifest.skel and usr.manifest.skel which causes image not to be
rebuilt if the files are changed.

Fix that up.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The idea of the patch is basically described in prevoius post:

https://groups.google.com/d/msg/osv-dev/RL2S3AL9TNE/l4XZJo3-lI0J

Whis this patch, you will be able to install OSv apps into disk image on
"make all" stage.

These apps does not require to exist in OSv repository, you can install
apps which is on any git repository or svn repository, or on local
directory.

You'll need to write a config file to add apps, format of the file is
JSON.

Here's a sample of the file:
{
   "modules":[
      {
	 "name":"osv-mruby",
         "type":"git",
         "path":"https://github.com/syuu1228/osv-mruby.git",
         "branch":"master"
      }
   ]
}

If you add "module" on config file, make all calls script/module.py.

This scripts perform "git clone" to fetch repository to $(out)/module,
and invoke "make module" on each module.

"make module" should outputs bootfs.manifest/usr.manifest on module
directory, the script merge bootfs.manifest.skel/usr.manifest.skel and
module local manifests to single file
$(out)/bootfs.manifest/$(out)/usr.manifest.

Here's app Makefile example:

  https://github.com/syuu1228/osv-mruby/blob/master/Makefile



It have "module" target, and the target builds all binaries and
generates *.manifest.

Signed-off-by: Takuya ASADA <syuu@dokukino.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Default image format stays QCOW2 as before.  May be changed via img_format
variable:

  make img_fomat=raw all

This option is required by release-ec2 script as EC2 does not support QCOW2.
Also will be useful for other hypervisors like VMWARE.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Dead code, and boost::signals2::signal<> does the job better.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Make use of the python nbd client.

Signed-off-by: Benoit Canet <benoit@irqsave.net>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Consolidates generated files, and makes it easier to maintain the eclipse
index.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

I dismissed the idea to configure the bridge in default JVM
configuration because having a default like this would force
all apps to use particular slf4j-api version.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>

This patch simplifies the host-side work in the new /usr zfs filesystem
build process.

Previously, we copied the files to a temporary directory, used "cpio"
to archive them and sent its output to the guest with "netcat".

With this patch, we no longer have a temporary directory, and do not
need either cpio or netcat on the build machine.

Rather, mkzfs.py itself, using python (instead of a separate "nc" process),
connects to the guest and sends it the files - still using the CPIO format.

Rather than arbitrarily sleep for 3 seconds before the host tries to
connect to the guest (which might not be enough for some, or a waste
of time for others), with this patch the host looks at the guest's output
and connects when it sees the message "Waiting for connection".

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>

The tst-zfs-simple.so test case has serverd its purpose for bringup.  As
OSv build relies on working ZFS now, there's no need to run the tests.

Furthermore, we have the full ztest stress test in the tree:

  bsd/cddl/contrib/opensolaris/cmd/ztest/ztest.c

which we can use if needed.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

The call to namei("/dir/file/") currently fails with ENOENT when
"/dir/file" exists. A more standard way is to return ENOTDIR
instead. This way calls to stat, open, rename, etc. will be in
line with the POSIX spec.

It is also useful to rename() implementation which needs to
differentiate behaviour between the case in which target does not
exist and the case in which it does but the path has trailing slash
and the last component is not a directory.

In addition to that the check was performed in an inconsistent matter
- only when dentry lookup failed. This change makes the check
performed always.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>

When converting a source file from .c to .cc, make will complain that the
old source file was missing.  This is annoying, especially when bisecting
or compile-testing a patch set.

Fix by removing the source file dependency.  Since the dependency is alread
specified explicitly by the makefile, no information is lost.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Add 'werror' option to Makefile that can be used to disable the -Werror
GCC option temporarily as follows:

  make werror=0

This is useful when fixing up build breakage caused by addition of new
default compiler warnings.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Some distributions have enabled gcc -Wformat-security by default.
This causes the build to raise format warnings even though -Wformat=0
is used.

Setting no-format-security for build consistency between distros.

Signed-off-by: Or Cohen <orc@fewbytes.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Benoit Canet <benoit@irqsave.net>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Benoit Canet <benoit@irqsave.net>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

For example, assuming the following file exists:

  /tmp/dir/x

..the following was sondiered absent:

  /tmp/dir/

..this was considered present:

  /tmp/dir

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>

Needed to link with boost_filesystem.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>

This patch adds version numbering to the loader. We use "git tag" for
generating an unique version number with a little bit of shell script
magic to make sure version number is always up-to-date.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

strlcpy() expects buffer length not string length.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>

Not supported by older qemus.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Drop sudo and global /zfs temporary directory.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

This drops the superuser requirement from the build process, and reduces
the build prerequisite list.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

This allows using the boost libraries in shared objects.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Dump registers on general protection fault for debugging purposes.  Even
if you have gdb available, getting to the exception frame is not always
possible after OSv has crashed.

Example output looks as follows:

registers:
RIP: 0x0000100000b7e913  RFL: 0x0000000000010202  CS:  0x0000000000000008  SS:  0x0000000000000010
RAX: 0xffffc000418ed278  RBX: 0xffffc00041b2c050  RCX: 0x0000000000000004  RDX: 0x0000000000000000
RSI: 0x0000000000000001  RDI: 0x43e0000000000000  RBP: 0x0000200008548d10  R8:  0xffffc000426e3010
R9:  0x0000000000000004  R10: 0x43e0000000000000  R11: 0xffffc00041b2c050  R12: 0xffffc000418ed1e8
R13: 0x0000000000000004  R14: 0x43e0000000000000  R15: 0xffffc00041b2c050  RSP: 0x0000200008548aa0
general protection fault

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Create a file system on top of the pool.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

For the moment, it only creates a zpool, not a filesystem.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>