Instead of always putting the tests into all images, this patch adds the
option of putting them only in some of the images, by making them into a
new module, modules/tests.

The default image (images/default.py) continues to require the tests
modules, so tests are included in the default image. Building with
"make image=tests" makes an image with only the tests and nothing else.
Other images (e.g., memcached, cassandra,...) currently do not require
the tests module, so the generated image does will not include the tests.

With this patch, "make image=memcached", for example, contains only the
bare minimum needed for memcached, and the resulting qcow image is just
18 MB, down from 69 MB when we included all the tests.

Fixes #160

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This reverts commit d914ee0d because it breaks
the build:

  I'm trying to compile the latest source code from git HEAD and I'm heaving
  the following error:

  make -r -C build/release/ all
  make[1]: Entering directory `/home/eduardo/workspace/osv/build/release'
  make[1]: *** No rule to make target
  `/home/eduardo/workspace/osv/build/release/gen/include/osv/version.h',
  needed by `/home/eduardo/workspace/osv/build.mk'.  Stop.
  make[1]: *** Waiting for unfinished jobs....
    GEN gen/include/bits/alltypes.h
  make[1]: Leaving directory `/home/eduardo/workspace/osv/build/release'
  make: *** [all] Error 2

Reported-by: Eduardo Piva <efpiva@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Since our kernel includes the entirety of libstdc++, there's no point of
specifying "-lstdc++" while linking shared objects (like tests and cpio.so):
This "linking" doesn't actually do any linking, it just specifies that the
libstdc++ library is needed, but we already have it, needed or not.

The list of objects-needing-libstdc++ was not only unnecessary, but probably
also out of date, since we stopped bothering to update it after libstdc++
was added into the kernel.

So this patch removes these -lstdc++ lines.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Force version.h generation to make sure the version number matches the
git version.

I noticed the problem after tagging v0.05-rc1 and noticing that OSv
happily reported the same version even after applying patches.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Current makefile cannot accept "make img_format=vmdk external all",
because QEMU does not support the format, but mkzfs.py and
upload_manifest.py requires to run OSv on QEMU.

With this patch, image conversion performs after mkzfs.py and
upload_manifest.py, prevents an error on QEMU execution.

Tested on QEMU/KVM and VMware, using these formats: raw / QCOW2 / VMDK.

Probably it can work on another formats such as VDI used by VirtualBox.

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The uname() function returns a fake Linux version number for application
compatibility.  Add a new osv::version() API that returns OSv version
that can be used by the management code.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
[ penberg: cleanups ]
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The best possible criteria for deflating balloons is heap pressure: Whenever
there is pressure in the JVM, we should give back memory so pressure stops.

To accomplish that, we need to somehow tap into the JVM. This patch register
a MXBean that will send us notifications about collections. We will ignore
minor collections and act upon major collections by deflating any existing
balloons.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch implements the JVM balloon driver, that is responsible for borrowing
memory from the JVM when OSv is short on memory, and giving it back when we are
plentiful. It works by allocating a java byte array, and then unmapping a large
page-aligned region inside it (as big as our size allows).

This array is good to go until the GC decides to move us. When that happens, we
need to carefuly emulate the memcpy fault and put things back in place.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently the parameter was read from the generated Makefile which was
not re-generated on incremental build. The fix is to move the default
to build.mk, this way the default will always be picked unless masked
by command line argument.

Fixes #153

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Fixes issue with JVM failing when started with a debugger
with the following message:

  NPT ERROR: Cannot find nptInitialize

Missing openjdk files in usr.manifest were a fertile source
of issues. This patch aims at making them less likely and
adding all files   except blacklisted files to the image.

This patch skips two files from JRE which are broken links
and inclusion of which would cause manifest upload failure:
 - jre/lib/audio/default.sf2
 - jre/lib/security/cacerts

These should be fixed incrementally.

Reported-by: Oded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

A previous patch renamed mutex.cc to spinlock.cc. This fixes the build.mk
dependency to make the code compile again... Sorry about that.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Commit 2ebfd915 ("build.mk: Generate a manifest of tests automatically")
separated native tests from Java tests but unfortunately broke the build
because it forgot to add dependency to the new java_tests variable.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Generate a manifest of tests automatically from the makefile
based on the value of $(tests) and $(java_tests) variables.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

In order to reuse the logic it needs to be extracted.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This test includes a large number of tests for the timerfd_*() functions
and many of their bizarre use-cases and corner cases.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It's just to painful to do this piecemeal.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Introduced variable fs_size_mb - FS size in megabytes.
Useful for debugging on EC2.

Reviewed-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently we use implementation from
bsd/sys/cddl/compat/opensolaris/kern/opensolaris_atomic.c
which is using global mutex.

This showes up on wait profile of ZFS writes:

(...)
   \-- 469.96 us (0.00%, #250) arc_do_user_evicts
       \-- 469.96 us (0.00%, #250) dbuf_do_evict
           |-- 355.93 us (0.00%, #154) dbuf_destroy
(...)
           |   \-- 76.60 us (0.00%, #53) zrl_add
           |                             atomic_cas_32
           |                             lockfree_mutex_lock
           |                             sched::thread::wait()
           |
           |-- 66.14 us (0.00%, #56) dbuf_clear
           |                         zrl_add
           |                         atomic_cas_32
           |                         lockfree_mutex_lock
           |                         sched::thread::wait()
           |
           \-- 47.89 us (0.00%, #40) dbuf_rele_and_unlock
                                     atomic_add_64_nv
                                     lockfree_mutex_lock
                                     sched::thread::wait()

This patch switches to amd64 implementation.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This adds initial virtio-scsi support. We have no scsi layer in osv, in
this implementation virtio-scsi works directly with the bio layer. It
translates BIO_READ, BIO_WRITE and BIO_FLUSH to SCSI CMD.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Now that backtrace() doesn't use libunwind, we can remove it.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

 - Add -DNDEBUG to the compiler flags when mode!=debug.
 - Prevent assert() from compiling out in kernel when mode=release

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Simplify core/mmu.cc and make it more portable by moving the page fault
handler to arch/x64/mmu.cc.  There's more arch specific code in
core/mmu.cc that should be also moved.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Separate /dev/random the virtio-rng driver and register virtio-rng as a
HW RNG entropy source.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Rename blacklisted tests, from tst-wake.cc et al. to misc-wake.cc.

The different name will cause these tests not to be automatically run
by "make check" - without needing the separate blacklist in test.py
(which this patch deletes).
After this patch, testrunner.so will also only run tests called tst-*,
so will not run the misc-* tests.

The misc-* tests can still be run manually, e.g.,
  run.py -e tests/misc-mutex.so

In addition to the previously blacklisted tests, this patch "blacklists"
(renames) a few additional tests which fail quickly, but test.py didn't
know because they didn't use the word "fail". An example is tst-schedule.so,
which existed immediately when not run on 1 vcpu. So this patch also
renames it to misc-schedule.so, so "make check" or testrunner.so won't
run this test.

Note that after this patch, testrunner.so is a new way to run all tests,
but it isn't working well yet because it still exposes new bugs that do not
exist in the separate tests (depending on your view point, this might be
considered a feature, not a bug, in testrunner.so...).

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch fixes the error codes in four error cases:

1. unlink() of a directory used to return EPERM (as in Posix), and now
   returns EISDIR (as in Linux).

2. rmdir() of a non-empty directory used to return EEXIST (as in Posix)
   and now returns ENOTEMPTY (as in Linux).

3. rmdir() of a regular file (non-directory) used to return EBADF
   and now returns ENOTDIR (as in Linux).

4. readdir() of a regular file (non-directory) used to return EBADF
   and now returns ENOTDIR (as in Linux).

This patch also adds a test, tst-remove.cc, for the various unlink() and
rmdir() success and failure modes.

Fixes #123.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add -Wno-maybe-uninitialized to compilation flags when mode=debug to
avoid bogus compilation errors.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Our implementation of operate() will try to fill as much as the address space
as possible with huge pages. If that fails, we should be able to fill the range
with small pages instead of failing. This test should make sure that in such
scenarios, the resulting mapping looks sane.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

gcc recommends -Og for debugging; follow its advice.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

That test goes together with thread detach, but I am also calling joins
to make sure we're not breaking them. It is unfortunate that this is quite
non-deterministic and we can't really surely test for failure. But on the
flip side, it did help me catch a couple of bugs in my implementation. So
it will eventually explode somewhere if a bug appears.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

    == Description of the problem ==

Currently modules can only specify files which need to be
copied. There is a need for a module to also declare ways it can be
run, so that we can automatically prepare a runnable image. It
should be easy to switch between run configurations.

Currently it is enough for image configuration to happen at the
time of image make process. In future this should be allowed
on the already built image.

We also need to be able to combine multiple modules in one
image. For example in addition to the main application one might want
to start various services like CLI, management API server, etc.

Java apps should be able to specify JVM arguments, which must be
passed to the JVM upon its creation, as well as regular run-java
arguments (classpath, main classes, main arguments, etc.)

    == Solution ==

This is not intended to be a permanent solution. The aim is to solve
immediate need to have a fully modularized build in a scalable way.

Every module has a new kind of file in its root directory which
holds its configuration. The file is named 'module.py' and is
a python script which uses osv's api for declaring run configurations.

Using python as config language has several advantages:
 - more expresiveness, unlike json it allows for expression reuse
 - it's easier to extend the config language
 - we don't need as much parsing, gluing, error checking, error
   reporting code because we have it already

There are currently two kinds of applications which can be declared:

   run(cmdline)  <- basic .so application
   run_java(jvm_args=[], classpath=[], args=[])  <- java applications

Run configurations can be declared as simple module attributes
which can be referenced from the image configuration file.

Image configuration

There is a new configuration file kind, which defines which modules
and which run configurations should be included in the image. Files
are located using path: ${OSV_BASE}/images/$(image-name).py

Syntax:

  require(module) <-- declares that module should be included in the image
  		      and returns an object which allows to access module's
		      attributes.

  run = []  <-- list of run configurations

Example:

  _mgmt = require('mgmt')
  run = [ _mgmt.shell ]

To use a particular image configuration run make like this:

  make image=fancy-tomcat

The default configuration is named 'default'.

This patch extracts mgmt into a module, which is embedded under
${OSV_BASE}/modules/mgmt

The purpose of ${OSV_BASE}/config.json has been changed. It does not
list modules which should be included anymore, image config file does
that. It's a module look-up configuration which tells the build where
to look for modules.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

We have an intermediate bare.img now. Rename usr.raw to bare.raw to make the
naming more consistent: bare.raw -> bare.img -> usr.img.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This will allow the users to populate /usr with simples binaries like haproxy
and their config at runtime.
It's intented to be combined with the loader ability to run multiple binaries
in a sequential way.
For example -e '/tools/cpiod.so; /usr/haproxy.so -f /usr/haproxy.conf'

Signed-off-by: Benoit Canet <benoit@irqsave.net>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This pave the way for the creation of bare images that the user can populate
using cpiod when he need it.

Signed-off-by: Benoit Canet <benoit@irqsave.net>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

A ';' at the end of a parameter mark the end of a program's arguments list.

The goal of this patch is to be able to split mkfs.so in to parts mkfs.so and
cpiod.so.

The patch uses a full spirit parser to escape "" and split commands around ';'.

Signed-off-by: Benoit Canet <benoit@irqsave.net>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>