This patch fixes chdir() on a normal file, which used to succeed (!?),
but now will fail as it should, with ENOTDIR.

The patch also adds an exhaustive test for chdir's success and error cases.
Before the latest chdir() patches, most of these tests would fail, and now
all of them succeed.

This test is standard C++ & Posix code, so it can be run also on Linux.
This is important for verifing that whatever we expect from OSv, Linux
really does the same.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Force version.h generation to make sure the version number matches the
git version.  I noticed the problem after tagging v0.05-rc1 and noticing
that OSv happily reported the same version even after applying patches.

Use the special ".PHONY" target suggested by Tomek to avoid breakage
from previous attempt that is documented in commit 996fdfde ('Revert
"build: Force version.h generation"').

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

A waitqueue is an object on which multiple threads can wait; other threads
can wake up either one or all waiting threads.  A waitqueue is associated
with an external mutex which the user must supply for both wait and wake
operations.

Waitqueues differ from condition variables in three respects:
- waitqueues do not contain an internal mutex.  This makes them smaller, and
  reduces lock acquisitions.  On the other hand the waker must hold the
  associated mutex, whereas this is not required with condition variables.
- waitqueues support sched::thread::wait_for()

waitqueues support wait morphing and do not cause excess lock contention,
even with wake_all().

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Fixes #116.

Because modules add files to which cannot be handled by our build
system incremental build fails with messages like this:

  make[1]: *** No rule to make target `/home/tgrabiec/src/osv \
     /apps/cassandra//upstream/jna-3.`

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This reverts commit 59563357 which
causes significant slowdown on first time boot to a QCOW2 image:

  [penberg@localhost osv]$ time ./scripts/run.py
  OSv v0.05-rc1-108-g0639dd8
  OpenJDK 64-Bit Server VM warning: Can't detect initial thread stack
  location - find_vma failed
  ^D^D^D
  [/]%

  real    0m9.954s
  user    0m4.958s
  sys     0m1.309s

  [penberg@localhost osv]$ git revert
  59563357
  [penberg@localhost osv]$ time ./scripts/run.py
  OSv v0.05-rc1-108-g0639dd8
  OpenJDK 64-Bit Server VM warning: Can't detect initial thread stack
  location - find_vma failed
  ^D^D^D
  [/]% ^D

  real    0m4.441s
  user    0m3.785s
  sys     0m0.530s

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add a simple manual test case for checking "/proc/self/maps" output.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Instead of always putting the tests into all images, this patch adds the
option of putting them only in some of the images, by making them into a
new module, modules/tests.

The default image (images/default.py) continues to require the tests
modules, so tests are included in the default image. Building with
"make image=tests" makes an image with only the tests and nothing else.
Other images (e.g., memcached, cassandra,...) currently do not require
the tests module, so the generated image does will not include the tests.

With this patch, "make image=memcached", for example, contains only the
bare minimum needed for memcached, and the resulting qcow image is just
18 MB, down from 69 MB when we included all the tests.

Fixes #160

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This reverts commit d914ee0d because it breaks
the build:

  I'm trying to compile the latest source code from git HEAD and I'm heaving
  the following error:

  make -r -C build/release/ all
  make[1]: Entering directory `/home/eduardo/workspace/osv/build/release'
  make[1]: *** No rule to make target
  `/home/eduardo/workspace/osv/build/release/gen/include/osv/version.h',
  needed by `/home/eduardo/workspace/osv/build.mk'.  Stop.
  make[1]: *** Waiting for unfinished jobs....
    GEN gen/include/bits/alltypes.h
  make[1]: Leaving directory `/home/eduardo/workspace/osv/build/release'
  make: *** [all] Error 2

Reported-by: Eduardo Piva <efpiva@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Since our kernel includes the entirety of libstdc++, there's no point of
specifying "-lstdc++" while linking shared objects (like tests and cpio.so):
This "linking" doesn't actually do any linking, it just specifies that the
libstdc++ library is needed, but we already have it, needed or not.

The list of objects-needing-libstdc++ was not only unnecessary, but probably
also out of date, since we stopped bothering to update it after libstdc++
was added into the kernel.

So this patch removes these -lstdc++ lines.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Force version.h generation to make sure the version number matches the
git version.

I noticed the problem after tagging v0.05-rc1 and noticing that OSv
happily reported the same version even after applying patches.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Current makefile cannot accept "make img_format=vmdk external all",
because QEMU does not support the format, but mkzfs.py and
upload_manifest.py requires to run OSv on QEMU.

With this patch, image conversion performs after mkzfs.py and
upload_manifest.py, prevents an error on QEMU execution.

Tested on QEMU/KVM and VMware, using these formats: raw / QCOW2 / VMDK.

Probably it can work on another formats such as VDI used by VirtualBox.

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The uname() function returns a fake Linux version number for application
compatibility.  Add a new osv::version() API that returns OSv version
that can be used by the management code.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
[ penberg: cleanups ]
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The best possible criteria for deflating balloons is heap pressure: Whenever
there is pressure in the JVM, we should give back memory so pressure stops.

To accomplish that, we need to somehow tap into the JVM. This patch register
a MXBean that will send us notifications about collections. We will ignore
minor collections and act upon major collections by deflating any existing
balloons.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch implements the JVM balloon driver, that is responsible for borrowing
memory from the JVM when OSv is short on memory, and giving it back when we are
plentiful. It works by allocating a java byte array, and then unmapping a large
page-aligned region inside it (as big as our size allows).

This array is good to go until the GC decides to move us. When that happens, we
need to carefuly emulate the memcpy fault and put things back in place.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently the parameter was read from the generated Makefile which was
not re-generated on incremental build. The fix is to move the default
to build.mk, this way the default will always be picked unless masked
by command line argument.

Fixes #153

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Fixes issue with JVM failing when started with a debugger
with the following message:

  NPT ERROR: Cannot find nptInitialize

Missing openjdk files in usr.manifest were a fertile source
of issues. This patch aims at making them less likely and
adding all files   except blacklisted files to the image.

This patch skips two files from JRE which are broken links
and inclusion of which would cause manifest upload failure:
 - jre/lib/audio/default.sf2
 - jre/lib/security/cacerts

These should be fixed incrementally.

Reported-by: Oded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

A previous patch renamed mutex.cc to spinlock.cc. This fixes the build.mk
dependency to make the code compile again... Sorry about that.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Commit 2ebfd915 ("build.mk: Generate a manifest of tests automatically")
separated native tests from Java tests but unfortunately broke the build
because it forgot to add dependency to the new java_tests variable.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Generate a manifest of tests automatically from the makefile
based on the value of $(tests) and $(java_tests) variables.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

In order to reuse the logic it needs to be extracted.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This test includes a large number of tests for the timerfd_*() functions
and many of their bizarre use-cases and corner cases.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It's just to painful to do this piecemeal.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Introduced variable fs_size_mb - FS size in megabytes.
Useful for debugging on EC2.

Reviewed-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently we use implementation from
bsd/sys/cddl/compat/opensolaris/kern/opensolaris_atomic.c
which is using global mutex.

This showes up on wait profile of ZFS writes:

(...)
   \-- 469.96 us (0.00%, #250) arc_do_user_evicts
       \-- 469.96 us (0.00%, #250) dbuf_do_evict
           |-- 355.93 us (0.00%, #154) dbuf_destroy
(...)
           |   \-- 76.60 us (0.00%, #53) zrl_add
           |                             atomic_cas_32
           |                             lockfree_mutex_lock
           |                             sched::thread::wait()
           |
           |-- 66.14 us (0.00%, #56) dbuf_clear
           |                         zrl_add
           |                         atomic_cas_32
           |                         lockfree_mutex_lock
           |                         sched::thread::wait()
           |
           \-- 47.89 us (0.00%, #40) dbuf_rele_and_unlock
                                     atomic_add_64_nv
                                     lockfree_mutex_lock
                                     sched::thread::wait()

This patch switches to amd64 implementation.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This adds initial virtio-scsi support. We have no scsi layer in osv, in
this implementation virtio-scsi works directly with the bio layer. It
translates BIO_READ, BIO_WRITE and BIO_FLUSH to SCSI CMD.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Now that backtrace() doesn't use libunwind, we can remove it.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

 - Add -DNDEBUG to the compiler flags when mode!=debug.
 - Prevent assert() from compiling out in kernel when mode=release

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Simplify core/mmu.cc and make it more portable by moving the page fault
handler to arch/x64/mmu.cc.  There's more arch specific code in
core/mmu.cc that should be also moved.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Separate /dev/random the virtio-rng driver and register virtio-rng as a
HW RNG entropy source.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Rename blacklisted tests, from tst-wake.cc et al. to misc-wake.cc.

The different name will cause these tests not to be automatically run
by "make check" - without needing the separate blacklist in test.py
(which this patch deletes).
After this patch, testrunner.so will also only run tests called tst-*,
so will not run the misc-* tests.

The misc-* tests can still be run manually, e.g.,
  run.py -e tests/misc-mutex.so

In addition to the previously blacklisted tests, this patch "blacklists"
(renames) a few additional tests which fail quickly, but test.py didn't
know because they didn't use the word "fail". An example is tst-schedule.so,
which existed immediately when not run on 1 vcpu. So this patch also
renames it to misc-schedule.so, so "make check" or testrunner.so won't
run this test.

Note that after this patch, testrunner.so is a new way to run all tests,
but it isn't working well yet because it still exposes new bugs that do not
exist in the separate tests (depending on your view point, this might be
considered a feature, not a bug, in testrunner.so...).

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch fixes the error codes in four error cases:

1. unlink() of a directory used to return EPERM (as in Posix), and now
   returns EISDIR (as in Linux).

2. rmdir() of a non-empty directory used to return EEXIST (as in Posix)
   and now returns ENOTEMPTY (as in Linux).

3. rmdir() of a regular file (non-directory) used to return EBADF
   and now returns ENOTDIR (as in Linux).

4. readdir() of a regular file (non-directory) used to return EBADF
   and now returns ENOTDIR (as in Linux).

This patch also adds a test, tst-remove.cc, for the various unlink() and
rmdir() success and failure modes.

Fixes #123.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add -Wno-maybe-uninitialized to compilation flags when mode=debug to
avoid bogus compilation errors.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Our implementation of operate() will try to fill as much as the address space
as possible with huge pages. If that fails, we should be able to fill the range
with small pages instead of failing. This test should make sure that in such
scenarios, the resulting mapping looks sane.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

gcc recommends -Og for debugging; follow its advice.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

That test goes together with thread detach, but I am also calling joins
to make sure we're not breaking them. It is unfortunate that this is quite
non-deterministic and we can't really surely test for failure. But on the
flip side, it did help me catch a couple of bugs in my implementation. So
it will eventually explode somewhere if a bug appears.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>