AHCI is supported on various VMM, e.g. Virtual Box, VMware Workstation.
Adding AHCI support enables OSv to run on them if the para-virtualized
block device is not present or not supported yet.

Tested on VirtualBox, VMware Workstation and QEMU.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

This adds a net_channel class the connects a packet producer with a
packet consumer.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

I am proposing a mechanism here that will allow us to have a better idea about
how much time do we spend booting, and also how much time each of the pieces
contribute to. For that, we need to be able to get time stamps really early, in
places where tracepoints may not be available, and a clock most definitely
won't.

With my proposal, one should be able to register events. After the system
boots, we will calculate the total time since the first event, as well as the
delta since the last event. If the first event is early enough, that should
produce a very good picture about our boot time.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It will allow to use automatic dependency management.

It slows down the build a bit. Incremental build takes 3 seconds
longer than previously.

First build takes longer due to downloading of maven artifacts.
This is once per machine.

The main purpose of this tool is to understand/analyze the ARC behavior/
performance on specific workloads.

$ scripts/run.py -e 'tests/misc-zfs-arc.so --help'
OSv v0.05-155-g1f04e49
Allowed options:
  --help                produce help message
  --set-max-target      set ARC max target to 80% of the system memory.
  --check-arc-shrink    check ARC shrink functionality
  --test arg            analyze ARC performance on a given testcase, e.g.
                        --test tst-001.so

* --set-max-target: Used to check performance when ARC max target is
higher than usual. Given that more data will be load into ARC, ZFS operations
that needs I/O would perform better. 80% was chosen as the low watermark
is 20%, so avoiding a bunch of memory pressure, thus more stability.

* --check-arc-shrink: Check the functionality of the function arc_shrink
from ARC.

* --test arg: Check ARC performance on a specified testcase, e.g.:
$ scripts/run.py -e 'tests/misc-zfs-arc.so --test tst-fs-link.so'

* Default run, i.e -e 'tests/misc-zfs-arc.so' provides four distinct
workloads:
1) Non-linear one where prefetch shouldn't be as effective.
2) Load all data into cache, then read it afterwards to check performance
on such cases, almost speed of main memory.
3) Linear workload where the amount of data is 1.5% the size of the system
memory, thus page replacement will be strongly used, and as the operation
is sequential, prefetch (readahead) must be effective. It leads to a high
cache hit ratio as blocks were read ahead of time.
4) Keep allocating memory through a populated anonymous mmaping to see
if shrink would take place to release memory back to the operating system.

Eventual reports and ARC stats are provided to ease the task of understanding
ARC performance on specific workloads.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Mainly created to be used as a tool that reproduces specific workloads,
so allowing us to understand how underlying components are performing,
e.g. Adjustable Replacement Cache (ARC) from ZFS.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch registers the ARC shrinker by using the event handler
list from BSD side. When ARC is initialized, it inserts the lowmem event
handler into an external event handler list. lowmem basically signals
the reclaiming thread which will then wake up to decide which approach
should be used to shrink the ARC.

The memory pressure on OSv is activated when the 20% watermark is
reached, so the shrink policy will decide which shrinker should
be called on such events.

bsd_shrinker_init is the responsible to find the lowmem event handler
from the external list, and integrate it into our shrinker infrastructure.

arc_lowmem needed few changes to return the amount of released memory
from the ARC.

Glauber and I tested the functionality by filling up the ARC up
to its target, then allocating as much memory as possible to see
if the ARC shrinker would take place to release memory back to
the operating system.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

this implements simple target detection borrowing from the
idea previously in Makefile.
Adds basic recognition of passed envs ARCH, CROSS_PREFIX,
along with CC, LD, CXX, HOST_CXX.
HOST_CXX is necessary because the compiler is used to
produce and run binaries at build time.

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

First step towards moving x64-specific stuff into arch
subdirectories:

move the submodules in external/ to external/x64/,

rename amd64->x64 in opensolaris,

rename x86_64->x64 in include/api,

adapt build.mk to reference the updated paths,
and improve detection of include dirs and libs in the
external submodules.

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Hopefully this will reduce porting efforts.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

This test is similar in spirit to misc-bdev-write, but instead of pushing
as many bios as we can, we'll push one bio at a time.

Example output for Xen:

OSv v0.05-118-g0f2973c
Min      50%      90%      99%      99.99%   99.999%  Max     [msec]
---      ---      ---      ---      ------   -------  ---
0.2344   0.3240   0.2847   0.8185   2.4095   6.5230   13.6572

Example output fo KVM:

OSv v0.05-118-g0f2973c
Min      50%      90%      99%      99.99%   99.999%  Max     [msec]
---      ---      ---      ---      ------   -------  ---
0.2626   0.3976   0.3273   0.4791   0.5993   7.6401   15.9672

(Hint: the current xen blkfront slowness is not related to RT latency...)

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Combined java/cli and java/cloudius packages into one.
Added java/cloudius.jar as a jar to the build.

Signed-off-by: Or Cohen <orc@fewbytes.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Now since we have support for parallel launching in the command line
we can implement support for it in the modules.

Example (image config):

 run = [
   api.require('netperf').default,
   api.require('mgmt').shell
 ]

Result:

$ scripts/run.py
OSv v0.05-132-g779307c
Starting netserver with host 'IN(6)ADDR_ANY' port '12865' and family AF_INET

[/]%

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This is useful to measure OSv boot speed, IOW, how fast are we without
CLI, Java, etc.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch fixes chdir() on a normal file, which used to succeed (!?),
but now will fail as it should, with ENOTDIR.

The patch also adds an exhaustive test for chdir's success and error cases.
Before the latest chdir() patches, most of these tests would fail, and now
all of them succeed.

This test is standard C++ & Posix code, so it can be run also on Linux.
This is important for verifing that whatever we expect from OSv, Linux
really does the same.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Force version.h generation to make sure the version number matches the
git version.  I noticed the problem after tagging v0.05-rc1 and noticing
that OSv happily reported the same version even after applying patches.

Use the special ".PHONY" target suggested by Tomek to avoid breakage
from previous attempt that is documented in commit 996fdfde ('Revert
"build: Force version.h generation"').

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

A waitqueue is an object on which multiple threads can wait; other threads
can wake up either one or all waiting threads.  A waitqueue is associated
with an external mutex which the user must supply for both wait and wake
operations.

Waitqueues differ from condition variables in three respects:
- waitqueues do not contain an internal mutex.  This makes them smaller, and
  reduces lock acquisitions.  On the other hand the waker must hold the
  associated mutex, whereas this is not required with condition variables.
- waitqueues support sched::thread::wait_for()

waitqueues support wait morphing and do not cause excess lock contention,
even with wake_all().

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Fixes #116.

Because modules add files to which cannot be handled by our build
system incremental build fails with messages like this:

  make[1]: *** No rule to make target `/home/tgrabiec/src/osv \
     /apps/cassandra//upstream/jna-3.`

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This reverts commit 59563357 which
causes significant slowdown on first time boot to a QCOW2 image:

  [penberg@localhost osv]$ time ./scripts/run.py
  OSv v0.05-rc1-108-g0639dd8
  OpenJDK 64-Bit Server VM warning: Can't detect initial thread stack
  location - find_vma failed
  ^D^D^D
  [/]%

  real    0m9.954s
  user    0m4.958s
  sys     0m1.309s

  [penberg@localhost osv]$ git revert
  59563357
  [penberg@localhost osv]$ time ./scripts/run.py
  OSv v0.05-rc1-108-g0639dd8
  OpenJDK 64-Bit Server VM warning: Can't detect initial thread stack
  location - find_vma failed
  ^D^D^D
  [/]% ^D

  real    0m4.441s
  user    0m3.785s
  sys     0m0.530s

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add a simple manual test case for checking "/proc/self/maps" output.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Instead of always putting the tests into all images, this patch adds the
option of putting them only in some of the images, by making them into a
new module, modules/tests.

The default image (images/default.py) continues to require the tests
modules, so tests are included in the default image. Building with
"make image=tests" makes an image with only the tests and nothing else.
Other images (e.g., memcached, cassandra,...) currently do not require
the tests module, so the generated image does will not include the tests.

With this patch, "make image=memcached", for example, contains only the
bare minimum needed for memcached, and the resulting qcow image is just
18 MB, down from 69 MB when we included all the tests.

Fixes #160

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This reverts commit d914ee0d because it breaks
the build:

  I'm trying to compile the latest source code from git HEAD and I'm heaving
  the following error:

  make -r -C build/release/ all
  make[1]: Entering directory `/home/eduardo/workspace/osv/build/release'
  make[1]: *** No rule to make target
  `/home/eduardo/workspace/osv/build/release/gen/include/osv/version.h',
  needed by `/home/eduardo/workspace/osv/build.mk'.  Stop.
  make[1]: *** Waiting for unfinished jobs....
    GEN gen/include/bits/alltypes.h
  make[1]: Leaving directory `/home/eduardo/workspace/osv/build/release'
  make: *** [all] Error 2

Reported-by: Eduardo Piva <efpiva@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Since our kernel includes the entirety of libstdc++, there's no point of
specifying "-lstdc++" while linking shared objects (like tests and cpio.so):
This "linking" doesn't actually do any linking, it just specifies that the
libstdc++ library is needed, but we already have it, needed or not.

The list of objects-needing-libstdc++ was not only unnecessary, but probably
also out of date, since we stopped bothering to update it after libstdc++
was added into the kernel.

So this patch removes these -lstdc++ lines.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Force version.h generation to make sure the version number matches the
git version.

I noticed the problem after tagging v0.05-rc1 and noticing that OSv
happily reported the same version even after applying patches.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Current makefile cannot accept "make img_format=vmdk external all",
because QEMU does not support the format, but mkzfs.py and
upload_manifest.py requires to run OSv on QEMU.

With this patch, image conversion performs after mkzfs.py and
upload_manifest.py, prevents an error on QEMU execution.

Tested on QEMU/KVM and VMware, using these formats: raw / QCOW2 / VMDK.

Probably it can work on another formats such as VDI used by VirtualBox.

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The uname() function returns a fake Linux version number for application
compatibility.  Add a new osv::version() API that returns OSv version
that can be used by the management code.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
[ penberg: cleanups ]
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The best possible criteria for deflating balloons is heap pressure: Whenever
there is pressure in the JVM, we should give back memory so pressure stops.

To accomplish that, we need to somehow tap into the JVM. This patch register
a MXBean that will send us notifications about collections. We will ignore
minor collections and act upon major collections by deflating any existing
balloons.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch implements the JVM balloon driver, that is responsible for borrowing
memory from the JVM when OSv is short on memory, and giving it back when we are
plentiful. It works by allocating a java byte array, and then unmapping a large
page-aligned region inside it (as big as our size allows).

This array is good to go until the GC decides to move us. When that happens, we
need to carefuly emulate the memcpy fault and put things back in place.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently the parameter was read from the generated Makefile which was
not re-generated on incremental build. The fix is to move the default
to build.mk, this way the default will always be picked unless masked
by command line argument.

Fixes #153

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Fixes issue with JVM failing when started with a debugger
with the following message:

  NPT ERROR: Cannot find nptInitialize

Missing openjdk files in usr.manifest were a fertile source
of issues. This patch aims at making them less likely and
adding all files   except blacklisted files to the image.

This patch skips two files from JRE which are broken links
and inclusion of which would cause manifest upload failure:
 - jre/lib/audio/default.sf2
 - jre/lib/security/cacerts

These should be fixed incrementally.

Reported-by: Oded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

A previous patch renamed mutex.cc to spinlock.cc. This fixes the build.mk
dependency to make the code compile again... Sorry about that.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Commit 2ebfd915 ("build.mk: Generate a manifest of tests automatically")
separated native tests from Java tests but unfortunately broke the build
because it forgot to add dependency to the new java_tests variable.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>