Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add a type-safe std::chrono interface to the wall-clock time. E.g., use
use osv::clock::wall::now() to get an std::chrono::time_point instead
of clock::get()->time() which returns an s64.

This patch also changes clock_gettime(CLOCK_REALTIME) and gettimeofday()
to use the new osv::clock::wall::now() interface.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

In <drivers/clock.hh> we had support for time literals looking like 100_ns,
100_us, 100_ms, 100_s. These simply translated to an integer number of
nanoseconds.

Added to <osv/clock.hh> similar literals, but using strongly-typed
std::chrono::duration, so that 100_ms translates to
std::chrono::milliseconds(100), and this is, as usual, automatically
scaled by the compiler to other units when other units (e.g., nanoseconds)
are required.

Because the literals of <drivers/clock.hh> and <osv/clock.hh> have the
same name, to use the latter you need to explicitly enable them in a scope
using the statement

    using namespace osv::clock::literals;

One the entire codebase is converted to use <osv/clock.hh> and std::chrono,
the old literals from <drivers/clock.hh> should be removed.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The timestamp typically will come from 'trace.py list' output.  This
change makes it easier to paste the timestamp. Listing prints the time
with '.' separating seconds from nanoseconds.  The options '--since'
and '--until' accept timestamp in nanoseconds so the dot must have
been removed. Now one only needs to append the 's' suffix which
changes the unit to seconds.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This:

  --since 10.0 --period 100ms

is now equivalent to:

  --since 10.0 --until 10.1

This option makes it easier to slice the time based on the output of
'trace.py list-timed' which prints trace timestamp and duration.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Removing 'osv trace summary' and 'osv trace duration' commands from
loader.py. The first is added as 'trace.py summary --timed', the
second one is added as 'trace.py list-timed'.

It was easier to move both in one commit as they are coupled with each
other.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The call-graph tree is already using call-sites, this makes also
`trace.py list -b` also use it.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Prints output like this:

Collected 36843 samples spanning 8.85 s

Tracepoint statistics:

  name              count
  ----              -----
  sched_wait        18420
  sched_wait_ret    18423

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

When output is not fully consumed because of piping, eg to `head`,
we get an ugly exception:

 Traceback (most recent call last):
   File "scripts/trace.py", line 274, in <module>
     args.func(args)
   File "scripts/trace.py", line 227, in list_timed
     bt_formatter(t.backtrace),
 IOError: [Errno 32] Broken pipe

Let's ignore it, it's normal condition.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

 - added command descriptions
 - improved clarity of options
 - extracted argument groups
 - changed command help to start with lower case letter

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It's a left over which is not used any more.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Our clock::time() function returns the current wall-clock time (number of
nanoseconds since the Unix epoch). We never clearly documented this, so this
patch adds this documentation.

Moreover, in kvmclock, we assumed that the host never adjusts its
wall-clock time, which is not a good assumption because it prevents us
from benefiting from NTP running on the host. As Avi Kivity explains:

   "For the wall clock, you need to sample the wall clock base every time,
    since it can be changed by the host."

So this patch changes kvmclock::time() to sample the wall-clock-at-boot
reported by the paravirtual clock, on every time() call, and not just once.

The downsite of this patch is that clock::get::time(), and therefore
nanotime(), become a bit slower on kvmclock, as they need to sample
the wall_clock_boot() every time, require a couple of barriers and
arithmetic operations. But note that we shouldn't even be using the
non-monotonic clock::get::time() for our timers - we should be using the
monotonic clock::get::uptime() - and when we switch to using that we'll
get back the bit of performance lost in this patch.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

httpserver.so does not build propery for many:

  LINK httpserver
  /usr/bin/ld: cannot find -lboost_system-mt
  /usr/bin/ld: cannot find -lboost_filesystem-mt
  /usr/bin/ld: cannot find -lboost_program_options-mt
  collect2: error: ld returned 1 exit status

drop it from osv.git build until it does.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The httpserver is part of the compilation of the mgmt.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This reverts commit a39cde78. I applied
the broken version.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The httpserver.so is added to the image and will be
placed under /usr/mgmt

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The httpserver is part of the compilation of the mgmt.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

In preparation for adding httpserver.so to mgmt module, update to latest
mgmt.git which fixes unsupported boost_program_options library
dependency to boost_program_options-mt.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Instead of always putting the tests into all images, this patch adds the
option of putting them only in some of the images, by making them into a
new module, modules/tests.

The default image (images/default.py) continues to require the tests
modules, so tests are included in the default image. Building with
"make image=tests" makes an image with only the tests and nothing else.
Other images (e.g., memcached, cassandra,...) currently do not require
the tests module, so the generated image does will not include the tests.

With this patch, "make image=memcached", for example, contains only the
bare minimum needed for memcached, and the resulting qcow image is just
18 MB, down from 69 MB when we included all the tests.

Fixes #160

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This reverts commit d914ee0d because it breaks
the build:

  I'm trying to compile the latest source code from git HEAD and I'm heaving
  the following error:

  make -r -C build/release/ all
  make[1]: Entering directory `/home/eduardo/workspace/osv/build/release'
  make[1]: *** No rule to make target
  `/home/eduardo/workspace/osv/build/release/gen/include/osv/version.h',
  needed by `/home/eduardo/workspace/osv/build.mk'.  Stop.
  make[1]: *** Waiting for unfinished jobs....
    GEN gen/include/bits/alltypes.h
  make[1]: Leaving directory `/home/eduardo/workspace/osv/build/release'
  make: *** [all] Error 2

Reported-by: Eduardo Piva <efpiva@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Our need for "make external" was plagued with problems. First and formost,
it was annoying - people need to be told to run "make all external" instead
of just "make". And as a Vlad recently discovered, it strangely relied on a
debug-mode build to have happened, and also "make clean" doesn't undo what
"make external" did.

"make external" had a glorious past, but today all that was left of it
is 3 measly tests which we compile from external/glibc-testsuite: malloc,
getcwd and clock. These tiny tests - 3 out of 796 tests (!!) in the full
glibc test suite - aren't really important enough to cause all this
"make external" mess. So this patch removes them, and the need for a
"make external".

If we'll ever want to bring back these tests, I think we need to delete the
partial external/glibc-testsuite directory, and bring the *full* glibc
test suite of 796 tests, into a separate module. The glibc test suite can be
downloaded together with the glibc source code - they are all files named
tst-*.c in the glibc source tree.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

After the creation of the "java" and "mgmt" modules, we still left with a
bunch of files in the main usr.manifest.skel which really belong in the
modules/{java,mgmt}/usr.manifest.

This patch moves various shared libraries and fonts which Java needs
(the fonts are needed for AWT, including headless AWT), as well as
various JNI shared objects and our java.so and runjava.jar, into the
"java" module, and moves a couple of tools (ifconfig, lsroute) and
configuration file (/etc/inputrc used by the management module into
the "mgmt" module.

This reduces small OSv appliances, like "make image=memcached", which
don't need Java, by 13 MB. We are still left with 69 MB, most of which
are tests, so the next item on the todo is to move the tests into a
"test" module.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

tst-vfs.cc currently stat()s the file
	/usr/lib/jvm/jre/lib/amd64/headless/libmawt.so
And dies if it doesn't exist.

Since Java is now optional in our images, it's not a good idea to check
for such a file, which might not exist (e.g., "make image=tests check"
will fail). This patch changes it to check a filename that is certain to
exist, like namely the test itself - /tests/tst-vfs.so.

If we wanted to have a pathname with more components, the test should
be rewritten to create this pathname, say /a/a/a/a/a/a/a/a/a/a, and then
test stat on that newly created file. It cannot rely on such a file to
pre-exist.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Since our kernel includes the entirety of libstdc++, there's no point of
specifying "-lstdc++" while linking shared objects (like tests and cpio.so):
This "linking" doesn't actually do any linking, it just specifies that the
libstdc++ library is needed, but we already have it, needed or not.

The list of objects-needing-libstdc++ was not only unnecessary, but probably
also out of date, since we stopped bothering to update it after libstdc++
was added into the kernel.

So this patch removes these -lstdc++ lines.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

For every OSv specific ifdef we remove in ZFS, God ressurects a kitten.

After some recent additions and fixes, that piece of code can now be
compiled out. It is the reclaimer code, so it is very welcome.

But beware: that does not means we are reclaiming yet. That only means that we
wired up the ZFS ARC reclaiming process to the BSD notification system.

We now need to somehow wire that notification system with the OSv shrinking
infrastructure. That is the easy part. And after that, of course, balance the
calls between ARC and balloon. That is the hard part.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

ZFS will perform some checks to determine if the current calling "process"
is the reclaimer. Export the address of the reclaimer thread so that test
can work.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

There is (currently commented out) code in ZFS that checks things like:

    if (curproc == pageproc) {
        /* Do something really great */
    }

The problem is that with our current implementation of curproc (designed for Xen)
that will break, because we will return a pointer to a in-stack variable that is
created on-demand and only contains the pid of the process.

Returning the thread address will make those checks works, but we will be forced
to give up on accessing fields inside it altogether. If we *really* must, we can
have a structure that have the fields in the same offset as our thread class.

But our thread class is defined in a .hh file, so *good luck* calculating the
offset of a field (say, id) at compile time so we can include in this other .h
file that contains exclusively C code. Since xen is the only user of the PID test,
and our resistance to changing the xen code is quite low (if at all), I'll just go
ahead and change it: storing the address of the process itself should allow us to
do compare tests the way ZFS does and get everything working.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

First of all, I am sorry. I am sorry Avi, Dor, Pekka, God, Dennis Ritchie, et caterva.
I am so very sorry. This is probably one of the ugliest things ever written by a C
programmer in the history of programming.

The story is: ZFS defines its own mutex of type kmutex_t, which is basically just a OSv
in our implementation. In a piece of code currently commented out (not for long), it calls:

    msleep(&needfree, &arc_reclaim_thr_lock, 0, "zfs:lowmem", 0);

The problem is that our msleep implementation expects a "struct mtx" which is our own
wrapper around mutex (Maybe that should be changed? Does anybody remember why it was
done this way?)

Keep in mind that we are going to great lenghts not to change ZFS (ifdefing code out
is generally fine), so the casting solution could not be used. I've tried to change the
for-ZFS definitions of mutex in the BSD glue code, but then, after a couple of hours
I was still resolving conflicts with all the other parts that would break because they
were expecting a certain type that was now changed.

I eventually set for the current ugly but functional solution: code msleep in a
way that it can accept any kind of mutex. That is really ugly because by "any
kind of mutex" I really mean any kind of crap the user passes and good bye type
safety altogether. But it works with minimal changes, and more importantly, with
all the changes being in *our* glue code.

If anybody have other ideas, I would be happy to try them out. But at this time,
I believe that to be the best compromise.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Force version.h generation to make sure the version number matches the
git version.

I noticed the problem after tagging v0.05-rc1 and noticing that OSv
happily reported the same version even after applying patches.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Current makefile cannot accept "make img_format=vmdk external all",
because QEMU does not support the format, but mkzfs.py and
upload_manifest.py requires to run OSv on QEMU.

With this patch, image conversion performs after mkzfs.py and
upload_manifest.py, prevents an error on QEMU execution.

Tested on QEMU/KVM and VMware, using these formats: raw / QCOW2 / VMDK.

Probably it can work on another formats such as VDI used by VirtualBox.

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch is not needed anymore since alarm() implementation was
extended with missing features.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>