Fix pthread_cond_timedwait to set the absolute timer using a timepoint,
instead of the old s64.

Moreover, now that we have both a wall-clock and monotonic clock, we
can support pthread_condattr_setclock, so this patch also adds this
support. OpenJDK 8, for example, cannot run without this support (it
assumes that if the OS supports CLOCK_MONOTONIC, it can also configure
condition variables to use it).

Unfortunately supporting pthread_condattr_setclock - the only condition-
variable attribute that really exists - grows the pthread condition
variable structure :(

Fixes #168.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Replace the old function condvar::wait(mutex*, uint64_t) with one taking
a timepoint. This timepoint can use any clock which the timer supports,
namely osv::clock::uptime or osv::clock::wall (as usual, wall-clock timers
are not recommended, and are converted to an uptime timer at the point
of instantiation).

Leave a C-only function condvar_wait(convar*, mutex*, s64) but comment on
what it takes.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Drop the nanotime() function.

Change the few remaining callers to use the appropriate osv::clock or
std::chrono replacements.

We already got rid in previous patches of most references to nanotime()
by switching from absolute times to relative times.

The direct equivalent of the old nanotime() function, where we actually
need the number of nanoseconds since the UNIX epoch, is the rather
verbose expression osv::clock::wall::now().time_since_epoch().count(),
or the shorter clock::get()->time().

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Drop the s64 literals _ms, _ns, etc., from <drivers/clock.hh>.
Fix a few places which still use the old literals.

The std:chrono::duration version from <osv/clock.hh> remains -
but remember you need to "using namespace osv::clock::literals"
to use them.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Fix tst-wait-for.cc to use the new <osv/clock.hh> APIs.

This test uses std::abs on a time duration, and unfortunately C++11 fails
to implement std::abs on an std::chrono::duration. This patch also adds
support for this (in the form of a trivial template function) to
<osv/clock.hh>.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Fix sbwait implementation to use the new <osv/clock.hh> APIs and the
monotonic clock.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reimplement the BSD functions getmicrotime(9), getmicrouptime(9)
and variable "ticks", using the new clock APIs.

getmicrotime() returns the system time ("wall clock"), while getmicrouptime
and ticks return the time since boot.

I believe this is the correct implementation according to the FreeBSD
documentation, but our previous implementation didn't quite do this and
it also worked ;-) The previous implementation pretended, according to
getmicrouptime() and get_ticks(), that the system is up since 1970,
and yet the variable "time_uptime" (which FreeBSD has) is never updated,
and is fixed at 1 second :-)

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Fix msleep implementation to use the new <osv/clock.hh> APIs and the
monotonic clock.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Change callout implementation to use the new <osv/clock.hh> APIs and the
monotonic clock.

Since _callout.h now uses the C++ type osv::clock::uptime::time_point,
it can only be used from C++ code. All the relevant code is already C++.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Change alarm() implementation to use the new <osv/clock.hh> APIs and the
monotonic clock.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Delete the sched::thread::sleep_until() function. All users of this
function actually wanted a relative time, not absolute time, and can
use the simpler new sched::thread::sleep() instead.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

C++11's std::chrono allows clearly differentiating between absolute times
(std::chrono::time_point) and relative times (std::chrono::duration),
and we are no longer at risk of confusing absolute and relative time.

This patch allows various functions to also take relative times (duration)
instead of absolute times (time_point), making them easier to use.
All of them use the monotonic clock's current time (osv::clock::uptime::now)
to convert relative to absolute times:

1. Shortcut tmr.set(10_ms) for tmr.set(osv::clock::uptime::now() + 10_ms)

2. Similarly, we add a duration version of condvar::wait(), which can now
be used as condvar::wait(mutex, 10_ms).

3. Add a new function sched::thread::sleep(10_ms).

These new variants are not only shorter, they also make it unnecessary for
the caller to understand what clock it should use for relative waits,
or mistakenly use the wrong clock.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

In tst-timerfd, test timerfd with monotonic clock, in addition to the
existing test with the realtime clock.

This patch also changes this test to only use Linux APIs, not anything
OSv-specific, so it can also be run on Linux.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Switch the timerfd to use std::chrono types, and add support also for the
monotonic clock.

Fixes #142.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Switch the thread scheduler from using the s64 type for durations and
the wall time, to the osv::clock::uptime::duration type (which is
std::chrono::nanoseconds) and monotonic clock.

Also, now that the per-thread CPU-time clock (thread::thread_clock())
returns an std::chrono::duration instead of s64, we no longer need the
fill_ts(s64) variant in libc/time.cc (if we leave it unused, we'll get a
compilation warning).

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Durations cannot be compared with unit-less integers: one cannot just
check d < 7: One needs to do something like d < 7_ms.

Unfortunately, this rule also applies to zero: d < 0 won't work, and users
need to add a unit to the zero. Of course, any unit would work: d < 0_ms,
or d < 0_ns, or any other time unit. For physicists, this requirement
makes perfect sense (even zero has units), but for most programmers, the
extra units would seem surprising and redundant.

This patch makes it possible to compare a std::chrono::duration to
0 without needing to add a unit, while still leaving comparison with a
non-zero integer an error.

This is convenient for us because existing code which compared s64
durations to unit-less 0, will continue to work unmodified after we
switch from s64 to std::chrono::duration.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

sched::timer_base, used to implement all forms of timeouts in OSv, is
currently based on the wall-clock time in s64 form. This is problematic
for two reasons:

1. The type s64 doesn't say which units should be used, whether the time
is absolute or relative or what is its epoch.

2. wall-clock time is a bad choice for short-term timers: If a thread
intends to sleep for a millisecond, and the wall-clock goes back a minute,
the thread will end up sleeping a whole minute.

So this patch changes the basis of sched::timer_base to strongly-typed
time points from a monotonic clock, osv::clock::uptime::time_point.

We also allow setting timers using the wall-clock time, but with a big
caveat: The expiration time is converted from wall to uptime clocks at
the moment of timer_base::set(), so if the wall-clock is adjusted later,
the wall time at expiration may not be exactly the one intended.

So that we don't have to change all timer users in this one patch, we
also temporarily implement the old weakly-typed timer_base::set(s64)
using the new mechanism. This variant will be removed in a later patch
in this series, when it is no longer used.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Unfortunately, sched.cc uses "using namespace osv" causing a big mixup
between ::clock (from <drivers/clock.hh>) and osv::clock (from
<osv/clock.hh>). In the next patch, we want to start using <osv/clock.hh>
in sched.cc, so we need to first clear this mixup by temporarily adding
an alias osv::clock::get(), to resolve the clock::get() used throughout
sched.cc.

In later patches in this series, we'll remove all the calls to clock::get(),
from sched.cc, so at the end of the series we revert this patch.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently, OSv's sched::timer mechanism supports only the wall clock
clock::get()->time(). In the following patches, we intend to support both
wall-clock and monotonic clock in timers, but only keep a single list of
timers (using the monotonic clock).

For this, we need a way to convert wall-clock time to monotonic uptime.
This patch adds to the clock driver a function clock::boot_time() which
returns the current offset between the wall time and the uptime, i.e., the
current estimate of the wall time at boot. Note that this estimate can
change if the wall time is adjusted (in the host, since OSv currently has
no mechanism for adjusting the clock from within).

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

In the previous patch we built clock_event::set(s64) on top of
set(duration). In this patch we add set(timepoint), for absolute time
in any clock supporting the now() static method. In the following patches,
we'll convert our timers to using the monotonic clock osv::clock::uptime,
and use timepoints of that clock in calls to clock_event::set().

By temporarily having both s64 and std::chrono:time_point versions
of set(), we make this patch series bisectable. At the end of this series,
the s64 variant will be removed.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

OSv's timer mechanism hinges on the Local APIC's (per-cpu) one-shot timer,
which delivers an interrupt after the requested number of nanoseconds.

The API to set this timer, clock_event::set(), took the absolute time of
the next interrupt. However, what it really needs is the duration in
nanoseconds until the next interrupt.

So this patch we change the basic clock_event::set() API to take a
duration, and implement the original clock_event::set(s64) - taking an s64
absolute wall-clock time - as a simple wrapper.  The next patch will add
more wrappers for set() taking absolute times from different clocks.
Later patches in this series will stop using the old set(s64) version,
until it is dropped in the end of the series.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Several source files include <drivers/clockevent.hh>, though this is a
very low-level feature which they don't actually use.

sched.cc does use <drivers/clockevent.hh>, but already gets it through
sched.hh, so also doesn't need to include it explicitly.

This patch removes the unnecessary includes.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Rename tests/misc-bsd-callout.c to tests/misc-bsd-callout.cc - we'll
need it in C++ in the upcoming patch set.

No changes were needed for this code to continue compiling.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

As we have recently discovered, some parallel GCs will move an object to two
different locations at times, and later on decide on which one to use. This
breaks our implementation if the final object is the second one to be copied,
because by then the original region is already mapped - so we won't fault, and
the unmapped region will not be the actual balloon, so we will have a bogus
fault

The core of this solution is to keep all the regions unmapped. Because they had
only garbage before, we know Java shouldn't read anything from it before it
writes something new. And when it does that, we declare that to be no longer a
balloon.

Movement is then split in two phases: the normal phase, and the finish phase.
In the finish phase we will remove the old VMA and create the new VMA again,
with heap characteristics.

Special care needs to be taken when "conciliating" the array: because we use
the difference between first faulting address and original array address to
calculate how many bytes we are skipping, we need to store that information
somewhere. We're using an unordered_map (hash) for that. We'll keep track of
all in-flight ballooned regions and hold the original address of the array.

When we detect movement *from* that region, we know it is the new location
and update the balloon object with the new address.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

To avoid using hard code values for the original anonymous vma that mapped the
region before we ballooned, let's store the original flags in the JVM vma. This
patch does not yet use it, but only lays down the infrastructure. User will
come in the next patch.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch introduces a separate operation, "conciliate", that calculates
the balloon parameters given an arbitrary address

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Up until now the JVM balloon explicitly forbade writes to its range, because
we didn't expect the JVM to ever write to it. But with the recently problem
Gleb discovered of double-copying of objects inside the Heap, we will use
writes to figure out when that object is no longer a balloon. But still,
we need to be able to go back to the JVM specific fault handler for that.

Therefore, we need write permissions in the VMA itself.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
[ penberg: use perm_rw as suggested by gleb ]
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

We should always remove the last balloon we've created. There are two main
reasons for that:

1) The older balloons are likely to be already in more tenured generations,
and will move less, whereas younger balloons could be in younger generations.
So by removing them like a stack, we'll avoid needless moves

2) The probe, when inserted, should stay for as long as we can.

That was always the intended behavior but I made the small mistake of inserting
them in the wrong order. Fix that.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

calling debug() and then abort() buffers a message that never
has a chance to reach the console early on;
pass the message to display directly to abort() instead.

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

BUILD SUCCESSFUL

Total time: 35.396 secs
make -r -C build/release/ all
make[1]: Entering directory `/home/zwu/osv/build/release'
  CXX loader.o
  CXX runtime.o
  CXX drivers/vga.o
  CXX bsd/net.o
  CXX bsd/porting/networking.o
/home/zwu/osv/bsd/porting/networking.cc: In function ‘int osv::if_set_mtu(std::string, u16)’:
/home/zwu/osv/bsd/porting/networking.cc:43:32: error: missing braces around initializer for ‘char [16]’ [-Werror=missing-braces]
cc1plus: all warnings being treated as errors
make[1]: *** [bsd/porting/networking.o] Error 1
make[1]: Leaving directory `/home/zwu/osv/build/release'
make: *** [all] Error 2

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

  CC bsd/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/zvol.o
  CXX core/mmu.o
/home/zwu/osv/core/mmu.cc: In instantiation of ‘void mmu::map_level<PageOp, ParentLevel>::operator()(mmu::hw_ptep, uintptr_t, uintptr_t) [with PageOp = mmu::virt_to_phys_map; int ParentLevel = 4; uintptr_t = long unsigned int]’:
/home/zwu/osv/core/mmu.cc:323:5:   required from ‘void mmu::map_range(uintptr_t, size_t, PageOp&, size_t) [with PageOp = mmu::virt_to_phys_map; uintptr_t = long unsigned int; size_t = long unsigned int]’
/home/zwu/osv/core/mmu.cc:623:43:   required from here
/home/zwu/osv/core/mmu.cc:383:13: error: comparison between signed and unsigned integer expressions [-Werror=sign-compare]
...

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch addresses a corner-case in our zfs_inactive which can potentially
leak a znode object.

*** Some background on znode/zfs_inactive ***
- Used to deallocate fs-specific data.

- Before destroying the znode, a DMU transaction is created to sync the znode
to the backing store *if* its z_atime_dirty is set (Not relevant to this
patch though).

- When removing a link, zfs_remove sets the field zp->z_unlinked of the
underlying znode if the number of links reached 0 (Simply put, not present in
the fs anymore).

*** The problem ***
The actual problem shows up when zfs_inactive is used on znodes with the
unlinked field set.

The code wrapped around by this patch was previously added to speed up the call
to vrecycle, whose name partially explains itself. Its first functionality is
to eliminate all activity associated to the vnode, then put the vnode back into
a list of free vnodes.

OSv VFS layer doesn't support vrecycle, but our zfs_inactive is acting as if it
were supported. Another thing is that vrecycle call was also removed.

*** Solution ***
Let's fix this problem by simply wrapping around the test which prevented
zfs_inactive from working properly on unlinked znodes, thus leaking references
to the underlying mount point afterwards.

The commentary added into zfs_inactive also explains why these changes are
needed. It would also make things easier when people look at it in the future,
and try to understand why things are the way they are.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The zfs_remove() function calls zfs_dirent_lock, which in turn calls
zfs_zget() which bumps up the underlying znode reference count once.

However, neither zfs_remove() or zfs_rmdir() release the reference count
after using it. This prevents zfs_zinactive() which is used to destroy
the znode object from working properly. Another consequence is that each
znode holds a reference to the underlying mount point, keeping it busy
for unmount.

Fix the znode refcnt by calling zfs_zinactive after znode usage.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

In the Http server implementation, some of the code was
based on an example from the boost library.

The boost license itself was not included in the license
directory.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

There were 2 places with ioctl definitions, Xen netfront driver
was compiled with IOCTL definitions from wrong place.

Fixed by changing include and deleting file with improper definitions

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add mruby on apps, support make all image=mruby.

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Functions which do not modify input parameters are easier to analyze
and more friendly for its users.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

To start the JVM with debugger server just run:

  sudo scripts/run.py -nv --jvm-debug

If you want the JVM to wait for debugger client:

  sudo scripts/run.py -nv --jvm-suspend

Listens on port 5005.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>