Lightweight waitqueue and wait_for() primitive.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Dor Laor writes:

  - Ballooning causes to issues and we like to disable it:

    1. Parallel GC
        2 GC threads are trying to move memory around and they
        do it optimistically without locking. This fools our memmov logic.
        Gleb and Glauber have a solution (to use GCmonitor) but will take
        time to implement.

    2. Logger clash
        The existing GCmonitor bean access the logger early and we don't
        have the tomcat logger handy. Tomek is working on a solution but
        it will take few days to complete.

Therefore, disable the JVM balloon for v0.05.

Acked-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Allow arbitrary predicates to be waited on.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Add sockbuf::sb_cc_wq for using instead of msleep(&sb->sb_cc).  This paves
the way for lockless wakeups for net channels, as the wait primitive is now
thread::wait_for() instead of msleep().  In addition this reduces lock
acquisition and improves netperf bandwidth by about 10%.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

This was used to ensure all socket-using code was converted to C++, but not
cleaned up later.

Clean it up now.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

A waitqueue is an object on which multiple threads can wait; other threads
can wake up either one or all waiting threads.  A waitqueue is associated
with an external mutex which the user must supply for both wait and wake
operations.

Waitqueues differ from condition variables in three respects:
- waitqueues do not contain an internal mutex.  This makes them smaller, and
  reduces lock acquisitions.  On the other hand the waker must hold the
  associated mutex, whereas this is not required with condition variables.
- waitqueues support sched::thread::wait_for()

waitqueues support wait morphing and do not cause excess lock contention,
even with wake_all().

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

This adds a facility to wake a thread, but with the intention that it will
acquire a certain lock after waking, and while the waker holds the lock.
This is implemented using the regular wait morphing code (send_lock() and
receive_lock()), but with additional mutual exclusion to allow regular
wake()s in parallel.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Add a signal catching class (signal_catcher).

Use signal_catcher::wait() to wait for a signal.
Use sched::thread::wait_for(..., signal_catcher&, ...) to wait for a signal
in addition to other events.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Waiting is complicated; you can wait for a condition variable, a timer,
wait queues and signals (when we have them), semaphores, and more.

When you need to wait for multiple events, things become even more
complicated as some sort of abstraction is needed.  poll() for example
supports waiting for multiple objects that conform to the fd interface,
or for a timeout.

To allow more flexible waiting, this patch adds a poll-like interface
to the scheduler, except that the interfaces are all compile time.

We add a new function wait_for() that accepts a variable number of waitable
objects.  A waitable object is an object that supports the three methods:

 - poll(): check if waiting is complete
 - arm(): prepare for waiting
 - disarm(): finish waiting

to make a waitable object usable with wait_for, these functions have to be
supplied.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Remove the extra check on size just like the remark above implies.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The check is moved to bdev_read and bdev_write.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

setup_features() is called twice, once in virtio_driver constructor and
once in virtio device driver constructor. Doing it in driver constructor
is enough.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

This would be useful for device with mmio bar, e.g. AHCI controller.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

The OSv loader has a "--verbose" option to enable debug() output to be
printed to the console and not just to a memory buffer. However, needing to
pass this option in the "-e" string is annoying, and even worse when you want
to use the default command line and didn't even want an explict "-e" option.

So this patch adds a --verbose to run.py, which can be used whether or not
a "-e" option is given. For example:

    run.py --verbose
or
    run.py --verbose tests/tst-timerfd.so

As a shorthand for --verbose, I used "-V". Unfortunately, lowercase "-v"
was already taken (by the "vhost" option).

Fixes #173.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

As issue #128 explains, it is annoying that once the "-e" option was used
in run.py, it overrides the default command line, so that running run.py
again without -e reruns the same modified command line, and not the default
one as usual.

This simple patch changes this behavior: On every run of run.py, the command
line is set - if "-e" is given it is set to the given command line, but if
"-e" is absent, the command line is reset to the default one. This is
basically the behavior that people always wanted from run.py.

This patch, and the behavior it performs, are much simpler than any of
the alternatives proposed in issue #128.

Please don't complain that this can only be used for development, and not
for EC2, for creating images to distribute, and so on. After all, this is a
patch to *run.py*, and it is obviously not relevant to anyone who is running
an OSv image not through run.py. This patch only makes run.py more convenient
to use.

Fixes #128.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

sysinfo() until now was a stub. This patches fills some of the information
we can already provide, namely the uptime in seconds (through the recently
added osv::clock::uptime()), the total and free memory (through the recently
added memory::stats::free()/total() functions), and number of processes
(always 1 on OSv).

This patch also changes a couple of types from unsized names (like "unsigned")
to fixed-sized ones (like "u32"), to match the definition of this structure
on Linux. It won't make any difference on x86_64, but perhaps in some future
architecture it will.

Tested-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

sysinfo() used sizeof(struct sysinfo) to decide how many zeros to write.
The problem is that our struct was much larger than Linux's (we had a much
larger padding at the end), so if we run code compiled on Linux, that code
would allocate a small structure on the stack and then zero a much larger
structure, causing the stack to be corrupted.

Drop the "reserved" field making our structure larger. It's not needed.

Fixes #174.

Tested-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Fix object::lookup_addr to lookup correct symbol.
It should returns the nearest symbol which is s_addr < addr, but it
compares opposite way.

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

This patch introduces support for MTU option as described in
RFC2132, chapter 5.1. Interface MTU Option

Amazon EC2 networking uses this option in some cases and it gives
throughput improvement of about 250% on big instances with 10G networking.

Netperf results for hi1.4xlarge instances, TCP_MAERTS test, OSv runs netserver:

Send buffer size     Throughput w/ patch (Mbps)     Throughput w/o patch (Mbps)     Improvement (%)

32                   4912.29                        1386.28                         254
64                   4832.01                        1385.99                         249
128                  4835.09                        1401.46                         245
256                  4746.41                        1382.28                         243
512                  4849.04                        1375.23                         253
1024                 4631.8                         1356.69                         241
2048                 4859.59                        1371.92                         254
4096                 4864.99                        1383.67                         252
8192                 4627.07                        1364.05                         239
16384                4868.73                        1366.48                         256
32768                4822.69                        1366.63                         253
65536                4837.67                        1353.87                         257

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Fixes #116.

Because modules add files to which cannot be handled by our build
system incremental build fails with messages like this:

  make[1]: *** No rule to make target `/home/tgrabiec/src/osv \
     /apps/cassandra//upstream/jna-3.`

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Total time: 30.77 secs
make -r -C build/release/ all
make[1]: Entering directory `/home/zwu/osv/build/release'
  CXX bsd/sys/xen/gnttab.o
In file included from /home/zwu/osv/bsd/sys/xen/hypervisor.h:40:0,
                 from /home/zwu/osv/bsd/sys/xen/gnttab.cc:29:
/home/zwu/osv/bsd/machine/xen/hypercall.h: In function ‘int HYPERVISOR_set_trap_table(const trap_info_t*)’:
/home/zwu/osv/bsd/machine/xen/hypercall.h:146:9: error: unable to find string literal operator ‘operator"" STR’
/home/zwu/osv/bsd/machine/xen/hypercall.h: In function ‘int HYPERVISOR_mmu_update(mmu_update_t*, unsigned int, unsigned int*, domid_t)’:
/home/zwu/osv/bsd/machine/xen/hypercall.h:154:9: error: unable to find string literal operator ‘operator"" STR’
/home/zwu/osv/bsd/machine/xen/hypercall.h: In function ‘int HYPERVISOR_mmuext_op(mmuext_op*, unsigned int, unsigned int*, domid_t)’:
......
make[1]: *** [bsd/sys/xen/gnttab.o] Error 1
make[1]: Leaving directory `/home/zwu/osv/build/release'
make: *** [all] Error 2

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This reverts commit 59563357 which
causes significant slowdown on first time boot to a QCOW2 image:

  [penberg@localhost osv]$ time ./scripts/run.py
  OSv v0.05-rc1-108-g0639dd8
  OpenJDK 64-Bit Server VM warning: Can't detect initial thread stack
  location - find_vma failed
  ^D^D^D
  [/]%

  real    0m9.954s
  user    0m4.958s
  sys     0m1.309s

  [penberg@localhost osv]$ git revert
  59563357
  [penberg@localhost osv]$ time ./scripts/run.py
  OSv v0.05-rc1-108-g0639dd8
  OpenJDK 64-Bit Server VM warning: Can't detect initial thread stack
  location - find_vma failed
  ^D^D^D
  [/]% ^D

  real    0m4.441s
  user    0m3.785s
  sys     0m0.530s

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add a simple manual test case for checking "/proc/self/maps" output.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add a '/proc' directory to OSv image and mount procfs on it.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch adds a simple Linux compatible procfs filesystem.  It
currently implements a "/proc/self/maps" file which is looked up by
OpenJDK during startup.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add procfs_maps() function to core/mmu.cc that returns all the VMAs
formatted for Linux compatible "/proc/<pid>/maps" file.

This will be called by the procfs filesystem.

Limitations:

  * Shared mappings are not identified as such.
  * File-backed mmap offset, device, inode, and pathname are not
    reported.
  * Special region names such as [heap] and [stack] are not reported.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add 'struct file' to VOP_READ API. This is needed for procfs which
generates file contents at open() time and read() must operate on it,
not the vnode.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently, RLIMIT_STACK is 64 KB.  This is too small for the JVM which
limits stack size to RLIMIT_STACK if a VMA range that covers address
found in '__libc_stack_end' in /proc/self/maps.

In preparation for procfs support, switch to pthread_attr_getstacksize()
for RLIMIT_STACK.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add pthread_attr_getstacksize() in preparation for fixing RLIMIT_STACK.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

As noted by many of you, we need a mechanism to control whether or not the
balloon mechanism is enabled. A trivial way is to allow for a boot option that
will control it, and that may (or may not) be the right route in the future.

But (at least) right now the safest way to proceed is to view -Xmx as a
declaration of intent. In that case, we disable the balloon, leaving it enabled
only for people who didn't went through the trouble of calculating a value.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This code should not be here. I am 100 % positive that I removed it in my
testings, but I must have forgotten to git add the remove before I sent out the
patch and it ended up in tree.

This is simply a test leftover, it will have the effect of having threads to
loop forever and never waiting because the initial value won't be 0.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

isa-serial.hh don't need to include vga.hh, but it's required on
console.cc, so move it.

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>