Since commit 54d6472ec67ea73e23b30b1534320fbe637, we need the Maven "shade"
plugin. This is yet another dependency we need to list in README.md...

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Stripping tests breaks "make check":

  TEST tst-af-local-stripped.so OSv v0.05-386-gdf0fd92
  run_main(): cannot execute /tests/tst-af-local-stripped.so. Powering off.
  Test tst-af-local-stripped.so FAILED
  make: *** [check] Error 1

Fix the problem by not stripping tests.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently files which are part of 'java' module are built as part of
the main osv build. These files need to be ready before the module is
processed because the module relies on them.

Because we didn't have explicit dependency between processing modules
and building java artifacts the build failed when parallel jobs were
used. The proper fix is to move all java stuff to modules/java so that it
would be built as part of the module build. Before that happens here's
a quick fix which adds explicit dependency to build.mk.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Commit 211bb7fb introduced masking of
system classes from the isolate point of view in order to prevent
runjava dependencies form becoming visible, which might cause
conflicts if application would like to use the same classes but of
different version. Currently the dependencies include cglib and asm
which are also used by spring framework and hibernate. Making them
visible would force the applications to use our version of these
libraries, which can be older or newer than the required one.

Unfortunately this causes extension classes from jre/lib/ext to be
unavailable to the application. They are part of the system class
loader, not bootstrap. One of the side effects is that security
providers are not found (See issue #208).

This fix chooses another approach to fix the problem. Instead of
masking dependencies we move them under io.osv.* namespace so that
they will not cause conflicts. For this task I use apache plugin for
maven which automatically renames packages in our uber runjava jar.

Fixes #208.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

With this patch we strip all .so included in our build images,
regardless if it's a debug or a release build.

Basically any object ended with .so is stripped and sent to the
image without the -strip suffix. The code will check if there isn't
already an stripped object and if timestamp matches it, so it won't
strip unecessary files.

Everything outside build/target is ignored.

Fixes #133

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Eduardo Piva <efpiva@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

In the process of making the lzloader, I noticed that boot.S and boot16.S
uses tabs.

Changes done using regexp.

Signed-off-by: Eduardo Piva <efpiva@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

On build.mk two image targets are created: lzloader.elf and lzloader-stripped.elf.

The lzloader-stripped.elf is not the imaged included on loader.img. For this process
we need to call the fastlz/lz tool and link it with fastlz/fastlz.o, so those dependencies
are added.

We're calling our lzloader from boot16.S, hence we need to compile all fastlz related
stuff as -m32, so we have 32 bits binary code. This design was used in order to
do as minimal changes to the OSv core as possible and since on boot.S we have
all page tables, GDT and so on, all this are also compressed and remain on the
same address as before lzloader was introduced.

With this design it's easier to revert this patch (if needed) and also
we could add a make target to build OSv without compression (with such scenario exists)
with minimal effort.

The address 0x1800000 is choosen so we still have enough memory to uncompress the kernel
and at the same times make OSv boot with less then 32Mb of memory. If some day this
is not enough to decompress the kernel, build will fail.

Signed-off-by: Eduardo Piva <efpiva@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This script is called from build.mk to check if enough space exists for kernel
decompression. If not, break the build.

Signed-off-by: Eduardo Piva <efpiva@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The lzloader itself is very simple, it read all data that was
compressed using the lz command line tool and uncompress it on
the address the kernel should be running, i.e.: 0x200000.

The MAX_BUFFER is the address range from 0x200000 to 0x0x1800000.

_binary_loader_stripped_elf_lz_start is an extern symbol to the
compressed kernel image. This is created during the build process
using objcopy command line tool.

Signed-off-by: Eduardo Piva <efpiva@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Fastlz library provides two default tools for compression and uncompression:
6pack and 6unpack. We are not using those tools since they work with chuncks
and checksums, because they assume we have a restriction on buffer size
(because the most common usage for compression tools is reading from disk
and writting it back the compressed image).

Since we're working with all the image in memory and our output buffer
is the memory itself, we can uncompress all the data at once, making it
a much simpler and faster solution.

But, to do that, we must compress the image in the same way, that's why
this simple command line was written.

Signed-off-by: Eduardo Piva <efpiva@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Added source code of fastlz from fastlz.org.

The original library is written in C, but it was also C++ compatible,
hence in this patch I added as a C++ source code and without
preprocessor and extern "C" code.

Signed-off-by: Eduardo Piva <efpiva@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

_flags is currently being used to determine which page_ops will be assigned to
the anonymous VMA, however, the variable flags received as a parameter is the
one that must be used for that purpose.

This problem was found while searching for the root of the general protection
faults that were often happening when running our test suite.
Bisect pointed to the commit 55693e5c
The GPF was probably happening due to code that should only proceed with
completely initialized anonymous VMAs.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Reviewed-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Current code injects vnc parameters into XL domain configuration
file improperly thus broking run.py for Xen

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Make it explicit that the use of common sense is required when applying
the rules of the style guide.  Also point out explicitly that in
multiple variable declarations, you're supposed to "violate" an earlier
rule.

Reviewed-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

C++ coding convention is to bind '*' and '&' to the type,
not to the variable.

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Zero is a regularly used value, so let's instead use a rare one to compare the
content of the symbols against. Addressing some stylistic issues as well.

Follow the new output:
$ sudo scripts/run.py -e 'tests/tst-resolve.so'
OSv v0.05-348-g8b39f8c
Target value:   0x05050505
Success:        nonexistant = 0x05050505
Success:        debug = 0x05050505
Success:        condvar_wait = 0x05050505
The time:       1392076964
success.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

When multiple threads concurrently use a function which has a static
variable with a constructor, gcc guarantees that only one thread will run
the constructor, and the other ones waits.  It uses __cxa_guard_acquire()/
release()/abort() for implementing this guarantee. Unfortunately these
functions, implemented in the C++ standard library, use the Linux futex
syscall which we don't implement. The futex system call is only used in
the rare case of contention - in the normal case, the __cxa_guard_*
functions uses just atomic memory operations.

This patch implements the bare minimum we need from futex to make the
__cxa_guard_* functions work: All we need is FUTEX_WAIT and FUTEX_WAKE,
with 0 timeout in FUTEX_WAIT and wake all in FUTEX_WAKE.

It's nice how the new waitqueues fit this implementation like glove to
a hand: We could use condvars too, but we anyway need to do the wake()
with the mutex locked (to get the futex's atomic test-and-wait), and in
that case we can use waitqueues without their additional internal mutex.

This patch also adds a test for this bug. Catching this bug in a real
application is very rare, but the test exposes it by defining an function-
static object with a very slow constructor (it sleeps for a second), and
calls the function from several threads concurrently.
Before this patch the test fails with:
   constructing
   syscall(): unimplemented system call 202. Aborting.

Fixes #199.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It can reduce the duplicated code

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Build failed on my Fedora 20 due to lack of maven package.

Reviewed-by Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

When control flow reaches at the bottom inner loop in namei(), the pointer p
will point to either a '\0' or a '/' character because of the upper inner loop
break condition:


        for (i = 0; i < PATH_MAX; i++) {
            if (*p == '\0' || *p == '/') {
                break;
            }
            name[i] = *p++;
        }

So the "while" loop will never be executed and we can eliminate it as dead
code.

Reviewed-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add --sata option to use AHCI driver instead virtio-blk for QEMU. It
makes no sense to use sata device instead of virtio-blk device. But this
is mainly for test purpose.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

AHCI is supported on various VMM, e.g. Virtual Box, VMware Workstation.
Adding AHCI support enables OSv to run on them if the para-virtualized
block device is not present or not supported yet.

Tested on VirtualBox, VMware Workstation and QEMU.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently, only MSI-X is support in our PCI layer. Devices like AHCI
controller support MSI interrupt only. This paves the way for AHCI
driver.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

MSI support is supported in pci layer now. Enable MSI support interrupt_manager.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The first BAR is not present does not mean the entire BAR are not
present. Some implementation of AHCI controller only has BAR6 present
with BAR1 to BAR5 empty.

Keep probing if a BAR is not present.

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Or Cohen <orc@fewbytes.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch adds support for epoll()'s edge-triggered mode, EPOLLET.
Fixes #188.

As explained in issue #188, Boost's asio uses EPOLLET heavily, and we use
that library in our management http server, and also in our image creation
tool (cpiod.so). By ignoring EPOLLET, like we did until now, the code worked,
but unnecessarily wasted CPU when epoll_wait() always returned immediately
instead of waiting until a new event.

This patch works within the confines of our existing poll mechanisms -
where epoll() call poll(). We do not change this in this patch, and it
should be changed in the future (see issue #17).

In this patch we add to each struct file a field "poll_wake_count", which
as its name suggests counts the number of poll_wake()s done on this
file. Additionally, epoll remembers the last value it saw of this counter,
so that in poll_scan(), if we see that an fp (polled with EPOLLET) has
an unchanged counter from last time, we do not return readiness on this fp
regardless on whether or not it has readable data.

We have a complication with EPOLLET on sockets. These have an "SB_SEL"
optimization, which avoids calling poll_wake() when it thinks the new
data is not interesting because the old data was not yet consumed, and
also avoids calling poll_wake() if fp->poll() was not previously done.
This optimization is counter-productive for EPOLLET (and causes missed
wakeups) so we need to work around it in the EPOLLET case.

This patch also adds a test for the EPOLLET case in tst-epoll.cc. The test
runs on both OSv and Linux, and can confirm that in the tested scenarios,
Linux and OSv behave the same, including even one same false-positive:
When epoll_wait() tells us there is data in a pipe, and we don't read it,
but then more data comes on a pipe, epoll_wait() will again return a new
event, despite this is not really being an edge event (the pipe didn't
change from empty to not-empty, as it was previously not-empty as well).

Concluding remarks:

The primary goal of this implementation is to stop EPOLLET epoll_wait()
from returning immediately despite nothing have happened on the file.
That was what caused the 100% CPU use before this patch. That being said,
the goal of this patch is NOT to avoid all false-positives or unnecessary
wakeups; When events do occur on the file, we may be doing a bit more
wakeups than strictly necessary. I think this is acceptable (our epoll()
has worse problems) but for posterity, I want to explain:

I already mentioned above one false-positive that also happens on Linux.
Another false-positive wakeup that remains is in one of EPOLLET's classic
use cases: Consider several threads sleeping on epoll() on the same socket
(e.g., TCP listening socket, or UDP socket). When one packet arrives, normal
level-triggered epoll() will wake all the threads, but only one will read
the packet and the rest will find they have nothing to read. With edge-
triggered epoll, only one thread should be woken and the rest would not.
But in our implementation, poll_wake() wakes up *all* the pollers on this
file, so we cannot currently support this optimization.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Instead of binding all msix interrupts to cpu 0, have them chase the
interrupt service routine thread and pin themselves to the same cpu.

This patch is based on the patch from Avi Kivity <avi@cloudius-systems.com>
and used some ideas of Nadav Har'El <nyh@cloudius-systems.com>.

It improves the performance of the single thread Rx netperf test by 16%:
before - 25694 Mbps
after  - 29875 Mbps

New in V2:
 - Dropped the functor class - use lambda instead.
 - Fixed the race in a waking flow.
 - Added some comments.
 - Added the performance numbers to the patch description.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It appears that in GDB, (mmu::vma*)0 does not work, and one needs to enclose
the type's name in single quotes: ('mmu::vma'*)0. This broke the vma_list
function in scripts/loader.py, and caused an exception in "osv mmap" and
other commands using the vma_list function.

This patch adds the missing single-quotes.

I don't understand how this code ever worked for anybody...
I'm using gdb-7.6.1 from Fedora 19, if it matters.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>