This patch adds two more load-balancing tests to tests/misc-loadbalance.cc:

1. Three threads on two cpus. If load-balancing is working correctly, this
   should slow down all threads x1.5 equally, and not get two x2 threads
   and one x1.

   Our performance on this test are fairly close to the expected.

2. Three threads on two cpus, but one thread has priority 0.5, meaning it
   should get twice the CPU time of the two other threads, so fair load
   balancing is to keep the priority-0.5 thread on its own CPU, and the
   two normal-priority threads together on the second CPU - so at the end
   the priority-0.5 thread will get twice the CPU time of the other threads.

   Unfortunately, this test now gets bad results (x0.93,x0.94,x1.14
   instead of x1,x1,x1), because our load balancer currently doesn't take
   into account thread priorities: It thinks the CPU running the
   priority-0.5 thread has load 1, while it should be considered to have
   the load 2.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

they are not enabled yet in the build system.

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Module definition is enough.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

the rcx register holds the amount of bytes we need to copy during the copy
operations. We will advance it by the fixup, which means that it should never,
ever, go bellow 0. I saw that happening due to a bug, and took me some time to
figure that out - the result is memory corruption, which can manifest in weird
ways. Should that assert be in place, I would have noticed sooner

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently it would throw ArrayIndexOutOfBounds when 'java' is called without arguments.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

also updates the isa defs with some AArch64 values.

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Useful in suppressing unimportant branches. Example:

Using percentage:

 trace.py --min-hits 10%

Using absolute count:

 trace.py --min-hits 3

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This option will allow to introduce more grouping criterias than just
by thread. It replaces -m|--merge-threads option. Now by default there
is no grouping. Samples can be grouped by thread by passing '-g
thread'.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This fixes openssl issue manifesting itself with inability to
establish an SSL connection. Openssl refused to create new connection
due to insufficient entropy to generate session id. There was no
entropy because RAND_poll() didn't manage to read anything from any of
the randomness sourcess, which by default are:

 /dev/urandom
 /dev/random
 /dev/srandom

We do, however, have /dev/random on OSv. The problem was that when
openssl tries different sources it skips the ones which it already
tried by comparing stat.st_ino and stat.st_dev as returned by
fstat(). Because devfs didn't implement getattr operation both these
fields were always 0 so even though /dev/random was present it was not
considered.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It's needed by openssl. It supposed to return NULL if the environemnt is not
trusted. Our implementation just delegates to getenv().

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Our README.md stated that gcc 4.7 is required (but 4.8 is recommended), but
4.7 will no longer work now that sched.hh uses, in class wait_object,
constructor inheritence. That feature was only added in 4.8 (see
http://gcc.gnu.org/gcc-4.8/changes.html

).
So this patch fixes the README.md to say that gcc 4.8 is required.

Gcc 4.8 is almost a year old, no reason why we shouldn't use it (and the
C++11 features that were still missing in 4.7).

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Until now, our manifest did not support an empty directory, so we resorted
to uploading a file ".keep" inside the directory, which is kind of ugly
(this file remains in the final image).

Instead, in upload_manifest.py, if the given file is a directory (any
directory...), put it in the cpio as a directory, which causes an empty
directory to be created.

So now our images will no longer have files like /tmp/.keep.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Now that we can print formatted stuff into abort, printing the address of the
fault can be helpful

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Zero req->mhdr.num_buffers in Tx path when MRG_RXBUF is negotiated as virtio spec demands.

Reviewed-by Zhi Yong Wu <zwu.kernel@gmail.com>
Reviewed-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add support for directories.

Found by streaming openjdk rpm2cpio output to OSv.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add symbolic link support to cpiod.so parser.  As OSv does not yet
support symlinks, the files are never created on the filesystem.

This is needed for streaming rpm2cpio to OSv guest.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Make cpiod.so more robust by checking the file type explicitly.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Specify file type in the cpio header.  This is needed to make cpiod.so
more robust by explicitly checking file type.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This reverts commit 860d6c53.  It makes
cpiod.so completely ignore zero-length files.  This breaks 'make check'
because the tmp directory is no longer created, for example.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The serializer was reading from the destination buffer instead of the
source string. As a result all string arguments appeard as empty
strings in the trace.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Looks like an omission. You'd run into this if you had multiple
mappings for the same file on the guest.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It is really helpful to know who is who when you are printing a backtrace
for all threads.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

If c_filesize is zero, bail out immediately from parse_one(). Otherwise
the rdbuf()->pubsetbuf() call will wreak havoc on the stream position
and cause the next header to have corrupt CPIO magic.

Found by streaming openjdk rpm2cpio output to OSv.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Cc: Benoît Canet <benoit@irqsave.net>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add CPIO header magic number checking to fail fast if there's something
wrong with the CPIO format.

Found by code inspection.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Cc: Benoît Canet <benoit@irqsave.net>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

I was setting up an OSv with network today, following README instructions
and noticed that to run the test an invoke command is missing.

Maybe it's a syntax change from older versions.

Signed-off-by: Eduardo Piva <efpiva@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

3c16063a forgot to make recv() as having C linkage.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

don't copy a virtio virtio_net_hdr to a local buffer in receiver().
This prevents CPU's optimizations.
Cast the mbuf's beginning to the virtio_net_hdr and use the relevant
fields where needed allowing a proper load prediction (as a result of
branch prediction).

Reviewed-by: Dor Laor <dor@cloudius-systems.com>
Reviewed-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

BacktraceFormatter expects a string, not a syminfo, as Python was kind enough
not to point out.

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Caching allocations can hide use-after-free bugs.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Easier to add tracepoints, etc.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Using malloc() fails with the debug allocator, and can fail for the normal
allocator as well since it does not guarantee physical memory.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Some tracepoints are called after their arguments are freed.  Reverse the
order.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The debug allocator does not recycle virtual addresses in order to
improve detection of use-after-free bugs.  This creates a lot of intermediate
pages as address space is consumed, leading to a very rapid OOM.

Fix by reclaiming intermediate pages that point nowhere.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

unpopulate<> (used by vdepopulate()) maintains an std::stack<> of pages
to be freed, in order to defer the tlb flush (that must be done after
pages are unlinked from the page table, but before they are actually freed).

However, the debug allocator uses vdepopulate() to ensure freed memory is
made inaccessible (so detecting use-after-free bugs).  This causes endless
recursion.

Fix by using a fixed-size queue of free pages and flushing it if it fills up.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

If a char* tracepoint argument points into demand paged memory, we may
try to access it while it is unmapped, in a non-preemptible context.

Fix by handling the fault gracefully.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch reverts commit 831b017e,
which stopped stripping the tests in order to fix "make check", and
instead just fixes the latter.

The fix is trivial: "make check" finds the list of tests not from the
image or the manifest, but from the list of files in build/release/tests.
So all we need to do is to subtract from that list the list of *-stripped.so.

As to the question *why* to strip the tests:

In the past, several people raised the desire to include our tests on some
production images (not just the "make image=tests" one). The rationale behind
this is that if the image encounters unexpected problems on some cloud, we
can run the included tests to see if something if it can find a problem.

I'm not sure this is a compelling reason to include tests in every image,
but let's at least not rule the option to include them on some image.

The difference in size between the stripped and unstripped tests is huge:
Currently, unstripped tests total 18 MB, while the stripped are just 2.2 MB.
That's more than 8 times smaller! We can easily imagine that in the future
we'll have more tests - perhaps many, many more tests, making this difference
even more significant.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>