Combined java/cli and java/cloudius packages into one.
Added java/cloudius.jar as a jar to the build.

Signed-off-by: Or Cohen <orc@fewbytes.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Thread names

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Conflicts:
	core/sched.cc
	libc/timerfd.cc

Useful for debugging.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

idle, load balancer, init.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Useful for debugging and pthread_setname_np().

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Because bitwise "&" was used instead of logical "&&" the condition
was always false:

  (e & EPOLLET) & !warned = 0x80000000 & 1 = 0

And the warning "EPOLLET ignored" was never printed.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

put all of .ctors, .init_array, .ctors.*, init_array.*
into the init_array section.
This fixes the build for mixed ctors / init_array tooling
and dependencies.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Automatically:
 s/\t/    /g
 s/ +$//

Manually: align constant definitions in block.h

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

A previous patch made it possible to run "make image=rogue" when a
module rogue exists, without creating a images/rogue file.

This patch expands this feature to be even more powerful:

First, it allows choosing alternative command lines of a particular
module. For example:

        make image=mgmt.shell

Creates an image with the mgmt module (and its dependencies), and its
command line will be mgmt's "shell" command line (i.e., run just the
shell without the web server) instead of the "default" command line.

Second, instead of specifying just one module to put in the image (plus its
dependencies), we now allow giving a comma-separated list of unrelated
modules, all of which are put in the image. For example:

        make image=memcached,netperf,tests

The default command line of the image will be to run in parallel the
"default" command line of each of the modules, if it exists. In the above
example, the "tests" module has no command line labeled "default", so only
those of memcached and netperf will run, but tests' files will still be
included in the image.

As above, a non-default command line of the module can also be chosen,
for example:

        make image=memcached,mgmt.shell

Will build an image with the memcached and mgmt modules, and will run
in parallel memcached's "default" command line and mgmt's "shell"
command line.

If you want to include some module's files in the image, but not run
its command line, use the special word "none". For example:

        make image=memcached,netperf.none

will create an image with both memcached and netperf's files, but only
run memcached's command line.

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Commit 622d8bba re-enabled code to register vm_lowmem event,
so the respective deregister should also be re-enabled.

'For every OSv specific ifdef we remove in ZFS, God ressurects a kitten.'
by Glommer.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Now since we have support for parallel launching in the command line
we can implement support for it in the modules.

Example (image config):

 run = [
   api.require('netperf').default,
   api.require('mgmt').shell
 ]

Result:

$ scripts/run.py
OSv v0.05-132-g779307c
Starting netserver with host 'IN(6)ADDR_ANY' port '12865' and family AF_INET

[/]%

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

I have previously seen situations in which we had a lot more requests than we
can handle, leading to an OOM. I've initially thought that this was because
since there are only max_requests (usually 32) requests, new requests would
overwrite the old bio queue, leading to some of the bio_done functions never
being called.

A closer look at the code was enough to show me wrong, since at the end of the
ring consumption, we do iterate over old requests, processing them.

The real problem then, is that we can generate memory a lot faster than we can
process disk IO. To give an idea about order of magnitude, when a spike would
happen in Xen block (the very problem I am tracking), causing processing to be
interrupted by ~1s, 18k requests would accumulate in the queue. Since after
draining the ring we can only requeue 32 new requests, the end result is that
the 18k requests are never really processed. And at the new spike, 18k more
sit, and so on.

The solution is to adopt a strategy similar to virtio. We'll add a condition
variable to the queue and block new requests while we are processing new ones.

With this patch applied, misc-bdev-write.so runs to completion, but
unfortunately still quite slow.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
[ penberg: Fix mixed formatting ]
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

They are going to work exactly the same. But with std::list instead

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch make some of the blkfront structures a bit more C++ friendly. I am
creating a C++ version of the device private structure, and the goal here is to
convert it slowly until all or most fields are inside it. At that point, I
expect it to be easy to turn the functions into methods, etc. I am sorry, I
would really like to do a mass translation here, but there are urgent bugs
requiring my attention.

Some notes:
 * I am converting blkfront only, and leaving netfront with the old code. It
   shouldn't be hard, I just deferred the risk of mistakes to a future
   oportunity.
 * you will notice that lines that are added by me are not following the
   original file's identation. This is deliberate. As much as we have in our
   guidelines to preserve the original ident, since we are slowly converting
   those files, our identation should eventually win. Think of it as a form
   of cultural domination. (and shouldn't bother viewers if tabs=4 spaces)

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Until now, whenever we added a new module in apps.git, e.g., apps/tomcat,
we also had to add a file images/tomcat.py in osv.git, so that
"make image=tomcat" would work.

This patch makes it unnecessary to have a images/tomcat.py - when one does
"make image=tomcat" and images/tomcat.py is absent, we try to build an
image containing the single module tomcat and its dependencies (which
is defined as usual in apps/modules.json and apps/*/module.py).

Non-default images, which contain more than one unrelated modules (e.g., both
mgmt and tests) can still be defined in files in images/).

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

qemu-nbd, which is used by imgedit.py, is contained in qemu-utils
package, not installed by default.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This is useful to measure OSv boot speed, IOW, how fast are we without
CLI, Java, etc.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Xen boot is not totally silent. The devices usually print through
device_printf, which right now is hooked at printf. Changing that to debugf
already gets rid of most messages. However, netfront by itself manually prints
some messages with printf. They are changed as well.

Please note that the messages in netfront are changed directly to debug,
instead to device_printfs as well. This is because although we just hook that
to a printer, device_printf is supposed to prepend the messages with device
information.  We don't do it now, but could do it in the future. That is why
the original code used printf directly, and I am keeping that intent.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add to our coding conventions, CODINGSTYLE.md, three coding conventions
which were mentioned by Avi Kivity in a review, and also fit most of our
code. These conventions are:

 1. No (void) in function declarations in C++ code.
 2. No space between function name and argument list.
 3. No parantheses around return value.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It prints the command lines without actually running them.

--
Changes since V2:
 - changed option to --dry-run from --just-print (Glauber)
 - in addition to qemu and xen, suppress imgedit.py invocation too
 - when printing wrap arguments containing spaces with quotes

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

I assume this was required for `make check` (without image=) to work.
Let's have 'make check' use 'tests' image by default unless overwritten.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

IP routes read interface MTU on IP address configuration so in case
of DHCP, MTU has to be propagated before other parameters.

This patch fixes the last big MTU-related problem we see on EC2.
Having this patch applied OSv TCP performance on EC2 is pretty much
equal to Linux instances and limited by EC2 networking infrastructure.

Buffer size    Osv          Ubuntu 12.04 HVM
TCP TX
32             5129.27      5128.11
64             5129.18      5128.26
128            5129.33      5133.59
256            5130.63      5128.64
512            5129.4       5133.04
1024           5129.7       5131.87
2048           5128.27      5132.73
4096           5129.57      5133.04
8192           5129.18      5132.16
16384          5129.02      5131.77
32768          5129.36      5132.55
65536          5129.29      5132.05

TCP RX
32             298.26       292.31
64             571.4        540.72
128            1069.91      1014.22
256            1953.42      1925.36
512            3282.79      3261.96
1024           4963.05      5002.54
2048           5127.83      5121.33
4096           5124.61      5127.28
8192           5129.35      5136.58
16384          5128.05      5124.87
32768          5130.63      5135.68
65536          5132.22      5137.49

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

For this patch series to be bisectable, the previous patches left behind
some features, or temporarily added features, that we couldn't remove
until they are no longer use. Now we can finally delete these unused
features:

1. After the previous patches stopped using clock::get()->time() in sched.cc,
we can drop the alias osv::clock::get() we added temporarily just to make
it easier to compile the first patches in this series.
2. Drop the clock_event::set(s64) variant.
3. Drop the timer_base::set(s64) variant.

Fixes #81.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Remove unused #include of <drivers/clock.hh>.
Except the clock drivers and <osv/clock.hh>, no source file now now
include this header. Rather, <osv/clock.hh> should be used. Code including
<sched.hh> will also get <osv/clock.hh> automatically.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Fix pthread_cond_timedwait to set the absolute timer using a timepoint,
instead of the old s64.

Moreover, now that we have both a wall-clock and monotonic clock, we
can support pthread_condattr_setclock, so this patch also adds this
support. OpenJDK 8, for example, cannot run without this support (it
assumes that if the OS supports CLOCK_MONOTONIC, it can also configure
condition variables to use it).

Unfortunately supporting pthread_condattr_setclock - the only condition-
variable attribute that really exists - grows the pthread condition
variable structure :(

Fixes #168.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Replace the old function condvar::wait(mutex*, uint64_t) with one taking
a timepoint. This timepoint can use any clock which the timer supports,
namely osv::clock::uptime or osv::clock::wall (as usual, wall-clock timers
are not recommended, and are converted to an uptime timer at the point
of instantiation).

Leave a C-only function condvar_wait(convar*, mutex*, s64) but comment on
what it takes.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>