Run a thread in a background to scan pagecache for accessed and
propagate them to ARC. The thread may take anywhere from 0.1% to 20%
of CPU time. There is no hard science behind how current CPU usage is
determined, it uses page access rate to calculate how hard pagecache
should be scanned currently. It can be improved by taking eviction rate
into account too.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Just so the symbol exists. We expect people to run their programs in foreground,
but if linked without lazy bindings, the symbol may be required.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

According to reality, the idea that rep movsb is the preferred way to implement
memcpy for x86 in the presence of the rep_good flag is false. This
implementation performs better in the misc-memcpy benchmark for pretty much all
sizes.

I have also tested a simple loop with byte-by-byte copy, and the duff's
mechanism. For the Duff, I am seeing a weird bug when it is implemented
together with our memcpy. But It is off course possible to implement it up to
256 separately for analysis, which is what I did.

What can be seen in the results below is that all versions start faster than
rep movsb for very small objects, but the loop starts to be slower for sizes as
low as 32-bytes.  Duff is slower for 64-byte elements, but this patch is faster
for all sizes measured.  We can copy 64i bytes in 5.6ns, 128 bytes in 7.7ns and
256 bytes in 13.3ns while the original numbers would be 11ns, 11ns, and 13.8
ns.

Balloon Safety:

Balloon memcpys are 128Mb in size. Even for partial copy, they are at least in
the kb range. So I am not expecting any funny interaction with this, nor
anticipating the need to insert fixups here.

Full Results:

Original
========
4,11.066000,13.217000,11.313369,0.527048
8,29.427999,31.054001,29.797934,0.540056
16,11.065000,11.147000,11.088465,0.030663
32,11.065000,11.199000,11.093401,0.043994
64,11.065000,11.508000,11.115365,0.092626
128,12.866000,13.137000,12.914132,0.066646
256,13.896000,14.252000,13.937533,0.067841
512,15.955000,16.304001,16.006964,0.073594
1024,20.072001,20.301001,20.122099,0.052627
2048,28.306999,28.577999,28.377703,0.063443
4096,44.785999,45.087002,44.899033,0.068806
8192,77.783997,78.370003,77.918457,0.113472
16384,150.259003,183.679001,158.534668,5.947755
32768,1049.886963,1053.098022,1051.364380,0.851499

Loop
====
4,3.152000,3.734000,3.347033,0.185811
8,4.467000,5.336000,4.936766,0.221336
16,6.655000,8.262000,7.695767,0.377303
32,19.788000,20.438000,19.960333,0.221289
64,25.996000,29.969999,29.217133,0.828447
128,44.501999,45.562000,45.335640,0.244315
256,85.459000,95.369003,91.925179,3.409483
512,14.925000,15.014000,14.939700,0.024197
1024,19.042999,19.143000,19.060701,0.028286
2048,27.277000,27.386000,27.306065,0.035528
4096,43.750000,43.902000,43.789631,0.038810
8192,76.699997,76.872002,76.769691,0.040407
16384,149.393997,164.602005,157.051132,4.324330
32768,1045.287964,1047.580933,1046.380493,0.617742

Duff
====
4,3.602000,4.120000,3.722167,0.163732
8,4.631000,4.725000,4.643835,0.028509
16,7.205000,7.316000,7.213567,0.022538
32,11.838000,12.613000,12.032168,0.285366
64,21.681000,22.173000,21.754402,0.088584
128,41.331001,41.651001,41.452267,0.066087
256,80.431000,80.927002,80.737724,0.106475

This patch
==========
4,3.602000,3.895000,3.636133,0.071126
8,3.602000,3.679000,3.607600,0.015768
16,3.859000,3.981000,3.875433,0.032632
32,4.888000,4.994000,4.899767,0.025539
64,5.663000,6.404000,6.001000,0.158665
128,7.737000,8.168000,7.881701,0.156874
256,13.301000,17.438999,14.937235,0.880874
512,14.925000,15.226000,14.975132,0.072150
1024,19.042999,19.412001,19.099068,0.095145
2048,27.278000,32.022999,27.617165,1.007376
4096,43.750000,44.146000,43.844494,0.094062
8192,76.698997,83.873001,77.137794,1.266063
16384,153.483994,168.636002,160.516830,3.837175
32768,1047.878052,1068.301025,1052.600586,4.441750

Signed-off-by: Glauber Costa <glommer@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Use backtick quotes for inline code.

Signed-off-by: Zifei Tong <zifeitong@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

How log does our memcpy take? misc-memcpy will let you find out

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

When running on a KVM host without EPT, OSv used to crash if ran 1GB
or less memory. The crash was in early acpi initialization, which called
AcpiOsMapMemory, which called linear_map() on the small chunk of physical
memory requested by ACPI.

It turns out that the chunk requested was just 4096 bytes, which caused
our linear_map() implementation to break down a huge-page into small-pages.
This didn't work - I'm don't know yet what exactly is the bug or why it
only shows up when the host doesn't use EPT, or if the memory is smaller
than 1GB...

But in any case, since we know our complete linear map always uses huge
pages, I think it makes no sense to map a single small page, and we can
just map a whole huge page. This will cause linear_map() not to try to
break up any huge pages, and this bug goes away.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Currently the code overwrites large pte with a pointer to intermediate
page before populating intermediate page's ptes with correct values. It
happen to work with EPT since TLB is still valid, but without EPT this
is not the case.

Fixes #316.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

vec_sz in net::txq::try_xmit_one_locked(net_req*) should be initialized to 1 and not to 0
since there is always net_hdr element and vec_sz is incremented only for fragments not taking
into an account the header element.

This miscalculation caused vqueue->add_buf() return false while vqueue->avail_ring_has_room(vec_sz)
was returning true when the amount of elements in the avail ring was equal to vec_sz
since the actual size of the _sg_vec was "vec_sz + 1".

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

trace.py allows to slice samples to a time range using --since,
--until and --period. Currently the default unit for timestamps was
nanoseconds. However the timestamp pretty much always comes from the
'trace list' output, which displays timestamp in seconds with a dot
separating the nanosecond precision part. That required explicit unit
designating by appending 's' suffix to the timestamp when passing to
--since and --until to

This change makes --since and --until use seconds as a default unit,
so that the timestamp copied from trace list can be used without any
change, eg:

 $ scripts/trace.py --since 1400607382.775557041

Default unit of --period (nanoseconds) remains unchanged.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

In AArch64 the I-bit set in DAIF means "interrupts disabled".
Therefore, the assertion check must be inverted in page_fault.

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

by marking as @function we get them to show up in the backtrace.

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

preadv, pwritev.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This test catches a few bugs we had in our C++ locale support (see
issue #314). After the previous patches, this test will succeed.

This test verifies that:

1. Trying to use an unsupported locale (we only support "C") throws an
   exception, as required, rather than crashing as before.

2. std::isspace(' ',std::locale) returns true, as it should (previously
   our ctype array was shifted by one, so this returned false!)

3. istream's input operator (operator>>) should stop on a space.
   Previously, it didn't, because we didn't recognize the space.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Fixes #314.

In two's complement, the lowest signed 8-bit number is -128, not -127.
By wrongly starting to generate the ctype array starting with -127 instead
of -128, we got all the locale ctype questions shifted by one character;
For example character 32 (' ') was not considered a space, but 31 was!

We didn't see this bug because our C library isspace() and friends are
currently implemented without using the locale framework (which is fine,
as we only support the "C" locale anyway). However, this bug is
apparent in C++, as explained in issue #314: std::isspace() returns the
wrong answer, and C++ facilities which use this under the hood - such
as reading from an istream which is supposed to stop at a space - also
got broken.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

When an application attempts to use an unknown locale, don't abort but
rather fail normally, as Linux does on an unknown locale.

Theoretically, Posix specifies newlocale() can also fail with EINVAL if
the category_mask is malformed. However, the only reasonable usage of this
function we support is when base=NULL (or "C" locale) and locale is again
"C", and then we just ignore the category_mask. If that is not the case,
we can just complain with ENOENT (meaning we couldn't find the named
locale).

In any case, callers like std::locale() don't actually care why newlocale()
failed, and anyway assume a failure means the localename wasn't recognized.

Reviewed-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The __newlocale() function in runtime.cc used a mixture of spaces and
tabs. Reindent it, with no changes to actual content (that would come
later).

Reviewed-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

If thread is not started its cpu pointer will be NULL. Handle this
case properly.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

the thread_control_block structure needs to be different
between x64 and AArch64;
For AArch64's implementation for local execution, try to
match the layout in glibc and the generated code.

Do not align .tdata and .tbss sections with .tdata : ALIGN(64)
or it will affect the TLS loads.

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>
Cc: Glauber Costa <glommer@cloudius-systems.com>
Cc: Will Newton <will.newton@linaro.org>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Java uses PROT_READ page to synchronize threads, so it worthwhile to be
able to catch this as fast as possible without taking vma_list_mutex.
The patch does it by checking that pte is not marked as cow during write
fault on present pte since cow or PROT_READ are the only reasons for pte
to be write protected. The problem is that to get pte we need to walk
the page table, but access to a page table is currently protected by
vma_list_mutex. The patch uses RCU to free intermediate page table levels
which makes it possible to get to pte without taking vma_list_mutex lock.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

The function should be called intermediate_page_post(), otherwise it is
never called.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Java uses accesses to PROT_NONE region to stop threads sometimes, so it
worthwhile to be able to catch this as fast as possible without taking
vma_list_mutex. The patch does it by setting reserved bit on all ptes in
PROT_NONE VMA which causes RSVD bit to be set in a page fault error code.
It is enough to check it to know that access is to a valid VMA, but
permission is lacking.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Currently address we get from pte includes reserved bits and setting
zeroes available and reserved bits.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Later patches will use reserved bit, so we want to be sure that it is
available.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

At the moment, defining DEBUG_VFS fails OSv compilation. The patch
ensures OSv compiles and correct debug logs are emitted.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Prasad Joshi <prasadjoshi.linux@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently covering three cases: 1 thread allocation, n threads allocation,
and n threads allocations with frees in a different cpu. We should improve
with more cases.

This test was designed to run on Linux as well. Let's keep it this way.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

According to http://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html

,
the versions of standard libraries have a strong matchup with versions
of compilers, so linkage errors occur when using libstdc++ in *external*
submodule when using GCC 4.9.0.

As there is such a mixup in the build system, the environments of
standard libraries in link time should be switchable to support GCC
whose version doesn't match up with external's.

*_env variables are introduced. To build a image with standard libraries
in the host, run `make build_env=host'. For fine-grained settings, use
gcc_lib_env and cxx_lib_env.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Boqun Feng <boqun.feng@linux.vnet.ibm.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Linked to wiki page.

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

As Nadav pointed out during review, this macro could use a bit more work, to
use a single parameter instead of one. That is what is done in this patch.
Unfortunately just pasting __COUNTER__ doesn't work because of preprocessor
rules, and we need some indirection to get it working. Also, visibility
"hidden" can go because that is already implied by "static". The problem then
becomes the fact that gcc does not really like unreferenced static variables,
which is solved by the "used" attribute. From gcc docs about "used":

   "This attribute, attached to a variable with the static storage, means that
    the variable must be emitted even if it appears that the variable is not
    referenced."

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>
Cc: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Fixes https://github.com/cloudius-systems/mgmt/issues/33



If a user runs 'java xyz', instead of throwing the stacktrace, a simple
message informing the user about the missing class, should suffice.

Signed-off-by: Jaspal Singh Dhillon <jaspal.iiith@gmail.com>
Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

If a thread which invokes flush_tlb_all() is migrated between the
call to flush_tlb_local() and send_allbutself(), the CPU onto which it
was migrated would not get its TLB flushed.

Spotted during code inspection.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

memory_order_acquire does not prevent previous stores from moving past
the barrier so if the _migration_lock_counter incrementation is split
into two accesses, this is eligible:

tmp = _migration_lock_counter;
atomic_signal_fence(std::memory_order_acquire); // load-load, load-store
<critical instructions here>
_migration_lock_counter = tmp + 1; // was moved past the barrier

To prevent this, we need to order previous stores with future
loads and stores, which is given only by std::memory_order_seq_cst.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Take the migration lock for pinned threads instead of a separate
check whether they are pinned or not.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Instead of forcing a reload (and a flush) of all variables in memory,
use the minimum required barrier via std::atomic_signal_fence().

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Proper memory ordering should be implied to loads and stores of _begin
field. Otherwise they may be reordered with the appropriate stores and
loads to/from the _ring array and in a corner case when the ring is full
it may lead to ring data corruption.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Reported-by: Nadav Har'el <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Jani Kokkonen <jani.kokkonen@huawei.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>