protect x64 machine code in #ifdef __x86_64__

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>

stub console_init for AArch64.

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>

the functions debug_early, debug_early_u64 and
debug_early_entry can be used very early, before premain.

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>

Signed-off-by: Jani Kokkonen <jani.kokkonen@huawei.com>
Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>

rename processor::halt_no_interrupts to cli_hlt, which
logically matches existing processor::sti_hlt.

Make use of arch::halt_no_interrupts in core/power.cc,
enclose x64-specific code in ifdefs, as well as putting
code on AArch64.

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>

virt_to_phys just returns the virtual address again
for AArch64 for now, while mmu_unmap will abort.

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>

use %wn when referring to the non-extended register n,
and %n when referring to the extended register n,
emit the right memory barrier instructions, get the
semantics of atomic_store_rel hopefully right.

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>

this contains still prototyped code, which if reached either
hangs (setjmp, longjmp), aborts (all that requires mmu::),
or implements differently (allocating with malloc instead of mmap)
This is enough libc AArch64 support for reaching the end of premain.

Signed-off-by: Claudio Fontana <claudio.fontana@huawei.com>

When halt() is called very early, before smp_launch(), it crashes when
calling crash_other_processors() because the other processors' IDT was
not yet set up. For example, in loader.cc's prepare_commands() we call
abort() when we failed to parse the command line, and this caused a
crash reported in issue #252.

With this patch, crash_other_processors does nothing when other processors
have not yet been set up. This is normally the case before smp_launch(),
but note that on single-vcpu VM, it will remain the case throughout the run.

Fixes #252.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

You'll run into this when you have nested trace samples
with undefined backtrace elements.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Useful to figure out which thread is waiting on what

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This adds a script for uploading OSv release to Capstan S3 bucket using
"s3cmd sync".

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

This adds a script for building a Capstan repository for an OSv release
with the following images:

  - OSv core (no API, no CLI)
  - Default (with API, CLI)
  - Cassandra virtual appliance (with API)
  - Tomcat virtual appliance (with API)
  - Iperf virtual appliance (with API)

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

This adds a script for creating Capstan compatible images that can be
uploaded to a S3 bucket for distribution.

To use it to build a base image, for example, type:

  $ ./scripts/build-capstan-img cloudius/osv-base empty "OSv base image"

which builds a Capstan image named "cloudius/osv-base" with "make
image=empty" and places QEMU and VirtualBox images under
"build/capstan/cloudius/osv-base" together with an index file.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Add cursor update callback to libtsm, update cursor position register on VGA device using the callback.
Fixes #220.

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Indirect descriptors for Xen PV disks is a newly
introduced protocol feature that increases
block devices performance in case of long transfers.

Detailed description:
http://blog.xen.org/index.php/2013/08/07/indirect-descriptors-for-xen-pv-disks/



Measurement results (RAM-backed storage):

Xen w/o indirect descriptors:

1046.251 Mb/s
1097.931 Mb/s
1149.672 Mb/s
1137.696 Mb/s
1135.481 Mb/s
1249.163 Mb/s
1115.417 Mb/s
1118.063 Mb/s
Wrote 11344.750 MB in 10.00 s = 1134.165 Mb/s

Xen w/ indirect descriptors:

bdev-write test offset limit: 250000000 byte(s)
1234.715 Mb/s
1360.013 Mb/s
1323.663 Mb/s
1336.916 Mb/s
1342.617 Mb/s
1332.882 Mb/s
1302.094 Mb/s
Wrote 13233.250 MB in 10.04 s = 1318.639 Mb/s

KVM:

bdev-write test offset limit: 250000000 byte(s)
674.729 Mb/s
698.736 Mb/s
627.677 Mb/s
630.209 Mb/s
742.977 Mb/s
759.205 Mb/s
681.476 Mb/s
655.717 Mb/s
713.231 Mb/s
688.478 Mb/s
Wrote 6872.750 MB in 10.00 s = 687.124 Mb/s

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

This is a refactoring commit to simplify future indirect descriptors code.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

This is a refactoring commit to simplify future indirect descriptors code.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

This is a refactoring commit that isolated some xenstore access
logic to make it reusable

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Test misc-bdev-rw introduced.

The test writes buffers of various lengths to block device,
reads data back and verifies data read is the same as data written.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Xen block driver now supports new featute called indirect descriptors.
This feature allows to put more data into each ring cell but it activates
for "long" reads and writes only - longer than 11 pages.

With this patch test by default runs 2 scenarious:
  * 1 page buffers
  * 32 pages buffers

Also introduced command line parameter to specify
size of buffers explicitly.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Add explicit conversion form bytes array to string which returned by
Popen.communicate().

Use Python3 style print function.

Signed-off-by: Zifei Tong <zifeitong@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Remove the embedded driver, and start using the one from drivers
sub-system. Still relies on the manual creation of /etc/mnttab as
the upload manifest wasn't yet processed.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Previously, zfs device was being only provided to allow the use of
commands needed to create the zpool, and so the file system.
At that time, doing so was quite enough, however, making zfs
device, i.e. /dev/zfs part of every OSv instance would allow us
to use commands that will help analysing, debugging, tuning
the zpool and file systems there contained.

The basic explanation is that those commands use libzfs which in
turn relies on /dev/zfs to communicate with the zfs code.

Commands example:
zpool, zfs, zdb. The latter one not being ported to OSv yet.
This patch will also be helpful for the ongoing ztest porting.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

/etc/mnttab is required by libzfs to get running properly, so let's
create it as an empty file.

ryao from zfsonlinux and openzfs told me that an empty /etc/mnttab is used
on Linux. Also reading the libzfs code shows that /etc/mnttab mostly used for
management of the file itself, nothing that would prevent some libzfs
functionality from working.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The tst-dns-resolver.so fails spuriously. Blacklist it until the problem
is fixed to keep Jenkin builds running.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The root dataset and ZFS are mounted at the mkfs phase, but they aren't
unmounted aftwards.

Running mkfs with VERBOSE flag enabled shows the following:
Running mkfs...
VFS: mounting zfs at /zfs
zfs: mounting osv from device osv
VFS: mounting zfs at /zfs/zfs
zfs: mounting osv/zfs from device osv/zfs

The first mount happens when issuing:
{"zpool", "create", "-f", "-R", "/zfs", "osv", "/dev/vblk0.1"}, &ret);
It creates a pool called osv and mounts the root dataset at /zfs

The latter mount happens when issuing:
{"zfs", "create", "osv/zfs"}
It creates a file system called zfs at the pool OSv and automatically
mounts it at the root dataset mountpoint.

No data inconsistency problem was seen up to now because both mkfs.so and
cpiod.so do an explicit sync() at the end, thus ensuring everything was
correctly flushed out to the stable storage.
There is an expression in Dutch that says: prevention is better than cure.
Thus, this patch changes cpiod.so to unmount both mount points when the
/zfs/zfs prefix was passed. It cannot be done at mkfs.so itself because
cpiod.so is called afterwards at the same OSv instance.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Python3 no longer allow implicitly conversion form bytes to string,
add explicit decode() to convert input bytes.

Tested with both Python2 and Python3.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Zifei Tong <zifeitong@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Gleb has noticed that the ARC buffers can go unshared too early. This will
happen because we call the UNMAP operation on every put(). That is certainly
not what we want, since the buffer only has to be unshared when the last
reference is gone.

Design decisions:
1) We obviously can't use the arc natural reference count for that, since
bumping it would make the buffer unevictable.
2) We could modify the arc_buf structure itself to add another refcnt (minimum
4 bytes).  However, I am trying to keep core-ZFS modifications to a minimum,
and only to places where it is totally unavoidable.

Therefore, the solution is to add another hash, which will hash the whole
buffer instead of the physaddr like the one we have currently. In terms of
memory usage, it will add only 8 bytes per buffer (+/- 128k each buffer), which
makes for a memory usage of 64k per mapped Gb compared to the arc refcount
solution. This is a good trade off.

I am also avoiding adding a new vop_map/unmap style operation just to query the
buffer address from its file attributes (needed for the put side). Instead, I
am conventioning that an empty iovec means query, and a filled iov means
unshare.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>

debugf() used to write log message with respect to the length of format
string. This will cause the messages wrongly truncated.

Also change confusing variable names: exchange 'fmt' and 'msg'.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Zifei Tong <zifeitong@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Spotted by code review. Gleg had spotted one improper use of "i", but
there was another. In this case we iterate over nothing, and i is always 0.
It is uninitialized to begin with, and the code works just because it is
being set to 0 by luck.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>

We have seen bugs with mmap shared/file handling for small files. This patch tests
some of the corner scenarios to find those problems.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>

There is a problem with the way ZFS currently handles its buffers, which is
actually a limitation of our allocator: buffers smaller than a page won't be
page aligned even if we ask for it. Therefore, if the buffer we are mapping
falls into this category, we will map the wrong location.

The way I solved this problem was so stupid, that in retrospect I can't even
believe I did it: when the file would run out of size, we would truncate the
file. This is obviously wrong because reading a file is not expected to change
its size in any circumstance, and if anybody relied in the actual size, we will
be crashing something. This is the bug that plagued Cassandra.

Not truncating, however, brings back the original problem. One solution I have
considered is to always allocate at least a page for data allocations (leaving
metadata alone), but that would deviate from ZFS and harm many-small-files
workloads.

However, During testing, I have noticed though that ZFS will allocate small
buffers only when the file itself is small. This means that we can just avoid
using the special shared mapping for small files - which makes sense anyway.

For instance, if we have a file that is 128k + 1byte (remember 128k is ZFS's
maximum buffer size), both buffers will be large enough to be aligned. And if I
that ever fails to hold, we will now see an assertion hit instead of a random
bug. In time, we should fix our allocator to provide alignment guarantees.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>

Currently all mapping are keyed on ARC buffer start when mapping is
added, but on remove pointer into ARC buffer is used, so remove may
leave no longer valid mappings in the database. This patch fixes it
by using a pointer into ARC as a key, the same pointer that is used
during removal.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>

Option -F should always be used with -X. Without this flag if output
is smaller than the screen then no output will be shown.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>