There is (currently commented out) code in ZFS that checks things like:

    if (curproc == pageproc) {
        /* Do something really great */
    }

The problem is that with our current implementation of curproc (designed for Xen)
that will break, because we will return a pointer to a in-stack variable that is
created on-demand and only contains the pid of the process.

Returning the thread address will make those checks works, but we will be forced
to give up on accessing fields inside it altogether. If we *really* must, we can
have a structure that have the fields in the same offset as our thread class.

But our thread class is defined in a .hh file, so *good luck* calculating the
offset of a field (say, id) at compile time so we can include in this other .h
file that contains exclusively C code. Since xen is the only user of the PID test,
and our resistance to changing the xen code is quite low (if at all), I'll just go
ahead and change it: storing the address of the process itself should allow us to
do compare tests the way ZFS does and get everything working.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

First of all, I am sorry. I am sorry Avi, Dor, Pekka, God, Dennis Ritchie, et caterva.
I am so very sorry. This is probably one of the ugliest things ever written by a C
programmer in the history of programming.

The story is: ZFS defines its own mutex of type kmutex_t, which is basically just a OSv
in our implementation. In a piece of code currently commented out (not for long), it calls:

    msleep(&needfree, &arc_reclaim_thr_lock, 0, "zfs:lowmem", 0);

The problem is that our msleep implementation expects a "struct mtx" which is our own
wrapper around mutex (Maybe that should be changed? Does anybody remember why it was
done this way?)

Keep in mind that we are going to great lenghts not to change ZFS (ifdefing code out
is generally fine), so the casting solution could not be used. I've tried to change the
for-ZFS definitions of mutex in the BSD glue code, but then, after a couple of hours
I was still resolving conflicts with all the other parts that would break because they
were expecting a certain type that was now changed.

I eventually set for the current ugly but functional solution: code msleep in a
way that it can accept any kind of mutex. That is really ugly because by "any
kind of mutex" I really mean any kind of crap the user passes and good bye type
safety altogether. But it works with minimal changes, and more importantly, with
all the changes being in *our* glue code.

If anybody have other ideas, I would be happy to try them out. But at this time,
I believe that to be the best compromise.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Force version.h generation to make sure the version number matches the
git version.

I noticed the problem after tagging v0.05-rc1 and noticing that OSv
happily reported the same version even after applying patches.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Current makefile cannot accept "make img_format=vmdk external all",
because QEMU does not support the format, but mkzfs.py and
upload_manifest.py requires to run OSv on QEMU.

With this patch, image conversion performs after mkzfs.py and
upload_manifest.py, prevents an error on QEMU execution.

Tested on QEMU/KVM and VMware, using these formats: raw / QCOW2 / VMDK.

Probably it can work on another formats such as VDI used by VirtualBox.

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This patch is not needed anymore since alarm() implementation was
extended with missing features.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

synch_port::msleep: merge time-out and non-time-out cases
into one conditional branch to avoid code duplication.

This both simplifies the code and makes future implementation of
interruption handling code for interruptable sleeps easier.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Move alarm cancellation logic to a separate function
cancel_alarm_ll(). This fucntion will be called on
thread destruction to avoid post-mortem alarms.

Also move alarm scheduling logic to a separate function
set_alarm_ll() to make code symmetric.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The uname() function returns a fake Linux version number for application
compatibility.  Add a new osv::version() API that returns OSv version
that can be used by the management code.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
[ penberg: cleanups ]
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Pekka noticed that his addr2line responds with '?? ??:0' on
unknown symbol, whereas mine responds with '??\n??:0'.

Reported-by: Pekka Enberg <penberg@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It is easier to use that timestamp as argument to --since and --until
of the prof command, which accept full timestamp in nanoseconds.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Affects output of 'trace.py list -b' and 'osv trace' GDB
commands. Makes the trace more readable when filename is also
included.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The new 'prof' command shows a hit profile of arbitrary tracepoint. Many
options accepted by 'prof-wait' are also accepted by this command.

Example:

  scripts/trace.py prof -t sched_wait

=== Thread 0xffffc0003eaeb010 ===

(100.00%, #7696) All
 |-- (99.51%, #7658) sched::thread::do_wait_until
 |    |-- (83.38%, #6417) condvar::wait(lockfree::mutex*, unsigned long)
 |    |    condvar_wait
 |    |    |-- (81.21%, #6250) cv_timedwait
 |    |    |    txg_delay
 |    |    |    dsl_pool_tempreserve_space
 |    |    |    dsl_dir_tempreserve_space
 |    |    |    dmu_tx_try_assign
 |    |    |    dmu_tx_assign
 |    |    |    |-- (81.19%, #6248) zfs_write
 |    |    |    |    vfs_file::write(uio*, int)
 |    |    |    |    sys_write
 |    |    |    |    pwritev
 |    |    |    |    writev
 |    |    |    |    __stdio_write
 |    |    |    |    __fwritex
 |    |    |    |    fwrite
 |    |    |    |    0x100000005a5f
 |    |    |    |    osv::run(std::string, int, char**, int*)

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

See

  scripts/trace.py prof-wait -h

The command is using sched_wait and sched_wait_ret tracepoints to
calculate the amount of time a thread was waiting. Samples are
collected and presented in a form of call graph tree.

By default callees are closer to the root. To inverse the order pass
-r|--caller-oriented.

If there is too much output, it can be narrowed down using
--max-levels and --min-duration options.

The presented time spectrum can be narrowed down using --since and --until
options which accept timestamps.

Example:

  scripts/trace.py prof-wait --max-levels 3 trace-file

=== Thread 0xffffc0003eaeb010 ===

12.43 s (100.00%, #7696) All
 |-- 12.43 s (99.99%, #7658) sched::thread::do_wait_until
 |    |-- 10.47 s (84.22%, #6417) condvar::wait(lockfree::mutex*, unsigned long)
 |    |    condvar_wait
 |    |    |-- 6.47 s (52.08%, #6250) cv_timedwait
 |    |    |    txg_delay
 |    |    |    dsl_pool_tempreserve_space
 |    |    |    dsl_dir_tempreserve_space
 |    |    |    dmu_tx_try_assign
 |    |    |    dmu_tx_assign
 |    |    |
 |    |    |-- 2.37 s (19.06%, #24) arc_read_nolock
 |    |    |    arc_read
 |    |    |    dsl_read
 |    |    |    traverse_visitbp
 |    |    |
 |    |    |-- 911.75 ms (7.33%, #3) txg_wait_open
 |    |    |    dmu_tx_wait
 |    |    |    zfs_write
 |    |    |    vfs_file::write(uio*, int)
 |    |    |    sys_write
 |    |    |    pwritev
 |    |    |    writev
 |    |    |    __stdio_write
 |    |    |    __fwritex
 |    |    |    fwrite
 |    |    |    0x100000005a5f
 |    |    |    osv::run(std::string, int, char**, int*)

By default every thread has a separate tree, because duration is best
interpreted in the context of particular thread. There is however an
option to merge samples from all threads into one tree:
-m|--merge-threads. It may be useful if you want to inspect all paths
going in/out to/from particular function. The direction can be changed
with -r|--caller-oriented option. Function names is passed to --function
parameter.

Example: check where zfs_write() blocks:

  scripts/trace.py prof-wait -rm --function=zfs_write trace-file

7.46 s (100.00%, #7314) All
 zfs_write
 |-- 6.48 s (86.85%, #6371) dmu_tx_assign
 |    |-- 6.47 s (86.75%, #6273) dmu_tx_try_assign
 |    |    dsl_dir_tempreserve_space
 |    |    |-- 6.47 s (86.75%, #6248) dsl_pool_tempreserve_space
 |    |    |    txg_delay
 |    |    |    cv_timedwait
 |    |    |    condvar_wait
 |    |    |    condvar::wait(lockfree::mutex*, unsigned long)
 |    |    |    sched::thread::do_wait_until
 |    |    |
 |    |    |-- 87.87 us (0.00%, #24) mutex_lock
 |    |    |    sched::thread::do_wait_until
 |    |    |
 |    |    \-- 6.40 us (0.00%, #1) dsl_dir_tempreserve_impl
 |    |         mutex_lock
 |    |         sched::thread::do_wait_until
 |    |
 |    \-- 7.32 ms (0.10%, #98) mutex_lock
 |         sched::thread::do_wait_until
 |
 |-- 911.75 ms (12.22%, #3) dmu_tx_wait
 |    txg_wait_open
 |    condvar_wait
 |    condvar::wait(lockfree::mutex*, unsigned long)
 |    sched::thread::do_wait_until

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Useful for calculating time during which thread
was scheduled out because of wait().

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Prints hierarchical data in a form of tree:

Example:

Node0
 |-- Node1
 |    |-- Node2
 |    |
 |    \-- Node3
 |
 \-- Node4

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

For all options check:

  scripts/trace.py -h

Example: list all traces:

  scripts/trace.py list trace-file

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Usage:

  osv trace save <filename>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Having traces persisted in a file has the follwing advantages:

 - trace collection is decoupled from trace analysis. trace analysis
   tools no longer have to be part of GDB scripts. This is important
   for EC2 instances, to which we cannot connect with GDB. In this
   case trace data can be downloaded using other channels, but tools
   remain unchanged.

 - trace analysis doesn't have to be performed in the same OSv session.
   After trace is collected OSv can be shut down or left running without
   further disturbance.

The trace is saved in a binary format of the following structure:

  <format-version>
  <N_tp>
  <tracepoint 1>
  ...
  <tracepoint N_tp>
  <trace 1>
  <trace 2>
  ...
  <trace N_t>

Where:

  <tracepoint X> = <tp_key><name><signature><format>
  <trace X> = <tp_key><thread><time><cpu>(<backtrace_addr>*)<0><data>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Speeds things up slightly.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

If there are fewer frames than backtrace_len the remaining
addresses will be zeros. Let's not print them.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Useful for ad-hoc printing.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The code "file_ts" converting a std::chrono::duration into a struct timespec,
was unnecessarily constrained to take as input a std::chrono::nanoseconds,
whereas it can actually work with durations of any resolution (the
duration_cast will convert it to the correct units), so this patch makes
this generalization which, I think, makes the code clearer.

At first I decided to move this general function into <osv/clock.hh> and
make it a bit more friendly (returning a timespec, instead of writing into
one), but at the end, I decided not to move it. typespec is a dangerous
type because though it does specify the clock's resolution (always
nanoseconds) it doesn't specify the clock's epoch, so one can make mistakes
like calling clock_gettime(CLOCK_MONOTONIC), and pass the returned timespec
to pthread_cond_timedwait, which expects CLOCK_REALTIME not CLOCK_MONOTONIC.
So we should avoid timespec whenever we can, and I don't think we'll need
the fill_ts() function anywhere outside time.cc.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

To prevent similar mistakes as in commit cc6b3a3c ("vga: Handle line
feed"), add the backup files generated by 'patch' to gitignore.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

I messed up in commit cc6b3a3c ("vga: Handle line feed"). Clean it up.

Reported-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

When trying to run a nonexistant program, e.g., "run.py -e xyz", sometimes
(especially in a debug build) we would see the spurious message:

    program xyz returned -16384

The bug is simple: when osv::run returns null, meaning the program was not
found, it does not set the "ret" value. Our code checked if ret!=0 before
checking if osv::run actually found the program, which is wrong. Simply
changing the order of the code solves this bug.

Fixes #156.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add --vga on cmdline, to switch vga console.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

To use VGAConsole class, add empty input_ready() and readch().

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Handle line feed in the VGA.  To access termios flag, add a constructor.

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Physical memory base address is 0xffffc00000000000, so VGA base address
should be 0xffffc000000b8000.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

We are currently only answering requests for CLOCK_REALTIME, but we could
easily handle:

    * CLOCK_REALTIME_COARSE, which is effective the same as CLOCK_REALTIME
      but faster. In our case, all time sources are equally fast
    * CLOCK_PROCESS_CPUTIME_ID and CLOCK_THREAD_CPUTIME_ID, since we can
      easily get runtimes for our threads and publish that.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The "APIC base" message is not very useful to users. Drop it.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently, OSv prints out the following at boot:

  acpi 0 apic 0
  acpi 1 apic 1
  acpi 2 apic 2
  acpi 3 apic 3

replace that with a simpler message:

  4 CPUs detected

We do lose the ACPI ID -> CPU ID mapping but it is not terribly
important for users.

Suggested-by: Nadav Har'El <nyh@cloudius-systems.com>
Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>