The current code the trace_log - a 4MB array which stores tracepoint data -
is a global array. In other words, as explained in issue #96, it is part
of the BSS, which always adds 4MB memory usage to OSv - whether or not
tracepoints are actually enabled.

This patch changes trace_log to be a dynamically-allocated array, allocated
when the first tracepoint is enabled (e.g., with a --trace=...  boot
option).

This will also allow us (in the future) to choose the size of the trace log
at run time rather than compile time.

This patch reduces 4 MB of BSS (from 13MB to 9MB), and correspondly lowers
our minimum memory use by 4 MB. Whereas the minimum memory requirement of
the "rogue" image used to be 59 MB, now it is 55 MB.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

list_time can currently be sorted by duration. Since we read things from the
buffer in buffer order to construct the trace list, the list is already
quasi-sorted by its timestamp key: it is per-cpu sorted, but globally we have
not a lot of guarantees and some samples will get reordered.  It would be
useful to have a list that is strongly sorted as well.

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

I think the current code is broken in 'GB' case because it trims
memory string twice.

Spotted during skimming over the file.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It is useful when trying to detect IP assigned to guest using the arp
table: `arp -n`

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The first entry in a backtrace tends to be the point in which we arrived at the
trace, which is a logtrace function. That only clutters output, because that is
a long named function that is basically adding no meaningful information

While we could pop the first element unconditionally, that would not be robust
against function inlining by the compiler. We can also add more trace functions
in the future, so I'm popping elements in a loop until we're out of trace stuff
in the beginning of the backtrace.

Backtrace traces are a lot easier to follow with this patch.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Currently when object file cannot be located GDB fails with enigmatic:

 Python Exception <type 'exceptions.TypeError'> cannot concatenate
 'str' and 'NoneType' objects

It's better to read as much as we can and warn about what we failed to
locate.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

We need to add mgmt/ to the search list. Otherwise it fails like this:

 gdb$ osv syms
 None 0x100000000000L Python Exception <type 'exceptions.TypeError'>
 cannot concatenate 'str' and 'NoneType' objects:
 Error occurred in Python command: cannot concatenate 'str' and
 'NoneType' objects

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

Useful for debugging and pthread_setname_np().

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>

A previous patch made it possible to run "make image=rogue" when a
module rogue exists, without creating a images/rogue file.

This patch expands this feature to be even more powerful:

First, it allows choosing alternative command lines of a particular
module. For example:

        make image=mgmt.shell

Creates an image with the mgmt module (and its dependencies), and its
command line will be mgmt's "shell" command line (i.e., run just the
shell without the web server) instead of the "default" command line.

Second, instead of specifying just one module to put in the image (plus its
dependencies), we now allow giving a comma-separated list of unrelated
modules, all of which are put in the image. For example:

        make image=memcached,netperf,tests

The default command line of the image will be to run in parallel the
"default" command line of each of the modules, if it exists. In the above
example, the "tests" module has no command line labeled "default", so only
those of memcached and netperf will run, but tests' files will still be
included in the image.

As above, a non-default command line of the module can also be chosen,
for example:

        make image=memcached,mgmt.shell

Will build an image with the memcached and mgmt modules, and will run
in parallel memcached's "default" command line and mgmt's "shell"
command line.

If you want to include some module's files in the image, but not run
its command line, use the special word "none". For example:

        make image=memcached,netperf.none

will create an image with both memcached and netperf's files, but only
run memcached's command line.

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Now since we have support for parallel launching in the command line
we can implement support for it in the modules.

Example (image config):

 run = [
   api.require('netperf').default,
   api.require('mgmt').shell
 ]

Result:

$ scripts/run.py
OSv v0.05-132-g779307c
Starting netserver with host 'IN(6)ADDR_ANY' port '12865' and family AF_INET

[/]%

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Until now, whenever we added a new module in apps.git, e.g., apps/tomcat,
we also had to add a file images/tomcat.py in osv.git, so that
"make image=tomcat" would work.

This patch makes it unnecessary to have a images/tomcat.py - when one does
"make image=tomcat" and images/tomcat.py is absent, we try to build an
image containing the single module tomcat and its dependencies (which
is defined as usual in apps/modules.json and apps/*/module.py).

Non-default images, which contain more than one unrelated modules (e.g., both
mgmt and tests) can still be defined in files in images/).

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It prints the command lines without actually running them.

--
Changes since V2:
 - changed option to --dry-run from --just-print (Glauber)
 - in addition to qemu and xen, suppress imgedit.py invocation too
 - when printing wrap arguments containing spaces with quotes

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Functions which do not modify input parameters are easier to analyze
and more friendly for its users.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

To start the JVM with debugger server just run:

  sudo scripts/run.py -nv --jvm-debug

If you want the JVM to wait for debugger client:

  sudo scripts/run.py -nv --jvm-suspend

Listens on port 5005.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The OSv loader has a "--verbose" option to enable debug() output to be
printed to the console and not just to a memory buffer. However, needing to
pass this option in the "-e" string is annoying, and even worse when you want
to use the default command line and didn't even want an explict "-e" option.

So this patch adds a --verbose to run.py, which can be used whether or not
a "-e" option is given. For example:

    run.py --verbose
or
    run.py --verbose tests/tst-timerfd.so

As a shorthand for --verbose, I used "-V". Unfortunately, lowercase "-v"
was already taken (by the "vhost" option).

Fixes #173.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

As issue #128 explains, it is annoying that once the "-e" option was used
in run.py, it overrides the default command line, so that running run.py
again without -e reruns the same modified command line, and not the default
one as usual.

This simple patch changes this behavior: On every run of run.py, the command
line is set - if "-e" is given it is set to the given command line, but if
"-e" is absent, the command line is reset to the default one. This is
basically the behavior that people always wanted from run.py.

This patch, and the behavior it performs, are much simpler than any of
the alternatives proposed in issue #128.

Please don't complain that this can only be used for development, and not
for EC2, for creating images to distribute, and so on. After all, this is a
patch to *run.py*, and it is obviously not relevant to anyone who is running
an OSv image not through run.py. This patch only makes run.py more convenient
to use.

Fixes #128.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Add a 'trace extract' command that hides the gdb magic and is
future-proof when we change the trace transport protocol.

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This adds a '--graphics' option to run.py that enables QEMU's graphics
mode using SDL.  This is needed for testing OSv's VGA support.

Requested-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Drop echoing: "SCRIPT, tap0"

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Print the thread pointer, not the address of the thread pointer.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The timestamp typically will come from 'trace.py list' output.  This
change makes it easier to paste the timestamp. Listing prints the time
with '.' separating seconds from nanoseconds.  The options '--since'
and '--until' accept timestamp in nanoseconds so the dot must have
been removed. Now one only needs to append the 's' suffix which
changes the unit to seconds.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

This:

  --since 10.0 --period 100ms

is now equivalent to:

  --since 10.0 --until 10.1

This option makes it easier to slice the time based on the output of
'trace.py list-timed' which prints trace timestamp and duration.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Removing 'osv trace summary' and 'osv trace duration' commands from
loader.py. The first is added as 'trace.py summary --timed', the
second one is added as 'trace.py list-timed'.

It was easier to move both in one commit as they are coupled with each
other.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

The call-graph tree is already using call-sites, this makes also
`trace.py list -b` also use it.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Prints output like this:

Collected 36843 samples spanning 8.85 s

Tracepoint statistics:

  name              count
  ----              -----
  sched_wait        18420
  sched_wait_ret    18423

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

When output is not fully consumed because of piping, eg to `head`,
we get an ugly exception:

 Traceback (most recent call last):
   File "scripts/trace.py", line 274, in <module>
     args.func(args)
   File "scripts/trace.py", line 227, in list_timed
     bt_formatter(t.backtrace),
 IOError: [Errno 32] Broken pipe

Let's ignore it, it's normal condition.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

 - added command descriptions
 - improved clarity of options
 - extracted argument groups
 - changed command help to start with lower case letter

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It's a left over which is not used any more.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Instead of always putting the tests into all images, this patch adds the
option of putting them only in some of the images, by making them into a
new module, modules/tests.

The default image (images/default.py) continues to require the tests
modules, so tests are included in the default image. Building with
"make image=tests" makes an image with only the tests and nothing else.
Other images (e.g., memcached, cassandra,...) currently do not require
the tests module, so the generated image does will not include the tests.

With this patch, "make image=memcached", for example, contains only the
bare minimum needed for memcached, and the resulting qcow image is just
18 MB, down from 69 MB when we included all the tests.

Fixes #160

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

Pekka noticed that his addr2line responds with '?? ??:0' on
unknown symbol, whereas mine responds with '??\n??:0'.

Reported-by: Pekka Enberg <penberg@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>

It is easier to use that timestamp as argument to --since and --until
of the prof command, which accept full timestamp in nanoseconds.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>