Currently we start both the iperf3 server(s) and client(s) afresh each time
we want to make a bandwidth measurement. That's not really necessary as
usually a whole batch of bandwidth measurements can use the same server.
Split up the iperf3 directive into 3 directives: iperf3s to start the
server, iperf3 to make a measurement and iperf3k to kill the server, so
that we can start the server less often. This - and more importantly, the
reduced number of waits for the server to be ready - reduces runtime of the
performance tests on my laptop by about 4m (out of ~28minutes).
For now we still restart the server between IPv4 and IPv6 tests. That's
because in some cases the latency measurements we make in between use the
same ports.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
iperf3 generates statistics about its run on both the client and server
sides. They don't have exactly the same information, but both have the
pieces we need (AFAICT the server communicates some nformation to the
client over the control socket, so the most important information is in the
client side output, even if measured by the server).
Currently we use the server side information for our measurements. Using
the client side information has several advantages though:
* We can directly wait for the client to complete and we know we'll have
the output we want. We don't need to sleep to give the server time to
write out the results.
* That in turn means we can wrap up as soon as the client is done, we
don't need to wait overlong to make sure everything is finished.
* The slightly different organisation of the data in the client output
means that we always want the same json value, rather than requiring
slightly different onces for UDP and TCP.
The fact that we avoid some extra delays speeds up the overal run of the
perf tests by around 7 minutes (out of around 35 minutes) on my laptop.
The fact that we no longer unconditionally kill client and server after
a certain time means that the client could run indefinitely if the server
doesn't respond. We mitigate that by setting 1s connect timeout on the
client. This isn't foolproof - if we get an initial response, but then
lose connectivity this could still run indefinitely, however it does cover
by far the most likely failure cases. --snd-timeout would provide more
robustness, but I've hit odd failures when trying to use it.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
In practical terms, passt doesn't benefit from the additional
protection offered by the AGPL over the GPL, because it's not
suitable to be executed over a computer network.
Further, restricting the distribution under the version 3 of the GPL
wouldn't provide any practical advantage either, as long as the passt
codebase is concerned, and might cause unnecessary compatibility
dilemmas.
Change licensing terms to the GNU General Public License Version 2,
or any later version, with written permission from all current and
past contributors, namely: myself, David Gibson, Laine Stump, Andrea
Bolognani, Paul Holzinger, Richard W.M. Jones, Chris Kuhn, Florian
Weimer, Giuseppe Scrivano, Stefan Hajnoczi, and Vasiliy Ulyanov.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
...instead of doing it after the test. Now that we have pre-built
guest images, we might also have old JSON files from previous,
interrupted test runs.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
This can be used for generic cell values with an arbitrary scale.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
To test log files on a tmpfs mount, we need to unshare the mount
namespace, which means using a context for the passt pane is not
really practical at the moment, as we can't open a shell there, so
we would have to encapsulate all the commands under 'unshare -rUm',
plus the "inner" pasta command, running in turn a tcp_rr server.
It might be worth fixing this by e.g. detecting we are trying to
spawn an interactive shell and adding a special path in the context
setup with some form of stdin redirection -- I'm not sure it's doable
though.
For this reason, add a new layout, using a context only for the host
pane, while keeping the old command dispatch mechanism for the passt
pane.
We also need a new setup function that doesn't start pasta: we want
to start and restart it with different options.
Further, we need a 'pint' directive, to send an interrupt to the
passt pane: add that in lib/test.
All the tests before the one involving tmpfs and a detached mount
namespace were also tested with the context mechanism. To make an
eventual conversion easier, pass tcp_crr directly as a command on
pasta's command line where feasible.
While at it, fix the comment to the teardown_pasta() function.
The new test set can be semi-conveniently run as:
./run pasta_options/log_to_file
and it checks basic log creation, size of the log file after flooding
it with debug entries, rotations, and basic consistency after
rotations, on both an existing filesystem and a tmpfs, chosen as
it doesn't support collapsing data ranges via fallocate(), hence
triggering the fall-back mechanism for logging rotation.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
If we start another server on the same port right away, we might fail
to bind the port. A small delay appears to be needed -- I'm not
entirely sure why at this point.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Unfortunately, this partially counters recent efforts by David to
speed up these tests, but it looks like iperf3 clients don't reliably
terminate, in some rare cases I couldn't isolate yet.
For the moment being, reintroduce the time-based wait approach, now
using the configurable test duration, and terminate the servers at
the end of it, in case they're stuck. There's no point in keeping
the 'sleep 2' later, so drop that, and while at it, make sure that
the stuck servers have time to flush the JSON output before we use
it.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
It appears that if we run throughput tests with one-second periodic
reports, the sending side of the vhost channel used for SSH-based
command dispatch occasionally stops working altogether. I haven't
investigated this further, all I see is that output is truncated
at some point, and doesn't resume.
If we use gzip compression (ssh -C) this happens less frequently,
but it still happens, seemingly indicating the issue is probably
related to vhost itself.
Disable periodic reports in iperf3 clients. The -i options were
actually redundant, so remove them from both test files as well as
from test_iperf3().
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
An iperf3 client might fail to send the control message indicating
the end of the test, if the kernel buffer doesn't accept it, and exit
without having sent it, as the control socket is non-blocking. Should
this happen, the server will just wait forever for this message,
instead of terminating.
Restore some of the behaviour that went away with the
"test: Rewrite test_iperf3" patch: instead of waiting on servers to
terminate, wait on the clients. When they are done, wait 2 seconds,
and then send SIGINT to the servers, which make them still write
out the JSON report before terminating.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
If we don't, guest command dispatch will fail altogether, given that
we use cat(1) on the enter file, which contains spaces.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Instead of using the 'temp' and 'tempdir' DSL directives to create
temporary files, use fixed paths relative to __STATEDIR__. This has two
advantages:
1) The files are automatically cleaned up if the tests fail (and even if
that doesn't work they're easier to clean up manuall)
2) When debugging tests it's easier to figure out which of the temporary
files are relevant to whatever's going wrong
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
The test scripts create a bunch of temporary files to keep track of
internal state. Some are made in /tmp with individual mktemp calls, some
go in the passt source directory, and some go in $LOGDIR. This can
sometimes make it messy to clean up after failed test runs.
Start cleaning this up by creating a single "state" directory ($STATEBASE)
in /tmp for all the state or temporary files used by a single test run.
Clean it up automatically in cleanup() - except when DEBUG==1, because
those files can be useful for debugging test script failures.
We create subdirectories under $STATEBASE for each setup function, exposed
as $STATESETUP. We also create subdirectories for each test script and
expose those to the scripts as __STATEDIR__.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
We're creating a system for tests to more reliably execute commands in
various contexts (e.g. host, guest, namespace). That transition is going
to happen over a number of steps though, so in the meantime we need to deal
with both the old-style issuing of commands via typing into and screen
scraping tmux panels, and the new-style system for executing commands in
context.
Introduce some transitional helpers which will issue a command via context
if the requested context is initialized, but will otherwise fall back to
the old style tmux panel based method. Re-implement the various test DSL
commands in terms of these new helpers.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
test_iperf3() is a pretty inscrutable mess of nested background processes.
It has a number of ugly sleeps needed to wait for things to complete.
Rewrite it to be cleaner:
* Use the construct (a & b & wait) to run 'a' and 'b' in parallel, but
then wait for them both to complete before continuing
* This allows us to wait for both the server and client to finish, rather
than sleeping
* Use jq to do all the math we need to get the final result, rather than
jq followed by some complicated 'bc' mangling
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
Currently all the throughput tests are run for 30s. This is reflected in
both the actual parameters given to the iperf commands, but also in the
matching sleeps in test_iperf3.
Allow this to be adjusted more easily with a new parameter to test_iperf3.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
[sbrivio: Reflect new parameter in comment to test_iperf3()]
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
These two commands in the DSL to run an iperf client and server are always
used together, and some of the parameters must match between them. The
iperf3s must also be run more or less immediately after iperf3c, since
iperf3c will run a client in the background after a sleep and requires a
server to be running before it will work.
A bunch of things can be made cleaner if we make a single DSL command that
runs both sides of the test. For now make the combined command work
exactly like the two commands together did, warts and all.
This does lose the ability for the DSL scripts to give additional options
to the iperf3 server, but we weren't using that anyway.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
The iperf based test commands create a bunch of .bw and .pid files for
each iperf client and server. The server side .bw files are cleaned
up afterwards, but the pid files are not, and none of the client side
files are cleaned up. The latter doesn't really matter when the
client is run on ephemeral guests, but sometimes we run it in a
namespace that shares the filesystem with the host.
Clean up all of these files after the tests.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
Currently test/run uses wildcards to run all of the tests in a directory.
However, that wildcard list is filtered down by the "onlyfor" directives
in the test files... usually to a single file.
Therefore, just explicitly list the files we *really* want to run for this
test mode. This makes it easier to see at the top level what tests will
be executed, and to change that list temporarily while debugging specific
failures.
This means the "onlyfor" directive no longer has any purpose, and we can
remove it. "onlyfor" was also the only used of the $MODE variable, so we
can remove that too.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
The top level listing control of which tests to run is in test/run, however
it uses the test() function which runs an entire directory of test files,
filtered by some criteria. This makes it awkward to narrow down to a
subset of tests when debugging a specific failure.
To make this easier, have test() take an explicit list of test files to
run, and have the caller in test/run handle the directory traversal. The
construct we use for this is pretty awkward to handle the fact that we're
in the source tree root directory rather than test/ at this point in
test/run. Later cleanups will improve that.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
The test scripts support a "req" directive which requires one test script
to be run before another. It's implemented by doing a topological sort
based on these directives in the runner scripts, which is about as awkward
as you'd expect in Bourne shell.
It turns out we only use this functionality in one place - to make the
"make install" test run after the plain "make" test. We also already have
a simpler way of making sure tests run in a specific order: just put them
into the same test script file.
So, remove support for the "req" directive and just fold the build/all and
build/install test scripts together.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
A number of passt/pasta testcases have initial steps which are just about
building images or other assets we need for the test proper. Repeating
these for each test run can be quite costly.
This patch makes a start on moving this sort of test asset building to
a separate phase before running the tests proper. For now just add a
Makefile to handle the asset building (although it doesn't build
anything yet), and make the path where we'll be building the assets
available to the tests.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
...there are no 'test' directives in demo, and this causes a
script failure.
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Now that we have pane_status to check the success of commands issued to
panes, we can more easily check for the success of the 'which' commands
used to check tool availability, rather than constructing, then parsing
special "skip" output.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
When we use pane_wait to wait for a command issued to a tmux pane to finish
we have no idea whether the command succeeded or not. This means that the
test scripts can keep running long after the point something vital has
failed, making it difficult to work out what went wrong.
Add a new pane_status command that checks for success of the issued command
and use it in most places instead of pane_wait. We still need explicit
pane_wait where we're gathering explicit output with pane_parse, because
the way we check the status with 'echo $?' means we lose track of that
output.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
[sbrivio:
- instead of quitting the script, make a test fail if a command
issued in a pane fails during a test, and loop until the status code is
numeric in pane_status() as a hack to make it a bit more robust
- retain usage of pane_wait() in iperf3 and teardown functions as we
interrupt iperf3, passt, and pasta, so a non-zero exit code is expected
- drop bogus ns_{1,2}_wait() calls in teardown_two_guests(), those
functions were never implemented
- use pane_status() for "guest" test directives too
]
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Most commands issued during the testing scripts aren't explicitly checked
for errors. Therefore, if they fail, the shell will just keep on
executing. This makes it difficult to figure out where things started
going wrong if things fall over.
Run the whole script with the set -e mode so that it will exit in the case
of any (unchecked) failing command. To make this work we do need to add
explicit checks / fallbacks for some commands which we expect to fail.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
[sbrivio: use sh -e instead of setting -e later, so that we don't miss
anything before set -e is issued]
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
The DEBUG option for tests/run enables debugging options to passt/pasta,
however that doesn't help with debugging the test scripts themselves, which
are fairly fragile.
Extend the DEBUG option so it also prints information on each command in
the test scripts to make it easier to work out where things are falling
over.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
...showing setup steps, some peculiarities as --net option, and a
general side-to-side comparison with slirp4netns(1), including
"quick" TCP and UDP throughput and latency benchmarks.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
For distribution tests, we'll repeat some tests frequently. Add a
'def' directive that starts a block, ended by 'endef', whose
execution can then be triggered by simply giving its name as a
directive itself.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
