Commit graph

184 commits

Author SHA1 Message Date
David Gibson
feb8946ff5 test: Simplify data handling for transfer tests
Many of our tests are based around performing transfers of sample data
across passt/pasta created links.  The data flow here can be a bit
hard to follow since, e.g. we create a file transfer it to the guest,
then transfer it back to the host across several different tests.
This also means that the test cases aren't independent of each other.

Because we don't have the original file available at both ends in some
cases, we compare them by generating md5sums at each end and comparing
them, which is a bit complicated.

Make a number of changes to simplify this:
  1. Pre-generate the sample data files as a test asset, rather than
     building them on the fly during the tests proper
  2. Include the sample data files in the mbuto guest image
  3. Because we have good copies of the original data available in all
     contexts, we can now simply use 'cmp' to check if the transfer
     has worked, avoiding md5sum complications.
  4. Similarly we can always use the original copy of the sample data
     on the send side of each transfer, meaning that the tests become
     more independent of each other.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-29 12:21:19 +02:00
David Gibson
0a15b467d4 test: Use --config-net for namespace setup
The setup functions for passt_in_ns and two_guests perform some fairly slow
dhclient calls to configure the network in the namespace before starting
the guest.  This isn't really part of the tests, just necessary for the
operations later.

We can simplify and speed this up a bit by using pasta's '--config-net'
option to configure the networking for us.  As a bonus this means we have
at least a minimal test of the --config-net option itself.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-29 12:21:13 +02:00
David Gibson
5b899dce7a test: More robust wait for pasta/passt to be ready
When we start passt or pasta, it may take a short time to be ready to
handle packets, especially if running under valgrind.  We have a
number of semi-arbitrary fixed sleeps to account for this.

We can do this more robustly by exploiting the fact that pasta/passt
doesn't write its pidfile until it's ready to go, so if we wait for
the pidfile to be created, we can proceed with confidence.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-29 12:21:07 +02:00
David Gibson
33983de46b test: Remove unnecessary sleeps from shutdown tests
These are hangovers from older ways of shutting down the pasta/passt
processes and no longer serve any purpose.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-29 12:21:01 +02:00
David Gibson
05a2c7ae3c test: Add wait_for() shell helper
Add a shell helper function to wait for some command to succeed - typically
a test for something to be done by a background process.  Use it in the
context code which waits for the guest to respond to ssh-over-vsock
connections.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-29 12:20:44 +02:00
Stefano Brivio
e0f415c025 test/lib: Don't try to write to perf.js when running demos
...it doesn't actually exist, and this error now causes the demo to
stop.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-24 00:11:10 +02:00
Stefano Brivio
2e93cb6ed8 test/lib: Drop perf_report_append() from perf_report
It's not used anymore. While at it, fix the function name in the
comment to perf_report_append_js().

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-24 00:09:23 +02:00
Stefano Brivio
b3549093f7 test/demo: Avoid using port 5201 on the host
That's the default port for iperf3, which also means that it's quite
likely in use on my test machine. Use different port numbers: recycle
the scheme we use in tests for passt and pasta's demo, use 5221-5224
(a bit shorter) for the slirp4netns container in Podman's demo.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-24 00:07:18 +02:00
Stefano Brivio
6d08bfc3e0 test/demo: Use relative paths to change directories when possible
A cd to __STATEDIR__ results in a rather long command, that's not
very readable. Jump between directories using .. and relative paths,
once we're there.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-24 00:05:20 +02:00
Stefano Brivio
85de88ff31 test/passt.mbuto: Don't fail on missing guest public key
We won't necessarily run mbuto as part of regular tests: it can also
be used for demos or out-of-tree tests.

To keep the profile simple, leave the whole sshd setup there, which
is otherwise harmless, but don't fail if guest-key.pub is missing in
the current directory.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-23 17:55:10 +02:00
Stefano Brivio
d6f865a40a test/distro: Update workarounds for Ubuntu 22.04 on s390x
If we use dhclient without creating a complete network configuration,
systemd-resolved will stop working after a while, and this sometimes
happens while we're still installing packages.

Disable it, together with systemd-networkd, while taking care of
removing the dhclient hook that prevents overriding /etc/resolv.conf.

While at it, it looks like removing snapd and needrestart actually
takes more time than keeping them: drop that line.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-23 02:46:24 +02:00
Stefano Brivio
cff565a1f6 test/lib: Wait for DHCPv4 before starting DHCPv6 client in two_guests test
I'm not sure why, but dhclient hangs otherwise. This reflects what we
do in the passt_in_ns setup steps.

Eventually, this whole block could go away if we let pasta configure
this network namespace with --config-net.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-23 02:46:24 +02:00
Stefano Brivio
1134ce88fe test/perf: Wait for neper servers in guest to be ready before starting client
Starting tcp_rr, tcp_crr, udp_rr servers in the guest takes a bit
longer than starting the corresponding clients on the host, and we
end up starting clients before servers unless we add a delay there.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-23 02:46:24 +02:00
Stefano Brivio
d1dbc4f992 test/lib: Wait for kernel to free up ports used by iperf3 before reusing them
If we start another server on the same port right away, we might fail
to bind the port. A small delay appears to be needed -- I'm not
entirely sure why at this point.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-22 16:54:09 +02:00
Stefano Brivio
856b04490a test/lib: Run also iperf3 clients in background, revert to time-based wait
Unfortunately, this partially counters recent efforts by David to
speed up these tests, but it looks like iperf3 clients don't reliably
terminate, in some rare cases I couldn't isolate yet.

For the moment being, reintroduce the time-based wait approach, now
using the configurable test duration, and terminate the servers at
the end of it, in case they're stuck. There's no point in keeping
the 'sleep 2' later, so drop that, and while at it, make sure that
the stuck servers have time to flush the JSON output before we use
it.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-22 16:54:09 +02:00
Stefano Brivio
df29ebfe02 test/perf: Disable periodic throughput reports to avoid vhost hang
It appears that if we run throughput tests with one-second periodic
reports, the sending side of the vhost channel used for SSH-based
command dispatch occasionally stops working altogether. I haven't
investigated this further, all I see is that output is truncated
at some point, and doesn't resume.

If we use gzip compression (ssh -C) this happens less frequently,
but it still happens, seemingly indicating the issue is probably
related to vhost itself.

Disable periodic reports in iperf3 clients. The -i options were
actually redundant, so remove them from both test files as well as
from test_iperf3().

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-22 16:54:09 +02:00
Stefano Brivio
9f8b783d4a test/lib: Wait on iperf3 clients to be done, then send SIGINT to servers
An iperf3 client might fail to send the control message indicating
the end of the test, if the kernel buffer doesn't accept it, and exit
without having sent it, as the control socket is non-blocking. Should
this happen, the server will just wait forever for this message,
instead of terminating.

Restore some of the behaviour that went away with the
"test: Rewrite test_iperf3" patch: instead of waiting on servers to
terminate, wait on the clients. When they are done, wait 2 seconds,
and then send SIGINT to the servers, which make them still write
out the JSON report before terminating.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-22 16:54:09 +02:00
Stefano Brivio
119bb265a3 test/lib: Restore IFS while executing directives in def blocks
If we don't, guest command dispatch will fail altogether, given that
we use cat(1) on the enter file, which contains spaces.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-22 16:54:09 +02:00
Stefano Brivio
a39398e840 test/perf: Switch performance test duration to 10 seconds instead of 30
It looks like the workaround for the virtio_net TX hang issue is
working less reliably with the new command dispatch mechanism, I'm
not sure why. Switch to 10 seconds, at least for the moment.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-22 16:53:55 +02:00
Stefano Brivio
df3a35c203 test/perf: Always use /sbin/sysctl in tcp test
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-22 16:53:35 +02:00
Stefano Brivio
fafdda083f test/passt_in_ns: Consistent sleep commands before starting socat client
There are some 'sleep 1' commands between starting the socat server
and its corresponding client to avoid races due to the server not
being ready as we start sending data.

However, those don't cover all the cases where we might need them,
and in some cases the sleep command actually ended up being before
the server even starts.

This fixes occasional failures in TCP and UDP simple transfer tests,
that became apparent with the new command dispatch mechanism.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2022-09-22 16:53:35 +02:00
Stefano Brivio
ae51d2dac1 test/perf: Check for /sbin/sysctl with which(1), not simply sysctl
Otherwise, we're depending on having /sbin in $PATH. For some reason
I didn't completely grasp, with the new command dispatch mechanism
that's not the case anymore, even if I have /sbin in $PATH in the
parent shell.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-22 16:53:35 +02:00
Stefano Brivio
b323e5f439 test: term: When checking if status line is a number, hide errors
We use the [ "$x" -eq "$x" ] syntax to check if $x is a number. The
behaviour is clearly implied by POSIX, but some shells might actually
report the (intended) error, and dash floods script.log with
"Illegal number" error messages. Hide them.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-14 20:49:08 +02:00
David Gibson
9fc476af6e test: Simpler termination handling for UDP tests
Because UDP is connectionless we don't have an in-built end-of-stream
signal for our connectivity tests.  We work around this by explicitly
adding an end marker to our sample data and killing the listening end once
it is seen.

However, socat has some built-in options - null-eof and shut-null - which
can be used to signal the end of stream with a zero-length UDP packet.
Use these to simplify how the UDP tests are implemented.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 11:14:29 +02:00
David Gibson
6f2db0e34b test: Move perf.js report file to $LOGDIR/web
The tests generate a performance report in $BASEPATH/perf.js and
hooks/pre-push copies it to the website.  To avoid cluttering the working
directory, instead put perf.js in $LOGDIR/web, since it's a test output
artefact.  Update hooks/pre-push to copy from its new location.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 11:12:41 +02:00
David Gibson
4e9b50f7b6 test: Move video processing files to $STATEBASE
The asciinema video handling creates a number of temporary files (.uncat,
.start, .stop) which currently go into the source tree.  Put them in the
temporary state directory to avoid clutter.

The final processed output is now placed into test_logs/web/ along with the
corresponding .js file with links, since they're essentially test
artefacts.  hooks/pre-push is updated to look for those files in the new
location when updating the web site.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 11:12:41 +02:00
David Gibson
ec6af08ca7 demo: Move pidfiles to state directory
Avoiding putting them in bare /tmp means they will be automatically
cleaned up with everything else.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 11:12:41 +02:00
David Gibson
d813660107 test: Move pidfiles and nsholder sockets into state directory
Currently they go in the passt source tree with a fixed names, which means
their presence can mess with subsequent test runs.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 11:12:41 +02:00
David Gibson
b23adf7132 test: Store pcap files in $LOGDIR instead of /tmp
The capture files are more or less a different form of log output from the
tests, so place them in $LOGDIR.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 11:12:41 +02:00
David Gibson
07aa24499e test: Move pause temporary file to state directory
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 11:12:41 +02:00
David Gibson
1c36c8d3f8 test: Use paths in __STATEDIR__ instead of 'temp' and 'tempdir' directives
Instead of using the 'temp' and 'tempdir' DSL directives to create
temporary files, use fixed paths relative to __STATEDIR__.  This has two
advantages:
  1) The files are automatically cleaned up if the tests fail (and even if
     that doesn't work they're easier to clean up manuall)
  2) When debugging tests it's easier to figure out which of the temporary
     files are relevant to whatever's going wrong

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 11:12:41 +02:00
David Gibson
c71a5e8528 test: Don't redundantly regenerate small test file in pasta/tcp
In what looks like a copy/paste error, pasta/tcp generates its small test
file twice.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 11:12:41 +02:00
David Gibson
54c5addeaa test: Move context temporary files to state dir
Currently the context command dispatch subsystem creates a bunch of
temporary files in $LOGDIR, which is messy.  Store them in $STATEDIR which
is for precisely this purpose.  The logs from each context still go into
$LOGDIR.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 11:12:41 +02:00
David Gibson
e4ecb6d795 test: Move passt_test_log_pipe to state directory
We use this fifo to send messages to the information pane.  Put it in the
state directory so it doesn't need its own cleanup.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 11:12:41 +02:00
David Gibson
8cb7d5b461 test: Create common state directories for temporary files
The test scripts create a bunch of temporary files to keep track of
internal state.  Some are made in /tmp with individual mktemp calls, some
go in the passt source directory, and some go in $LOGDIR.  This can
sometimes make it messy to clean up after failed test runs.

Start cleaning this up by creating a single "state" directory ($STATEBASE)
in /tmp for all the state or temporary files used by a single test run.
Clean it up automatically in cleanup() - except when DEBUG==1, because
those files can be useful for debugging test script failures.

We create subdirectories under $STATEBASE for each setup function, exposed
as $STATESETUP.  We also create subdirectories for each test script and
expose those to the scripts as __STATEDIR__.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 11:12:41 +02:00
David Gibson
1e4a15398f test: Actually run cleanup function
We install a cleanup() function with 'trap' in order to clean up temporary
files we generate during the tests.  However, we deinstall it after
run_term, which means it won't run in most of the cases where it would be
useful.  Even if "run from_term" exits with an error, that error will be
hidden from the run_term wrapper because it's within a tmux session, so we
will return from run_term normally, uninstall the trap and never clean up.

In fact there's no reason to uninstall the trap at all, it works just as
well on the success exit path as an error exit path.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 11:12:41 +02:00
David Gibson
a9b18300b4 test: Remove unused variable FFMPEG_PID_FILE
FFPMPEG_PID_FILE is set (creating a temporary file), then never used.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 11:12:41 +02:00
David Gibson
594c2f9371 test: Group tests by mode then protocol, rather than the reverse
For example, passt/dhcp rather than dhcp/passt.  This is more
consistent with the two_guests and other test groups, and makes some
other cleanups simpler.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 11:12:41 +02:00
David Gibson
4381278b7d test: Use new-style command issue for passt_in_ns tests
Put the pieces together to use the new style context based dispatch for
the passt_in_pasta tests.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 05:32:00 +02:00
David Gibson
87a57156cf test: Use context system for two_guests tests
Now that we have all the pieces we need for issuing commands both into
namespaces and into guests, we can use those to convert the two_guests to
using only the new style context command issue.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 05:32:00 +02:00
David Gibson
56e755ba11 test: Use context system for guest commands
Extends the context system in the test scripts to allow executing commands
within a guest.  Do this without requiring an existing network in the guest
by using socat to run ssh via a vsock connection.

We do need some additional "sleep"s in the tests, because the new
faster dispatch means that sometimes we attempt to connect before
socat has managed to listen.

For now, only use this for the plain "passt" tests.  The "passt_in_ns" and
other tests have additional complications we still need to deal with.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 05:32:00 +02:00
David Gibson
a8a437abd6 test: Extend context system to run commands in namespace for pasta tests
Extend the context system to allow commands to be run in a namespace
created with unshare, and use it for the namespace used in the pasta tests.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 05:32:00 +02:00
David Gibson
a8c32c85d5 test: Add nsholder utility
In our test scripts we need to do some ugly parsing of /proc and/or pstree
output in order to get the PIDs of processes running in namespaces so that
we can connect to those namespaces with nsenter or pasta.

This is actually a pretty tricky problem with standard tools.  To determine
the PID from the outside of the namespace we need to know how the process
of interest is related to the unshare or pasta process (child? one of
several children? grandchild?) as well as then parsing /proc or ps output.
This is slightly awkward now, and will get worse with future changes I'd
like to make to have processes are dispatched.

The obvious solution would be to have the process of interest (which we
control) report its own PID, but that doesn't work easily, because it is in
a PID namepace and sees only its local PID not the global PID we need to
address it from outside.

To handle this, add a small custom tool, "nsholder".  This takes a path
and a mode parameter.  In "hold" mode it will create a unix domain socket
bound to the path and listening.  In "pid" mode it will get the "hold"ing
process's pid via the unix socket using SO_PEERCRED, which translates
between PID namespaces.  In "stop" mode it will send a message to the
socket causing the "hold"ing process to clean up and exit.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 05:32:00 +02:00
David Gibson
2f87265d8b test: Use new-style contexts for passt pane in the pasta and passt tests
Convert the pasta and passt tests to use new-style context execution
for the things that run in the "passt" frame.  Don't touch the
passt_in_ns or two_guests tests yet, because they run passt inside a
namespace which introduces some additional complications we have yet
to handle.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 05:32:00 +02:00
David Gibson
d9b9c252cd test: Issue host commands via context for most tests
Convert most of the tests to use the new-style system for issuing commands
for all host commands.  We leave the distro tests for now: they use
the same pane for both host and guest commands which we'll need some more
things to deal with.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 05:32:00 +02:00
David Gibson
c2f248588b test: Integration of old-style pane execution and new context execution
We're creating a system for tests to more reliably execute commands in
various contexts (e.g. host, guest, namespace).  That transition is going
to happen over a number of steps though, so in the meantime we need to deal
with both the old-style issuing of commands via typing into and screen
scraping tmux panels, and the new-style system for executing commands in
context.

Introduce some transitional helpers which will issue a command via context
if the requested context is initialized, but will otherwise fall back to
the old style tmux panel based method.  Re-implement the various test DSL
commands in terms of these new helpers.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 05:32:00 +02:00
David Gibson
a32df9b6f4 test: Allow a tmux pane to watch commands executed in contexts
We're moving to a new way of the tests dispatching commands to running in
contexts (host, guest, namespace, etc.).  As we make this transition,
though, we still want the user to be able to watch the commands running
in a context, as they previously could from the commands issued in the
pane.

Add a helper to set up a pane to watch a context's log to allow this.  In
some cases we currently issue commands from several different logical
contexts in the same pane, so allow a pane to watch several contexts at
once.  Also use tail's --retry option to allow starting the watch before
we've initialized the context which will be useful in some cases.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 05:32:00 +02:00
David Gibson
3c47b65a64 test: Context execution helpers
For the tests, we need to run commands in various contexts: in the host,
in a guest or in a namespace.  Currently we do this by running each context
in a tmux pane, and using tmux commands to type the commands into the
relevant pane, then screen-scrape the output for the results if we need
them.

This is very fragile, because we have to make various assumptions to parse
the output.  Those can break if a shell doesn't have the prompt we expect,
if the tmux pane is too small or in various other conditions.

This starts some library functions for a new "context" system, that
provides a common way to invoke commands in a given context, in a way that
properly preserves stdout, stderr and the process return code.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 05:32:00 +02:00
David Gibson
4fbd0a8ff9 test: Correctly match "background" with "wait" commands
Our test DSL has a number of paired commands to run something in the
background in a pane, then later to wait for it to complete.  However, in
some of the tests we have these mismatched - starting a command in one
pane, then waiting for it in another.

We appear to get away with this for some reason, but it's not correct and
future changes make it cause more problems.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-13 05:32:00 +02:00
David Gibson
e5e10aff81 test: Rewrite test_iperf3
test_iperf3() is a pretty inscrutable mess of nested background processes.
It has a number of ugly sleeps needed to wait for things to complete.

Rewrite it to be cleaner:
  * Use the construct (a & b & wait) to run 'a' and 'b' in parallel, but
    then wait for them both to complete before continuing
  * This allows us to wait for both the server and client to finish, rather
    than sleeping
  * Use jq to do all the math we need to get the final result, rather than
    jq followed by some complicated 'bc' mangling

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
2022-09-07 11:01:22 +02:00