We have a number of delays when we switch to new layouts that were
added to make the tests visually easier to follow, together with
blinking status bars. Shorten the delays and avoid blinking the
status bar if $FAST is set to 1 (no demo mode).
Shorten delays in busy loops to 10ms, instead of 100ms, and skip the
one-second fixed delay when we wait for the status of a command.
Cut the duration of throughput and latency tests to one second, down
from ten. Somewhat surprisingly, the results we get are rather
consistent, and not significantly different from what we'd get with
10 seconds.
This, together with Podman's commit 20f3e8909e3a ("test/system:
pasta_test_do add explicit port check"), cuts the time needed on my
setup for full test run from approximately 37 minutes to...:
$ time ./run
[exited]
PASS: 165, FAIL: 0
Log at /home/sbrivio/passt/test/test_logs/test.log
real 15m34.253s
user 0m0.011s
sys 0m0.011s
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
Tested-by: David Gibson <david@gibson.dropbear.id.au>
When using the old-style "pane" methods of executing commands during the
tests, we need to scan the shell output for prompts in order to tell when
commands have finished. This is inherently unreliable because commands
could output things that look like prompts, and prompts might not look like
we expect them to. The only way to really fix this is to use a better way
of dispatching commands, like the newer "context" system.
However, it's awkward to convert everything to "context" right at the
moment, so we're still relying on some tests that do work most of the time.
It is, however, particularly sensitive to fancy coloured prompts using
escape sequences. Currently we try to handle this by stripping actual
ESC characters with tr, then looking for some common variants.
We can do a bit better: instead strip all escape sequences using sed before
looking for our prompt. Or, at least, any one using [a-zA-Z] as the
terminating character. Strictly speaking ANSI escapes can be terminated by
any character in 0x40..0x7e, which isn't easily expressed in a regexp.
This should capture all common ones, though.
With this transformation we can simplify the list of patterns we then look
for as a prompt, removing some redundant variants.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
In practical terms, passt doesn't benefit from the additional
protection offered by the AGPL over the GPL, because it's not
suitable to be executed over a computer network.
Further, restricting the distribution under the version 3 of the GPL
wouldn't provide any practical advantage either, as long as the passt
codebase is concerned, and might cause unnecessary compatibility
dilemmas.
Change licensing terms to the GNU General Public License Version 2,
or any later version, with written permission from all current and
past contributors, namely: myself, David Gibson, Laine Stump, Andrea
Bolognani, Paul Holzinger, Richard W.M. Jones, Chris Kuhn, Florian
Weimer, Giuseppe Scrivano, Stefan Hajnoczi, and Vasiliy Ulyanov.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
This can be used for generic cell values with an arbitrary scale.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
To keep this simple, only support tests that have corresponding setup
and teardown functions implied by their path. For example:
./run passt/ndp
will trigger the 'passt' setup and teardown functions.
This is not really elegant, but it looks robust, and while David is
considering proper alternatives, it should be quite useful.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
We use the [ "$x" -eq "$x" ] syntax to check if $x is a number. The
behaviour is clearly implied by POSIX, but some shells might actually
report the (intended) error, and dash floods script.log with
"Illegal number" error messages. Hide them.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
The asciinema video handling creates a number of temporary files (.uncat,
.start, .stop) which currently go into the source tree. Put them in the
temporary state directory to avoid clutter.
The final processed output is now placed into test_logs/web/ along with the
corresponding .js file with links, since they're essentially test
artefacts. hooks/pre-push is updated to look for those files in the new
location when updating the web site.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
We use this fifo to send messages to the information pane. Put it in the
state directory so it doesn't need its own cleanup.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
The test scripts create a bunch of temporary files to keep track of
internal state. Some are made in /tmp with individual mktemp calls, some
go in the passt source directory, and some go in $LOGDIR. This can
sometimes make it messy to clean up after failed test runs.
Start cleaning this up by creating a single "state" directory ($STATEBASE)
in /tmp for all the state or temporary files used by a single test run.
Clean it up automatically in cleanup() - except when DEBUG==1, because
those files can be useful for debugging test script failures.
We create subdirectories under $STATEBASE for each setup function, exposed
as $STATESETUP. We also create subdirectories for each test script and
expose those to the scripts as __STATEDIR__.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
Extends the context system in the test scripts to allow executing commands
within a guest. Do this without requiring an existing network in the guest
by using socat to run ssh via a vsock connection.
We do need some additional "sleep"s in the tests, because the new
faster dispatch means that sometimes we attempt to connect before
socat has managed to listen.
For now, only use this for the plain "passt" tests. The "passt_in_ns" and
other tests have additional complications we still need to deal with.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
We're creating a system for tests to more reliably execute commands in
various contexts (e.g. host, guest, namespace). That transition is going
to happen over a number of steps though, so in the meantime we need to deal
with both the old-style issuing of commands via typing into and screen
scraping tmux panels, and the new-style system for executing commands in
context.
Introduce some transitional helpers which will issue a command via context
if the requested context is initialized, but will otherwise fall back to
the old style tmux panel based method. Re-implement the various test DSL
commands in terms of these new helpers.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
We're moving to a new way of the tests dispatching commands to running in
contexts (host, guest, namespace, etc.). As we make this transition,
though, we still want the user to be able to watch the commands running
in a context, as they previously could from the commands issued in the
pane.
Add a helper to set up a pane to watch a context's log to allow this. In
some cases we currently issue commands from several different logical
contexts in the same pane, so allow a pane to watch several contexts at
once. Also use tail's --retry option to allow starting the watch before
we've initialized the context which will be useful in some cases.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
The DEMO_XTERM and CI_XTERM variables defined in test/lib/term aren't used
anywhere. Remove them.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
Having all those 'echo $?' is rather distracting in demos.
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Sefano Brivio <sbrivio@redhat.com>
When we use pane_wait to wait for a command issued to a tmux pane to finish
we have no idea whether the command succeeded or not. This means that the
test scripts can keep running long after the point something vital has
failed, making it difficult to work out what went wrong.
Add a new pane_status command that checks for success of the issued command
and use it in most places instead of pane_wait. We still need explicit
pane_wait where we're gathering explicit output with pane_parse, because
the way we check the status with 'echo $?' means we lose track of that
output.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
[sbrivio:
- instead of quitting the script, make a test fail if a command
issued in a pane fails during a test, and loop until the status code is
numeric in pane_status() as a hack to make it a bit more robust
- retain usage of pane_wait() in iperf3 and teardown functions as we
interrupt iperf3, passt, and pasta, so a non-zero exit code is expected
- drop bogus ns_{1,2}_wait() calls in teardown_two_guests(), those
functions were never implemented
- use pane_status() for "guest" test directives too
]
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
pane_parse() attempts to grab the output from the last command issued
into a tmux pane. It strips out control characters using tr, which in
particular includes the final \r\n. However, this won't fully strip
out terminal escape sequences. In particular this breaks if the shell
in the pane is bash, with enable-bracketed-paste enabled in readline.
That issues terminal sequences to enable and disable bracketed paste
mode around every shell prompt.
We can work around this because these escapes are followed by a \r
(CR). More generally, it seems reasonable to assume that any terminal
shenanigans followed by a CR, but not an LF is supposed to be hidden.
So, use sed to strip everything before the second last CR. We still
need the tr to remove the final \r\n from the string (sed processes a
line at a time, and doesn't consider the CRLF part of the buffer it's
processing).
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
[sbrivio: modify regexp to keep foo\r\r\n unchanged, by matching on at
least one CR and a non-CR afterwards: that's the usual output pattern
for bash on Debian 8 and Debian 9]
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
run_term() uses tmux set-option -g to globally set the default shell.
Unfortunately this hits a chicken-and-egg problem that's common with many
of tmux's session options. If there isn't already a tmux server running,
we can't connect to set the option. If we attempt this after starting our
session (and therefore the server), then the session will already be
started with the previous default shell.
In any case it's not a good idea to set tmux global options, since that
might interfere with whatever else the user is doing in tmux. So, instead
set the default-shell option locally to the session after starting it. To
make sure we get the right shell for our initial script, explicitly invoke
/bin/sh to interpret it.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
The semantics of tmux's update-environment option are a bit confusing.
It says it means the given variables are copied into the session
environment from the source environment, but it's not entirely clear
what the "source" environment means.
From my experimentation it appeast to be the environment from which
the tmux *server* is launched, not the one issuing the 'new-session'
command. That makes it pretty much useles, certainly in our case where
we have no way of knowing if the user has pre-existing tmux sessions.
Instead use the new-session -e option to explicitly pass in the variables
we want to propagate.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
For demos, cool-retro-term(1) looked fancier, but several threads of
that and ffmpeg(1) are just messing up with performance testing.
The CI videos started getting really big as well, and they were
difficult to read.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
...showing setup steps, some peculiarities as --net option, and a
general side-to-side comparison with slirp4netns(1), including
"quick" TCP and UDP throughput and latency benchmarks.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
We might have highlighting and slightly different prompts across
different distributions, allow a more reasonable set of prompt
strings to be accepted as prompts.
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
