Commit graph

273 commits

Author SHA1 Message Date
Stefano Brivio
81128241d6 tcp: Set TCP_TAP_FRAMES back to 32
Now that we fixed the issue with small receiving buffers, we can
safely increase this back and get slightly lower syscall overhead.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-10-04 22:21:21 +02:00
Stefano Brivio
683043e200 tcp: Probe net.core.{r,w}mem_max, don't set SO_{RCV,SND}BUF if low
If net.core.rmem_max and net.core.wmem_max sysctls have low values,
we can get bigger buffers by not trying to set them high -- the
kernel would lock their values to what we get.

Try, instead, to get bigger buffers by queueing as much as possible,
and if maximum values in tcp_wmem and tcp_rmem are bigger than this,
that will work.

While at it, drop QUICKACK option for non-spliced sockets, I set
that earlier by mistake.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-10-04 22:20:43 +02:00
Stefano Brivio
e1a2e2780c tcp: Check if connection is local or low RTT was seen before using large MSS
If the connection is local or the RTT was comparable to the time it
takes to queue a batch of messages, we can safely use a large MSS
regardless of the sending buffer, but otherwise not.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-10-04 22:20:43 +02:00
Stefano Brivio
f6bff339a9 tcp: Adjust usage of sending buffer depending on its size
If we start with a very small sending buffer, we can make the kernel
expand it if we cause the congestion window to get bigger, but this
won't reliably happen if we use just half (other half is accounted
as overhead).

Scale usage depending on its own size, we might eventually get some
retransmissions because we can't queue messages the sender sends us
in-window, but it's better than keeping that small buffer forever.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-10-04 22:20:43 +02:00
Stefano Brivio
2408ddffa3 tcp: Derive MSS announced to guest/namespace from configured MTU if present
...and from the sending socket only if the MTU is not configured.

Otherwise, a connection to a host from a local guest, with a
non-loopback destination address, will get its MSS from the MTU of the
outbound interface with that address, which is unnecessary as we know
the guest can send us larger segments.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-29 16:46:58 +02:00
Stefano Brivio
9657b6ed05 conf, tcp: Periodic detection of bound ports for pasta port forwarding
Detecting bound ports at start-up time isn't terribly useful: do this
periodically instead, if configured.

This is only implemented for TCP at the moment, UDP is somewhat more
complicated: leave a TODO there.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-27 11:23:44 +02:00
Stefano Brivio
904b86ade7 tcp: Rework window handling, timers, add SO_RCVLOWAT and pools for sockets/pipes
This introduces a number of fundamental changes that would be quite
messy to split. Summary:

- advertised window scaling can be as big as we want, we just need
  to clamp window sizes to avoid exceeding the size of our "discard"
  buffer for unacknowledged data from socket

- add macros to compare sequence numbers

- force sending ACK to guest/tap on PSH segments, always in pasta
  mode, whenever we see an overlapping segment, or when we reach a
  given threshold compared to our window

- we don't actually use recvmmsg() here, fix comments and label

- introduce pools for pre-opened sockets and pipes, to decrease
  latency on new connections

- set receiving and sending buffer sizes to the maximum allowed,
  kernel will clamp and round appropriately

- defer clean-up of spliced and non-spliced connection to timer

- in tcp_send_to_tap(), there's no need anymore to keep a large
  buffer, shrink it down to what we actually need

- introduce SO_RCVLOWAT setting and activity tracking for spliced
  connections, to coalesce data moved by splice() calls as much as
  possible

- as we now have a compacted connection table, there's no need to
  keep sparse bitmaps tracking connection activity -- simply go
  through active connections with a loop in the timer handler

- always clamp the advertised window to half our sending buffer,
  too, to minimise retransmissions from the guest/tap

- set TCP_QUICKACK for originating socket in spliced connections,
  there's no need to delay them

- fix up timeout for unacknowledged data from socket

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-27 01:28:02 +02:00
Stefano Brivio
3c839bfc46 tcp: Drop TODO about sequence collision attacks
A random initial sequence number based on a secret has already been
there for a while.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-27 01:28:02 +02:00
Stefano Brivio
dd581730e5 tap: Completely de-serialise input message batches
Until now, messages would be passed to protocol handlers in a single
batch only if they happened to be dequeued in a row. Packets
interleaved between different connections would result in multiple
calls to the same protocol handler for a single connection.

Instead, keep track of incoming packet descriptors, arrange them in
sequences, and call protocol handlers only as we completely sorted
input messages in batches.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-27 01:28:02 +02:00
Stefano Brivio
522878e6bb tcp: Decrease TCP_TAP_FRAMES to 8
This significantly improves fairness in serving concurrent connections.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-27 01:28:02 +02:00
Stefano Brivio
e9961cecfc pasta, tcp: Update comment about spliced connection states
...we now have SPLICE_FIN_{FROM,TO,BOTH} too.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-27 01:28:02 +02:00
Stefano Brivio
9b6769d53b tcp: Don't reset connection from ESTABLISHED state on EPOLLHUP
That might just mean we shut down the socket -- but we still have to
go through the other states to ensure a orderly shutdown guest-side.

While at it, drop the EPOLLHUP check for unhandled states: we should
never hit that, but if we do, resetting the connection at that point
is probably the wrong thing to do.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-16 08:50:02 +02:00
Stefano Brivio
62bace390b pasta, tcp: Mask EPOLLIN and EPOLLRDHUP after sending FIN
Now that we dropped EPOLLET, we'll keep getting EPOLLRDHUP, and
possibly EPOLLIN, even if there's nothing to read anymore.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-16 08:50:02 +02:00
Stefano Brivio
492b58d64b pasta, tcp: Break splice() loop once we've written everything that was read
That's a guarantee that we don't need to retry writing.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-16 08:50:02 +02:00
Stefano Brivio
34dd4b28b0 pasta, tcp: Don't set SPLICE_FIN_BOTH state on EPOLLHUP
EPOLLHUP just means we shut down one side of the connection on
*one* socket: remember, we have two sockets here.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-16 08:50:02 +02:00
Stefano Brivio
e8540b3f26 pasta, tcp: Don't reset 'never_read' flag on write retries
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-16 08:50:02 +02:00
Stefano Brivio
7ecf693297 pasta, tcp: Don't set TCP_CORK on spliced sockets
...throughput isn't everything: this leads (of course) to horrible
latency with small, sparse messages. As a consequence, there's no
need to set TCP_NODELAY either.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-16 08:49:58 +02:00
Stefano Brivio
a7eb8bb2f6 tcp: Fix setting window from maximum ACK sequence in batch
If we're at the first message in a batch, it's safe to get the
window value from it, and there's no need to subtract anything for
a comparison on that's not even done -- we'll override it later in
any case if we find messages with a higher ACK sequence number.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-16 08:20:50 +02:00
Stefano Brivio
3be131280d pasta, tcp: Set pipe descriptor numbers to -1 after closing
...so that we don't try to close them again, even if harmless.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-16 08:19:39 +02:00
Stefano Brivio
d481578882 pasta, tcp: Drop EPOLLET for spliced, established connections
...tcp_handler_splice() doesn't guarantee we read all the available
data, the sending buffer might be full.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-16 08:17:18 +02:00
Stefano Brivio
45d9b0000e tcp: Read SO_SNDBUF unconditionally
Checking it only when the cached value is smaller than the current
window of the receiver is not enough: it might shrink further while
the receiver window is growing.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-16 08:14:01 +02:00
Stefano Brivio
474b8e6fb7 pasta: Clean up FIN connection flags once a connection is deleted
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-15 10:43:59 +02:00
Stefano Brivio
57d17292f9 pasta: Set spliced connection flag in epoll reference on compaction
...otherwise, we'll mix indices with non-spliced connections.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-15 10:41:31 +02:00
Stefano Brivio
9af8e0a1a7 tcp: Request retransmission with updated sequence also on partial write to socket
If we couldn't write the whole batch of received packets to the socket,
and we have missing segments, we still need to request their
retransmission right away, otherwise it will take ages for the guest to
figure out we're missing them.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-14 16:57:50 +02:00
Stefano Brivio
a616357c86 tcp: In ESTABLISHED state, acknowledge segments as they're sent to the socket
...instead of waiting for the remote peer to do that -- it's
especially important in case we request retransmissions from the
guest, but it also helps speeding up slow start. This should
probably be a configurable behaviour in the future.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-14 16:57:50 +02:00
Stefano Brivio
621c589d36 tcp: Properly time out ACK wait from tap
Seen with iperf3: a control connection is established, no data flows
for a while, all segments are acknowledged. The socket starts closing
it, and we immediately time out because the last ACK from tap was one
minute before that.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-14 16:57:50 +02:00
Stefano Brivio
7c82ea4dd9 tcp: Don't mistake a FIN segment with no data for a Fast Retransmit request
It carries no data and usually duplicates the previous ACK sequence,
but it's just a FIN.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-14 16:57:50 +02:00
Stefano Brivio
c162f1e801 tcp: Check errno on sendmmsg() failure, not just the return value
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-14 16:57:50 +02:00
Stefano Brivio
2c009e8e6f tcp: Make sure sending window is initialised before sending to tap
Seen with iperf3: the first packet from socket (data connection) is
65520 bytes and doesn't fit in the window.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-14 16:57:50 +02:00
Stefano Brivio
e58828f340 tcp: Fixes for closing states, spliced connections, out-of-order packets, etc.
This fixes a number of issues found with some heavier testing with
uperf and neper:

- in most closing states, we can still accept data, check for EPOLLIN
  when appropriate

- introduce a new state, ESTABLISHED_SOCK_FIN_SENT, to track the fact
  we already sent a FIN segment to the tap device, for proper sequence
  number bookkeeping

- for pasta mode only: spliced connections also need tracking of
  (inferred) FIN segments and clean half-pipe shutdowns

- streamline resetting epoll_wait bitmaps with a new function,
  tcp_tap_epoll_mask(), instead of repeating the logic all over the
  place

- set EPOLLET for tap connections too, whenever we are waiting for
  EPOLLRDHUP or an event from the tap to proceed with data transfer,
  to avoid useless loops with EPOLLIN set

- impose an additional limit on the sending window advertised to the
  guest, given by SO_SNDBUF: it makes no sense to completely fill
  the sending buffer and send a zero window: stop a bit before we
  hit that

- handle *all* interrupted system calls as needed

- simplify the logic for reordering of out-of-order segments received
  from tap: it's not a corner case, and the previous logic allowed
  for deadloops

- fix comparison of seen IPv4 address when we get a new connection
  from a socket directed to the configured guest address

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-09 15:40:04 +02:00
Stefano Brivio
647a413794 tcp, udp: Restore usage of gateway for guest to connect to local host
This went lost in a recent rework: if the guest wants to connect
directly to the host, it can use the address of the default gateway.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-01 17:00:27 +02:00
Stefano Brivio
1e49d194d0 passt, pasta: Introduce command-line options and port re-mapping
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-01 17:00:27 +02:00
Stefano Brivio
1b1b27c06a tcp: Fixes for early data in SOCK_SYN_SENT, closing states, clamping window
More details here after rebase.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-09-01 16:49:21 +02:00
Stefano Brivio
d2272f74f7 tcp: Proper error handling for sendmmsg() to UNIX domain socket
As data from socket is forwarded to the guest, sendmmsg() might send
fewer bytes than requested in three different ways:

- failing altogether with a negative error code -- ignore that,
  we'll get an error on the UNIX domain socket later if there's
  really an issue with it and reset the connection to the guest

- sending less than 'vlen' messages -- instead of assuming success
  in that case and waiting for the guest to send a duplicate ACK
  indicating missing data, update the sequence number according to
  what was actually sent and spare some retransmissions

- somewhat unexpectedly to me, sending 'vlen' or less than 'vlen'
  messages, returning up to 'vlen', with the last message being
  partially sent, and no further indication of errors other than
  the returned msg_len for the last partially sent message being
  less than iov_len.

  In this case, we would assume success and proceed as nothing
  happened. However, qemu would fail to parse any further message,
  having received a partial descriptor, and eventually close the
  connection, logging:

	serious error: oversized packet received,connection terminated.

  as the length descriptor for the next message would be sourced
  from the middle of the next successfully sent message, not from
  its header.

  Handle this by checking the msg_len returned for the last (even
  partially) sent message, and force re-sending the missing bytes,
  if any, with a blocking sendmsg() -- qemu must not receive
  anything else than that anyway.

While at it, allow to send up to 64KiB for each message, the
previous 32KiB limit isn't actually required, and just switch to a
new message at each iteration on sending buffers, they are already
MSS-sized anyway, so the check in the loop isn't really needed.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-08-26 23:30:22 +02:00
Stefano Brivio
cc2ebfd5f2 tcp: Never send ACK because of pending unacknowleged data when sending SYN
With a kernel older than 5.3 (no_snd_wnd set), ack_pending in
tcp_send_to_tap() might be true at the beginning of a new connection
initiated by a socket. This means we send the first SYN segment to the
tap together with ACK set, which is clearly invalid and triggers the
receiver to reply with an RST segment right away.

Set ack_pending to 0 whenever we're sending a SYN segment. In case of a
SYN, ACK segment sent by the caller, the caller passes the ACK flag
explicitly.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-08-24 18:27:24 +02:00
Stefano Brivio
f2e3b9defd tcp: Drop EPOLLET for non-spliced connections
Socket-facing functions don't guarantee that all data is handled before
they return: stick to level-triggered mode for TCP sockets.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-08-24 18:24:11 +02:00
Stefano Brivio
539dcf5add tcp: Fast re-transmit, more fixes for closing states and no_snd_wnd
...and while at it, fix an issue in the calculation of the last IOV
buffer size: if we can't receive enough data to fill up the window,
the last buffer can be filled completely.

Also streamline the code setting iovec lengths if cached values are
not matching.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-08-04 01:35:45 +02:00
Stefano Brivio
0017bc3c3e tcp: Always allow ACKs when pending, fixes for no_snd_wnd and closing states
We won't necessarily have another choice to ACK in a timely fashion
if we skip ACKs from a number of states (including ESTABLISHED) when
there's enough window left. Check for ACKed bytes as soon as it makes
sense.

If the sending window is not reported by the kernel, ACK as soon as
we queue onto the socket, given that we're forced to use a rather
small window.

In FIN_WAIT_1_SOCK_FIN, we also have to account for the FIN flag sent
by the peer in the sequence.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-08-04 01:29:59 +02:00
Stefano Brivio
c62490ffa8 tcp: Lower TCP_TAP_FRAMES to 32
Sending 64 frames in a batch looks quite bad when a duplicate ACK
comes right at the beginning of it. Lowering this to 32 doesn't
affect performance noticeably, with 16 the impact is more apparent.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-08-04 01:28:21 +02:00
Stefano Brivio
dc169643a4 tcp: Full batched processing for tap messages
Similar to UDP, but using a simple sendmsg() on iovec-style buffers
from tap instead, as we don't need to preserve message boundaries.

A quick test in PASTA mode, from namespace to init via tap:

 # ip link set dev pasta0 mtu 16384
 # iperf3 -c 192.168.1.222 -t 60
   [...]
 [ ID] Interval           Transfer     Bitrate
 [  5]   0.00-60.00  sec  80.4 GBytes  11.5 Gbits/sec                  receiver

 # iperf3 -c 2a02:6d40:3cfc:3a01:2b20:4a6a:c25a:3056 -t 60
   [...]
 [ ID] Interval           Transfer     Bitrate
 [  5]   0.00-60.01  sec  39.9 GBytes  5.71 Gbits/sec                  receiver

 # ip link set dev pasta0 mtu 65520
 # iperf3 -c 192.168.1.222 -t 60
   [...]
 [ ID] Interval           Transfer     Bitrate
 [  5]   0.00-60.01  sec  88.7 GBytes  12.7 Gbits/sec                  receiver

 # iperf3 -c 2a02:6d40:3cfc:3a01:2b20:4a6a:c25a:3056 -t 60
   [...]
 [ ID] Interval           Transfer     Bitrate
 [  5]   0.00-60.00  sec  79.5 GBytes  11.4 Gbits/sec                  receiver

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-07-27 01:35:58 +02:00
Stefano Brivio
fd5050ccba tcp: Limit TCP_INFO getsockopt() syscalls
There's no need to constantly query the socket for number of
acknowledged bytes if we're far from exhausting the sending window,
just do it if we're at least down to 90% of it.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-07-27 00:50:53 +02:00
Stefano Brivio
8af961b85b tcp, udp: Map source address to gateway for any traffic from 127.0.0.0/8
...instead of just 127.0.0.1.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-07-26 18:20:01 +02:00
Stefano Brivio
0279ec8eae tcp: Fix re-send mechanism to tap on ACK timeout
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-07-26 14:25:16 +02:00
Stefano Brivio
74677bddb2 tcp: Simplify ACK accounting, skip some useless operations on tap handling
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-07-26 14:23:34 +02:00
Stefano Brivio
39ad062100 tcp: Introduce scatter-gather IO path from socket to tap
...similarly to what was done for UDP. Quick performance test with
32KiB buffers, host to VM:

$ iperf3 -c 192.0.2.2 -N
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  8.47 GBytes  7.27 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  8.45 GBytes  7.26 Gbits/sec                  receiver

$ iperf3 -c 2a01:598:88ba:a056:271f:473a:c0d9:abc1
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  8.43 GBytes  7.24 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  8.41 GBytes  7.22 Gbits/sec                  receiver

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-07-26 14:20:36 +02:00
Stefano Brivio
86b273150a tcp, udp: Allow binding ports in init namespace to both tap and loopback
Traffic with loopback source address will be forwarded to the direct
loopback connection in the namespace, and the tap interface is used
for the rest.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-07-26 14:10:29 +02:00
Stefano Brivio
16b08367a5 tap: Fill the IPv6 flow label field to represent flow association
This isn't optional: TCP streams must carry a unique, hard-to-guess,
non-zero label for each direction. Linux, probably among others,
will otherwise refuse to associate packets in a given stream to the
same connection.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-07-26 07:30:57 +02:00
Stefano Brivio
60dee2705b tcp: Don't open a new connection from tap if both SYN and ACK are set
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-07-21 17:48:33 +02:00
Stefano Brivio
49631a38a6 tcp, udp: Split IPv4 and IPv6 bound port sets
Allow to bind IPv4 and IPv6 ports to tap, namespace or init separately.

Port numbers of TCP ports that are bound in a namespace are also bound
for UDP for convenience (e.g. iperf3), and IPv4 ports are always bound
if the corresponding IPv6 port is bound (socket might not have the
IPV6_V6ONLY option set). This will also be configurable later.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-07-21 17:44:39 +02:00
Stefano Brivio
b508079c4c tcp: Replace source address also if it's the same as the guest address
...not just for loopback addresses, with the address of the default
gateway. Otherwise, the guest might receive packets with source and
destination set to the same address.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-07-21 12:05:58 +02:00
Stefano Brivio
1642a04f48 tcp: Increase maximum window scaling factor from 8 to 9
This is actually reasonable in terms of memory consumption and
allows for better performance with local services.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-07-21 12:05:03 +02:00
Stefano Brivio
4667226bb0 tcp: Fix partial (ACK) message coalescing, ACK timeout, MSG_MORE flag setting
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-07-17 17:57:11 +02:00
Stefano Brivio
33482d5bf2 passt: Add PASTA mode, major rework
PASTA (Pack A Subtle Tap Abstraction) provides quasi-native host
connectivity to an otherwise disconnected, unprivileged network
and user namespace, similarly to slirp4netns. Given that the
implementation is largely overlapping with PASST, no separate binary
is built: 'pasta' (and 'passt4netns' for clarity) both link to
'passt', and the mode of operation is selected depending on how the
binary is invoked. Usage example:

	$ unshare -rUn
	# echo $$
	1871759

	$ ./pasta 1871759	# From another terminal

	# udhcpc -i pasta0 2>/dev/null
	# ping -c1 pasta.pizza
	PING pasta.pizza (64.190.62.111) 56(84) bytes of data.
	64 bytes from 64.190.62.111 (64.190.62.111): icmp_seq=1 ttl=255 time=34.6 ms

	--- pasta.pizza ping statistics ---
	1 packets transmitted, 1 received, 0% packet loss, time 0ms
	rtt min/avg/max/mdev = 34.575/34.575/34.575/0.000 ms
	# ping -c1 spaghetti.pizza
	PING spaghetti.pizza(2606:4700:3034::6815:147a (2606:4700:3034::6815:147a)) 56 data bytes
	64 bytes from 2606:4700:3034::6815:147a (2606:4700:3034::6815:147a): icmp_seq=1 ttl=255 time=29.0 ms

	--- spaghetti.pizza ping statistics ---
	1 packets transmitted, 1 received, 0% packet loss, time 0ms
	rtt min/avg/max/mdev = 28.967/28.967/28.967/0.000 ms

This entails a major rework, especially with regard to the storage of
tracked connections and to the semantics of epoll(7) references.

Indexing TCP and UDP bindings merely by socket proved to be
inflexible and unsuitable to handle different connection flows: pasta
also provides Layer-2 to Layer-2 socket mapping between init and a
separate namespace for local connections, using a pair of splice()
system calls for TCP, and a recvmmsg()/sendmmsg() pair for UDP local
bindings. For instance, building on the previous example:

	# ip link set dev lo up
	# iperf3 -s

	$ iperf3 -c ::1 -Z -w 32M -l 1024k -P2 | tail -n4
	[SUM]   0.00-10.00  sec  52.3 GBytes  44.9 Gbits/sec  283             sender
	[SUM]   0.00-10.43  sec  52.3 GBytes  43.1 Gbits/sec                  receiver

	iperf Done.

epoll(7) references now include a generic part in order to
demultiplex data to the relevant protocol handler, using 24
bits for the socket number, and an opaque portion reserved for
usage by the single protocol handlers, in order to track sockets
back to corresponding connections and bindings.

A number of fixes pertaining to TCP state machine and congestion
window handling are also included here.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-07-17 11:04:22 +02:00
Stefano Brivio
90078ebc59 tcp: Add support for kernels not exporting tcpi_snd_wnd via TCP_INFO
Before commit 8f7baad7f035 ("tcp: Add snd_wnd to TCP_INFO"), the
kernel didn't export tcpi_snd_wnd via TCP_INFO, which means we don't
know what's the window size of the receiver, socket-side.

To get TCP connections working in that case, ignore this value if
it's zero during handshake, and use the initial window value as
suggested by RFC 6928 (14 600 bytes, instead of 4 380 bytes), to
keep network performance usable.

To make the TCP dynamic responsive enough in this case, also check
the socket for available data whenever we get an ACK segment from
tap, instead of waiting until all the data from the tap is dequeued.

While at it, fix the window scaling value sent for SYN and SYN, ACK
segments: we want to increase the data pointer after writing the
option, not the value itself.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-06-08 02:20:28 +02:00
Stefano Brivio
8b39b0b47f tcp: Fix window size in initial SYN, ACK segment to guest
During handshake, the initial SYN, ACK segment to the guest, send as
a response to the SYN segment, needs to report the unscaled value for
the window, given that the handshake hasn't completed yet.

While at it, fix the endianness for the window value in case TCP
parameters can't be queried via TCP_INFO and we need to use the
default value.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-06-05 14:54:12 +02:00
Stefano Brivio
8ce188ecb0 tcp: Properly initialise parameters for SO_ACCEPTCONN getsockopt()
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-05-21 11:14:51 +02:00
Stefano Brivio
bd5aaaac7f tcp: Actually enforce MAX_CONNS limit
and, given that the connection table is indexed by socket number,
we also need to increase MAX_CONNS now as the ICMP implementation
needs 2^17 sockets, that will be opened before TCP connections are
accepted.

This needs to be changed later: the connection table should be
indexed by a translated number -- we're wasting 2^17 table entries
otherwise. Move initialisation of TCP listening sockets as last
per-protocol initialisation, this will make it easier.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-05-21 11:14:50 +02:00
Stefano Brivio
7e3e36b2c2 tcp: Close socket on EPOLLHUP or EPOLLRDHUP in non-data state
If the peer doesn't shut down orderly, this might happen: just close
the socket then.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-05-21 11:14:42 +02:00
Stefano Brivio
e07f539ae0 udp, passt: Introduce socket packet buffer, avoid getsockname() for UDP
This is in preparation for scatter-gather IO on the UDP receive path:
save a getsockname() syscall by setting a flag if we get the numbering
of all bound sockets in a strict sequence (expected, in practice) and
repurpose the tap buffer to be also a socket receive buffer, passing
it down to protocol handlers.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-04-30 14:52:18 +02:00
Stefano Brivio
605af213c5 udp: Connection tracking for ephemeral, local ports, and related fixes
As we support UDP forwarding for packets that are sent to local
ports, we actually need some kind of connection tracking for UDP.
While at it, this commit introduces a number of vaguely related fixes
for issues observed while trying this out. In detail:

- implement an explicit, albeit minimalistic, connection tracking
  for UDP, to allow usage of ephemeral ports by the guest and by
  the host at the same time, by binding them dynamically as needed,
  and to allow mapping address changes for packets with a loopback
  address as destination

- set the guest MAC address whenever we receive a packet from tap
  instead of waiting for an ARP request, and set it to broadcast on
  start, otherwise DHCPv6 might not work if all DHCPv6 requests time
  out before the guest starts talking IPv4

- split context IPv6 address into address we assign, global or site
  address seen on tap, and link-local address seen on tap, and make
  sure we use the addresses we've seen as destination (link-local
  choice depends on source address). Similarly, for IPv4, split into
  address we assign and address we observe, and use the address we
  observe as destination

- introduce a clock_gettime() syscall right after epoll_wait() wakes
  up, so that we can remove all the other ones and pass the current
  timestamp to tap and socket handlers -- this is additionally needed
  by UDP to time out bindings to ephemeral ports and mappings between
  loopback address and a local address

- rename sock_l4_add() to sock_l4(), no semantic changes intended

- include <arpa/inet.h> in passt.c before kernel headers so that we
  can use <netinet/in.h> macros to check IPv6 address types, and
  remove a duplicate <linux/ip.h> inclusion

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-04-29 17:15:26 +02:00
Stefano Brivio
db1fe773a3 tcp: Avoid SO_ACCEPTCONN getsockopt() by noting listening/data sockets numbers
...the rest is reshuffling existing macros to use the bits we need in
TCP code.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-04-29 17:15:26 +02:00
Stefano Brivio
48afbe321e tcp: Preserve data sent during SOCK_SYN_SENT state
Seen with iperf3 server on tap side: connection state is SOCK_SYN_SENT,
we haven't got an ACK from the tap yet (that's why we're not in
ESTABLISHED), but a data packet comes. Don't read this data until we
reach the ESTABLISHED state, by keeping EPOLLIN disabled until that
point.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-04-29 17:12:59 +02:00
Stefano Brivio
38b50dba47 passt: Spare some syscalls, add some optimisations from profiling
Avoid a bunch of syscalls on forwarding paths by:

- storing minimum and maximum file descriptor numbers for each
  protocol, fall back to SO_PROTOCOL query only on overlaps

- allocating a larger receive buffer -- this can result in more
  coalesced packets than sendmmsg() can take (UIO_MAXIOV, i.e. 1024),
  so make sure we don't exceed that within a single call to protocol
  tap handlers

- nesting the handling loop in tap_handler() in the receive loop,
  so that we have better chances of filling our receive buffer in
  fewer calls

- skipping the recvfrom() in the UDP handler on EPOLLERR -- there's
  nothing to be done in that case

and while at it:

- restore the 20ms timer interval for periodic (TCP) events, I
  accidentally changed that to 100ms in an earlier commit

- attempt using SO_ZEROCOPY for UDP -- if it's not available,
  sendmmsg() will succeed anyway

- fix the handling of the status code from sendmmsg(), if it fails,
  we'll try to discard the first message, hence return 1 from the
  UDP handler

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-04-23 22:22:37 +02:00
Stefano Brivio
6488c3e848 tcp, udp: Replace loopback source address by gateway address
This is symmetric with tap operation and addressing model, and
allows again to reach the guest behind the tap interface by
contacting the local address.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-04-22 17:03:43 +02:00
Stefano Brivio
1f7cf04d34 passt: Introduce packet batching mechanism
Receive packets in batches from AF_UNIX, check if they can be sent
with a single syscall, and batch them up with sendmmsg() in case.

A bit rudimentary, currently only implemented for UDP, but it seems
to work.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-04-22 13:39:36 +02:00
Stefano Brivio
9f80499313 tcp: Don't dereference IPv4 addresses
...sometimes they're not valid pointers.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-03-20 22:19:15 +01:00
Stefano Brivio
48ca38c606 passt: Run in background, add message logging with severities
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-03-18 12:58:07 +01:00
Stefano Brivio
cd14bff5ea tcp: Add struct for TCP execution context, move hash_secret to it
We don't need to keep small data as static variables, move the only
small variable we have so far to the new struct.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-03-17 10:57:41 +01:00
Stefano Brivio
bb9fb9e2d1 tcp: Introduce hash table for socket lookup for packets from tap
Replace the dummy, full array scan implementation, by a hash table
based on SipHash, with chained hashing for collisions.

This table is also statically allocated, and it's simply an array
of socket numbers. Connection entries are chained by pointers in
the connection entry itself, which now also contains socket number
and hash bucket index to keep removal reasonably fast.

New entries are inserted at the head of the chain, that is, the most
recently inserted entry is directly mapped from the bucket.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-03-17 10:57:40 +01:00
Stefano Brivio
4f675d63e8 tcp: Ignore out-of-order ACKs from tap instead of resetting connection
We might receive out-of-order ACK packets from the tap device, just
like any other packet.

I guess I've been overcautious and regarded it as a condition we
can't recover from, but all that happens is that we have already seen
a higher ACK sequence number, which means that data has been already
received and discarded from the buffer. We have to ignore the lower
sequence number we receive later, though, because that would force
the buffer bookkeeping into throwing away more data than expected.

Drop the ACK sequence assignment from tcp_tap_handler(), which was
redundant, and let tcp_sock_consume() take exclusive care of that.

Now that tcp_sock_consume() can never fail, make it a void, and
drop checks from callers.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-03-17 10:57:39 +01:00
Stefano Brivio
a418946837 tcp: Add siphash implementation for initial sequence numbers
Implement siphash routines for initial TCP sequence numbers (12 bytes
input for IPv4, 36 bytes input for IPv6), and while at it, also
functions we'll use later on for hash table indices and TCP timestamp
offsets (with 8, 20, 32 bytes of input).

Use these to set the initial sequence number, according to RFC 6528,
for connections originating either from the tap device or from
sockets.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-03-17 10:57:36 +01:00
Stefano Brivio
8bca388e8a passt: Assorted fixes from "fresh eyes" review
A bunch of fixes not worth single commits at this stage, notably:

- make buffer, length parameter ordering consistent in ARP, DHCP,
  NDP handlers

- strict checking of buffer, message and option length in DHCP
  handler (a malicious client could have easily crashed it)

- set up forwarding for IPv4 and IPv6, and masquerading with nft for
  IPv4, from demo script

- get rid of separate slow and fast timers, we don't save any
  overhead that way

- stricter checking of buffer lengths as passed to tap handlers

- proper dequeuing from qemu socket back-end: I accidentally trashed
  messages that were bundled up together in a single tap read
  operation -- the length header tells us what's the size of the next
  frame, but there's no apparent limit to the number of messages we
  get with one single receive

- rework some bits of the TCP state machine, now passive and active
  connection closes appear to be robust -- introduce a new
  FIN_WAIT_1_SOCK_FIN state indicating a FIN_WAIT_1 with a FIN flag
  from socket

- streamline TCP option parsing routine

- track TCP state changes to stderr (this is temporary, proper
  debugging and syslogging support pending)

- observe that multiplying a number by four might very well change
  its value, and this happens to be the case for the data offset
  from the TCP header as we check if it's the same as the total
  length to find out if it's a duplicated ACK segment

- recent estimates suggest that the duration of a millisecond is
  closer to a million nanoseconds than a thousand of them, this
  trend is now reflected into the timespec_diff_ms() convenience
  routine

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-02-21 11:55:49 +01:00
Stefano Brivio
105b916361 passt: New design and implementation with native Layer 4 sockets
This is a reimplementation, partially building on the earlier draft,
that uses L4 sockets (SOCK_DGRAM, SOCK_STREAM) instead of SOCK_RAW,
providing L4-L2 translation functionality without requiring any
security capability.

Conceptually, this follows the design presented at:
	https://gitlab.com/abologna/kubevirt-and-kvm/-/blob/master/Networking.md

The most significant novelty here comes from TCP and UDP translation
layers. In particular, the TCP state and translation logic follows
the intent of being minimalistic, without reimplementing a full TCP
stack in either direction, and synchronising as much as possible the
TCP dynamic and flows between guest and host kernel.

Another important introduction concerns addressing, port translation
and forwarding. The Layer 4 implementations now attempt to bind on
all unbound ports, in order to forward connections in a transparent
way.

While at it:
- the qemu 'tap' back-end can't be used as-is by qrap anymore,
  because of explicit checks now introduced in qemu to ensure that
  the corresponding file descriptor is actually a tap device. For
  this reason, qrap now operates on a 'socket' back-end type,
  accounting for and building the additional header reporting
  frame length

- provide a demo script that sets up namespaces, addresses and
  routes, and starts the daemon. A virtual machine started in the
  network namespace, wrapped by qrap, will now directly interface
  with passt and communicate using Layer 4 sockets provided by the
  host kernel.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
2021-02-16 09:28:55 +01:00