Stefano Brivio 0515adceaa passt, pasta: Namespace-based sandboxing, defer seccomp policy application

To reach (at least) a conceptually equivalent security level as
implemented by --enable-sandbox in slirp4netns, we need to create a
new mount namespace and pivot_root() into a new (empty) mountpoint, so
that passt and pasta can't access any filesystem resource after
initialisation.

While at it, also detach IPC, PID (only for passt, to prevent
vulnerabilities based on the knowledge of a target PID), and UTS
namespaces.

With this approach, if we apply the seccomp filters right after the
configuration step, the number of allowed syscalls grows further. To
prevent this, defer the application of seccomp policies after the
initialisation phase, before the main loop, that's where we expect bad
things to happen, potentially. This way, we get back to 22 allowed
syscalls for passt and 34 for pasta, on x86_64.

While at it, move #syscalls notes to specific code paths wherever it
conceptually makes sense.

We have to open all the file handles we'll ever need before
sandboxing:

- the packet capture file can only be opened once, drop instance
  numbers from the default path and use the (pre-sandbox) PID instead

- /proc/net/tcp{,v6} and /proc/net/udp{,v6}, for automatic detection
  of bound ports in pasta mode, are now opened only once, before
  sandboxing, and their handles are stored in the execution context

- the UNIX domain socket for passt is also bound only once, before
  sandboxing: to reject clients after the first one, instead of
  closing the listening socket, keep it open, accept and immediately
  discard new connection if we already have a valid one

Clarify the (unchanged) behaviour for --netns-only in the man page.

To actually make passt and pasta processes run in a separate PID
namespace, we need to unshare(CLONE_NEWPID) before forking to
background (if configured to do so). Introduce a small daemon()
implementation, __daemon(), that additionally saves the PID file
before forking. While running in foreground, the process itself can't
move to a new PID namespace (a process can't change the notion of its
own PID): mention that in the man page.

For some reason, fork() in a detached PID namespace causes SIGTERM
and SIGQUIT to be ignored, even if the handler is still reported as
SIG_DFL: add a signal handler that just exits.

We can now drop most of the pasta_child_handler() implementation,
that took care of terminating all processes running in the same
namespace, if pasta started a shell: the shell itself is now the
init process in that namespace, and all children will terminate
once the init process exits.

Issuing 'echo $$' in a detached PID namespace won't return the
actual namespace PID as seen from the init namespace: adapt
demo and test setup scripts to reflect that.

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>

2022-02-21 13:41:13 +01:00

24 KiB

Raw Blame History

While functional and tested to some extent, this project is still in early development phase: don't use in production or critical environments yet.

passt: Plug A Simple Socket Transport

passt implements a translation layer between a Layer-2 network interface and native Layer-4 sockets (TCP, UDP, ICMP/ICMPv6 echo) on a host. It doesn't require any capabilities or privileges, and it can be used as a simple replacement for Slirp.

pasta: Pack A Subtle Tap Abstraction

pasta (same binary as passt, different command) offers equivalent functionality, for network namespaces: traffic is forwarded using a tap interface inside the namespace, without the need to create further interfaces on the host, hence not requiring any capabilities or privileges.

It also implements a tap bypass path for local connections: packets with a local destination address are moved directly between Layer-4 sockets, avoiding Layer-2 translations, using the splice(2) and recvmmsg(2)/sendmmsg(2) system calls for TCP and UDP, respectively.

Motivation
Features
Interfaces and Environment
Services
Addresses
Protocols
Ports
Demo
Continuous Integration
Performance
Try it
Contribute
Security and Vulnerability Reports

Motivation

passt

When container workloads are moved to virtual machines, the network traffic is typically forwarded by interfaces operating at data link level. Some components in the containers ecosystem (such as service meshes), however, expect applications to run locally, with visible sockets and processes, for the purposes of socket redirection, monitoring, port mapping.

To solve this issue, user mode networking, as provided e.g. by libslirp, can be used. Existing solutions implement a full TCP/IP stack, replaying traffic on sockets that are local to the pod of the service mesh. This creates the illusion of application processes running on the same host, eventually separated by user namespaces.

While being almost transparent to the service mesh infrastructure, that kind of solution comes with a number of downsides:

three different TCP/IP stacks (guest, adaptation and host) need to be traversed for every service request
addressing needs to be coordinated to create the pretense of consistent addresses and routes between guest and host environments. This typically needs a NAT with masquerading, or some form of packet bridging
the traffic seen by the service mesh and observable externally is a distant replica of the packets forwarded to and from the guest environment:
- TCP congestion windows and network buffering mechanisms in general operate differently from what would be naturally expected by the application
- protocols carrying addressing information might pose additional challenges, as the applications don't see the same set of addresses and routes as they would if deployed with regular containers

passt implements a thinner layer between guest and host, that only implements what's strictly needed to pretend processes are running locally. The TCP adaptation doesn't keep per-connection packet buffers, and reflects observed sending windows and acknowledgements between the two sides. This TCP adaptation is needed as passt runs without the CAP_NET_RAW capability: it can't create raw IP sockets on the pod, and therefore needs to map packets at Layer-2 to Layer-4 sockets offered by the host kernel.

The problem and this approach are illustrated in more detail, with diagrams, here.

pasta

On Linux, regular users can create network namespaces and run application services inside them. However, connecting namespaces to other namespaces and to external hosts requires the creation of network interfaces, such as veth pairs, which needs in turn elevated privileges or the CAP_NET_ADMIN capability. pasta, similarly to slirp4netns, solves this problem by creating a tap interface available to processes in the namespace, and mapping network traffic outside the namespace using native Layer-4 sockets.

Existing approaches typically implement a full, generic TCP/IP stack for this translation between data and transport layers, without the possibility of speeding up local connections, and usually requiring NAT. pasta:

avoids the need for a generic, full-fledged TCP/IP stack by coordinating TCP connection dynamics between sender and receiver
offers a fast bypass path for local connections: if a process connects to another process on the same host across namespaces, data is directly forwarded using pairs of Layer-4 sockets
with default options, maps routing and addressing information to the namespace, avoiding any need for NAT

Features

Protocols

✅ IPv4
- ✅ all features, except for
- ❌ fragmentation
✅ IPv6
- ✅ all features, except for
- ❌ fragmentation
- ❌ jumbograms
✅ TCP
- ✅ Window Scaling (RFC 7323)
- ✅ Defenses against Sequence Number Attacks (RFC 6528)
- ⌚ Protection Against Wrapped Sequences (PAWS, RFC 7323)
- ⌚ Timestamps (RFC 7323)
- ❌ Selective Acknowledgment (RFC 2018)
✅ UDP
✅ ICMP/ICMPv6 Echo
⌚ IGMP/MLD proxy
⌚ SCTP

Portability

Linux
- ✅ starting from 4.18 kernel version
- ✅ starting from 3.13 kernel version
🛠 build-time selection of AVX2 instructions (as much as possible)
⌚ musl and uClibc-ng
⌚ FreeBSD, Darwin
⌚ NetBSD, OpenBSD
⌚ Win2k

Security

✅ no dynamic memory allocation (sbrk(2), brk(2), mmap(2) blocked via seccomp)
✅ root operation not allowed outside user namespaces
✅ all capabilities dropped, other than CAP_NET_BIND_SERVICE (if granted)
✅ with default options, user, mount, IPC, UTS, PID namespaces are detached
✅ no external dependencies (other than a standard C library)
✅ restrictive seccomp profiles (22 syscalls allowed for passt, 34 for pasta on x86_64)
✅ static checkers in continuous integration (clang-tidy, cppcheck)
🛠️ rework of TCP state machine (flags instead of states), TCP timers, and code de-duplication
🛠️ clearly defined packet abstraction
🛠️ ~5 000 LoC target
⌚ fuzzing, packetdrill tests
⌚ stricter synflood protection
💡 your ideas here

Configurability

✅ all addresses, ports, port ranges
✅ optional NAT, not required
✅ all protocols
✅ pasta: auto-detection of bound ports
🛠 run-time configuration of port ranges without autodetection
🛠 configuration of port ranges for autodetection
💡 your ideas here

Performance

✅ maximum two (cache hot) copies on every data path
✅ pasta: zero-copy for local connections by design (no configuration needed)
✅ generalised coalescing and batching on every path for every supported protocol
✅ 4 to 50 times IPv4 TCP throughput of existing, conceptually similar solutions depending on MTU (UDP and IPv6 hard to compare)
🛠 vhost-user support for maximum one copy on every data path and lower request-response latency
⌚ multithreading
⌚ raw IP socket support if CAP_NET_RAW is granted
⌚ eBPF support (might not improve performance over vhost-user)

Interfaces

✅ qemu, libvirt support with qrap wrapper
✅ out-of-tree patches for qemu and libvirt available
🛠 bug-to-bug compatible slirp4netns replacement (rootless Podman, RootlessKit)
🛠 native qemu, libvirt support
⌚ drop-in replacement for VPNKit (rootless Docker)

Availability

✅ convenience unofficial packages for Debian, RPM-based distributions on x86_64 (static builds)
✅ testing on non-x86 architectures
🛠 official OpenSUSE packages
⌚ packages for Debian, Fedora, etc.

Services

✅ built-in ARP proxy
✅ minimalistic DHCP server
✅ minimalistic NDP proxy with router advertisements and SLAAC support
✅ minimalistic DHCPv6 server
⌚ fine-grained configurability of DHCP, NDP, DHCPv6 options

Interfaces and Environment

passt exchanges packets with qemu via UNIX domain socket, using the socket back-end in qemu. Currently, qemu can only connect to a listening process via TCP. Two temporary solutions are available:

a patch for qemu
a wrapper, qrap, that connects to a UNIX domain socket and starts qemu, which can now use the file descriptor that's already opened

This approach, compared to using a tap device, doesn't require any security capabilities, as we don't need to create any interface.

pasta runs out of the box with any recent (post-3.8) Linux kernel.

Services

passt and pasta provide some minimalistic implementations of networking services:

ARP proxy, that resolves the address of the host (which is used as gateway) to the original MAC address of the host
DHCP server, a simple implementation handing out one single IPv4 address to the guest or namespace, namely, the same address as the first one configured for the upstream host interface, and passing the nameservers configured on the host
NDP proxy, which can also assign prefix and nameserver using SLAAC
DHCPv6 server: a simple implementation handing out one single IPv6 address to the guest or namespace, namely, the the same address as the first one configured for the upstream host interface, and passing the nameservers configured on the host

Addresses

For IPv4, the guest or namespace is assigned, via DHCP, the same address as the upstream interface of the host, and the same default gateway as the default gateway of the host. Addresses are translated in case the guest is seen using a different address from the assigned one.

For IPv6, the guest or namespace is assigned, via SLAAC, the same prefix as the upstream interface of the host, the same default route as the default route of the host, and, if a DHCPv6 client is running in the guest or namespace, also the same address as the upstream address of the host. This means that, with a DHCPv6 client in the guest or namespace, addresses don't need to be translated. Should the client use a different address, the destination address is translated for packets going to the guest or to the namespace.

Local connections with passt

For UDP and TCP, for both IPv4 and IPv6, packets from the host addressed to a loopback address are forwarded to the guest with their source address changed to the address of the gateway or first hop of the default route. This mapping is reversed on the other way.

Local connections with pasta

Packets addressed to a loopback address in either namespace are directly forwarded to the corresponding (or configured) port in the other namespace. Similarly as passt, packets from the non-init namespace addressed to the default gateway, which are therefore sent via the tap device, will have their destination address translated to the loopback address.

Protocols

passt and pasta support TCP, UDP and ICMP/ICMPv6 echo (requests and replies). More details about the TCP implementation are available here, and for the UDP implementation here.

An IGMP/MLD proxy is currently work in progress.

Ports

passt

To avoid the need for explicit port mapping configuration, passt can bind to all unbound non-ephemeral (0-49152) TCP and UDP ports. Binding to low ports (0-1023) will fail without additional capabilities, and ports already bound (service proxies, etc.) will also not be used. Smaller subsets of ports, with port translations, are also configurable.

UDP ephemeral ports are bound dynamically, as the guest uses them.

If all ports are forwarded, service proxies and other services running in the container need to be started before passt starts.

pasta

With default options, pasta scans for bound ports on init and non-init namespaces, and automatically forwards them from the other side. Port forwarding is fully configurable with command line options.

Demo

pasta

passt

Continuous Integration

Test logs here.

Performance

Try it

passt

build from source:
```
  git clone https://passt.top/passt
  cd passt
  make
```
- alternatively, static builds for x86_64, with or without AVX2 instructions, as of the latest commit are also available for convenience here and here. Convenience, non-official packages for Debian (and derivatives) and RPM-based distributions are also available there. These binaries and packages are simply built with:
```
  CFLAGS="-static" make avx2
  make pkgs
  make static
  make pkgs
```
have a look at the man page for synopsis and options:
```
  man ./passt.1
```
run the demo script, that creates a network namespace called passt, sets up sets up a veth pair and and addresses, together with NAT for IPv4 and NDP proxying for IPv6, then starts passt in the network namespace:
```
  doc/demo.sh
```
from the same network namespace, start qemu. At the moment, qemu doesn't support UNIX domain sockets for the socket back-end. Two alternatives:
- use the qrap wrapper, which maps a tap socket descriptor to passt's UNIX domain socket, for example:
```
  ip netns exec passt ./qrap 5 qemu-system-x86_64 ... -net socket,fd=5 -net nic,model=virtio ...
```
- or patch qemu with this patch and start it like this:
```
  qemu-system-x86_64 ... -net socket,connect=/tmp/passt.socket -net nic,model=virtio
```

alternatively, you can use libvirt, with this patch, to start qemu (with the patch mentioned above), with this kind of network interface configuration:

  <interface type='client'>
    <mac address='52:54:00:02:6b:60'/>
    <source path='/tmp/passt.socket'/>
    <model type='virtio'/>
    <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
  </interface>

and that's it, you should now have TCP connections, UDP, and ICMP/ICMPv6 echo working from/to the guest for IPv4 and IPv6
to connect to a service on the VM, just connect to the same port directly with the address of the network namespace. For example, to ssh to the guest, from the main namespace on the host:
```
  ssh 192.0.2.2
```

pasta

build from source:
```
  git clone https://passt.top/passt
  cd passt
  make
```
- alternatively, static builds for x86_64, with or without AVX2 instructions, as of the latest commit are also available for convenience here and here. Convenience, non-official packages for Debian (and derivatives) and RPM-based distributions are also available there. These binaries and packages are simply built with:
```
  CFLAGS="-static" make avx2
  make pkgs
  make static
  make pkgs
```
have a look at the man page for synopsis and options:
```
  man ./pasta.1
```
start pasta with:
```
  ./pasta
```
you're now inside a new user and network namespace. For IPv6, SLAAC happens right away as pasta sets up the interface, but DHCPv6 support is available as well. For IPv4, configure the interface with a DHCP client:
```
  dhclient
```
and, optionally:
```
  dhclient -6
```
and that's it, you should now have TCP connections, UDP, and ICMP/ICMPv6 echo working from/to the guest for IPv4 and IPv6
to connect to a service inside the namespace, just connect to the same port using the loopback address.

Contribute

Mailing Lists

Submit, review patches, and discuss development ideas on passt-dev
Ask your questions and discuss usage needs on passt-user

Bug Reports and Feature Requests

File a bug for those

Chat

Somebody might be available on IRC

Security and Vulnerability Reports

Please send an email to passt-sec, private list, no subscription required

24 KiB Raw Blame History

passt: Plug A Simple Socket Transport

pasta: Pack A Subtle Tap Abstraction

Motivation

passt

pasta

Features

Protocols

Portability

Security

Configurability

Performance

Interfaces

Availability

Services

Interfaces and Environment

Services

Addresses

Local connections with passt

Local connections with pasta

Protocols

Ports

passt

pasta

Demo

pasta

passt

Continuous Integration

Performance

Try it

passt

pasta

Contribute

Mailing Lists

Bug Reports and Feature Requests

Chat

Security and Vulnerability Reports

24 KiB

Raw Blame History