one137

Theme: 

Homelab Networking

See Services and network layout.

Useful commands

ip address
ip route

In case of subnet conflict, explicitly route IPs 192.168.1.96/28 (16 IPs, 96-111) through the VPN interface: sudo route add -net 192.168.1.96 -netmask 255.255.255.240 -interface utun4

Flush DNS chache: sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

List all tcp/udp listening ports and their proceses: sudo ss -tulpn or netstat if ss is unavailable Display the kernel routing table: netstat -r

Docker Networking

Using a star network, where each container is isolated from one another but all that need incoming connections are connected to NPM.

Containers that need outgoing VPN connection directly share ct-wireguard-client’s container network.

Outdated example:

graph TD
ct-ddclient --> |172.18.0.2| ddclient_default["ddclient_default\n(Gateway: 172.18.0.1)"]

mongodb_internal["mongodb_internal\n(Gateway: 172.25.0.1)"] --> |172.25.0.3| ct-npm
ct-mongo --> |172.25.0.2| mongodb_internal

wg-client_default["wg-client_default\n(Gateway: 172.21.0.1)"] --> |172.21.0.3| ct-npm
ct-wg-client --> |172.21.0.2| wg-client_default
ct-transmission --> ct-wg-client

Speeds tests

fio writes

fio tests from Debian:

  • fio --name=write_test --size=1G --bs=4k --rw=write --direct=1 --directory=...
    • /home/USER/fio: 100MB/s
    • /mnt/nas-video/fio: 43MB/s
    • /mnt/nas-xfs-zvol/fio: 35 MB/s
  • no --direct flag:
    • All up to 3’600 MB/s

iperf3 transfer

iperf3 tests:

  • Debian - Proxmox: 33.8 Gbps
  • macOS - Debian:
    • 942 Mbps RJ-45
    • 776 Mbps Wifi living room
    • 82 Mbps Wifi office

Internet connexion (speedtest.net)

  • Wifi: 95 MB/s (760 Mbps)
  • rj45: 118 MB/s (940 Mbps)

Plain file transfer

Transfer times of movie.mkv, 2.6GB, 2765358436 bytes:

From/TomacOS WifimacOS rj45ProxmoxDebianTNAS SSHTNAS SMBTNAS SMB Un-encryptTNAS NFS
macOS Wifi1s–53s52s52s34s
macOS rj45––24s24s33s24s
Proxmox29s24s1s3s10s9s24s
Debian29s24s3s1s3s / 10s9s2s
TNAS SSH29s24s61s / 20s / 3s61s / 3s0s
TNAS SMB27s24s15s / 3s16s / 1s–
TNAS SMB Un-encrypt2s–
TNAS NFS32s / 2s–
  • 1s = 2’637 MB/s
  • 3s = 879 MB/s
  • 9s = 293 MB/s (2.3 Gbps)
  • 15s = 176 MB/s (1.4 Gbps)
  • 24s = 110 MB/s (0.9 Gbps) ⇐ rj45 limit
  • 27s = 98 MB/s (0.8 Gbps) ⇐ Wifi 5/ac limit
  • 34s = 78 MB/s
  • 52s = 51 MB/s

Summary

macOS

  • RJ-45
    • Reads and writes always optimal (close to 1Gbps)
    • Except writes to TrueNAS by SSH (SMB no prob)
  • Wifi
    • Reads mostly optimal (SMB best, ~780 Mbps)
    • Writes are ok on SMB, 25% slower than reads
    • Writes are BAD on SSH, 80% slower than reads Intra VMs & Non-encrypted TrueNAS
  • Reads/writes are extremely fast (7 Gbps)
  • TrueNAS Reads/writes possibly instantaneous (12+ Gbps) even when not cached? Encrypted TrueNAS
  • Reads can be as slow as 1.5 Gbps
  • Cached read are instantaneous (12 Gbps)
  • Writes are average (2.3Gbps), both SSH/SMB
  • With 8 vCPU
    • Reads 33% faster (16s → 11s)
    • Writes 40% faster (9 → 5.3s)
  • With 12 vCPUs instead of 4
    • Reads 50% faster (8s)
    • Write 55% faster (3.9s)
  • Adding RAM doesn’t change speeds (as expected)

Backlinks