Virtualization

Hardware virtualization (VMs)

Proxmox is currently used as the VM hypervisor.

OS virtualization (containers)

LXC or Docker

Differences between VMs and Containers

VMs provide full hardware virtualization. A full, independent, isolated OS runs in the VM. As far as the OS is concerned, it interacts with a real computer (BIOS, CPU, Storage, Keyboard, etc.), even though most of these components are virtual ones managed by the host’s hypervisor (vCPUs, etc.).

Containers only provide an isolated environment for applications (as opposed to for an OS). Importantly, the kernel used by the containers is that of the host’s. This makes containers much lighter-weight than VMs.

Containers are therefore usually faster (less abstraction) but provide less isolation (kernel sharing).

Docker, LXC or VM for a new service?

1. Use Docker when possible
2. If not, consider LXC first
   • In theory it can offer: more “isolation” (behaves more like a full VM, no Docker net integration, etc.), hardware passthrough capabilities, its own IP directly on LAN.
   • In practice, the few times I used them, it was a pain and it always ended up being better to switch to either a Docker or a VM.
   • Running them without privileges makes them hard to use and less useful but running them privileged is dangerous.
   • One definite use case is to test/play with OSes (the few that have an LXC template) without having to go through the relatively long installation process.

• Otherwise, create a VM

CPU and RAM allocation

For CPU cores in Proxmox, you don’t need to match the cumulative number of virtual cores (vCPUs) assigned to VMs with the physical cores available on your host system. Proxmox uses CPU time scheduling to share the available physical cores among VMs. You can actually “overcommit” CPU resources, meaning you can assign more virtual cores to your VMs than you have physical cores.

However, there are some important considerations:

• While you can overcommit, assigning too many vCPUs can lead to performance degradation when VMs compete for CPU time
• A good starting point is staying within 1:1 to 2:1 ratio (virtual to physical cores) for production environments
• The optimal ratio depends on your workload patterns and whether VMs are consistently CPU-intensive

For RAM, the situation is different:

• RAM is a finite resource that cannot be as flexibly shared between VMs
• You generally should not allocate more RAM to VMs than physically available on the host (and minus what the host OS needs)
• While some memory overcommitment is possible using features like KSM (Kernel Samepage Merging) and ballooning, it’s risky and can severely impact performance if VMs actually try to use all their allocated memory
• A safe approach is to leave about 1-2GB for the Proxmox host system and only allocate the remaining RAM to VMs

Source: Claude.ai (parts)

RAM

VM memory ballooning:

• Not recommended for TrueNAS
• For others, may use as “extra memory”
• For example, set total RAM = 1.3 * minimum RAM

Check usage:

free -h # general
arc_summary # ZFS details
docker stats --no-stream # containers

Limit available memory for a Docker container to 512MB:

    mem_limit: 512m
    memswap_limit: 512m

Investigate: journalctl --since "24 hours ago" | grep -i "memory\|oom\|killed"