ZFS is one of the key features of ProxMox and why it is so popular.

ProxMox natively supports clone, destroy, snapshot and replicate features of ZFS. It can be installed on top of ZFS pool, which makes it easy to roll back a bad update or missconfig, etc. On contrary XCP-Ng has only started it’s ZFS journey and it has a lot of rough edges. No ability to install it on top of ZFS, no support for encryption, nor there are snapshot/destroy/replicate features included. It treats it as a regular file system, and not CoW file system ZFS is. Let’s just hope it will change with time (although it’s hard to predict anything due to how Dom0 and Xen works in general).

If you’d need to test a lot of storage backends, look no further than ProxMox. It supports ZFS, NFS, CIFS, Gluster, Ceph, LVM, LVM-thin, iSCSI/kernel, iSCSI/user space and ZFS ofver iSCSI. If it doesn’t support your storage backend natively (something like MooseFS or BeeFS), no worries, just install it’s agent from the terminal and mount it as you would mount it on a regular linux system. Then go to ProxMox WebUI and tell it: this is your new storage backend, use it wisely. XCP-Ng has a limited options when it comes to storage (especially local), and is not nearly as flexible.

ProxMox supports up to 31 network interface per VM, where as XCP-Ng only supports 8.

This was a deal breaker for us as a hosting company, and for few of our clients that are doing similar things. Imagine the situation where you have to create a separate VLAN/subnet for each client, so they don’t interfere with others, for security purposes, etc. So when I had to install few virtualised firewalls the choice of virtualisation platform was a no brainer.

P.S. although now, we moved on to BHYVE running on top of vanilla FreeBSD, I’ll answer why in another blog post or a video soon.

ProxMox 6 is more or less a standard linux distribution, built on top of Debian 10. They just include a more modern kernel, web interface for KVM and LXC, and some internal CLI abstraction tools that make interaction with KVM and LXC easier for the end user. It makes it a better choice for home labbers: at one point my media centre PC was a ProxMox host and media consumption machine at the same time. I’ve just created a new user (after a default ProxMox install on top ZFS), installed Gnome 3, couple of web browsers and media players and it was good to go. It also worked like a charm for a long time before I could afford something bigger/better or enterprise grade.

This makes it real easy to integrate with Ansible and other automation tools.

XCP-Ng on the other hand is a very custom, purposely built system that has a Dom0, which makes it nearly impossible to do something similar to the above and it’s a completely different discussion for another day.

ProxMox has 2 ways of controlling it: CLI and WebUI (well, technically there is also API, but I consider it to be a part of the WebUI) and it’s more straight forward to configure.

XCP-Ng on the other hand has 3 management interfaces: XCP-Ng center, XenOrchestra and CLI (XenAPI is a part of CLI not XenOrchestra). At this point you would think, but more choice is better, no? And you would be right in the most cases, but not here. You see not everyone has a Windows OS to install XCP-Ng Center on, and it’s a real pain for such people, because XenOrchestra and XCP-Ng center are not replacing each other, they rather complement each other. I’ll give you an example: most of the networking and cluster configuration can only be done from within the XCP-Ng Center, but most of the backups, VM management, resource monitoring, etc is done in the XenOrchestra. It confuses a hell of a lot of people and is inconvenient to switch between the management interfaces all the time.

But in all fairness, once the stack is configured, I rarely touch the XCP-Ng Center, so think of it as a initial point of config. Nonetheless it can be very annoying from time to time, for new users specifically.

Native linux container support is another selling point of ProxMox, I wish there was something s in XCP-Ng. LXC containers are like linux VMs, but they are more lightweight, easier and faster to deploy, and much more performant.

There is an option to install addon to XenOrchestra that will show docker containers in the web interface, but containers themselves would be running inside of CoreOS VM, and that can impact the performance on the low end machines.

XCP-Ng has a limit on a number of snapshots inherited from XenServer, it’s around 30 per VM (can’t find the exact number in the docs right now). And it is low by today’s standards.

ProxMox running on top of ZFS is capable to keep 18446744073709551616 snapshots (or 2^64 for those interested). So by the time we reach that limit, there will probably be the YFS (yottabyte file system), that will come to the rescue (hopefully somebody else than Oracle will develop it XD).

It’s 2020 now, and XCP-Ng still can’t manage to create virtual disks for VMs that are bigger than 2TB. I know there is development going on right now to overcome this issue, but at least devs listened to the community. Citrix XenServer forums are full of the posts with feature requests to add a support for disks bigger than 2TB, but all of them are either unanswered or ignored.

ProxMox doesn’t have such issue, your disk is limited only by the free space on the datastore or by the storage backend, if you are using FAT32 for example (I know, I know, it was a joke, just use modern and well maintained FS, and there will be nothing to worry about).

This feature is a very useful part of ProxMox. When the Guest OS has a weird firewall implementation that you are just not in the mood to learn, or it doesn’t have a firewall at all, just click on the VM/Container you want to configure, go to firewall settings and start filtering that traffic. It also supports rate limiting.

In XCP-Ng you can only limit the Guest OS interface speed, no firewall rules applied.

Every ProxMox installation includes a web interface by default. When it comes to a ProxMox cluster and 1 server has a fault, just punch in the IP address of another server in a cluster, including HTTPS and port 8006, and you’ll have a full control over the whole cluster once again.

XCP-Ng is not that easy. After the default installation you can deploy a free version of XenOrchestra (which is very limited to say the least) inside of the VM, or install XCP-Ng center on one of your windows machines. But in the end you’ll require both XCP-Ng Center and XenOrchestra anyway, if you’d want any useful features.

There is a delta replication option in XCP-Ng, but it only works 1 way, which is a shame.

For example: we have a VM on ProxMox at location “A”, we need to replicate it to location “B”, and we’ll use ZFS replication for that. VM is 5GB. First time we’ll have to transfer all of that, but every time after that, only the bits that changed are transferred. This is 1 way replication, there is a same option in XCP-Ng, so all good so far. The difference comes when you need to start that VM at location “B”, change some files, work on it, etc (for the DR scenarios or something similar), then location “A” comes back online, and the only thing we need to do is shutdown the VM at location “B” and migrate it back to “A”. Only the bits that were changes are transferred. Where with XCP-Ng you’ll have to transfer back a whole VM again. 5GB is not so bad, but imagine it was 10TB? Now, that’s a disaster.