ProxMox vs XCP-Ng
Let’s begin our journey to the ProxMox vs XCP-Ng discussion.
This post will go over the product history, then move on to the web-interface usability, and then cover all the features included and a roadmap for new features. Lastly I’ll let you know what I don’t like about each product and how to choose one over the other.
It all started with XenOrchestra: initial project started in 2009 by Olivier Lambert. XO was born because of one question: On which host is my VM running, goddammit? Then, more needs appear: – Does my Dom0 have too much VMs? – Does my Dom0 have enough RAM for new VM? – Common admin tasks from Web interface.
Coup de grace by the end of 2017, Citrix introduced even further limitations to the free version (3 hosts in a pool) and moved many useful features (live storage migration, dynamic memory) from the free version to the paid commercial plan. Conclusion — By the end of 2017, XenServer was not usable from the sources and the free version was badly crippled. Now, we were worried, pissed, and more than ever determined to find a solution for us and the community. And so it was born: XCP-ng — a fork of XenServer, a true open source hypervisor solution! After a very suscessful Kickstarter campaign that has reinforced us in our determination, we released the first version of XCP-ng on the 31st of March and we are now committed to improve the solution over time.
Started in 2005 as a self-funded open source startup. Development of Proxmox VE started when Dietmar Maurer and Martin Maurer, two Linux developers, found out OpenVZ had no backup tool and no management GUI. KVM was appearing at the same time in Linux, and was added shortly afterwards. The first public release took place in April 2008, and the platform quickly gained traction. It was one of the few platforms providing out-of-the-box support for container and full virtualization, managed with a web-based user interface similar to commercial offerings.
Web console for VM Management
Basic VM Admin
Live VM migration
Live Disk migration
VM High Availability
Integrated VM console view
Two Factor Auth
Both projects are mature enough for you to find the answers to any question you may have. ProxMox even includes offline web docs with every installation.
XCP-Ng has a lot of docs on the web thanks to Citrix Xen Server Forums and Docs, I often find myself answers to all questions there.
Documentation from the project itself is decent, but it’s more of a quick start guide, rather than a manual. One of the good things is that XCP-Ng and XenOrchestra are published on GitHub, so it’s easier to file bug reports and track them. Another good one is that they use Discource as their forum engine, but you need to register there, if you’d like to search (!?) through already created topics.
Started in 2005 ProxMox has enough docs to cover almost all of the use cases. And as stated before, they include (and update) the manual in HTML format with every installation. So the docs are good.
What’s not so good is their Git system (seems like they wrote the WebUI for it from scratch, but I might be wrong), which makes it harder to search for commits, look for undocumented features, etc.
Advantages, compared to each other
XCP-Ng is advertising heavily, which is a good thing. That way they’ll get more customers. And from what I can see, the project is getting more and more traction.
One time I had to download a fresh ProxMox ISO from their webiste, so I googled “Download ProxMox ISO”, but the google ad suggested to use XCP-Ng instead. What an awkward situation.
There is an easy way to migrate resources between clusters (even live VMs, if you’d like to): be it two clusters next to each other or in 2 different parts of a globe. The feature is supported out of the box, where with ProxMox you’d have to play with a software from 3rd party developer on github, or manually setup a ZFS replication job, to get a similar outcome.
XCP-Ng cluster supports up to 64 machines (ProxMox only goes up to 32, depending on the networking infrastructure). Adding and removing machines from the cluster is much easier than it is in ProxMox. In XCP-Ng Center, you just need to right click with your mouse and choose the remove from the pool option, followed by are you sure? Where in ProxMox there is a whole procedure (https://sysadmin-community.com/remove-node-from-cluster-proxmox/) with it’s own limitations.
What’s backed up in ProxMox can only be restored to ProxMox, same goes for import, you cannot import a VM from VirtualBox or ESXi. XCP-Ng doesn’t have this issue. It can export any VM into an OVA archive that will be understood by most virtualisation systems out there, and it can import VMs from the same and more formats.
Configuring LAGG/LACP, management, VLAN and other interfaces in XCP-Ng is a breeze. Every VLAN afterwards (and every network interface for that matter) has a name that will help you understand what it stands for later on down the road (which is helpful if you didn’t touch the system in a while, or if you inherited it from someone). ProxMox is not even near that polished in this regard, and you’ll have to read a lot on Linux networking standards to understand how to separate management interface from a regular one. On the other hand you’ll learn a lot and will be more competent in a long run, but the question is: do you really need it? Because not everyone does.
Where XCP-Ng really shines (it’s XenOrchestra really, but I consider it a native part of the project now) is backups. You can do backups, snapshots, exports, delta backups and do all sorts of stuff on schedule or manually for that matter. Flexibility is through the roof. ProxMox greatly lacks such simple features, which makes me sad. Delta backups are not possible to do, so every backup you make, will take another 100% of the space VM takes (if VM is 10gig and the disk is half full, every backup will be 5gig, period). To make the matters even worse, snapshotting is not available on schedule. I had to install another piece of 3rd party software to achieve this:
Resource listing filter is just off the charts great in XenOrchestra. It was developed to host and display a lot of VMs, Disks, Networks, etc. Filter supports regular expressions, running VMs, stopped VMs, VMs that are not backed up and much more. In contrast ProxMox only supports searching through the VMs in a list.
I often have to test new virtualisation technologies due to a field of my work, which requires nested virtualisation and XCP-Ng makes it super easy to activate it. There is a quick point and click solution that can turn this feature on and off.
I had much more success achieving huge uptimes with XCP-Ng. Sometimes I would even forget that there is a virtualisation stack that I need to keep an eye on, which is a great thing, because it leaves me more time to do other things, like writing this blog post, or making videos, or taking some time off and simply enjoying the moment. ProxMox on the other hand is like a baby, it requires attention. One evening I was playing an online game with a friend (we used a TeamSpeak server that I am hosting) and suddenly he said that my audio started to get flaky (and considering my audio setup it shouldn’t). After some troubleshooting it appeared to be a networking issue: ProxMox corosync (the clustering technology it uses on the backend) started to push so much traffic through the VPN tunnel that my connection had a 60% packet loss. Corosync crashed and started flooding a network with packets. This was the last time I considered clustering ProxMox nodes over the internet. If you have at least a glimpse of networking issues in your infrastructure don’t use ProxMox clustering, it will stab you in the back when you’d least expect.
ZFS is one of the key features of ProxMox and why it is so popular.
ProxMox natively supports clone, destroy, snapshot and replicate features of ZFS. It can be installed on top of ZFS pool, which makes it easy to roll back a bad update or missconfig, etc. On contrary XCP-Ng has only started it’s ZFS journey and it has a lot of rough edges. No ability to install it on top of ZFS, no support for encryption, nor there are snapshot/destroy/replicate features included. It treats it as a regular file system, and not CoW file system ZFS is. Let’s just hope it will change with time (although it’s hard to predict anything due to how Dom0 and Xen works in general).
If you’d need to test a lot of storage backends, look no further than ProxMox. It supports ZFS, NFS, CIFS, Gluster, Ceph, LVM, LVM-thin, iSCSI/kernel, iSCSI/user space and ZFS ofver iSCSI. If it doesn’t support your storage backend natively (something like MooseFS or BeeFS), no worries, just install it’s agent from the terminal and mount it as you would mount it on a regular linux system. Then go to ProxMox WebUI and tell it: this is your new storage backend, use it wisely. XCP-Ng has a limited options when it comes to storage (especially local), and is not nearly as flexible.
ProxMox supports up to 31 network interface per VM, where as XCP-Ng only supports 8.
This was a deal breaker for us as a hosting company, and for few of our clients that are doing similar things. Imagine the situation where you have to create a separate VLAN/subnet for each client, so they don’t interfere with others, for security purposes, etc. So when I had to install few virtualised firewalls the choice of virtualisation platform was a no brainer.
P.S. although now, we moved on to BHYVE running on top of vanilla FreeBSD, I’ll answer why in another blog post or a video soon.
ProxMox 6 is more or less a standard linux distribution, built on top of Debian 10. They just include a more modern kernel, web interface for KVM and LXC, and some internal CLI abstraction tools that make interaction with KVM and LXC easier for the end user. It makes it a better choice for home labbers: at one point my media centre PC was a ProxMox host and media consumption machine at the same time. I’ve just created a new user (after a default ProxMox install on top ZFS), installed Gnome 3, couple of web browsers and media players and it was good to go. It also worked like a charm for a long time before I could afford something bigger/better or enterprise grade.
This makes it real easy to integrate with Ansible and other automation tools.
XCP-Ng on the other hand is a very custom, purposely built system that has a Dom0, which makes it nearly impossible to do something similar to the above and it’s a completely different discussion for another day.
ProxMox has 2 ways of controlling it: CLI and WebUI (well, technically there is also API, but I consider it to be a part of the WebUI) and it’s more straight forward to configure.
XCP-Ng on the other hand has 3 management interfaces: XCP-Ng center, XenOrchestra and CLI (XenAPI is a part of CLI not XenOrchestra). At this point you would think, but more choice is better, no? And you would be right in the most cases, but not here. You see not everyone has a Windows OS to install XCP-Ng Center on, and it’s a real pain for such people, because XenOrchestra and XCP-Ng center are not replacing each other, they rather complement each other. I’ll give you an example: most of the networking and cluster configuration can only be done from within the XCP-Ng Center, but most of the backups, VM management, resource monitoring, etc is done in the XenOrchestra. It confuses a hell of a lot of people and is inconvenient to switch between the management interfaces all the time.
But in all fairness, once the stack is configured, I rarely touch the XCP-Ng Center, so think of it as a initial point of config. Nonetheless it can be very annoying from time to time, for new users specifically.
Native linux container support is another selling point of ProxMox, I wish there was something s in XCP-Ng. LXC containers are like linux VMs, but they are more lightweight, easier and faster to deploy, and much more performant.
There is an option to install addon to XenOrchestra that will show docker containers in the web interface, but containers themselves would be running inside of CoreOS VM, and that can impact the performance on the low end machines.
XCP-Ng has a limit on a number of snapshots inherited from XenServer, it’s around 30 per VM (can’t find the exact number in the docs right now). And it is low by today’s standards.
ProxMox running on top of ZFS is capable to keep 18446744073709551616 snapshots (or 2^64 for those interested). So by the time we reach that limit, there will probably be the YFS (yottabyte file system), that will come to the rescue (hopefully somebody else than Oracle will develop it XD).
It’s 2020 now, and XCP-Ng still can’t manage to create virtual disks for VMs that are bigger than 2TB. I know there is development going on right now to overcome this issue, but at least devs listened to the community. Citrix XenServer forums are full of the posts with feature requests to add a support for disks bigger than 2TB, but all of them are either unanswered or ignored.
ProxMox doesn’t have such issue, your disk is limited only by the free space on the datastore or by the storage backend, if you are using FAT32 for example (I know, I know, it was a joke, just use modern and well maintained FS, and there will be nothing to worry about).
This feature is a very useful part of ProxMox. When the Guest OS has a weird firewall implementation that you are just not in the mood to learn, or it doesn’t have a firewall at all, just click on the VM/Container you want to configure, go to firewall settings and start filtering that traffic. It also supports rate limiting.
In XCP-Ng you can only limit the Guest OS interface speed, no firewall rules applied.
Every ProxMox installation includes a web interface by default. When it comes to a ProxMox cluster and 1 server has a fault, just punch in the IP address of another server in a cluster, including HTTPS and port 8006, and you’ll have a full control over the whole cluster once again.
XCP-Ng is not that easy. After the default installation you can deploy a free version of XenOrchestra (which is very limited to say the least) inside of the VM, or install XCP-Ng center on one of your windows machines. But in the end you’ll require both XCP-Ng Center and XenOrchestra anyway, if you’d want any useful features.
There is a delta replication option in XCP-Ng, but it only works 1 way, which is a shame.
For example: we have a VM on ProxMox at location “A”, we need to replicate it to location “B”, and we’ll use ZFS replication for that. VM is 5GB. First time we’ll have to transfer all of that, but every time after that, only the bits that changed are transferred. This is 1 way replication, there is a same option in XCP-Ng, so all good so far. The difference comes when you need to start that VM at location “B”, change some files, work on it, etc (for the DR scenarios or something similar), then location “A” comes back online, and the only thing we need to do is shutdown the VM at location “B” and migrate it back to “A”. Only the bits that were changes are transferred. Where with XCP-Ng you’ll have to transfer back a whole VM again. 5GB is not so bad, but imagine it was 10TB? Now, that’s a disaster.
https://xcp-ng.org/docs/roadmap.html (last time I’ve checked this was an empty page)
Features that are missing from both projects
– Storage encryption. None of the projects natively support REST encrypted volumes. ProxMox is a little better, because you can use encrypted ZFS datasets, but only on a secondary zpool due to compatibility issues with GRUB.
– Less resource usage: DOM0 inside XCP-Ng will use anywhere between 2 and 5 gigabytes of RAM. ProxMox wastes most of it’s resources for the corosync and pve-cluster processes. I would love to see some resource usage tuning on both sides, it’s especially crucial for projects like tiny-mini-micro from STH.
– Installation directly onto a flash drive: both ProxMox and XCP-Ng are writing logs, RRD graphs, etc to the local storage, which makes it difficult to install them onto the regular USB flash drive, it will simply wear out very soon. I would like to see an option to move such random writes to an external storage dataset, like in FreeNAS/TrueNAS (yet another reason I moved on to FreeBSD13 as my new hypervisor).
– Pop-up about a missing subscription in ProxMox and XenOrchesra OpenSource Edition top bar notification. I know, what my setup is. Stop screaming about it please, it’s a homelab.
Final thoughs and why you'd choose one over the other
As always the choice is yours, and there is no clear winner here.
Warning! Sarcasm alert!
If you are a windows admin, XCP-Ng will be easier to use for you, you’ll have your buttons and windows and all that good stuff that you can’t live without. ProxMox on the other hand is rather for a red-eyed debian sysadmins, that love to live in the shell and are afraid of a day light.
But with all seriousness the stack choice will depend on the problem you want to solve. Is it just an old fashion iSCSI on a SAN? Then choose XCPNG without any hesitation. It has a better support for iSCSI, with multipathing, snapshots and so on. Need LXC containers and a lot of storage choices (both local and remote), hardware pass-through, custom network settings, easy access to Firewall for each VM/container? That’s a job that ProxMox will handle very well.
Let me know what is your hypervisor of choice down in the comments below, point out anything I missed in this video or anything that was fixed in any one of the projects recently, I’ll be updating video description or pinned comment to reflect the latest changes.
Consider donating towards our project, to support a future content release