10Gb/s Ethernet: what I actually did to get it working in my home

Archives Categories Blogroll Having learned enough about 10Gb/s Ethernet to be comfortable about setting it up in my house, it was time to bite the bullet: order it from the ISP, buy some kit, and get started.I already had 2.5Gb/s working. The apartment has structured cabling -- each room has one or more RJ45 sockets in the wall, and there's a patch panel downstairs by our front door that has a matching patch socket for each wall socket. So when we moved in, I simply set things up so that there was a 2.5Gb/s switch down by the patch panel, and wired everything together there. Most of our stuff works over WiFi, of course, but I needed a wired backbone to connect the excessive number of computers in my study both to each other, and to the outside world.What did I need to do?Simplifying a bit, I had this 2.5Gb/s setup: The ISP connection came into the apartment in the living room. It went through a router/firewall machine I'd set up myself (more on that later), then via a 2.5Gb/s switch to the main WiFi AP and also to a wall socket. Down at the patch panel, I had a 2.5Gb/s switch, which was connected to the patch socket corresponding to the router's wall socket. Another connection from that switch went to the patch socket corresponding to the wall socket in my study. In the study, I had another 2.5Gb/s switch that handled internal networking. There are a few other things dotted around, of course -- extra APs and what-have-you -- but that's the core, and I'll focus on that to keep things simple.Would I be able to get it all upgraded to work with 10Gb/s? The most important question was the structured cabling in the walls; was it CAT-5E or CAT-6, or even CAT-6A? Remember from the last post, 10GBASE-T might work over short runs of -5E (even though officially it's not meant to be able to).

It probably would run over -6, because that's generally OK up to 55 metres or so, and I don't think any of the runs in the house are longer than that. And it would be fine over -6A, which is good for 100-metre runs.I was unable to find out exactly which type I had (the parts of the cables that are visible to me don't have any kind of marking to say), so I decided to do a staged rollout.The first step was to set up the wired network within my study as 10Gb/s. There were two important things to wire up; my primary desktop, perry, and a Proxmox cluster I have running in an 11" rack. The setup I had was just one 2.5Gb/s switch sitting on top of the rack, linked to the wall, to the cluster machines, and to perry.Now, getting the Proxmox cluster up to high-speed internal networking was a non-starter. The machines there are all old ones -- it's essentially a retirement home for mini-PCs I used to use for other things 1. They're mostly gigabit ethernet, with one 2.5Gb/s one.But getting perry up to 10Gb/s was an important goal, as that's where I do most of my work. I also wanted to have space for a second machine that I'm planning to set up to do training/inference without tying up perry's GPU, and that would also need fast networking.I wanted to have things running reasonably cool (after all, the PC itself and its GPU pump out quite enough heat already when doing a training run), so DAC felt like the right way to go. I bought a reasonably cheap managed 10Gb/s switch 2, a MikroTik CRS305-1G-4S+IN, with a single 10GBASE-T adapter to allow me to connect it to the wall socket. I tend to name anything on my network with its own IP, so this became nigel.

Next, a 10Gb/s SFP+ PCIe card -- an Asus XG-C100F -- for perry and a DAC cable to connect the two.For the Proxmox cluster, I decided to stick with the old 2.5Gb/s unmanaged switch, a TRENDnet TEG-S5061. I'd originally bought that one because it was the cheapest 2.5Gb/s on Amazon with decent reviews, and had completely forgotten that it had one major feature -- an SFP+ 10Gb/s port for the uplink! So another short DAC to connect that to the MikroTik, and the study network "backbone" was 10Gb/s.Of course, no two computers in there could actually communicate at that speed, as only perry was 10Gb/s-capable -- but I could have all of the Proxmox machines talking to perry at the same time at full speed. I did some tests with iperf3 to make sure that it was all working as expected; I couldn't test very thoroughly, but I was able to get about 4Gb/s total throughput, which was reassuring: two machines at 1Gb/s plus one at 2.5Gb/s should be a touch less than 4.5Gb/s.The next step was to check the possibilities for the connection down to the patch panel. I bought a Ubiquiti 10G Ethernet dongle, and took my laptop, laura 3, down there.The news was good! Running an iperf3 test between perry and laura down the structured cabling, I was able to get just less than 10Gb/s from laura to perry, and about 7Gb/s from perry to laura. The slower receive speed at the laura end worried me, but when I checked ps it became obvious what was going on. I could see the ksoftirqd kernel process running at 100%, so some single-core thing was maxing out.

The Ethernet dongle was connected over USB, of course, and that meant it needed to do much more work on the CPU for each incoming "data has arrived" interrupt than a PCIe card like the one on perry. That meant that laura could only receive data at a rate that one core could handle, which happened to be 7Gb/s. laura is a ThinkPad optimised for lightness and long battery life, not CPU power, so single-core performance is not great, and it hit a wall.But the 10Gb/s speed in the other direction was enough to make me comfortable that the structured cabling could handle that speed, which was excellent news -- probably I had either short runs of CAT-6, or CAT-6A in there, though conceivably I was just getting very lucky with CAT-5E.The downside was the heat. The USB dongle got too hot to comfortably hold while it was running, and while I wasn't able to check the SFP+ module in the MikroTik during the test, when I came back upstairs again I touched it and it was even hotter. I decided that that was something to keep an eye on for later (and as you'll see, it did become a recurring theme).For now, it was time to do the rest of the upgrade.Downstairs at the patch panel, it was a simple choice. All of the connections were RJ45, of course, and I only needed four. So the MikroTik CRS304-4XG-IN was the obvious choice.The final place where I needed to do some upgrades was at the ISP end. The box that our provider gave us had just one 10Gb/s port -- a 10GBASE-T RJ45 one. Now, I don't generally trust ISP routers that much, so I've always had my own router sitting between them and the home network -- a dual-port mini-PC running a locked-down Arch installation 4. My old one was dual-2.5Gb/s, so that needed an upgrade.

I settled on a Protectli VP2440, which has two SFP+ 10Gb/s cages, plus two normal 2.5Gb/s RJ45s. I didn't need the latter, but it was the cheapest option with 10Gb/s in their range, and I've always been very happy with their hardware and customer service.However, I was a little concerned about thermals. As I mentioned, the SFP+ module in the MikroTik in the study got very hot when I did my test. I'd need dual SFP+ modules for the Protectli -- one for the WAN port connected to the ISP box, and the other for the wall socket to go down to the patch panel. Might it overheat?The good thing about Protectli is that you can just ask them. I dropped them a line, and got a reply the next day from a customer support rep saying that he believed it would be fine, but he just wanted to double-check with one of their techs. The following day, he followed up to say that the tech had confirmed that it would be OK.Promising! And because of that, plus their 30-day money-back guarantee, I decided to go for it.A few days later, the new router arrived. I named it reggie, set it up with my normal router Arch installation, plugged it into the ISP box and the wall... and it worked just fine!So the setup at this point was: ISP box to WAN on reggie the router. LAN on reggie to wall socket. Patch panel socket corresponding to that wall socket to port 0 on the downstairs RJ45-only switch, nelly. nelly port 1 to the patch panel corresponding to my study's wall socket. (Other ports to other things I'm disregarding for simplicity.) Wall socket in the study to the RJ45 SFP+ module in port 0 on nigel. nigel port 1: DAC to an SFP+ network card on perry, my workstation. nigel port 2: DAC to the SFP+ 10Gb/s uplink on the old TRENDnet 2.5Gb/s switch to handle the Proxmox cluster.

At the same time I decided to move the main WiFi AP (winona, a Ubiquiti U6 Enterprise) that was previously next to the router over to my study -- so that was hanging off the TRENDnet switch.After a bit of bedding in, I decided I wanted to move winona back to the same place as the router -- it's more central so it provides better WiFi coverage from there. So I got another CRS304-4XG-IN -- the 10GBASE-T MikroTik switch, like the one by the patch panel -- so that the first part of the above topology became: ISP box to WAN on reggie the router. LAN on reggie to the new switch (norman) port 0. Port 1 on norman to the wall socket (thence down to the patch panel). Port 2 on norman to the WiFi AP via a PoE injector. All of this is sitting in a sideboard next to the dining table with no ventilation. That's probably close to a pathological case for hot-running network infrastructure like this, so... how about those thermals?In which I consider replacing my airfryer with an SFP+ moduleI like to keep track of what is going on with my zoo of computers, so I run Telegraf on all of them. This collects stats like the CPU temperature, system load, disk space, CPU and network use, and so on. They send this to an InfluxDB instance on a Proxmox VM (varro, if you're keeping track).When I set all of this up, I also wanted to monitor the switches. MikroTik switches expose their stats over SNMP, so with a bit of help from various LLMs I was able to augment the Telegraf config on reggie to also scrape that data and send it to varro.I use Grafana to get all of this stuff into various dashboards, and one of them is the temperatures of the networking hardware. Firstly, reggie -- the Protectli router with two SFP+ cages, each of which has a 10GBASE-T module.