klaxzy[dot]net/var/log

ieee 802.3ae-2002 nikola[at]klaxzy.net:/var/infra/10g-upgrade_ home_lab

running self-hosted services requires network connectivity, there is no escape from networking, in my opinion. a good network foundation is key to the stable and smooth operation of any modern infrastructure. in the era of cloud and virtual services, the network level is often invisible to end users as well as people who manage all the underlying network infrastructure.

today, I still believe that networking is an important part of the infrastructure, which is nicely reflected in my hobbies as well. now, I host my own infrastructure (and try to keep it modern) at home, and it's a nice little project that keeps me busy and happy.

currently, my setup looks like this (connection diagram).

in the early stages (it was 2014 if i remember correctly) i would just use a decent WiFi router, back then that was ASUS router with asuswrt-merlin[0] firmware, and connect the server directly to it. then, i have discovered a pihole at around 2015, got an Raspberry Pi with the only possibility to connect via LAN, plus i already had a proliant server, qnap nas and then another LAN device that required LAN connectivity. then i thought, let me get a switch. the next state was getting a switch, and i got a cisco switch (SG200-26) and started to grow my network slowly including the LAN connection to all my workstations since I got very bad experience with WiFi plus back then there was the key reinstallation [KRACK][1] vulnerability with WPA2 that just kept me using physical cabel. everything was in one network and i haven't been using much of the security features of that cisco only to introduce vlans in 2019 when i got my first firewall (UniFi gateway[*]) and then later added even more segmentation with the opensense coming in the game.

*i have replaced unifi stuff very quickly from the moment i got it, primarily because of the lack of flexibility, weird and confusing terminology and the "ecosystem" nature. all the equipment got sold at swapfest @c-base.

image: ieee802.3 and ieee802.3q dataformat comparison, source: internetworking, technische grundladen und anwendungen

why to upgrade? the cisco SG200-26 served me well for years, it was a decent little switch, honestly. however, over the time ports started flapping up and down, crc errors popping up like weeds, and the switch just giving up and putting ports into err-disabled mode. the hardware was basically dying, choking on the all the heavy backup traffic that wasn't built to handle gracefully. originally, i got this cisco switch without working power supply, the switch was refusing to start and I have diagnosed that the power supply capacitor was swollen and electrolyte has escaped, it was an easy fix and i was really proud to make it work again. meanwhile, the model was EOL[2] by 2023 already and UI became painfully slow + all the port issues, etc... *cisco small business sg200 series had an EOS (end of sale) after just 6 years which i find very short for that period and the successor sg350 was targeted for more larger corps rather than small business, everything after that became meraki which essentially became all cloud dependent. so, clap-clap to cisco serving me for more than 10 years, i couldn't save it again, and honestly i felt like that is time for an upgrade. i built couple of servers, 2 NAS devices, lots of VMs, containers, personal data backups and syncs, cisco was just a bottleneck. plus, i already got Intel NUC with 2.5Gibt NIC, firewall with 2.5Gig interfaces and 3xIntel 10G 2P x520 adapters rescued from servers that were thrown away by the ship company I worked for (yes, shame on you HAL for throwing away all those Dell server carelessly)

image: das netz, source: self, Deutsches Technikmuseum Berlin

then, there was a question of powering some of the PoE capable devices like Netgear AP and temperature sensor device. the thought on having PoE switch gave me the fantasy of removing those extra PoE injectors that were consuming power sockets.

when cisco announced EOL i developed the fear of having something that cannot be patched anymore therefore i put all my network management in the separate vlan without external internet access and restricted inbound traffic. this vlan still exists and all the network management is still there!

10g decision & oss dilemma first of all, why not cisco again? well, cisco is fully-blown cloud dependent bro now with their meraki series and catalyst was simply expensive. then, cisco is not open source, i mean there are some really honorable mentions, however here SONiC could be a choice but looking at their hardware support list the switches there cost 3K+ EUR, and those are mostly enterprise grade hardware switches and those are not easy to find used. then the consumer oss software could be vyos, opx, openwrt or milk-v but those are still niche products with not that broad hardware support.

i already run oss on my other networking hardware, openwrt on my WAX220 APs and opnsense on my firewall, however moving to a oss switch os seams like a real hustle.

fully open hardware ethernet switches are still uncommon compared to opensource routers or firewalls. open source routers and firewalls became practical much earlier because routing can run efficiently on general purpose consumer CPUs, switching at 10/25/100/400+ Gbps is different, it usually requires highly specialized ASICs with enormous packet processing throughput and very low latency, that is very expensive for home use. switch ASICs are usually proprietary (broadcom, marvell, etc) & ASIC SDKs are often closed. so “open switch” usually means, open NOS, open management plane, open or whitebox hardware, ONIE environment and standardized APIs.

image: openwrt one open source & open hardware router, source: openwrt.org

# the openness of internet infrastructure can be an illusion. while it is possible to use open source os, software, and hardware in local networks, there exists no completely open source path across the entire internet. Even if you're using open source solutions from A to B, it is inevitable that some data packets will traverse networks through proprietary vendors like Cisco or Juniper, making the Internet itself a hybrid of open and closed systems. #

i needed to find a good afordable solution. d-link was a very good option, but expensive. tp-link came out with the offer of 2.5&10Gbit ports and PoE, then the idea of utilizing the 10G intel x520 and upgrade NAS devices came instantly but i was missing 10G ports to support the core. as someone who worked as network engineer in the past i got an idea to get a core switch to support the 10G backbone, here the decision fell on mikrotik CRS305-1G-4S+[very catchy name]. additionally i got designed the network to get another 8 port 2.5Gbit swithc with 10Gbit uplinks for my home office which is in a different room then the main server and network rack, again tp-link was the best option. what i find very surprising is the fact that when you plan your network upgrade somehow you slip the idea of getting new cables and modules, this is the unexpected cost that honestly caught me unprepared. dealing with 10G network requires better cabling, also SFP+ modules or in the best case fiber-optic to support the bandwidth. luckily, the moment i rescued those Intel 10G NICs is also saved bunch of SFP+ DAC cables which ended up being a perfect fit. the only thing i got putchased was SFP+ BASE-T modules to support connectivity between main access switch and my 8 port switch via 20m CAT6 ethernet cable. image: transceiver block diagram, source: broadcom, newcom-acorn joint workshop presentation

what i got in the end:

tp-link sg3218xp-m2: 16x2.5Gbit ports, 8 of those support PoE+ and 2x10Gbit SFP+ ports 410EUR

mikrotik crs305-1g-4s+ [used]: 4xSFP+ ports with switchos 70EUR tp-link sg3210x-m2: small version of sg3218xp-m2 with 8x2.5Gbit and 2xSFP+ ports 193ER SFP+ DAC & BASE-T cables&modules: enough to support the cabling and interconnection 80EUR (approx.) noctua nf-a4x20 flx fans: to replace factory fans on the PoE switch 30EUR intel x520 10G NIC x 3: dual port 10G NIC that i used in servers and NAS, previously rescued from the garbage 0EUR

hacks

tp-link fans: tp-links arrived the first shocker was the the noise, sg3218xp-m2 was really noisy especially while booting, reminds a lot on pretty much any serious enterprise network switch from cisco, brocade, hp or juniper. even when fully booted the noise was just unareable, especially for me and the fact that this is going to be in the rack in hallway of my flat, no way that this is buzzing there 24/7. i tried not to panic and found the good solution to lower the noise. i've found couple of articles on the internet that mainly use noctua fans and i got a pair of NF-A4x20 FLX, the improvement was drasticall with very low thermal degradation.

core switch fans: one small design flaw is that the crs305-1g-4s+ switch can overheat in high traffic utilization especially when SFP+ base T modules are used, those can generate 90°C easily. I modified the case and added 2(two) 4mm fans to cool down the modules, after the modification system temperature went down from 70°C to approx 38°C.

configuration

tp-link switches are locally managed and i do not use neither cloud or locally hosted omada controller, mikrotik is also local only config.

all network equipment management has its own vlan with many restrictions like no outbound traffic, except the ntp to local and external server. also inbound is very controlled for https and ssh only from few IPs that i use to manage them from.

i also used the chance to redo my network configuration so this was not a migration really, rather the new configuration with some upgrades and addons.

i have 8 vlans configured in total, all are terminated on opnsense firewall, inter-vlan communication is restricted with firewall rules on the firewall and intra-vlan comms are unrestricted, here i use local server or device firewall when needed. additionally i use security features like stp, storm control, arp & dhcp spoofing. PoE is also configured to deliver not more than needed for the device and save some energy. the management is available only via ssh with ssh key and/or https via nginx-proxy.

core switch is configured with all ports as trunks with port isolation allowing all vlans on all port, this gives me the flexibility to later decide to block certain vlans ports. stp is also active on the core switch on all ports.

MTU size gave me some troubles and doubts originally, I managed to configure [MTU 9000] on all devices I have only to discover that zenarmor and netmap do support up to 1500 bytes max[3] i've decided to keep it like that on all other devices. zenarmor relies on the netmap framework, which is optimized for fixed size 1500 byte buffers and lacks the dynamic memory handling required to safely process the variable, oversized packets of jumbo frames without causing crashes or data corruption. I hope they implement better support for MTU in the future.

i've used a single cable to connect core switch, servers, NAS and the switches. do i care about the redundancy, no i don't. normally in the enterprise setup N+1[4] is a religious doctrine and the way to go, however in my home setup i don't depend on anything that is hosted on servers, and any other failover or redundancy could introduce additional complexity and cost, what for? I actually have N+1 Human Redundancy. I don't need a second switch. I need a spare cable and a backup plan. That is the optimal engineering solution for a solo operator.