Cheap 10 GbE SFP+ switch

[averyfreeman]

I'm extremely interested in hearing if this goes through -

And thanks for expressing your approval of my switch choice! ;) It seemed like a really good switch from my research but I was unprepared for how truly great it has been.

I was running a lot of CX4 82598 gear with my 2-host ESXi setup and a D-Link DGS-3427 switch which was fine but never achieved line speed, best was 6.5Gbps iperf3 between hosted VMs at 9000 MTU

Before that, I had a Netgear XS708E (v1) with a few Intel X540s and a chelsio T520 and was rarely breaking 4Gbps over 10Gbase-T 1500 MTU (never bothered trying jumbo frames). Sold because it was all expensive mainly and needed to make rent, but it was apparently one of the best decisions I had made.

The XS708Ev1 is truly an awful switch, I wouldn't wish it on my worst enemy. Seriously, it's stunningly bad.

So this immediate peg to 9.4Gbps on my 7048P was really exciting for me, having been through two fairly crappy 10GbE experiences. I finally feel like I can relax and stop looking at network gear for a while, basically feeling like I've reached the pinnacle of 10GbE.

Hopefully I'll be able to stop myself from trolling eBay for ESXi-friendly QSFP gear. Stranger things have happened.

Edit: And re: Mikrotik being CPU-bound, I do believe that's what I've heard other people criticising. I think that's by design, both as a way they can keep the units cheap and make them more flexible with different OS choices. If they manage to make it perform as well as dedicated-ASIC HW, more power to them, but I have my doubts.

Bring on the benchmarks!!!

 

[danb35]

Well, wonder of wonders, it looks like my switch has finally shipped. Looking forward to seeing it.

 

[averyfreeman]

Mikrotik has been a leader in bringing low-power but effective switches to market. I have a CSS326-24G-2S+RM (24-Gbit + 2x SFP+) unit at home that draws 9W - no Fans. Some of the online tests have questioned the throughput capacity of the CSS326-24G-2S+RM, especially if you're using it as a Layer 3 router. However, for my use case, (Layer 2 only, no routing, few simultaneous users), the thing has been impossible to saturate. Similarly, I expect the above CRS305 to be perfect for the "high-use" lanes in a home or office. Especially now that more and more WiFi AP's are potentially to make use of NBase-T connections above 1Gbit/s.

Where might these tests be, do you have any references you could point us towards? I'd like to see these! :)

Also wish that Mikrotik would enable their hardware to auto-update itself. Given the sheer number of exploits that their hardware has been subject to, it would be great if the gear could download updates at their leisure and then execute the update at a given time.

Seems like it could be possible via OS updates that at some point this could be automated. Do you know if people have mentioned it to Mikrotik?

Also, any refs for exploits? Would like to see those, also! ;)

Thanks for all the great info.

 

[jgreco]

Well, wonder of wonders, it looks like my switch has finally shipped. Looking forward to seeing it.

Mikrotik suddenly reached out to me yesterday asking for a phone number for shipping. Normally that'd be non-interesting but in combination with your update, makes me think a cargo container of these things just arrived in the US.

 

[Constantin]

Seems like it could be possible via OS updates that at some point this could be automated. Do you know if people have mentioned it to Mikrotik? Also, any refs for exploits? Would like to see those, also! ;)

The most obvious reference is CVE. However, every update is chock a block full of references to updates, bug fixes, and vulnerabilities. Sometimes, Mikrotik tells its customers to update in the forums. Anyhow, I just wish there was a way to get the OS to download updates on a random basis (to keep the load manageable on the Mikrotik servers) and then allow the OS to get auto-updated, booted, etc. on a set schedule.

I'm not hugely concerned given that my Mikrotik sits behind a good firewall. But the brand seems to be a popular target. Maybe because they're Latvian, used in a lot of networks, etc. and hence a juicy target for certain state actors.

The CRS226-24G-2S+IN model is neat but I'm not happy how I usually have to reboot it after every FreeNAS reboot because something discombobulates the connection. I've made everything as standard as possible (even following @jgreco's advice re: MTU=1500), to no avail. Once the Mikrotik reboots, all is well.

 

[averyfreeman]

Mikrotik suddenly reached out to me yesterday asking for a phone number for shipping. Normally that'd be non-interesting but in combination with your update, makes me think a cargo container of these things just arrived in the US.

Rad!

 

[averyfreeman]

The most obvious reference is CVE. However, every update is chock a block full of references to updates, bug fixes, and vulnerabilities. Sometimes, Mikrotik tells its customers to update in the forums. Anyhow, I just wish there was a way to get the OS to download updates on a random basis (to keep the load manageable on the Mikrotik servers) and then allow the OS to get auto-updated, booted, etc. on a set schedule.

Ahh, yes, CVE, that's a very good idea. Thanks for mentioning them.

I'm not hugely concerned given that my Mikrotik sits behind a good firewall. But the brand seems to be a popular target. Maybe because they're Latvian, used in a lot of networks, etc. and hence a juicy target for certain state actors.

Just curious, what are you using? I've been on OPNSense since my pfSense web gui decided to become non-responsive after an update around dec.

I wonder if it has anything to do with Latvia being in a former Soviet NATO-member country... hmmm... gotta love geo-politics.

The CRS226-24G-2S+IN model is neat but I'm not happy how I usually have to reboot it after every FreeNAS reboot because something discombobulates the connection. I've made everything as standard as possible (even following @jgreco's advice re: MTU=1500), to no avail. Once the Mikrotik reboots, all is well.

Now that's weird. What would FreeNAS be doing to interact with it that, say, some other server might not? Makes no obvious sense...

 

[NAS_warrior]

Strongly support the idea using Mikrotik @ Home LAB
I have 8 already. unfortunately still not get one CRS309-1G-8S+IN as I was at sea when come out.
Indeed is in my future upgrade list but I still thinking between 309 and 317 (CRS317-1G-16S+RM)....
Honestly I moved from Cisco, Juniper & Aruba to Mikrotik for my home lab as do not cost me an arm & leg when upgrading.
Yes it's lacking some small options those "3 whales" have but they catching pretty fast.

 

[Constantin]

What would FreeNAS be doing to interact with [Mikrotik Switch - Ed.] that, say, some other server might not? Makes no obvious sense...

I agree that it makes no sense. Ifconfig shows the link as up on the FreeNAS side of things yet pings to the gateway (I'm afraid to call it a router after @jgreco's recent router explainer) result in 100% packet loss. I'm inclined to blame the Mikrotik switch since the issue clears as soon as the switch is reset (whereas rebooting the FreeNAS does nothing re: re-establishing connectivity).

It's entirely possible that the Mikrotik auto-negotiates something when it boots that the FreeNAS agrees to. Then, when the FreeNAS reboots, the Mikrotik doesn't notice the "new" FreeNAS state and then doesn't automatically re-negotiate a working set of parameters. I will have to dive into the details at some point. For now, the reboots of the FreeNAS rig are rare enough where it's not a big deal to hit both power switches. I also wonder if using a generic twinax may contribute to it.

As for my gateway, it's a Edgerouter with a minimal dual-WAN configuration. It my not break any speed records but its ludicrously faster than my Comcastic connection to the internet. I'm still looking for an alternative but unfortunately my town has no high-speed competition. The closest I came to that was a WISP (netblazr) but their closest transceiver with a sort-of LOS is below the minimum thresholds despite a Powerbeam 5AC Gen 2 antenna with IsoBeam 620 shroud on my end. Oh well.

 

[NAS_warrior]

Hey Constantin, as MT is full fledged Router and FW check you rules in FW section.
You may have something that drop the new connections on some state or when you have set protection from packet flood rate.
The last to my mind, but is the first to be checked - check your SPT config.

 

[averyfreeman]

Strongly support the idea using Mikrotik @ Home LAB :)
I have 8 already. unfortunately still not get one CRS309-1G-8S+IN as I was at sea when come out.
Indeed is in my future upgrade list but I still thinking between 309 and 317 (CRS317-1G-16S+RM)....
Honestly I moved from Cisco, Juniper & Aruba to Mikrotik for my home lab as do not cost me an arm & leg when upgrading.
Yes it's lacking some small options those "3 whales" have but they catching pretty fast.

I had an eye out for the Aruba S2500 for a while when wanting a cheap used SFP+-backbone switch, but then I noticed their support updates stopped in 2016... :/ Hopefully my powerconnect 7048P won't stop anytime soon :)

 

[averyfreeman]

I agree that it makes no sense. Ifconfig shows the link as up on the FreeNAS side of things yet pings to the gateway (I'm afraid to call it a router after @jgreco's recent router explainer) result in 100% packet loss. I'm inclined to blame the Mikrotik switch since the issue clears as soon as the switch is reset (whereas rebooting the FreeNAS does nothing re: re-establishing connectivity).

It's entirely possible that the Mikrotik auto-negotiates something when it boots that the FreeNAS agrees to. Then, when the FreeNAS reboots, the Mikrotik doesn't notice the "new" FreeNAS state and then doesn't automatically re-negotiate a working set of parameters. I will have to dive into the details at some point. For now, the reboots of the FreeNAS rig are rare enough where it's not a big deal to hit both power switches. I also wonder if using a generic twinax may contribute to it.

As for my gateway, it's a Edgerouter with a minimal dual-WAN configuration. It my not break any speed records but its ludicrously faster than my Comcastic connection to the internet. I'm still looking for an alternative but unfortunately my town has no high-speed competition. The closest I came to that was a WISP (netblazr) but their closest transceiver with a sort-of LOS is below the minimum thresholds despite a Powerbeam 5AC Gen 2 antenna with IsoBeam 620 shroud on my end. Oh well.

Assuming you're using the full-speed full-duplex of all your links, have you tried just setting them from autoneg to full? I was having an issue in ESXi where a vSwitch was auto-negotiating certain VMs to 100 mbps with my switch (the old D-Link) when it should have been 1000 mbps - just set it to be 1gbps all the time and the problem went away...

Of course, I guess it could complicate things if you have other equipment you might need to use that could be 10 or 100 if you turned off autoneg switch-wide, but you might be able to set it on a port-by-port basis and then only use your FreeNAS box with those ports.

 

[Constantin]

Hey Constantin, as MT is full fledged Router and FW check you rules in FW section.
You may have something that drop the new connections on some state or when you have set protection from packet flood rate.
The last to my mind, but is the first to be checked - check your SPT config.

The unit is currently set up as a "bridge" - I don't think I set any of these parameters... No firewall rules, no stateful packet inspection, if that is what you mean.

 

[Constantin]

Assuming you're using the full-speed full-duplex of all your links, have you tried just setting them from autoneg to full? I was having an issue in ESXi where a vSwitch was auto-negotiating certain VMs to 100 mbps with my switch (the old D-Link) when it should have been 1000 mbps - just set it to be 1gbps all the time and the problem went away...

I'll try that later. Thanks!

 

[danb35]

Well, my box arrived today. Looks nice, like the redundant power inputs (as well as PoE), and the strain relief clip for the power cords. The web GUI is, well, a little busy, but doubt it'll be a problem. Firmware update is nice and easy, just put it on the Internet and it will auto-update. Haven't done any 10G stuff with it yet--I'll have to check if any of my existing SFP+ modules will work with it, or if I'll need to order more from fs.com.

Edit: also will want to check out https://github.com/gitpel/letsencrypt-routeros to automate pushing certs to the box.

 

[averyfreeman]

Redundant power inputs (as well as PoE) ... strain relief ... auto-update ... letsencrypt-routeros

Wow sounds like it has a lot of nice features!

 

[danb35]

I'd also note, since it's come up (though I think in different threads than this one), that the pamphlet that comes with it refers to max power consumption with 4x RJ45 SFP+ modules--so apparently that configuration is supported. I don't plan to test it (I'd be using DACs or, more likely, fiber), but FWIW, they appear to support it.

 

[jgreco]

I'd also note, since it's come up (though I think in different threads than this one), that the pamphlet that comes with it refers to max power consumption with 4x RJ45 SFP+ modules--so apparently that configuration is supported. I don't plan to test it (I'd be using DACs or, more likely, fiber), but FWIW, they appear to support it.

I'm expecting that a lot of "new" switches will be trying to do that, as there's no good reason not to do so, and you can probably even jigger the power so that you can get better 10GBASE-T distance. It does add confusion to the mix though. :-/

 

[danb35]

Edit: also will want to check out https://github.com/gitpel/letsencrypt-routeros to automate pushing certs to the box.

Having played with it a little bit, it seems to work nicely--I just need to decide which *nix box I'm going to use to obtain the cert and deploy it to the switch. One caveat: the README currently calls for you to use DSA SSH keys, which are deprecated--I've submitted a PR to change that to RSA instead.

 

[jgreco]

Mikrotik suddenly reached out to me yesterday asking for a phone number for shipping. Normally that'd be non-interesting but in combination with your update, makes me think a cargo container of these things just arrived in the US.

Totally wrong.

I suddenly got an e-mail from MikroTik on Monday morning with a DHL tracking number. An hour later, DHL dropped off a box, apparently shipped directly from Latvia.

Now, I got this from them on an eval ask. I don't play favorites and I speak my mind, but I do try to be fair. So with that in mind, some observations.

Monday was busy and I had a meeting all day yesterday, so at this point I only have a few impressions:

1) "Nicely packaged but not Apple-level ridiculousness". The packaging box is tight but sufficient, no wasted space, minimal padding. As someone who stocks inventory, I *appreciate* this.

2) "Wow it feels like this is made out of tinfoil." My first impression. But in MikroTik's defense, I routinely work with heavy enterprise grade 1U switches like the Dell 7048F, 8024F, etc., and those things weigh a ton. Because this is a passively cooled device, the choice makes a lot of sense, and the beauty of the sheet metal venting reinforces that this wasn't "let's just build something cheap".

3) I immediately noticed one other little design feature that reinforced that. The supplied PSU has a right angle plug and the chassis has a cord hold-down stamped into the metal. Holy mackerel. Someone with real world experience designed this.

4) I guess I'm not impressed by the logo stamped into the top of the chassis as I think it's mostly going to be a victim or perpetrator of rack rash.

5) "Nice, DE9 serial console... POE/BOOT?" ... at which point I was like... naw... but yes, apparently this thing can actually be powered by PoE.

I get a strong impression that this was designed by people who have actually built networks.