Designing an Ethernet Switch ASIC(youtu.be)

Table of ContentsTable of Contents
AI disclosure: All bugs are 100% human made.It seems I have just built the world’s first open source switch ASIC and I am getting it back from the fab mid november.Switch floorplan render with area density view enabled, occupying 711.2 x 325um of area. The first version of this chip is currently taped out on the Tiny Tapeout gf26b shuttle chip, part of the second wafer.space run. Silicon bring-up is expected to start 2026-11-15.Here is its repository :Essenceia/ethernet_switch_asicOpen source sillicon for a 100Mbps unmanaged cut-through ethernet switch.Verilog50Buckle up and welcome to the tale of more madness.Recommended soundtrack for reading: Taishi - Reverie for Another SphereWhere are the open source networking ASICs?#Recent FCC decisions have put the central importance of networking equipment top of mind, and in doing so, have also put the total absence of any entirely open source networking equipment hardware at the top of my mind.Although Open Source Silicon is in its infancy we are currently seeing a number of projects being designed, tested, and for the most ambitious ones, even taped-out with some proven silicon already in the wild.Awesome Kian-V project, a silicon proven RISC-V SoC fabbed on wafer.space run 1. And yes, it runs Linux. Link to the original post.That said, the vast majority of the most ambitious projects are predominantly RISC-V SoCs.Surprisingly, there has been much less interest in building open hardware for networking equipment.Well .. unsurprisingly actually … networking equipment is far less “sexy” than CPUs and has thus received much less attention from the open source community. On the other hand, this means there’s a huge untaped boulevard of projects open to anyone with more time than common sense to build open source networking equipment chips!Because, on the other side of the great silicon divide, the world of open source networking equipment is actually quite rich, both in terms of its flourishing software ecosystem and the open PCB/electronics ecosystem, with a flurry of fully featured open source routers available. Yet, due to the current ecosystem’s limitations, under the hood, these are all still all running closed-sourced, blackbox proprietary chips for the central compute and routing tasks.So what would it take to build a router chip?It’s a very ambitious project, since a router involves a complex SoC needing both a powerful enough CPU to run the networking stack alongside specialized networking hardware, and analog frontends for the wired and wireless connections. Since building a full router outright is much too ambitious of a greenfield project for a single person to ever hope to pull off, let’s begin with a more approachable first step: building a switch. After all, although most people only have one router, you can have a number of smaller switches, and we also don’t have any open source chips for those.Oh and, I am not talking about building an FPGA switch, oh no, I am talking about building a switch ASIC, taping it out and then proving the silicon works.Switch Flavors#Before we start figuring out what kind of switch we want to build, let me give you a quick overview of the different flavors of switches out there. Readers deeply familiar with networking equipment can skip this part.Glossary#Physical Layer#Ethernet supports multiple physical layers, outlined in the 802.3 IEEE spec. Each clause in the spec defines the underlying medium characteristics for carrying the Ethernet protocol over the medium at a given bandwidth.Both a 10 Gb fiber Ethernet link and a 10 Mb coaxial cable can carry Ethernet packets and, to the layers above the physical layer, the packets will look exactly the same.Where they will differ is at the physical layer and, outlined in the spec is precisely how, with which timing and with what encoding data will be transmitted over the medium.This ensures that devices made by different manufacturers agree on what “talking Ethernet” looks like, making them interoperable.So one of the first questions I must answer is which physical layer my switch should support. This will define how much traffic I must route, how fast I should do so and how much bandwidth I can carry.Managed vs Unmanaged#The second question is: what type of switch do I want to build?There are two big families of switches: managed and unmanaged. As the name implies, managed switches can be configured and managed from the outside. This allows for much smarter routing, such as supporting different VLANs.While unmanaged switches are essentially unconfigurable pieces of networking equipment that you plug into your network and just work (until power surge do you part).Cut-through vs Store-and-Forward#The last switch characteristic is cut-through versus store-and-forward.Cut-through switches start forwarding a packet while the packet is still arriving, leading to much lower networking latency, whereas store-and-forward switches wait for the entire packet to arrive before checking that no corruption has occurred, and then only forward the packet if that check passes.Where issues arise is when a packet is corrupted, a cut-through switch might still forward it, propagating corrupted packets through the network since forwarding began before the FCS (frame check sequence) could be checked.Designing against constraints (again)#As usual with my ASIC design work, what I build is shaped as much by what I want to build as it is by the constraints of the silicon.Pins#My first major constraint is the pins: not only in their amount but also in their maximum bandwidth.Since I will be taping this first generation chip out over the Tiny Tapeout shuttle chip, using purely digital tiles, I’m constrained by the limits of these pins.In total this will afford me 8 input, 8 output, and 8 bi-directional GPIO pins, rated to run reliably at 50 MHz on both inbound and outbound data.The absence of any analog front end makes building anything IEEE-physical-layer-compliant directly a challenge, but there’s a way around this: using an external PHY (physical layer) chip and interfacing with it over the standardized RMII bus. Though this consumes 7 bits per Ethernet interface it makes my 50Mbps pins capable of sending and receiving over 100Mbps Ethernet (100BASE-TX).Tiny Tapeout shuttle chipIndexInputOutputBidirectional0ui[0]uo[0]uio[0]1ui[1]uo[1]uio[1]2ui[2]uo[2]uio[2]3ui[3]uo[3]uio[3]4ui[4]uo[4]uio[4]5ui[5]uo[5]uio[5]6ui[6]uo[6]uio[6]7ui[7]uo[7]uio[7]I will be targeting the widely available Microchip LAN8720A/LAN8720AI (which will be the only mention of AI in this article) PHY chip for this interface. I will be going more in-depth as to how this ASIC will be interfacing with this PHY later.Area#My second major constraint is my limited die area.I am paying for all this out of pocket afterall, and more area means a higher manufacturing cost. Now I am not trying to accumulate a pool of gold or anything, it’s just that, in compliance with Maslow’s hierarchy of needs, waffles rank higher than area, and higher manufacturing cost means less waffles. So just like any semi-conductor company, I am incentivised to keep my area budget under control.Maslow’s hierarchy of needs, distorted ASIC edition.Because store-and-forward requires the switch to store the entire packet before forwarding it, and because ethernet frames can reach upwards of +1.5k Bytes (and +9k for jumbo frames), they require massive amounts of storage. A workaround for that would be to store the packet to some off-chip memory but I don’t have the pin budget to afford that right now, so on-chip memory is my only option.The problem is, on-chip memory consumes a lot of area given the large area footprint of SRAM and the even larger area per stored bit footprint of flip-flops.Now if we were talking of a few bytes this could be negotiable, but for multiple 1.5k Bytes packets this is a dealbreaker, thus ruling store-and-forward out.Very very high resolution render of the switch ASIC floorplan using Tim Edward’s excellent OCD 256x8 SRAM IP, for scale. Using W orientation.Assuming I was using Tim Edward’s excellent OCD 256x8 SRAM IP, a single 256 Byte SRAM occupies 301.3 x 224.93 um (using W orientation in this implementation) and consumes just by itself 1/3 of the floorplan. If we want to store even a single full packet we would need 6 of such instances, and since we have 3 ports, we would then need to replicate that 3 times, so 18 instances in total. Now, we have a larger 1028 Byte version of this SRAM that is 301.3 x 515.81 um which nicely increases our storage area density. But again, if we were to instantiate six such macros this would occupy x3 the area than we currently have to spare.Now, the Tiny Tapeout shuttle chip does support me scaling up this design up at most another two factors of two, or 4x. But the cost would also scale by a factor of 4, at which point we are getting into full chip orders of magnitude of cost. And if I were to go down the full chip route, since I then have the possibility of having a lot more pins to play with, this re-unlocks the possibility of interfacing with much larger external memories, thus changing the landscape of what the correct technical tradeoff would be again.RTimothyEdwards/gf180mcu_ocd_ip_sram3.3V SRAM macros for GF180MCU, based on the original 5V SRAM macros.Verilog225Easy to use#Lastly, I want something that doesn’t require external software or configuration.Firstly because I would like external third party users in the community to be able to easily pick this ASIC up and start using it. And that becomes increasingly difficult as soon as I start involving custom software. I am aiming for something that is as close to plug and play as possible with ease of use being a measure of success.Not to say that the community won’t be able to compile and flash my custom embedded abominations, just that their willingness to do so decreases exponentially with each extra step.Secondly because I am once again working on a very tight schedule and that custom software support adds significant amounts of time to both the pre-tapeout and bringup workloads. (Let’s not forget we are still talking about a single person’s project here.) This would be especially true for this ethernet switch project since I would aim to be compatible with a widely adopted open source switch management software and protocol like SNMP for monitoring or SSH for monitoring and configuration.Support for these protocols is not at the level of something I can trivially implement in hardware, especially for SSH, and the best design decision would be to offload to a CPU. So either integrate an on chip CPU on the ASIC or, build a custom interface to an external MCU and offload all these requests to it. Both options involve a significant software effort, with the on chip CPU also promising a huge design effort and even more area usage. At which point, I am approaching the planned architecture of the router rather than the first generation of the switch.So, for both of these reasons, an unmanaged switch is the path I am taking.What are we building?#So to recap, I am building a:3-portFull-duplex100 Mbps bandwidthcut-throughunmanagedethernet switch.Now that we have figured out what we’re building, it’s time to focus on the fun part: how to build it! 🥳System Overview#The real Ethernet frame that is hidden from view#Most people have actually never seen a full ethernet packet.When you inspect Ethernet traffic on your computer via tcpdump or wireshark, you actually only see part of the Ethernet frame, with some parts missing. These parts are generally stripped out by your computer’s network interface before the packet is forwarded to the software realm, thus hiding them from user’s view.Random IPv4 packet captured over tcpdump (tcpdump -xx -e -v 'ether proto 0x0800') :16:36:51.298082 60:e9:aa:92:dc:7d (oui Unknown) > 5c:e9:31:1e:9d:00 (oui Unknown), ethertype IPv4 (0x0800), length 78: (tos 0x2,ECT(0), ttl 64, id 0, offset 0, flags [DF], proto UDP (17), length 64)
superpitchu.53332 > dfw25s53-in-f9.1e100.net.https: UDP, length 36
0x0000: 5ce9 311e 9d00 60e9 aa92 dc7d 0800 4502
0x0010: 0040 0000 4000 4011 2b98 c0a8 0085 4a7d
0x0020: 0369 d054 01bb 002c a93f 6fe8 5bd5 5378
0x0030: f4c9 ccae 477c 5083 0696 8fa4 4f62 57bf
0x0040: 41a9 78a3 fc60 7781 2b8b 38a1 2ccdThe missing parts are:Preamble + SFD: Before the start of the MAC header exists a sequence of 7 bytes of an alternating bit sequence signifying that a packet is about to start, called the preamble, followed by a 1-byte marker ending in a double asserted bit sequence called the SFD (Start Frame Delimiter) signaling that the next bits are the MAC header.Frame Check Sequence Footer: At the end of the packet there is a 4-byte footer called the FCS (Frame Check Sequence), used to check whether the Ethernet frame’s content was corrupted during transmission. This is checked by both store-and-forward switches and by most computers’ network interface cards and these drop all frames that fail this test before the packet is forwarded.Raw packet overviewAnd since, once they have been evaluated by the networking interface, both the Preamble+SFD and FCS serve no further relevant purpose, they are stripped out before the remaining packet bits are forwarded up the networking stack.Now, since we’re going through the Microchip PHY, we won’t actually be interfacing directly with the Ethernet physical layer, as it will be abstracting away the precise 100BASE-TX PHY behavior, though the full ethernet frame will remain intact (including Preamble+SFD+FCS)All our data transmission and reception will actually be done through the RMII bus interface.The RMII interface#LAN8720A to application device interface diagram taken directly from the LAN8720A/LAN8720AI datasheet. For our application the MAC and RMII block are going to be part of our ASIC and we will have 3 such interfaces in parallel. Like in this example, both the ASIC and all the 3 LAN8720A chips will be driven by the same external 50MHz reference clock signal.RX#rxv valid signal (crs_dv in the diagram above )rxer error signalrxd[1:0] two data signalsOn the RX (reception) side of RMII, we have four pins: two for data, one for data validity, and one to signal that an error has occurred on the line.
Interestingly, this valid signal isn’t a pure data-valid signal but an early data-valid signal, and it’s up to our ASIC to read the incoming data and correctly identify when the Ethernet frame data actually starts using the preamble and SFD. The valid signal actually asserts asynchronously a few cycles before the start of the preamble but, deasserts synchronously with the end of the Ethernet frame.TX#txv valid signal (txen in the diagram above )txd[1:0] two data signalsOn the TX (transmit) side, we only have three signals: two for data and one for valid, with no error signal. This makes sense, since the error signal is there to indicate issues on the medium, and since there is no medium between the ASIC and the PHY there is no need for it in the TX direction.Timing#Lastly the PHY chip’s datasheet outlines the expected timings for the interface signals.Unlike what my high level overview might have suggested, new data isn’t immediately available at the start of the clock cycle. Like in all real hardware there is an internal propagation delay. The same goes for how fast the data is allowed to transition after the start of a cycle. Because the PHY chip also contains flip-flops and is thus also subject to hold constraints, the previous cycle’s data must remain stable for a certain period of time after the clock edge before the new value can be propagated.RMII interface timing for the LAN8720A/LAN8720AI Microchip chip family. Documentation code: DS00002165C. I swear, this doc is so well written, it truly makes this chip such a pleasure to work with ❤️.So, implementing correct signaling alone isn’t enough to properly interface with this chip. If I want my ASIC to work I also really need to make sure my physical implementation’s resulting timing respects these constraints. This is both verified and enforced by specifying design constraint rules as part of the SDC (Synopsys Design Constraints) file.Now by designing around the constraints of the LAN8720A chip readers might be concerned that I am locking myself into a dependency on a single external chip.And actually that is kind of the case here, I am not proud of this but I caught this discrepancy too late, and not mentioning it in this article simply doesn’t align with the technically honest recollections I am looking to do here.Since RMII is regented by the RMII consortium, and since in the RMII specification they include the AC characteristics
I blindly assumed the LAN8720A would be compliant.Turns out the LAN8720A isn’t actually compliant.And, although in many cases the LAN8720’s timings are actually more constraining than the official spec, when it comes to the hold constraints on the RX pins it is actually much looser at 1.4ns less than the RMII’s consortium’s 2.0ns.Although this might still work in practice, it would be by pure luck and I only believe in engineered luck!RMII Specification, AC CharacteristicsSo note to self for future versions: I should set this to 2.0ns to guarantee that the ASIC would work with other RMII chips.Apart from that, the LAN8720A is widely available, easy to get dev boards for, well documented, and at 103 cents a piece, worth every penny.The official Microchip LAN8720A developpement daughterboard not only came with a real paper datasheet but it was the most precious tiny datasheet I have ever seen.ASIC pinout#Putting it together, here is what the final pinout of our ASIC will look like connected to all 3 PHY chips. We will be referring to these as PHY0, PHY1 and PHY2.Indexui (Input)uo (Output)uio (Bidirectional)0phy0_rx_data[0]phy0_tx_data[0]phy2_rx_data[0]1phy0_rx_data[1]phy0_tx_data[1]phy2_rx_data[1]2phy0_rx_vphy0_tx_vphy2_rx_v3phy0_rx_err-phy2_rx_err4phy1_rx_data[0]-tx_phase5phy1_rx_data[1]phy1_tx_data[0]phy2_tx_data[0]6phy1_rx_vphy1_tx_data[1]phy2_tx_data[1]7phy1_rx_errphy1_tx_vphy2_tx_vAddress Resolution#At its heart, a switch is actually a conceptually simple piece of networking equipment whose role is to read and forward incoming Ethernet frames to the correct ports.
Matching incoming packets destination MAC addresses to the correct port is accomplished by an internal structure called an address table.Like a software dictionary construct, it maps a destination MAC address to a switch port index.In order to build this correspondence, the switch reads incoming traffic’s source MACs, keeping track of which port each source MAC was seen on, in order to forward future traffic to the correct port.Putting it together: when an incoming Ethernet frame arrives, the switch first reads the destination MAC from the frame header and checks the address table for a matching entry. If there’s a hit, it forwards the packet to the specified port. And in cases where there is no hit, in a best effort to minimize packet drops, it broadcasts the frame on all ports except the one it came from.At least that is the high level overview, in practice, this is hardware, so things get a bit more hard (pun intended … no, I am not sorry).Filling the table#Recall that this switch is unmanaged, it needs to autonomously learn the correct ports corresponding to each device’s MAC address. It does this by reading the source MAC of all incoming Ethernet frames. Once it has the source MAC, it checks the address table to see if this MAC is already present. If no entry exists, it writes a new one.The requirement that all entries in the table be unique is not only a correctness requirement but also stems from two physical implementation constraints (see ASIC implementation rules for readers not already deeply familiar with ASIC design ):Most efficient utilization of the table: since the table is implemented in hardware not only is its size fixed but each additional entry is non-linearly more expensive. Since this first generation 3 port design only features 4 such entries it is imperative to make the most use of each entry, and duplicated entries are a pure waste.Timing: lookup of this address table is on the critical path. But by guaranteeing that each hit is unique, we can replace the priority mux selecting from individual entry hit to the table hit, with a lower logic depth and/or reduction tree. See the code below as an example for the entry port hit to table port hit.always @(*) begin
/* verilator lint_off CASEOVERLAP */
(* parallel_case *) // telling synthesizer it can optimize this from a priority mux to a parallel case
casez(mac_hit)