 Hello all, I'm Shreya Rangini, working at Samsung Semiconductor India R&D as a embedded system engineer and primarily responsible for the Ethernet IP driver development. Before we start our presentation, let me tell you a short story. Initially when I started working on Ethernet, there was a situation where I need to know the basic informations of the network interface. I started using if config. I was aware that if config gives some basic information, while if config is good for some basic information about network interface, but for some advanced operations and information, we need to have something better than this. It was then while exploring, I came across another utility in Linux named eTool, which we can help to extract many meaningful or internals of the network interface and also its open source. So we can add new features to it. Ever since I started working on eTool, it was fascinating me and I tried in improving the tool. This is the result of my work and experience with this tool in the past couple of years. In this talk, I would explain what all issues I encountered while working on developing and validation of Ethernet IP. And I have taken the example of design where Ethernet QOS IP and automotive SOC from Synopsys in this presentation. So how this IP is helping to resolve most of these issues and how we utilized eTool to explore and get the best out of Ethernet IP. Okay, let's begin. On behalf of my co-author Ravi Patel, I welcome you all to the presentation on eTool at Diagnostic Approach for Network Issues in Linux. Let me quickly walk you through the agenda of this presentation. First, we will see about introduction to Ethernet. We will see about standard Ethernet and we will see about automotive Ethernet. Then we will see the need of debugging and performance tuning of a network interface. What are all the issues we are commonly facing and what are all the tools available to debug those issues. Then we will see about MAC management counters, what are MAC management counters and how they are beneficial in debugging any issue. Then we will see why we have chosen eTool to implement this feature, what are the benefits of implementing this feature in eTool. Then we will see our work which will contains the implementation we have done in the user space and in the kernel space. Then I will explain you one real use case which we encountered while working in Ethernet. Then that I will show two use cases which will show the result of our work. Then finally I will conclude with the future scope. Okay, now let's have a look on Ethernet, standard Ethernet. We all know that Ethernet is a set of technologies and protocols that is primarily used in LANs. It is it provides a service up to data link layer. At the data link layer Ethernet divides the data stream received from the upper layers and encapsulates into frames before passing them on to the physical layer. Ethernet has gone through four generations like standard Ethernet, fast Ethernet, Gigabit Ethernet and 10 Gigabit Ethernet. Standard Ethernet, it can transmit data at the rate up to 10 Mbps speed and fast Ethernet have a speed limit to 100 Mbps. The 100 base TX standard has become the most popular due to its close compatibility with the 10-dacity Ethernet standard. The Gigabit Ethernet works at 1 Gbps speed which is 1000 base T. The most important difference between Gigabit Ethernet and the fast Ethernet include the additional support of full duplex operation in the MAC layer and at the data rates. 10 Gigabit Ethernet is the fastest and the most recent Ethernet standards. It is 10 times faster than the Gigabit Ethernet and it is entirely based on the use of optical fiber connections. Okay, now how automotive Ethernet differs from this. Automotive Ethernet is a subcategory of Ethernet as specified in IEEE 802.3. It operates over a single differential pair of wires and it is specifically designed for the low-radiated emissions and immunity requirements of the automotive industry. The distance which these standards operates is also much shorter than other Ethernet standards given the size of the vehicle they are intended to be used in. As we can see in this slide automotive Ethernet have crossed three generations from year 2010 to 2020. In 2010 it started with the use of Ethernet in cars for diagnostics and firmware updates. It used 100 base DX standard though this standard does not meet automotive requirements but this interface is only used for diagnostics while the car is in service location and exemption was made to allow its use. In 2015 Ethernet is used for camera driver assist and video infotainment connections. In this model Ethernet is used in point links and is noted being used as a shared medium for different interfaces. Currently in the new model switched 1g automotive Ethernet will interconnect all the domains in the car meaning that Ethernet will be a shared medium. The new anatomy not only helps to reduce cost and weight but also makes it much easier for the different systems in the car to cooperate. Nowadays people expect car to be like their extensions of their home where they can access all the devices they use at home like apps, social media and real time information many more. So automotive Ethernet will be the solution for all these requirements. That's a pretty much about automotive Ethernet. It's time to think about why what is the need of debugging and tuning of the network and what are all the tools available for this purpose. As we are working in Linux probably we must have faced various network problems. Being working with a high speed Ethernet IP the sheer volume of traffic on a super busy network makes it very tricky to diagnose any issues. Whenever an issue occurred we have to obtain all the required information using all needed tools to find out at what level of the network stack contains the bug. For example we can use if config or e-tool to get information about network interface which includes status of the link, speed of load parameters and interrupts occurred and number of packet got dropped during data transmission and so on. We can use IPERF to check the bandwidth availability and to validate whether the whole target is accessible or not we can use ping, trace route or dig with diagnose everything DNS. Another one popular command instead which we used to get all information about TCP connections. Hence being a software people it is very important to know about these tools as if maybe in real time if a product is out in the market maybe we don't have permission to connect JTAG or something to debug any issue occurred at that time all these software tools will help us to debug any issue. Okay let's see about MAC management counters. MAC management counter or MMC counter or the extension of the registered address space of the CSR module. It contains various registers to maintain the statistics of various received and transmitted packets. They also contain control registers for controlling the actions of registers like there are two status registers containing the interrupt generator during transmit and receive and interrupt enable registers for transmit and receive. The MMC counters are free running. There is no separate enable for the counters to start. If a particular MMC register or MMC counter is present in the RTL, it starts counting when the corresponding packet is received or transmitted. The width of each MMC counter is 32 bit by default. So you can individually change the width to 16 bit for a selected MMC counter during configuration. These MMC registers are very useful when it comes for debugging and monitoring different types of packets which is essential in automotive application. Okay, so though many tools are available for network tuning and debugging, why we are focusing on each tool? Each tool is very versatile. It can be used for various scenarios like configuring features such as TCP segmentation of loading, DXR extension of loading and etc. It is important to know about network tuning as we are dealing with high speed NICs like Gigabit Ethernet. As I said before, each tool is an open source utility and can be enhanced as per the requirements of the new hardware features. Each tool isn't concerned with the IP address of VLANs and subnets. Instead, each tool lets us to manage then configure the software drivers and the hardware settings that control the network interfaces. There are so many parameters that can be tuned by each tool. I have listed some of the few basic parameters like we can select the port. We can check and change the transmit and receive mode of operation, either full duplex or half duplex. We can change the offload parameters. We can check and change the speed of transmission. We can control the auto negotiation of the network speed and mode of operation. We can change the physical address and it can be used to set VACON LAN option. Another important feature of each tool is getting the statistics of the network interface. We can get more detailed information of network interface like number of packets transmitted and received and count of various error packets. There is a reason for highlighting this because we have implemented a new feature in each tool, which is like an extension for the statistics showed currently in each tool. We will see about that in detail in further slides. Okay, we can install each tool in most of the Linux distributions. First of all, as we have to do is to update the APT package. Then we can use sudo apt install each tool life and via option to install each tool in Ubuntu systems. And if we are using Fedora or CentOS system means we can use sudo dnf install python 3 each tool command to install each tool. Let's have a look on each tool architecture. Each tool is a command line utility that resides in the Linux user space. It provides consistent access to networking commands directly through bash. It runs directly from and integrates with bash while being interoperable with the regular way of accessing underlying configuration files and automation. As we see in this slide, each tool uses I octal interface to communicate with the kernel. Once the each tool I octal is called, it will check for the relevant each tool of sender respective driver and do the operation accordingly. Okay, so how does each tool is communicating with the kernel? In each tool interface, all informations are passed in the form of request and replace as defined as macros in each tool.h uapi header. The each tool command enables you to query or control the network driver and the hardware settings. As you can see in this slide, each tool request or divided into three categories like get set an action get request or the or sent by the user space application to retrieve some information. They do not contain any message specific attributes. They do not contain any message specific attributes kernel replies with the corresponding information by passing a reply message. If the data has to be modified, that can be done by passing set request with the corresponding attributes, which we have to modify replies to most of the set command will be error code. And if the kernel provides any additional data, it is sent in the form of reply. Most of the cases that command needs pseudo permission to apply the changes. The most commonly used get and set category request action is used in case if the user wants to perform any specific action. So as you see in this slide, for example, if if the get request has been sent from user space to kernel space, like if the user want to get the link info, the message will be sent like each tool message link info get and the response will be like each tool message link info get reply. If the user want to set any property in link info, means the message will be like each tool message link info set. This will be passed from user space to kernel space and the kernel replies with the appropriate notification reply message. And if the user wants to perform any action like here in this case, if the user wants to perform any cable test, the message is passed from the user space to kernel space is each tool message cable test act. And the kernel replies with the appropriate notification message. Okay, let's see how the IOP communication is happening in each tool. The flowchart shows how the user space command each tool ends up invoking the dev each tool on the kernel site. The interface between the user space and the functions is the IOP system call. As we see in this flowchart, each tool uses SAOC each tool IOP to communicate with the kernel. When an IOP call occurs, do IOP function gets invoked in the Linux kernel in which dev IOP function call is happening. Inside dev IOP function, all the IOP calls defined in the Linux kernel is added in a switch statement. So that when SAOC each tool gets called, that particular switch case will get executed. Dev each tool function will get called. All the each tool routines are added in the dev each tool function. Hence, depending on the function called particular each tool routine will get executed. Here we go. We will see what modifications we have done in each tool user space side to get the MMC registered details. We have declared a new option in each tool to get the MMC information like with a flag called showMMC and a function called doGMMC stats. This doGMMC stats function take command as an argument and will call another function called doGMMC. This inside this doGMMC function currently we have used getStringSet function which will do an IOP call to get the MMC registered information and the respective size of the statistic so that we can allocate enough memory. Now we will see the kernel side implementation what we have done to get the MMC registered details. As I said before, SAOC each tool will get called and along with the command of getStringSet. So inside dev it tool function it will check for the command. Since we have passed getStringSet function each tool getStringS function will get called. Inside each tool getStringS function we have added a separate case to call estimate getMMC stat function. This estimate getMMC stat function is defined inside estimateEatTool.c file. In the estimateEatTool.c file, first we have declared a separate structure for the MMC registers which will contain the bitwise information. This structure contains bit index and the bit name. Like this we will be adding a structure for all the registers with the bitwise information. And we have added a small function to convert the decimal value of the register into a 32-bit binary value. In the estimate getMMCStats function, first thing we are doing is read the register value. So first we will read the register MMC registered value. Then we will be passing it to the decimal to bin function to convert into a 32-bit binary value. Then we are checking for each bit, whether it is set or not. And according to that, we are passing the bit name and if it is set, it is we are making us enabled. And if it is not, we are making us disabled and we are copying all these strings into a common buffer. Finally, this buffer is passed back to the user space. So we saw our implementation what we have done in the kernel space and the user space to get the MMC registered details. What made us to implement this? While working in the Ethernet IP, one time we faced the issue like our if config shows these many errors on the TX side. But the pink flood test with a different packet size shows there is no packet drop, but the error status was increasing when the packet sizes more than 100 bytes. And we are having one more issue like in the MTL, we have set the disabled transmit status field. This caused an issue in updating the status field in the TX descriptor. So we are unable to find what error occurred. Does it mean the software won't be able to find what error occurred? It was then while exploring, we got to know about MMC counters. MMC counters have information about the interrupt status, interrupt mask and various other registers to show how many packets got dropped because of that particular error. But the current eased tool implementation does not support the support to show the registered details of the MMC control register or MMC interrupt status register interrupt mask register. We have added support to get the value and the bitwise information of the MMC. We have added extra support to show the register value and the bitwise information of the interrupt status register and MMC control register and interrupt mask register. So using this, we were able to find that underflow interrupt have occurred and we are able to find number of packets were dropped due to underflow error. Then this helps us to resolve the issue. As we saw our implementation, now we will see our first use case. We have taken MMC control register in DWC equest registers is the very first and the basic MMC register, which have bitwise information that this very useful for debug purpose. This MMC control register value is not displayed in the current tool implementation for displaying MMC registers. We have added additional support for the MMC control register so that user gets more details information about the critical register in the user space itself. As you can see in this slide, this is how the information looks like when we give a each tool command. We have given show MMC flag and the device name as E0. You can see it displayed the first the value of the MMC control register and shown the each bitwise information like particular bit is enabled or not. Next next use case we have taken is for MMC RX IPC interrupt mask. The current implementation of the E2 shows this value of the register as a big number as shown in this slide. It is very difficult for the user to decode and interpret the bitwise information. We have added additional support to get all bitwise information of this register. As you can see, we can we first we will get the value of the register then each bitwise information like a particular bit is enabled or disabled. So we saw our implementation, output and what made us to develop this. Now we will see our final conclusion and future scope for this work. We are able to enhance E2 user space utility to capture some of the critical debug registers of design where equals controller and present a readable content instead of just dumping the register values. We modified the design where EQS driver in the kernel by adding a required E2 hops hooks to get all the MMC register value and the bitwise information of MMC control register interrupt status register interrupt mask registers as defined in EQS IP. This implementation is helping us to get all the information via E2 command line. The user can simply use this tool to tool for their self diagnosis. Future scope involves implementing this functionality in E2 for other MMC registers and upstream in the source code to the Linux mainline community. Please feel free to ask any doubt regarding my session. I thank you all for joining the session and I thank whole Linux Foundation committee members for giving us this opportunity to showcase our work. I hope our work will help the users to reduce some of their burden in debugging this in some of the issues which needs MMC register details. Once again, I thank you all for attending this session. Have a great day.