 Welcome everybody to the next talk. It's about L2 BSA that is layer 2 bitstream access for broadband services. It's a complicated issue more on the political side than on the technical one. Let's start with the person that it shows. I have grown up in East Germany. The access to computer technology was not that easy. But I managed to have the Commodore 64. During the unification change I switched from the Commodore to Microsoft to MS-DOS and Windows. But a few years later, two years later, I had switched to Linux for a project in order to provide a religious website in Germany. One of the firsts about sects and about providing information about the dangers of religion. So we need a web server and we found a possibility to do this with Linux. The next problem was to get access to the internet in these times. I was part of the network AFOW in Germany. They tried to build an internet for really ordinary people and by ordinary people. They managed to bring the internet from the universities and from some companies to the masses and allow everybody to have access to the internet on a very simple and collegial base. So that was a grassroots activity I've been proud to be part of. And I see some of the people I have met in this time in this meeting here too. That's fine. Hi everybody. Going from this enthusiasm to a commercial activity started in 1996 when I co-founded an original ISP. And in this position I was responsible for building the infrastructure of real commercial broadband ISPs in Turingia. One was sold to another company and the second one is still existing and we are doing services for them. And one of the points we had in the second ISP lifetime was that we had to provide PPP. And we had a commercial appliance for this and this commercial appliance was not sufficient anymore. And we thought about changing it to an open source solution but none of the solutions were really applicable for our needs. It was too small or too unstable and somebody told me that the Russian internet is running on NPD on a free BSD on a so-called net graph infrastructure. And I was very pleased to have contact to these people and they told me they had no problems. So I switched the ISP to free BSD and NPD. The result was that we had corner panics about every two days and it took three quarters a year to fix the bugs in the corner and NPD software and so on. And it was my start with free BSD. So broadband in Germany after, oh yeah, that's a horrible story. Let's start with the early beginnings. Germany had an early start with networking. Of course using the telephone network, so the copper lines and of course they are using everything everybody else is doing. But in other areas of the world, for instance in Japan, they already noticed that copper is not the best way to do the last mile and they insist on switching to fiber. Like many other countries are preparing in the 90s to do so. But in Germany we had a problem. We had a government which was tightly coupled with factories who produced these copper infrastructure and therefore they decided that they do not switch to fiber. And still stick to the copper area. And it was so a massive shift in the decision making that it still persists to this day. In 1995, the same political party chose to privatize the infrastructure and sold it out to a new founded company, the German Telecom or Deutsche Telecom. And in order to start such a company, they got all the infrastructure as a gift. And so they created politically a monopoly. Nobody else was able to have access to the households. And the consequence was that the competitors which want to start in this market was faced with an impossible mission. And they choose the way, the only way they can do, they choose the way to go to the curb. And the consequence was that the unbundling of the last mile was regulated so that the Deutsche Telecom was forced to give access to the last mile to the copper into the households to competitors. And a few years later, after almost all in the world has changed to high speed internet using fiber, the German internet was still very slow and expensive. So they tried to build faster internet access and the technology was called vectoring. Vectoring is technology using all the signals from a bunch of copper cables and find out which cable has which signal, which signal influences which other line and then compute which signal might become from which household. But this technology needs to have access to all the copper lines in an area at once because they are doing signal processing on this. For the technical background, you may think of DSL as, yeah, it's not using the copper line as a wire, it uses as a guide to a wireless signal. The wireless signal is guided by the wire on the surface, but it's not shielded in any case. So if the two copper lines are using the same cable, these spring from one cable to the other and the vectoring technologies make use of this arrows in transmission and recalculate which signal is coming from which household. So in the consequence that means that we get a remodernapalization of the last mile and it took a long time up to we get a regulated access to the last mile again. Now, without layer one, only a layer two access because layer one is vectoring, it has to be done by a single company in a single device. So they had to find the solution to bring the layer two bit stream out to another ISP. The consequences of this who will mess with fiber is very simple. If you make a ranking how fast internet is available in which country, you will find Germany in this area here. It's fiber to the home or fiber to the building. And if you like, you might have a look at the other countries which are better or even worse than Germany. It's hard to be worse than Germany in digitalization. Currently to this day in Europe, only one country has a worse digitalization strategy than Germany or digitalization deployment in Germany. It's Albania. Every other country in Europe do a better job than our country here. Okay, back to the solution. So we had a monopoly recreated by vectoring. As I said before, that we need to have all the copper lines in one device. In order to make a break out of this lines, we had to name the line to the household and say, we need all the bits on this line. So please bring it to an gateway in order to provide services to this household by another ISP. So we had a one-to-one mapping between the port and the gateway and the technical solutions were simple. They only put a VLAN tag on the port and transport so a multiple tagged frame to the gateway. And there you have to process it. The consequence is obvious. At first, you have only 4,000 customers per gateway. You have only 4,000 VLAN tags you can use, so you can only number 4,000 ports. The second consequence is a little bit difficult. It's the consequence that you do not have the opportunity to have a smart network distribution or aggregation network. For instance, you are not able to provide IPTV over such a layer-to-bit stream access because you have to replicate all the traffic in the central unit. So you have to send the same stream again and again to each customer which is currently watching a special stream. Usually you would use multicast and make the replication as late as possible in the network. So you can't provide IPTV in this approach. It's an interesting point because it means that the competition is not able to provide triple play. You are only able to provide double play, so internet and voice, but not TV. The next problem is that if you have a DSL line that's synchronized to a special speed, say for instance 200 megabits, but on the gateway line we have a gigabit or 10 gigabits. So we send much more traffic to the aggregation network or transport network than we can send out on the last mile. So we had a problem with packet loss and the solution for this problem is very simple. It's strictly forbidden to send more data than can be sent to the customer. How do we know how much data we can send? That's by modifying the DHCP or PPP requests from the customer. They are adding a special option and saying the current synchronized speed is the following how much megabits up and down. So you have to provide the rules and traffic shaping. If you do not do this, if you send more traffic to the aggregation network of the other provider, then the DSL line is able to deal with, you have to pay a fine. The fine is about, it's a regular price, it's about 2,000 euro per day in customer. So there's a strong motivation to implement shaping. No, there's no hardware flow control. Forget it. Even if the DSL line is shaping the synchronized speed, which is very common and maybe happen every minute, you will not get any notification about this because DHCP requests or PPP resynchronization will not happen every time. If you build such a thing like we are doing, you have to calculate a little bit and say, okay, we have one million households and two ringers. So we need 260 interconnection points if we have 4,000 households per interconnection. But luckily we do not need this because we do not get all the households in the whole country here. Of course, only a very small percent. So we have only to have access to each region where a customer we want to serve is located. It's about 50 interconnections we need to take in German telecom operated areas. The rural areas are no problem because they are operated by us. So we only have to manage the connections to about six cities. We do not have any more cities in this area. Yeah, 100% market penetration would be very simple. But that's not the case here. So we have to use this approach. The main problem is that in order to terminate such an A10 MSP or L2BSA, you have to buy an appliance. That's the same situation we started in 2008. We switched to the freeBSD and we do not want to go this way again because these appliances are much, much more expensive. They are about a million euro per device. And the only support single mode of connectivity is PPPOE. I will explain why they are doing so. And we choose to use DHCP and IPv6 and multiple VLANs. But we run into problems which are designed to prevent DHCP and such services. But we have to deal with it. What happens on the wire? If you have a customer, the customer has a payload, a PAN, a customer service tag as a CBLAN saying that's traffic for internet or for voice or for management or something like this. And then you have the source and destination IP. The packet or the frame is transported from this area. The destination is coming first and the source port and the CBLAN and payload, of course. On the DSLAM where the last mile is terminated, the L2BSA provider has to insert the second tag, an S tag, service provider tag. So all the traffic in between here, the payload and the CBLAN, is completely opaque to the transporting service provider. That's a good thing. And now let's have a look on such a packet with real numbers. We have a packet in VLAN 140 which is internet in our case. And so the customer CBA which is configured to our ISP and provides access to this network in this way. Now the customer is connected to a German telecom DSLAM. They are injecting the SVLAN 27 and hand over the packet to us. Very simple. And if we send back a packet in VLAN 27, then we reach the customer. And so we have a B-directional one-to-one mapping. A few moments later, see this, the SVLAN has changed. They changed on every DSL resumed. They may change it even on other cases. We have noticed such changes. And the idea behind this is that every new synchronization is a new session which requires a new dynamic service provider VLAN. If we still assume that the old session is existing, we send to a drop-hole. Power traffic will go out on the wrong SVLAN and that's why it will be dropped because there is no port associated to this SVLAN anymore. It will happen suddenly without any warning. If you are using PPP, there is almost no problem because this situation will sort of itself. PPP has to keep alive in the LCP frame. So if you lost the connection, it will notice and will start a complete resumed. That's why you get new information from the new PPP session and new information about DSL speed. And you can still connect the customer over the new PPP session if you have DHCP or FNASM. That's our situation. So at any time, the SVLAN might change. In many cases, even if it's in DSL resync, even the customer does not notice it simply because in DSL modem, the real router is behind the DSL modem, the article, and the DSL modem will make a resync but will not notify the router of the customer that they had changed the synchronization because the whole equipment depends on the idea that the DSL line is well defined and well recognized and this is completely changed by choosing a random number at any time. So what we do is to simplify the approach by making it more complex as usual in technology. So we have from the customer side, we have the CVLAN and the transport provider will add an SVLAN. We want to have it cheap, so we do not use a single server power. In the connection, we use the server with multiple interfaces. At the moment, we have about 15 to 20 interfaces per server. We do not need to be at very high speeds. We do not have so much customers per interconnection. In order to aggregate this, we have added an AVLAN and aggregate VLAN so we can name the interface that allows us to make the cross configuration per interface. Then the magic happens. The magic is to rotate the VLANs. So if we have the VLANs in another order, we can start with the CVLAN, which was the inner one at first and can build a network network for a server. So we have a different cross configuration for each service in a static way. Then we can decouple the interface and then as the last step, we can decouple the line. So the dynamic one is on the end, and that's easier to handle. Let's have a look. That was the approach. Traffic is coming from an interface from the L2PSA provider. We have some patch-shaping, patch-node and for the cross. The diamond nodes are VLAN nodes. Here we have traffic-shaper. So for each line, we have a traffic-shaper. We are not allowed to send more traffic to the line than the DSL is in sync. So we have to aggregate over all the VLANs, all the services we have, and we have to provide a shaping of all these things in one step. That's why we are doing this on the interface level. And then that's the aggregation here. We have more than one line. This is the rotation step. Then we break out the customer VLANs. We have, for instance, a cross session for IPTV if we could use it or for voice. Or we have the simple session for Internet. Then we have to break out the aggregation line than the SVLAN from the provider. And then we can go to a bridge, and the bridge has a VLAN on our network, and we can go to the whole IP platform we already have. So we do not need to develop anything new here. What are the problems? Red are existing nodes. It's an NGE. And that's all to inspect it for DHCP in order to set up there. I have actually all, without any problem, the green nodes, the bridge nodes, they do their work, but only in the lab. Orange is a lot of the user-specific programs and DHCP break out looking on the DHCP options setting up the appropriate parameters for traffic shaping. On the other hand, we do not know which line is coming, which SVLAN is. We have a nomad here. So we postcript again, which is setting up a new VLAN as to the bridge. So we can use the bridge for differentiating the lines by mega-presses. Each end device should have a different mega-press. So we can use the bridge and say, oh, that's like so-and-so, and then it's SVLAN and so-and-so on. Let's have a look a little bit later. In order to bring this up, we had some modifications. We had to do some modifications to FreeBSD. First, a simple box that frames didn't pass through the chip or that the magical node has to come into the corner itself for the VLAN rotation. Then the problems with the bridge node, which was very hard because the NG bridge node has an arbitrary limit of 16 links to connect to. It's a little bit low for all purposes. So we have to bring it up to unlimited links. Then the next problem is that this creates so large, a net graph message that's not able to bring it in user space. In order to stop this from growing arbitrary large, we make the bridge node to stop learning from the uplink ports so that the MAC addresses are not floating the table of the bridge node. It's the only node in the graph world which is not production-ready because it's not multithreaded. All the activities through this node is on a single core. So the whole device, the whole server was limited to about 800 megabit to 1 gigabit, one core processing for all the nodes, all the lines. For course, there are some modifications necessarily because we need coloring the traffic and add a shader to a horn order so that the green traffic, usually voice traffic in shaping curve and changes to make the graph a little bit more compact. Our problems we had was the same VLAN and different VLANs, the same Mac and different VLANs. It's possible that a customer device is operating on multiple VLANs, for instance, un-tagged at the same time with the MAC address and we have to deal with it. We have overlapping MAC addresses from different devices because several CPEs, customer boxes, increase the MAC address by one for each VLAN if you have a lot of VLANs. You have a good chance to reach the MAC space of the next device especially if you buy devices on a palette with increasing serial numbers. Then the chances are very high that you have a lot of these devices in the situation. Then we had a problem that the customer are free to use a router on their own decision so they do not have to use a pre-prepared router from us and of course they have, in many cases, very interesting differences. The CPEs last devices have to be worked and they still have to work and then you are on the way to end first. Send the device to the customer which is working or have a long discussion with the customer and saying whatever he is doing wrong or if you are going to the network I can say make it happen so that the customer is lucky. That's the common approach. We have a problem with the customer, find the solution so that the customer has come up online and do not longer complain about the service. Then we have problems with German telecom, German telecom sometimes is able to say we have a DSL sync with zero megabits per second. Zero megabits per second means that we do not have to send out any data. That means that the DHCP response will not get an IP address at all costs it's not allowed to send it out. And of course a lot of broken interfaces especially if you have untact then you are violating IEEE and so on and so forth. Real-world example, that's the net graph of a running system at the moment. One with not so much lines is a new device here. On top we have the aggregator VLAN which adds for multiple lines incoming the A VLAN then we have the rotation ASC to CAS. Now we have the service VLAN at the first place and we can go out with VLAN 140 or anything else. So for 140 we go out and make a DHCP SNIVER with BPF node and then to a socket which is not shown here. Then we go out to make the A aggregation line so we have two lines here and for these we have the dynamic links to the bridge so that every line has the SVLAN, the ALAN, the CVLAN and so on. And that's the untact case. We don't have enough VLANs to rotate then we have the processes as an incomplete one. There is nothing to do so it can be. It's nothing special but you see that you have a problem to untact and detect into one bridge. There are others which are able to do the same as you. That's the net graph node for the services. We are seeing that for the internet traffic we only make a copy for the DHCP. And for this voice traffic we have a course in order to make the right text. We have color mark text coming in and we need to go out and say which priority class it has to be in order to fulfill the specification of the interaction. Let's go a little bit more down and see the more part to our infrastructure. That's our infrastructure here where we are complete. So we have high speed internet coverage and for the voice traffic we have each line a traffic shaper node a traffic classifying traffic to be sent out with priority and without priority. The reason behind this is that there is a penalty fee for sending out too much to limit this and of course we have to do it per stream. On the other hand, in the upper case up to the interface to the German telecom or the A to BSA line we have to make the shaping for the whole traffic so we split out the V lines for each of the customers and there we currently you see the speeds here the nodes in a way that they reflect the speeds so the name of the node is really shape 116-46 so that the net cap output is automatically providing the information with the site in the car node. All other traffic which is not yet classified goes out without any shaping. Then we have the patching so we have a classification saying for instance that's priority 4 traffic then we go in and patch there this is a simple patch there the bits for the VLAN tag so that we have the right information on the line and that's going out to the German telecom and they are providing it to the correct customer. So in practice that's from yesterday we have several max and several bridges the internet or the voice bridge and we have random chosen link names which is available on A VLANs or aggregation VLANs naming the line we have the SVLAN and we have the CVLAN and that's a typical output from our monitoring system can be very long that's why the problem is the NetCraft messages are to ask for conversion so you see a lot of different VLANs we are using here 180 is for business customers they are choosing to use their own devices they have to be handled a little bit different but okay if you have such a situation here with Antec that's fine but if you have a situation like here with a tact one most CBE sending out frames on the Antec VLAN despite it's not configured for instance they are sending out a network for OAM so Ethernet message traffic about 5 to 20 per second and they are coming with the same MAC address here and confusing the system each unwanted traffic is filtered out directly on the link node from the outside is first filtered for any unwanted traffic with the BPF node it works out of the box there is nothing special here okay now it's up to you we have two minutes the output of the graph is SVG we are using SVG on our monitoring system you can tag them or you can make a label on the link so you can hover over the graph and get more information but here 99 do's usually happen on an unconfigured device on the Antec VLAN but in all other cases it depends we have devices which are configured to choose three different VLANs before they are chosen on the Antec VLAN you have nodes here why do we stick to free BSD despite harsh welcome because I do not have any other option very simple we had to throw away the other systems and we are very sure that we can manage the problems in weeks but it takes months it was unpleasant time the main solution for the old situation was that we had a lot of problems with memory leak so I tweaked free BSD 8 I tweaked the memory management that free memory buffers are not freed but going into a small cyclic buffer so they are delayed freed and it solves almost all of the problems we had so there are possibilities what you are going to do if you have the source code network topology log is not a problem at the moment because we do not change topology that often so it is already answered of course it did not by the attunement of if german telecom is going to another provider and asking stream you might come up with the situation that you are saying ok we are using the regulated providing you exactly the same interconnection point as you are providing to us and the answer from german telecom is no you never do this we cannot handle this you have to provide us fixed SVLAN for each customer if you have such a situation in the other way we did not need this we could do it with a simple Cisco router because if everything is stable you can map the VLANs and tag VLANs to the service instances you are fine with this we consider this approach but the dynamics make it a mess the other point the question about why it is not layer 3 because layer 3 overlays we are not state of the art at the time where the standardization process is so the broadband forum which is mainly responsible for the internet between 1997 to 2002 in this time they had the first drafts out so we are still using the technology from 20 years ago that is an issue with standardization if you have a standardization process you will always choose the technology 20 years ago if you lost me I will upload the video it might be better so it is simply the problem of the time of standardization we have a standardization here 20 years ago and this time layer 3 overlays was not state of the art multi-threaded bridge why simply because I didn't look no, because 3p is the 13 started later than I started with this issue but it is really an interesting point to match up the code between both bridging interfaces I think it is not necessary to have two incomplete code bases which overlap major overlapping functionality each of the implementations are incomplete in other areas because they have different purposes no, we do not use layer 2 bitstream access for ourselves but we use layer 2 bitstream access for other providers like Deutsche Telecom how many can a 3PST box handle we are using 3PST for a category of grade 2 and we are able to handle without special tweaking we are able to handle about 12 megabits 12 gigabits per second the problem you have is that you have to distribute the code over the course so you have to inspect your server very carefully you have to have a look where each interface card is connected to which PCI bus you have to make sure that which PCI bus is connected to which CPU provides additional CPUs in order to get a direct link for each PCI express this lane to the CPU you have to have a look but in the normal situation where you have a processor with about 16 or 16 cores it is enough for this task you can get a gigabit per second I don't know if we have a lab at the moment for the L2BSA connection but because we are taking only gigabit lines from the interconnection simply because there are no more customers on the line so it is sufficient I think we have 16 lines it is about 8 I think that was the peak here