 Hello. Hello. Hello. Very good. Good. Does anyone else have the issue of trying to make slack shut up? You could just close it. I know that's not a thing we do here, but you could just close it. That's a terrifying idea. On occasion, like you can just be in a meeting. I don't know to tell you. I just believe, just believe. Greetings then. It never occurred to me that I could actually have the power to close it. Yeah, I have the worst problem when something goes chirp at me, I don't really, I don't know which one are there like six of them and half of the trips on the same. How do you live like this? Monsters all of you. Monsters are very good. We're a couple of minutes after the. Welcome everyone to the CNCF sig network. By monthly call. Today is Thursday, June 18th. We've got a singular item on the agenda. Very excited to be talking about this project today. A couple of points of order, if we may. The meeting minutes link is in the chat. If you have those up, or if you don't, please. If you're on the call, please go in there and list your name down. It's good to record attendance. Today's topic is a project review. Of BFE. And it looks like we have members of that open source project on the call today, quite clearly. How many of those do we have? Or who all is. Who all is from the project today? We've got at least, at least three of you. Miles. Miles. I'm from Baidu. Very good. Miles. And then is it Thai? Or tea. Hi, this is tea. This is tea here. Hello. Nice to meet you. I'm a fan of the way that you annunciate your, your first name. It sort of reminds me of, of mine. Yeah. All right, fair enough, enough of the horrible jokes that I have. I think everyone is drooling to get more familiar with, with BFE. And you have, we have the hour today. And so please, if you would guys, you know, to take us through, please introduce us to BFE. Okay. Can I, can I start now? Please. Yeah. Okay. Okay. I will make a brief introduction to BFE. Okay. BFE is an open source layer seven load balancer. And the BF stands for beyond the front end. It is retained in Golan. So it is more stable and easy to add new features. BFE was born in 2014 and released as an open source in 19, 19, 2019. And under Apache lessons 2.0. And BFE was evolving and improving in a very large scale environment. And the BFE serves more than 1,000 billion RTP requests per day in Baidu now. And the BFE now have more than 2, more than 3,400 GitHub stars now. And you can see that it is applied in many companies and organizations. That's like the city TV. It is the most famous TV station in China. And the lead to very famous commercial banks in China. And this is a very large ally company based in Sichuan province. And this is a very leading interest company. And this is a leading internet financial company. Both of them use BFE now. And BFE has some features and advantages. First is plug-in architecture. And then it's very easy to add new plug-in. And now more than 20 billion plug-in is ready for traffic management, security, observability, and architecture. And another is scalable. It can be horizontal scalable. And it is part of multi-pendency. And it can run on multiple platforms. It is also very easy to integrate with other projects. For example, Kubernetes, Prometheus, and others. BFE can support many protocols such as APS, speedy, R2P2 web sockets, and other protocol. And we plan to support GRPC and AP3 in the future. And BFE supports very advanced routing and balancing. We support specific language. We call condition expression to describe routing routes. BFE can also support global load balancer. And the ability of BFE is very, very powerful. It supports building metrics and logs and other ways to offer the ability. Here we show a brief example of how BFE forwarded traffic. For example, a request to demo.baidu.com. The first step is to determine the tenant. So by demo.baidu.com, BFE can determine if this traffic belongs to the tenant demo. The first step to determine the cluster. A cluster means a service from demo.baidu.com and slash static from this table. For each tenant, there is a forwarding table. There are forwarding routes. Every forwarding routes have two parts. Another is the destination cluster. So by this rule, the traffic is determined to send to S1, one of the cluster. And the sub-3, it should select a sub-cluster. In the three available zones, for S1, there are three sub-clusters. So in the step three, the traffic is determined to send to S1, three in the available zone, three. And the last step, select the instance, S1, three, one in this sub-cluster. This is how BFE for traffic. Let me introduce a more about a conditioning pressure. There are two. Quick question if I could. Before I, I will no doubt forget this question before we, you get to the end. Back on your other slide briefly is multi-tenancy defined by sub-domains. Is that kind of the unit of tenancy? Pardon? The way in which BFE supports multi-tenancy. Okay. We can determine the tenant by the domain name, all the virtual IP, all the other characteristics. Okay, got it. Some other characteristic of the... For example, the path, the flow URL path. In Baidu, we mainly use the domain name or the VIP. That is a virtual IP address. Thank you. Okay, that makes sense. Okay. So let me continue. There are two major concepts in conditioning pressure. One is the condition primitive. That is a built-in function for checking specific condition. This is an example. This primitive is an IEQ hosted in www.BFENetworks.com. Now we... BFE supports more than 40 condition primitives. I think it's very easy to add new condition primitive. Another concept is condition expression. It is a serious condition primitive with operators. This is an example. There are two primitives. One is IEQ hosted in www.BFENetworks.com. And another is IEQ method ingot. They are connected with the end. The advantage of a condition impression compared with regular expression, there are two advantages. One is it is very easy to understand. So I believe that you all can understand the two. Another is there is no risk for severe performance degradation, which is a very senior problem for regular expression. And in BFE, BFE supports the mobility in three ways. One is BFE provides various logs for troubleshooting and data analysis. For example, server logs, SS logs, TRS key logs. The second way is detailed building metrics for all subsystems. BFE has embedded web server for monitoring and reload. And BFE stores thousands of metrics for external observation. And it is very easy to add new metrics. And BFE is very easy to work with many monitoring systems, for example, primitives. And BFE also provides support for distributed tracing. BFE now can work well with Django, the machine elastic. Here is a demo for how BFE provides the building metrics. When BFE starts, you can assess a point, a monitor point. And this page is shown. There are two entries. One is monitor, another is reload. In this path, monitor, trace, process, state, many internal metrics can show here. For example, this is how many clients request active the number. Now it is zero, but in the running system, it's not zero. And with this parameter, format equal primitives, this format can change to suitable for primitives. Then the BFE platform is a very little complex with more than 10 components. And BFE, the forwarding engine, it is now open source. We plan to open source more components in the future. For example, this is a crypto service. This is a cash service. This is a program for processing logs. And here is a console for BFE. For example, API server and aggregator for data analysis. These components will be open sourced later. Here is how BFE works with Kubernetes. There are two scenarios. One is this for how to work in multi-IDC scenario. For example, there are two ADs, and BFE can load balance between more than one Kubernetes clusters. So here BFE works as a global load balancer. In the right, here is how BFE works as an ingressed proxy inside a Kubernetes cluster. BFE can forward traffic to the APPs in one Kubernetes cluster. Here we make a brief comparison between BFE and other similar systems. For example, traffic, engines, envoy. I assure you are very familiar with these systems. For the first system, the similarity is that they all support HTTPS and EP2 and support instance level load balancing. And all of them support pluggable architecture. And here is the difference. For example, for Hashtag, BFE supports passive Hashtag. That is, BFE do not send the Hashtag actively. But if the traffic forward to the real server, BFE detects some failure, then BFE starts the Hashtag process. And for traffic, it supports active Hashtag and engines support passive and envoy supports active passive and it is hybrid mode. And another is BFE supports cluster level load balancer. I show just here because it's a cluster level load balancer. And engines now don't have such a feature. And another is the following rules. Engine supports regular expression and BFE supports conditional express, just as I just introduced before. And both traffic and envoy support based on request content. And because both BFE and traffic is returning go long. So the cost for add a new feature is very low. For engines and envoy, they are returning C or C++. So the cost for add a new feature is high. Also, because of the use go language, both BFE and the traffic are related to some exception. For example, it is very easy to do management and go language provide recovery mechanism. But for engines and the void, we can do nothing with the wrong memory usage. And the debug such a bug is very time consuming. And for all of the ability, BFE provides rich internal status, just as envoy. But traffic and engines provide less internal status. And another is for support for configuration how to reload. All these three BFE traffic and envoy all support how to reload. But the engines, if do how to reload, the process will be started and active connections will be terminated. And the BFE open source is a very open community. We now have more than 40 contributors from various companies and organizations. And the five maintainers from Baidu, Kuaishou, and Baidans. And from July 2019, 10 versions have been released. So BFE have now rich features. For example, integrated with the mainstream layer for load balancer. And in here have support to integrated with the Kubernetes, Prometheus, Jager, Fenty, etc. And we added many layer seven plugins for traffic management, security, availability, etc. And here explain why we want contribute BFE to CNCF. BFE is built upon 80 years experience of running production load at Baidu. And we believe that BFE plus CNCF can enable the adoption of Kubernetes and the ecosystem in the enterprise environment. Okay, here's the last page. Now this website is ready for BFE at github, slag.com, and Twitter. Okay, this is all the material. So how about your question? This is fantastic, Miles. Before some of the questions start rolling in, I have to say that just the level of commitment that those of you who are on the call and it's what the laughter 2am coming up on 230am, is that right? Yeah, right now, yeah. Well, that speaks volumes right there. Very good. So great, great presentation, Miles. Thank you for this. I have a few questions, but, but rather I would open it up to the folks that are here on the phone with us. Questions for Miles, questions for the team. I have a couple with Nikolai. One would be, and actually I'm coming from my developer background here, so probably a little bit kind of slightly deeper dive. So when you say plugins, I know that goal is in general has some problems with plugins. Are these static plugins or dynamic plugins? Like, is it something that you plug in at compilation time, let's say? BF can support both static plugins and dynamic plugins. It is supported by Go language. Okay. Here we considered WebAssembly as a way to extend. I know that it's slightly problematic with Go language, but effectively, you know, this is a trend that's being, you know, very, very trendy. Do you mean WebAssembly? Yes. Okay. Currently BF don't support WebAssembly now. But if there are some requirements, I think we can support it later. Yeah, I mean, it's not a requirement, it's just, have you considered this as an option to write plugins? Because of course, Go is easier than C, for example, as you said in your presentation. But still, there are people that probably would prefer some other means to extend their proxies. Can you open the slide with the example of your language, like of your DSL, domain-specific language? Pardon? The slide where you were showing the requests, like the matching. Like the domain-specific language, DSL. Yes. Okay. Okay. Okay. Yes. Yes. Yeah. The conditional expression. Okay. Was this inspired by some existing standards or like industry, you know, even informal standards or something like this? Or this is completely, you know, invented for BFE? Very obviously, in fact, we designed it by ourselves. But later we find it very similar to I-Rus in F5 networks. Do you know F5 networks? Yeah. Yeah. So I-Rus is more complex and powerful than conditional expression. Conditional expression is easier to learn and to maintain, in my opinion. Well, I mean, at least for me, I mean trying to invent the thing from scratch is always harder because you cannot get it right at the first time. But okay. The project obviously is old enough. I might consider that like the learning curve is too high, right? The learning curve for learn how to conflict the BFE is a little bit higher than those other like other tools with DSL or something like that, right? Yeah. Do we have plans to like have plans to have a DSL later for BFE? The DSL support. DSL support? Yes. He asked whether we will support DSL or not later. What is a DSL? Sorry, I don't know. The domain specific language. Okay. Okay. It's made the conditional expression. Yeah. So can you please go to the slide with the architecture where you are showing the API server and- Architecture. Yeah, like the big picture with all the components for web UI and- Okay. Okay. The whole architecture. Find the slides. Yeah, but like my question was what is the protocol that is spoken between the control plane and the data plane? Yes. So these are all here between the BFE server and BFE API server. Is it like some- PBRPC? Oh, okay. Okay. It's just RPC. Okay. I'm sure because you should compare some with others. I'm sure that you're aware, like, you know, the way that envoy configuration started and this kind of evolved into this universal DPA, like the UDPA. At least effort for standardization. I mean, at least from my point of view, it would be great if the industry starts to kind of converge towards something more uniform, something that can be used across different implementations. And that's my probably my only remark here. I don't know how hard it is. It would be for you to be able to support something based on XDS. But I guess it would be something useful, I believe. Okay. And my final question here is, of course, coming from the service mesh world. I haven't seen any mention of some of the popular service measures out there. Have you had chance to try BFE with like being integrated with the service mesh of any kind? Oh, BFE, don't support the service mesh directly. I think envoy can do better work in this area. So BFE mainly work as ingress proxy under here as a traditional load balancer here. So BFE kind of will work well with envoy. For example here, envoy can serve as a service mesh proxy here. Okay. Okay, good. That's that's all for me. Thank you, Nikolai. Questions from others. Very good. I have a couple. One is maybe to follow up in part. Nikolai knocked out a couple of mine, I think, but the going back to the concept of plugins and your future facing support for a protocol like GRPC. Could you speak a little bit, I think you were in part speaking to the process by which those plugins would be written in go and my question is kind of reiterate I think a re ask part of what Nikolai was asking that is how are how separable our protocol is protocol support and kind of protocol filtering in terms of the code base for builds of BFE is there just sort of a single distribution of BFE that has all of the filters all of the plugins built in or what does it look like for someone to create GRPC support. Okay, do you mean how to add a new plugin? Yeah. Okay, let me show the GitHub page. So for for a module here's a there's a list of all plugins in BFE modules. This is path is GitHub by do BFE. I think all you can assess this webpage. So here, here are a list of all the modules here and for for module here's a plugin to support a set log. And here is the main entrance for model assess. I can we can here you can you can define a a stretch module assess and here it initialize it and you can make some such as this initialize the the module and here the the main function add the filter to to the plugin the the callback point add the filter for example to finish we can finish we have about about more than 10 callback points for add the callback functions for for the plugins to add functions. Nice. Nice. Very good. Okay. That does help and actually that last bit of what you said also also helps the dynamically reloadable configuration that BFE supports. Yeah. I think that that could be a delicate process. Could you describe that a little bit more about how to hot reload, you know, a new config is that is that just the configuration is that a displacement of the actively running process as well. Okay. It is a feature for goal goal language. It is a process a model. So so when a new config is reloaded, the process must be restarted for BFE. It applied utilize go routine. The programming program model is a thread model, multi thread model. So, next question. Thank you so much, Miles. I'm cutting. I'm interrupting just to ask a bunch, ask a bunch more questions. You're doing a great job of answering. So thanks for that. To in order to upgrade BFE itself, you know, the go binary itself. What is just very briefly what is that? What does that process look like in terms of ensuring that, you know, active traffic that that particular deployment is serving. Is like, is that a, is that a concern that BFE addresses directly, or is that a concern that that you would utilize other infrastructure to You can see this, this picture here. Over BFE, there's a layer for load balancer and the BFE server as a real server for load for load balancer. So traffic come from here layer for load balancer then forward to BFE, then BFE forward to traffic to the back end. Okay, fair enough. I think that answers my question that like, had the process of upgrading from one BFE server version to the next. Is it concerned that you would deal with probably at that layer for load balancer or that you would deal with sort of outside of BFE server. So traffic is first forward to send to layer for load balancer, then send to BFE. BFE server as a layer 7 load balancer. So, so I think this question will be. So if if BFE server itself do the do the loading update, like, like to do the loading update and then how the sessions, the BFE session does those sessions which goes through the BFE server will not be killed or will how how can you handle those like existing like connections, I think, is this what was asked me. By the way, by the way, just just for content, these questions are in part just me and man hopefully the rest of the folks on the call, just poking around that there is no right answer wrong answer rather for much of the things that I'm asking so. As a matter of fact, let me, yeah. I think I think I get the gist of kind of the BFE's capabilities in this regard, let me ask a different one about the global load balancing. And, and I think this maybe in part is a question around load balancing algorithms in general that BFE supports. Is there a smattering of load balancing algorithms that BFE supports, and then in context of the global load balancing capability. Is that location aware. Would you say about the global load balancer means BFE can forward the traffic across the boundary of a very long. And for some for some load balancer program that they can only send the traffic to the to the to the instance level, but the BFE can support the cluster level forwarding. Nice so BFE is cognizant of an availability zone, like it's it's aware of its location in that regard, or the location of the real servers that it's forwarding to. Yeah. In fact, BFE don't care about the location, but BFE can see there are, for example, three subcluster for this service. So, so, yeah. People can set weight here, weight one, weight two, weight three to distribute the traffic. Also BFE have have a automatic program to calculate the weight, but we don't have now didn't open source this component. Got it. I guess you have the algorithm of the load balancing algorithms that are kind of under discussion today that are up for consideration as part of this proposal. What are those load balance balancing algorithms. Load balancer algorithm. For the instance level load balancer we support run robin. WR or others for example, at least the connection. It's actually a scaling algorithm. Okay. Good deal. And just as a point of clarification, sort of like for the global load balancing capability. You know what, maybe I won't. Good. Thank you, Miles. When I think about it long enough, I think you answered my questions and that that's that's good. You can find some detailed descriptions of this feature in the hyper document at GitHub. Yeah, we will we will follow up all those questions with with detailed answers after this meeting and send send it to the group. Yeah. I'm going to let some others on the call interrupt me as I prod with a couple of other things. Just to confirm, I'll make a statement. BFE is most commonly deployed as an ingress controller. Is that accurate. No, no. Currently BFE serve as a tradition. Load balancer, but we are now adding support to make BFE serve as ingress proxy. In fact, it is, it is supporting now but we haven't open source it now. I think we can open source this feature in one quarter. Gotcha. Yeah, I probably shouldn't have tossed the word controller in there but of the so this. Okay. Yeah, okay. Yeah, that's a good follow up question kind of the compatibility with the sort of the how Kubernetes native or how deep does kind of BFE go in that regard I guess is a good sort of follow up to characterize that. The project has some grower. A healthy set of adoption. There's just some great statistics you guys are putting up about the project itself. Pretty cool. Any performance analysis that's been done on the BFE. Yeah, yeah, yeah. One of the weakness of BFE is the profanity is little less index, because there are there are two reasons. One is for go language. And not as a reason for go language but for index, the memory usage is very well optimized, but in BFE we use a standard network protocol stack from Google. So I think Google didn't do very, very, very deep optimized for the network stack. Another reason is for go language. We can't control the very underlying the real threat, but for index can use such a feature set as a CPU affinity. But we can't do open the mainstream about CPU affinity. So I think these two are the major reason for BFE counter to do not so well as index. But the problems about one to two. Yeah, BFE performance is about half or about 70% about two index. So I think it is enough for most scenario. Clearly it has been for the adopters that you guys have today. So yeah, some deployments are much more performance sensitive while many others aren't. In fact, in most scenarios, most people only use, for example, 10 to 20 machines to run such a program. So this overhead to use more machine is not a big problem, but the cost for people is a bigger problem. For here in China in Beijing, it is very difficult to look for people who knows index well, but it's very easier to find people who know how to write Golan program. So people cost is a very, very important consideration to use go language in this area. Thanks for covering part of the governance of the project that's helpful to see the, the number of maintainers and kind of the affiliation of those maintainers. You guys have a slack as well that you invite people into do you'd spoken a bit to, I think it might have been the second to last or the last slide about kind of the in context the donation to the CNCF. I think you, you might have spoken to a bit of the motivation. I'm here maybe I'll ask it again which is how do you anticipate that BFE will benefit from being within the CNCF and maybe and maybe sort of how do you anticipate that the CNCF will benefit from BFE being in the CNCF. Yeah. I'll be a house since I will benefit from BFE. It's mutually mutually. Oh, mutually. Okay. Okay. So, so I think ingress ingress process is very important for communities, right. So BFE provide another choice to, to, to, to implement the ingress proxy. For example, in back to here, some, some guys come to, to talk with me. So the index as a ingress proxy, but it is hard to, to, to make new features. So can you provide BFE for such a support. This is a very common scenario. And for BFE, I think the, the cognitive scenario is a very good, very, very large trend in the future. And BFE must, must follow this trend. So BFE must provide both support to communities and the cognitive. So this is the major reason why BFE come here to talk with all you guys. Nice. Very good. I'm going to be quiet and ask if others have questions. I have a quick one regarding goal length. So if you get to start BFE today, would you choose some other language like Rust, for example? Part of the language. Like, like, if you, if you write, rewrite like go BFE right now, where you choose like languages like Rust or something like that. Oh, okay. Rust is another very good, very good language. But one, one difficult is about the protocol stack. You know, the protocol stack is very complex. So very honestly, the protocol stack in BFE is from Google. So if Rust can have a very good protocol stack, we can consider rewrite BFE in Rust. But now, now in my opinion, the, the protocol stack is not very, very ready now for Rust. So, so Google is a very major source for new network protocol. So I believe that Google can support all advanced protocol very quickly in goal language. In fact, I have a personal community with some guys in goal team in Google. So, so, so, so, so, so this is a major reason why they choose goal language as the language of BFE. Thank you. Thanks. Other questions. One thing that I that I need to go back and review and, and maybe based on some of the questions miles and then tea that you guys have fielded today. I need to go back and look at the proposal and just ensure that there's clarity around which, which of the sort of BFE family of components are included in the proposal and sort of which ones aren't just which is probably already in there and pretty clearly spelled out I suspect. This page. Oh, sure. Yeah. And today, today we're really just like in terms of the what's proposed to be donated. Which part is proposed to be the CNCF like sandbox project or which one is not open source right now and whether they will be open source later or it will be like close source for commercial purpose. So can you can you point it can you point them out. Oh, okay. This component that is the for the engine of BFE is now open source. And we plan to open source more more component later. For example, here is a reader for log and here's the cash service and here the API server. We all plan to to open source later but we will close. We will not open source some key features for example some some some advanced plug in or other other supportive for very large, large system for large, very large scenario. Thanks for that helps clarify. Thanks for that. Other questions. Get him in quick before tea and miles fall asleep. Nice. Okay. Guys, thanks so much for staying up late for presenting this. I really enjoyed just hammering you guys with a bunch of questions it was. One was on, on my side of the table for the questions that Nikolai had some great questions tossed in there and one of them that was around you know web assembly kind of you know wasm and, and kind of an interest thought provoking question around the, you know, if you could do it all over what what architectural choices or technology choices might you make differently and, and just kind of in context of that. EBPF is a sort of another technology that comes to mind as a. This is mostly just as a as an interesting one relevant to the problems that BFE is solving itself. I guess. Guys, thanks very much. Anything from anyone else on the call otherwise we'll can, we'll end. Thank you much. Gentlemen will reach out with a few other questions will get you try to get you through the process promptly. Amy will make sure of that. Other than that, thanks so much for staying up we'll see everyone in a couple of weeks. Bye all. Thank you. Thank you.