 Getting nervous about talking my session because I'm talking about Neutron and I should talking about SDN, right? So So I made a title a little bit catching with SDN without SDN Everybody knows it here that we need to some software to setting or define network for the VMs and The reason why I add without SDN is that I don't we don't using the OpenFlow controller in our environment. So Yeah, everybody using that. Okay, let's start it. Well, I'm coming from down Kakao again. We're Down Kakao is the result of the two major IT Company in Korea one is down and one is Kakao like bottom of this Kakao is very Very popular mobile messenger in Glover's care, but no one is using there, right? well Anyway, Kakao talk has one one hundred seventy million users right now and the down is number two Korea's searching and poor poor Companies so mergers of two those two companies resulting number one in terms of stock prices, you know Korea Yeah, anyway, yeah Most of time we when we're running out of resources like CPU and memories and disk We just add a new the new resources to our existing one like system team prepares some servers using NKS, which is our parameter provisioners and They provision server to To our team or some other develop development development System team using the trying to using the cmdb API As much as possible so we can interact with the API and after the server is ready the network teams set up the switches and Lovers and villains to working with the networks after that with our team and my team using the chef server and knife the provision the open set controllers and Open-stack the computer. It's kind of easy, right? after the Two companies merged the VM. I just showed a graph about the rate of Deleting and creating VMs The interesting is that when you start the service from 2013 the VM delete rate is really high right because at that time we didn't Buy officially for the VM service. I'm the new hardware for our open-stack service We just using the existing one so we reading up the VM We request the users and developers to delete unused VMs, please So at that time they delete the VM, you know developer sometimes just creating the VM Just check the open-stack API is working or not. So at that time the deletion is really high, but right after we Stabilized our service the VM creation rate is getting high like this So the depletion the depletion rate is also getting accelerated too Well, I changed that number into more tangible Value which is prices we call it crane because in our service the service name is crane So we using the some sense of the expenses to our developer or inside our team and The crane itself which is equivalent to the Korean one and which is equivalent to $1.1 and Right right now we are using $40,000 per month when according to the AWS is to pricing table In but it doesn't count in counter the network and this could use this at all. Anyway The expenses is getting high like this The interesting is that the black one is M1 medium, which is only have two gigs memory and first of service developer trying to make the small VMs just for the testing but after we Stabilized our service developers and users trying to create more large VM like in Descript the VM large which have four gigs memory and eight CPU cores I'm sorry Well, I think it's just related with the human nature which have which want to have some more powerful VM and Excellent isn't getting Gross is getting accelerated after two major companies merges because the number of engineers is getting increases and Then number of developers trying to make new service and new pilot services and project to Prepare for the future service and and other ideas This list resource depletion space is getting accelerated and they simply make more jobs to Managing the resource management is managing the resource like system team and network teams. So system teams should prepare and Plenty more about how they deploy or how they prepare the server to the developers Include our our teams and they took teams or so thinking about the network to getting work Like feelings and loudness and some that they have more time to planning and deploying dead things to our Impulse and and also our teams but in terms in terms of our teams working. It's not that getting high Just like the graph here, you know the network teams and system teams workload is getting orange We are just why because we only test we have already tested the recipes for the servers so whenever the in price is ready, we just Click down the knife something something something then there's something is comes up for our surfaces So in our in terms of the road, we didn't have any challenges at that time But right after we thinking about more data centers in a global levels that include that not in a only Korean like using some services China some service in a America and sometimes we're using we set up the talk pops to get the to get the Trape from the outside of I said outside of the global so when you think about the Setting up the new data center is kind of nightmare, right? So system teams and they talk teams and even our teams should interact Yeah, interest interest so deeply we're talking a lot and we made a lot of discussion and we made a lot of meetings to make those things so and This thing only not only include the physical data center physical data Physical and that's all it includes the cloud public cloud provider like AWS and Google and workspace to and nowadays the The new type of this was isolated isolation. We call it container, right? We should prepare the container itself So just those things getting complicated too well The lessons we from experience We experience from the rapid growth. It's like this one. The thing is that growth doesn't come alone Which means? Yeah, info in terms of improv the growth includes scale up and scale out at the same time Skate up can define like this. Well If you have existing servers and stories, you can just add more servers and more stories to the existing one We call it we can call it some kind of scale up in an inflatable, right? and sometimes prepare more power to the our resources and This thing is not that difficult because when we scale up the racks or scale up the servers We know when to do and we can prepare and we can plan in something to our A scale of environment so but I don't think it's easy But it's not that hard when it compared to the scale of situation like this Like set up the new data centers and set up the new availability Jones from Everywhere is really hard job So this leads radical changes of everything like The way of preparing and the way of provisioning and the way of monitoring the way we are logging and the way the way of Logging and developing itself For example, we are using Splunk for our Central logging system, but when we're thinking about the volume and we should think about the price, right? so we translate that code into the open source based one like What's that? Let's take Let's do searchy and other the Kafka things we integrate them to them into the more So it gives the way of logging and monitoring and developing system and when you think about the container itself container doesn't have the C-slope type of Logging system, right? So we need to have really big and scale scale scale out possible way of logging system and For the open stack we learned something too and the thing is that resources for the OpenStack VM is come to you just finally CPU memory stories always experiencing Shortages and the thing is that they have some skewness like This part and this this zone sometimes CPU CPU is gone first and this zone is data Storage first so To make it even the every resource it could be Candidate for the rebalancing like you can move to another job or another availability system or racks and even class VM could be moved from or Interrex and the second thing is that IP is also Resources everyone understand and know that IP is resources, right? But when you're thinking about the top in terms of Nova scheduling Right now the IP is not a resources, right? even though from killer version or the later and you turn thinking about the scheduling over the IP is but at this moment we don't have it at all so But After we experience in some Rapid growth we understand that IP is very limited in our expectations like it has limited number of IPs Right, so when you set up the subnet like slash 32 or 25 then you limited number you have limited number of IP available, right? So when we making up the resources in terms of IP you should prepare more network or you should prepare more subnet and get ways In hours, too And the thing is that the second thing is that location of IP is also restricted by the physical location of the routers and The some nest right so when you create some IP inside of some net you cannot move that IP to another chance or another rack because it using the Using different some net or different gateways so That that's Making different like managing those kind of IPs through the racks or through the availability chance making tougher for us Okay, let's go to the neutral networks Well, we've been using provide a network so in terms of that so we don't have any problem because our network team prepared for us the neutral networks right right now we are using mr. To plug in we're studying the beta service from grizzly and we upgrade to now Juno and we preparing for the grain to the kilo But changing to the MLT plug-in is not that easy because At the time they don't thinking about the abstraction of layer 2 level So they have don't have the tables for that even though You should on give us some script to update the database. Sometimes it's not working Sometimes but nowadays is kind of good really matured So you can you don't have any problem with upgrading from some versions to to another version And now we are using this bridge The reason why we move move from obvious to this bridge is not because stability and not because performance because obvious these days you don't have any Disagree about obvious is really Not good or obvious is not that obvious performance is not good because it's really getting improved day by day So these days obvious obvious is this stability and the performances Yes, it's not that bad. Actually, it's really good But the reason why we move to this bridge is that only pure the operational overhead when we restarting or Reprovising our shape code to our compute node if we start the nearest bridge agent No, it's news obvious agent. But the thing is that When start of the news obvious agent it deletes the existing one and fetch is from new Which is not actually new Data flows for the data plane from the neutral API So it makes some disk disk connection like to or Maybe three second, but most of time it doesn't matter right but It's our inside of our our company. We had a in-house mysql Hs solution we call it MHA. It's really sensitive to the network So sometimes it's that this question this connection calls stone is and that the Spirit brain symptoms so the user's database can't work work on that So we move to the nearest bridge because it doesn't delete the villains and other things on the bridge at all so right after that took that took teams plan and set up the network to be then and some they gave us we make our remapping the Availability Jones and you turn that talks to the physical and that talks which is really easy for us After we learning multiple availability Jones we experiencing resource imbalance is through the Jones Naturally at that time at the first of the service we permit the developer can select the John so If things like this if you if some developer selects John one, we just don't have any IP It gets the error and then he's like the second one second John Which don't have the CPU resources and guess I guess error two and John three Actually, you don't have any resources. Yeah, it has the Aerobases too so at the start of the service we thinking choosing the develop availability Jones to the developer is not that Bad, but after times goes by because we hear a lot of complain from the developer because he cannot prepare the VM to certain Jones So this time the filtering scheduling is one helpful because it doesn't count the IP Numbers at all. So even though we have CP CP research in a certain certain John the IP is kind of different story. So To make it To make serve this one migration of VM is kind of solution to that But the thing is that when you migrate or move for VM from like John one to From John one and to the John two then the IP should Changes, right? But the it caused a lot of complaints from the developer itself because he changes it changes its IP So Yeah, we to cure the resource imbalance. We developed some network counter filter Which is checking the remaining IP over a Jones and at that time at this time Developer cannot select their their availability Jones only is coming from the scheduler algorithm And set the John and the algorithm set the job is a more IP count Like this we Always checking the IP available and IP users and the CP account and the memory counter ready for our services Yeah We're thinking it can solve the problem but we experienced more highly issue because we said more More than two billion on the same internet device like this after start just start off the services which we just said the Some villains on that In our company we have simple rules like one internet and one villain when some once some net on death, right? it's quite simple but Right after we imposing the scheduler on this on the on our system that two teams and our and our teams to thinking about and caring about the Jones of every compute servers to is kind of okay, but when when we Operating this like more than a John is kind of Really hard job because it's not continuous it happened it sometimes happen and that was him care about the What's that the? Both the trunking and in our side we should thinking about the villains on that so it causes a lot of problem Sometimes the trunking is not set correctly so we cannot provide the network for the VM and the thing is that still the migration of VM through the John is not with The IP on change is not possible. Anyway Well, let me think it about and we need need to think about network connectivity when you set up the IP before network the AIP protocol set up the AIP table in in your Compute or in your switches like which which IP has which back address so when application go to the Same some some net servers and it using the back address directly than using the IP address But when the client to reach it reach out to the different some net it attaches back address of it and you send it to the Broadcast terminator terminator right which we call it gateway or lovers and the lovers taking care of that packet and Send it to the next stop and the next stop lovers taking that packet and taking care of that The the receive the package so we send it to the destination These days well with People using the overlay network, so it gives the applications and it gives a VMs some kind of fascination. They are on the same network. I'm the same broadcast network Even though they are physically and geographically separated. They the tunneling tunneling technology stitching can stitch in them to Give an image may imagination. We are on the same some net, but the thing is that Even though we can solve the connectivity problem. We still thinking about such some net, right? Who has the gateways who has the broadcast? So We should think about remedy version 2.0 and we need to think about those requirement. IP can move Through the rag and through the John. So when we experience the resource imbalances, it can be solved by migration without changed IP and if we If we develop or set up new system for the network, it should have false resilience like some switches or some system is gone still the network is working and And And first thing is that it should check the dynamic status of the network. So some port is gone or some being is created. It should Informed by event or IP connection It should have that and the more important thing is is that for us is the simple research IP resource planning and management Every time we need is set of the VLAN I mean the available job we set up the VLANs and set up the gateway and set up the lowest which is Not good for us because the rapid speed is really Accelerated every time So we thinking Laura is kind of really good candidate because when we using the dynamic routing protocols through the routers you can Dynamically dynamically detect the situation and changes of your Networks and now it is a really high distributed, right? There's no central controller about the net About the routing network because they can exchange their information through the Routers using the dynamic routing protocol and you have a chain like you can link the two loud at the same time So one lower is gone. The second round can handle that traffic instead of the old one and The issue is that most of time we are doing Routing information in a range is like subnet. So The thing is that when we thinking about we minimize the some net to certain level then we can move like in a someday only IP is one then we thinking about We can move that IP use the routing network. So We come to route only IP which you have 32-bit network Generally using or describe like this slash 32 or full messed Network address because of it doesn't have any subnet. You don't have to think about L2 networks anymore right No, no L2 so no gateway no subnet and no other links information needed anymore When you and when you think when you're using the dynamic routing protocol The IP itself can move anywhere because it ever when it moved it at its location by through the Dynamic routing protocol, right? So Now you can make the simple IP planning like just using some net as your ranges So and it's very atomic resources so you can keep its IP when you migrate through the John How set it up? we set up in a compute node just like we provided the other types of networks in an open stack like We installed over compute over neutron news bridge agent neutron DTCP agent in every compute node and we create Neutron this was like that we even though we define some that is not for the gateway or something like that We're just using this as just ranges to our This was this was plenty and then the DHC server No, this is DHC agent create the DHC server with some certain IP Yeah, just like that and you just create the VM and Nova create the VM and Make the tech device through that and you get steam controller to make Nettos for the VM and then the neutron API controller command to make a command to a neutron bridge agent and Neutron DHS agent like this so Neutron links bridge agent create the links bridge for the VM and Neutron DTCP agent Create the that's for the VM and that and this time We changed the neutron DTCP DTCP agent a little bit which have which only have 32 bit subnet and it gets no It gets IP to the VM And up until now it's really similar with the neutron local mode network, right? So VMs VMs inside of it. I mean that the package inside of it can Communicate through the bridges, but it cannot come out to somewhere else so to make it work we installed the newsletter on that and Make a newsletter information to the outside world if you go to the If you want to go to the this VM just make it Just go to the this gateway and We make the default gateway of this this system as a Internet one so the packet is coming through the internet one and The only problem is that when the packet is coming back It doesn't know it can receive it can arrive here But you don't have any information to go there. So we make the host route to this to this gateway and This information through the dynamic routing protocol is advocate to Upper loudness, which is double red switches by the dynamic routing protocol. So it's getting done So it gets information about the VMs routing informations Well, the face one is that I use the RIP and OSPF for the RIP RIP for the computer node and OSPF for the Top of red switches. The reason why I picked the RIP is that it's kind of it's really light process and you do know that RIP has some problem with the Tracking down the next half right like 13 or 15, but it's kind of really light processing. So to save the Computing resources in here. I first time choose the RIP for that after a time up after a little while we feeling that heterogeneous setting to the compute node and the switches is kind of vergent to everyone. So we need to simplify those Protocol to something some standard and the most important the drawback of this design is that we make Internet one as a different gateway. So in terms of installing packages in a compute node This also follow the Internet one Actually, it's not a big problem. But when you're thinking about the service network quality, it's not that good, right? And the most Significant drawback is that you can see the management IP from the VM when you're running NC Like too, so you can see the computer IPs and you can see the open computer this port network port so it's kind of possible Security home. So we divided these using the namespaces. So we did we defined the switch namespace just like the Logical and software switches spaces for the VM network. So inside of it you cannot see the Hypervisors networks at all and inside of this global namespaces You cannot see the VMs networks either. So it's set. It's completely separated and the for the simplify Simplify the dynamic the routing protocol we using the BGP for the VMs and for the table of Lexus 2 Yeah, we're thinking about phase 3 2 like setting the same ace number and same peering IP for the For that this rack. So when you prepare new or prepare new rack or new compute node like this so you cannot use you can using the same peering IP for the BGP and Same same ace number for the BGP configurations the only drawback is that It's because using it's coming from because using the same peering IPs in inside of rack You cannot ping from here to the peering IP, but if you make if you make some new kind of managing software like if you want to ping to this type of this this rack you can go into the this Tableware switches and ping. Yeah, we actually doing that and and getting more extreme. It's like when we think it about using adapt or adapting tunneling peering Tunneling algorithm to the peering process you can use the public cloud Without changing anything. So We actually testing for that Yeah, how we service that is really simple when Usually created VM here it advocate at the point that is advertised is IP to the Router one and the Router one at the part advertising is IP to the TOR and TOR gets the information something like this. This IP is coming from the RT1 and the Backbone switches get the our information too and Because of we rebalancing or have some problem with the That compute node we move if we remove the IP. I mean the VM then The routing information is updated like this. So this time This IP is coming from the TOR too, right? It's happening dynamic happened in a dynamical way. You don't have to Check every process of migrations So what is service that simple IP playing like only IP ranges matter no more freelance and no more some that no more louder playing planning at all And because IP can move free through the racks or inside of DC the research Resource imbalancing of IP itself is there's no chance like that and for to vigilant vigilance like one Servo, I mean no One network is gone the other to take the that role and you propagate propagate through the dynamic routing protocol Sorry And it has this distributed nature. So Deciding the routing pass is very Distributed through the dynamic routing protocol. So no single point of failure at all. So it's very scared out and What we still have to serve is that applying this to the physical server, too We actually this testing this kind of network environment to the hypervisor only these days but we think it about applying this to the physical servers and The router setting setting up by API is a little bit Yeah, because we are now using only shred script to update the our switch switches with trying to set up the API and Trying to using the seed vis-p. So IP routing policy can be advertised through the routing protocol itself And the third one is ACL propagation using the API like using the flow spec some some kind of a standard for that. So ACL information can be propagated through the routers without any human intervention And the last thing is that even though we can move IP everywhere inside of the rack, we can still have the problem with the VM image itself. We didn't actually using the shared storage right now. So we we're planning set up the sort Shared storage for that Yeah, this is I'm a little bit so fast because I prepared so much. I'm sorry. So here's here's Q&A. We have five minutes Yeah, if you want to have if you if you have some questions using the mic over there, but the is for the recording. Yeah, I Can hear you clearly Yeah Do you do you have multiple tap or top of the rack switches or you only have one? For the opposite we have only one. Yeah, so you do have some level of Single-point of failures that you're kind of counting on. Yeah kind of you need to account for okay. Got it Also, we're thinking about using the multiple to our switches for that Yeah, and how large does your routing table get? I mean do you talk in The routing tables because you're talking a lot of routes to be updated Loads to be mapped because each IP is a route for you practically Yeah, how big is your routing tables gonna get get? You're asking me about the routing table size, right? Yeah on the on the switches and other out routers About the world because right now actually the network team is taking over the road itself Only thing I can mention about it IP routing table itself. Well, we think it about more than much how big is that more than 10 10 and a 1000 routing tables on the TOR switches and maybe the back one switches can handle up to the 100,000 routing tables Yeah, so it doesn't matter to us actually Hey, so related question there you must mark in your phase three that you're going to use using IBG P As opposed to the OSPF that you already had Have you found how do you found the IBG P handling equal cost multipath that OSPF handles much better Particularly with multiple 10 gig links The thing is that I actually Was that so just using OSPF instead of BGP? But the thing is that they talk things using BGP for that right now. So Actually, we didn't care up. I didn't care about the routing right now I'm already running a similar setup with about 500 500 routes, but entirely via OSPF And we stick explicitly stuck with OSPF because we successfully got equal cost multipath routing with multiple 10 10 gig routes between a pair of top of rack switches And we found that performance necessary and we're concerned that if we switch that to IBGP that we're going to lose that performance Really, maybe we should inform that information to our network team to you. Thank you for the information And yeah, so I saw that you use flow spec Right. Yeah. Yeah. Can you elaborate a little bit more? How do you use it? Why I? Just pick up the flow spec as an example for the propagating ACI We actually thinking about writing some code to the very to the variation of switches But when we using the when you using the juniper switches for that then flow space kind of best selection But for for us like we using heterogeneous Switches for inside our infrastructure. It's not a good solutions. Yeah Thank you All questions done or yeah, maybe maybe last one. I have quick two questions. So what network gears do you use? Network gears which vendor do you use for the top to your top of rack switches? We're using Maybe Cisco Yeah, I cannot remember why I said maybe maybe we're using Cisco Okay, that's fine. And the next person is you mentioned you use plunk and you're planning to change to some in-house tool Yeah, I didn't understand. I didn't get the name of the tools. You're using interest plunk. It's very Generic tools elastic search last okay, and There's plunk tool today works with your open stack setup because plunk originally was not meant for that But does it work with your STN and this kind of L3 based? Network that you have does it work with that? No, it's just related with the working itself. That's not with the STN itself And one more thing is that this is our performance graph through the our 32-bit networks it gives more than it using 10 gig Interface and it gives more than 7.5 giga VPS. Yeah, so it's not that bad, right? Yeah, when we're using the OSPF Okay, thank you