 Hello, everyone. My name is Kunal Gandhi, and today I'm going to talk about Adopt-In Designate for Managing Mission Critical eBay and PayPal, DNS, and this talk was actually going to be done by Bharata and Ron. They couldn't make it, so I'm going to do the talk instead of them. Most of the material on the slide they have worked on, so I've kept their name on the slides. So let's look at the agenda for today. So first of all, what we'll look at is, before designate PayPal and eBay were using OpenStack, we had integrated with OpenStack, but before that, before designate was started, we integrated OpenStack with our own proprietary DNS service. So we'll look briefly at what those features were. Then what we'll do is basically we'll do a designate overview on what components designate has. We'll look at the designate architecture. And then when we had to migrate from our proprietary DNS as a service to designate, we had to make enhancements to designate so that it meets our business needs. So we'll look at a list of those enhancements, and then when we deployed designate to production, we ran into a few challenges. We would like to share those challenges with you. And towards the end, what we'll do is we'll look at some future work that we need to work with the community in the upcoming designate releases so that designate can be this production ready for the other companies as well. So the first thing I wanted to talk about is the proprietary DNS as a service that we had at eBay and PayPal, which this was developed a few years ago. It was an in-house system written in Java. It was a RESTful API for DNS management. The way the DNS as a service was designed is it was designed to work with multiple providers. And the way we had designed DNS as a service a few years ago is the source of truth for the DNS as a service was actually the DNS server itself. It didn't store any data locally. So that was the source of truth, the DNS servers with their source of truth. Designate overview. So now let's look at how designate is and their components. So designate has various components. One of the first component designate has is the API. This component is responsible for taking all the API call request from clients and from Horizon. Designate central is the central piece as the name suggests it's actually the central piece of the system which is responsible to make changes on the DNS servers, the name servers and the database. Designate sync is actually a component that is a framework designed to add notification handlers on top so that it can listen to notification of various other components and open stack and take actions accordingly. Then there's a storage component in designate which is basically the central component talks to the storage component. It's the underlying database that's the source of truth for designate. And the back end component is basically your name server, the DNS name server that has the data. So these are the various components in designate. So let's look at an architecture overview of how designate looks like. This is the Juno release and the ISO release as well. The Kilo release a few things have changed but we will look at the Juno release and how it looks. So in this picture here if you notice the middle section has the API component. It has the sync component. It has the central component and there is a queue in the middle. That's the rapid MQ. So the API component is the component that takes all your API calls, the customer API calls. So this will handle various calls like your e-records, your reads, your writes, your deletes, any updates. So that will take all those calls and it will also take calls from horizon. The other thing, so that's the API component. So in the bottom you see there is the sync component. So that component is basically responsible for listening to all the notifications that are on the bus. So the sync component will basically have a bunch of consumers to consume all the messages from the bus. And then the right next to the sync component is the central component. The central component is responsible for consuming messages that is on the MQ and it will make changes to DNS on a DB. So a typical workflow is let's say for instance someone makes an API call. The API call which is the API server and what the API server does is all it does is basically it puts the message onto the MQ bus. So that MQ bus that you see in the middle is a dedicated MQ bus for designating. That's different from the other messaging buses. So the API call gets to the API server and the API server puts the message on the MQ bus. And then the central component is responsible to consume that message and it basically makes changes to DNS and the DB. So let's say for instance someone makes a call for a record creation. It goes to the API first. The API module will put the message onto the MQ bus. The central will consume it and it will make changes to DNS and DB. Similarly, what sync is responsible is it's responsible for listening on notifications from NOVA, from Neutron or from other components. And then it will basically put consume those messages and will create new messages and put it back on the MQ bus. And central just like it would consume messages from the API server will consume messages from the MQ bus that the sync puts onto the MQ bus as well. So that's what central does and it changes all the data in the DB as well as the DNS name server. So what I wanted to talk about is a use case for domains in eBay. So the way Designate is designed is that every tenant can have multiple domains. So a tenant can create domains and delete domains. So in case of Designate, for those who are not familiar with domains, domains is actually zones in DNS servers. So in DNS servers there are zones and here they are called domains. So any user that's part of a tenant can create domains, they can manage domains, they can delete domains. But the way we have it in eBay and PayPal is since it's a private cloud, our customers and our tenants are mainly business units. And our business, so we didn't want our business units to go ahead and create domains. So what we have is we don't allow all users to create domains, all users of a tenant cannot create domains. So what we have done is basically we have created a logical construct called VPC which stands for Virtual Private Cloud. What we have is we have a group of logical tenants that we group into one VPC. And each of this VPC has one admin tenant that can manage domains. So that way what we have is we have these multiple VPCs and each VPC has one tenant that's the admin tenant that can manage the domain. So that's how we have it at eBay and PayPal. So that's one change we had to make to Designate. And the second change that we had to make is adding handlers. So out of the box, I think Designate does not come with a, Sync does not come with a lot of handler implementation. It just comes with a framework and you sort of have to build handlers on top of that. So the first handler that we build is the NOVA handler. So the NOVA handler is basically what it does is whenever any instance is created or whenever any instance is deleted in NOVA, it actually puts a message onto the bus and the Sync notification handler will consume that message. So it will listen to instance.create.end event and instance.delete.end event. So that end actually means that, so like the way it's a standard thing where in NOVA and in other components also, whenever a flow is started, it will sort of send and it will put a notification that says eventtype.start and when it ends, it will say eventtype.end. So when we listen to end, we actually know that the event is completed. So when VM creation is completed, it says instance.create.end is a message that you see on the message bus. So the Sync notification handler will listen for that message and on deletion, it will listen for instance.delete.end. So when the Sync notification handler sees these two messages, what it does is it goes ahead and creates A and PTR records. So what it does is Sync will actually create a message for A record and create a message for PTR record and it will put it out of the bus and the central will consume it. So the notification handler, the way it's designed is when we actually, when NOVA creates the instance, it makes sure that it has these pieces on the message. So it's part of the message metadata. So we have the domain name. So we need to know the domain name on which we need to create the A record. We also need to know the host name of the VM. So we'll use the host name and we need the IP address. So we use the IP address and the host name and the domain name to create an A record and then the PTR record. The other thing that we do is we also store the instance ID in the database. So the instance ID is the ID that NOVA keeps track of what instance it is. So we keep that also in the designate database. So the reason why we keep that is when an instance is deleted, a message is put onto NOVA, puts a message onto the RabbitMQ bus saying that an instance is deleted. But at that time, all we have is the instance ID. We don't have the IP address. We don't have the host name. We don't have domain name. We don't have any of the information. So what we do is we look up the designate database to find out which instance ID is trying to get deleted and accordingly we'll look at all the records, the A record and the PTR record that maps that instance ID and accordingly we'll delete that. So that's why we keep track of the instance ID. So this is how we handle VM creation and VM deletion in designate. We create A and PTR records for that. The next thing is that we added a neutron handler. So when we say neutron handler, what we did is at eBay and PayPal, what we have is we use Elbas. So Elbas is a component that is part of Neutron and it is used to create and delete whips, pools and entities. So when someone calls the Neutron Elbas API to create a whip, what we do is Neutron actually puts a message on the bus saying that a whip has been successfully created and in that case what we do is we specify the whip address, the IP address and the whip name that the consumer wants and the whip domain name. So that's the domain name for the whip. So we put all of that in the message and what sync does is basically it consumes that message and what it will do is it will create A and PTR records for the whips and when someone tries to delete, calls a delete a whip API, what sync does is basically what Neutron does is basically it will put, similarly it will put a delete message onto the bus and then sync notify will consume that message and it will automatically delete the A and PTR record as well. That's corresponding to the whip. So that's the Neutron enhancements that we made, CNAME CREATIONS. So the default designate behavior is as we mentioned earlier is domains are owned by tenants and user under that tenants create the domains. The users and tenants, the users belonging to that tenant can only create records under that domain. So A, PTR, CNAME records can only be created by users of that tenant that have created in the domains that the tenant owns. But in our case at eBay and PayPal, what our requirement was is that specifically for CNAME, we wish to create CNAME such that a tenant A can create a CNAME for an A record that is created by tenant B in a different domain. So we had that requirement. So what we had to do is we had to add new extensions to the API where CNAME creation was allowed by any user for any domain. So that is the enhancement that we had to make. Import zones. So whenever, so as we saw in the earlier slides, our proprietary eBay, PayPal, DNS as a service used the DNS as a source of truth. It didn't have its own local database for source of truth. But the designate is a little different. What designate does is it has its own database which is used as a source of truth and it also, so it updates at two places. As we saw in the architecture diagram, it updates the DNS server, the name server, also as well as the DB. So for all the reads, what it does is it does not go to the DNS server. It just goes to its local database to pull up all the data for all the reads. So for that, for designate the source of truth is actually the DB. So what we had to do is when we had to migrate from proprietary eBay, PayPal, DNS server, DNS as a service, I'm sorry, to designate, we had to import all the data from the DNS server into the database, the designate database. So for that, what designate has is they have an API for import zone. We tried using that API, excuse me, we tried using that API, but the way the API is designed is it writes one record at a time. So the way it is designed is it reads one record from the DNS name server and it will write one to the database. It will do that one at a time. So at eBay and PayPal, we have a lot of zones and each zones have thousands and thousands of records. So the API call would take several days and it would fail a lot. So since it was the case, what we did is we developed our own tool called FastImport and what that FastImport does is it will read all the records in the domain at once. So it will go domain by domain. It will read all the records for a given domain at once. It will write all the records to the database. And what it will also do is when it reads all the records from the database, from the domain and writes to the database, what it will also do is it will go to the NOVA database and try to find the instance ID and save that and designate. And so if you looked at, we discussed about this thing a few slides ago where when designate creates, when designate listens to notifications from NOVA, it stores the instance ID so that it's easier for it to delete the corresponding NPTR records looking up the instance ID. So what we wanted to do is when we import the data from the DNS server, we also wanted to make sure that we correlate all the NPTR records that we import to the instance ID in the NOVA and save that in designate. So after the migration, what we wanted is once we have switched over completely to designate, we wanted all the delete calls to go through successfully. So that's the reason why what we did is the import tool actually did that. The other thing that import tool does is there is also a sync API, a sync domain API. So basically designate has an API where you call and you say this is the domain you want to sync. So it will sync that between the database and the name server. So what we do is doing the migration. We migrate the database. We correlate the data between the designate database and the NOVA database and then we call the sync domain API to sync the data from the database to the name server. So that way both the systems are in sync. Challenges with designate Juno. So when we migrated from, so this was the migration. So once we finished the migration and now we were on Juno, what we had is we started taking production traffic and this is we ran into a few issues. One of the issues that we ran into is the database concurrency issue. So the concurrency issue is basically the way we, the concurrency issue is when two messages when designate central has to process two messages, either it's for A or PTR, but if they are for the same domain, what would happen is that we would see database deadlocks. So this issue, we had this one issue. The second issue is the import zone designate API is very slow. We looked at it at a previous slide on why it is very slow, but this was the second issue that we found. And CNIM creations, we discussed that also. Basically CNIM creations were not supported. These were the challenges that we faced. I would like to double click on the database issue. So the next slide we will look at that. So what we did is we did a lot of load testing. What we did is we came up with a load testing tool to reproduce this database concurrency issue. So the way we did the load testing is we made API calls to designate at the same time because if you look at it, designate central is where the problem was. Designate central gets requests from the message bus, but the message bus gets it from two places. One is from the API and other is from sync. The only way to get data from the sync is to basically do NOAA boots and delete. So what we did is we had our load testing tool. Excuse me, we had our load testing tool, which made API calls as well as at the same time what it did is it issued a parallel NOAA boot and NOAA delete commands. And we wrote the tools such that could configure a number of concurrent calls we want to make at the same time. So we did that and then we found out a few things. So one thing we found out is we were able to reproduce the database concurrency issue within three threads. So when there were three messages that sync, sorry, central had to process at the same time, we ran into concurrency issues. And the root cause of the issue is there's a field called serial number on the domain table. So it's on the domain table. There's a field called serial number. And that serial number is the same number that you see on a DNS server zone. So that number gets, I think, incremented each time and you update anything on the zone or the domain. So what the designate does is it increments it in the database and it will increment it on your DNS server also. And what happens is that once, so if you have two changes for the same domain, what is going to happen is it's going to try to increment the serial number at the same time. So that causes deadlock because each time when you update, increment the serial number, it has to lock the row in the database. But if you have two devices trying to do that, you will have to, there will be a deadlock. So this is why the deadlock was caused. So the fix for the database concurrency is there are various back-end DNS servers and some back-end DNS servers use the serial number. Some DNS servers don't use the serial number. So the back-end database property server that we use does not use the serial number. So what we did is, for us, a quick workaround fix was to not update the serial number. So that is one option. So if your back-end database does not update the serial number and is not needed, you won't have to worry about it. You just have to make sure that your code does not update the increment the serial number. But certain back-end DNS servers, what they do is they require the serial number to update, bind and baud DNS. They require them to be updated. So the fix for that is actually in designate kilo. So if you upgrade to designate kilo, this problem doesn't come because they have rewritten the way they update the database. Future work. So future work is basically you want to adapt kilo and liberty. There are a few changes that, enhancements that we need to make beyond kilo and liberty. We have created blueprints for them, and we have shared it with the community on Launchpad. So one of the blueprints that we have created is IXFR support. So currently what happens is that the changes between DNS and the slaves are propagated via AXFR. So AXFR actually means that you sync the entire zone. And IXFR is incremental zone transfers. So what happens is that with the full zone transfers, if your zone is very big, it will take a really long time to sync. So let's say if your zone has thousands of records, you don't want to do a full sync each time anything changes on the zone or the domain. So with that, we need IXFR support. So we have created a blueprint for IXFR support. The other thing that we have is granular access at various levels. So right now, the granular access is at the tenant level. So basically if a tenant creates a domain, all the users of that tenant can do anything with that domain, like create A records, delete A records, change anything. But what we would like to do is basically we would like to have more granular access. We would like to have granular access at domain level itself. We want to say that only even if this tenant has these many domains, excuse me, we don't want all the users to have access to all the domains. So we want domain level access. We want record type access also. So we just want to say that certain users can create C names, but they cannot create A records and PTR records. So we want record type level access also. What also we want is we want access at record name and record value level. So let's say for instance if your record name starts with the word test, you only want to give access to certain users with that. So we have created a blueprint for that. What we have done is basically the Brooklyn has a very powerful feature where you actually have an API on Designate to specify the exact role by a regular expression. So we have that blueprint in the community. And the next thing we want is transaction support. So we would like to have APIs which basically you could create for a given domain, you could create APTRC name records at the same time, but we want transaction support. So basically what we would like to have is we would, if we could create APTRC name, all those records at the same time, multiple of those, and if any one of them fails, you just want to roll back everything. So we have a blueprint for that as well. So we have any questions? Thank you very much. Thank you very much. And I want to know about the motivation to use Designate because that is an incubated project. What is the motivation to use? Designate is actually used in production by a lot of companies actually. Can you add to that? I think if I'm not mistaken, if some folks from HP, what I heard actually they are already using in production. So of course there are some local patches that I did talk to HP team as well, but most of the fixes are in kilo. So if you are going to be in kilo, I think it will be in better up. So a lot of companies are using it really. Specifically, and most of the folks involved in Designate are from Rackspace and HP, and we also contributed a bunch of code to that. Of course it's not in upstream, you know, right? Because if you look at what Kunal explained, to make it operationalize, actually it's not a trivial task of just go and take it and then put it into production. Like any other open stack component, when we started and sometime until sx, I don't know how many people worked on sx, but it's a lot of firefighting. Still a lot of firefighting, but it's definitely better than those days. So we are still in those days. But it's not an option. Because if you start early, your migration path will be less. For example, we have unified DNS that we wrote it internally three years back when we architected. We had three different DNS infrastructure. Our goal was to bring all this DNS infrastructure into Common API so that actually you don't care whether you are using Bind or any other vendor or any other solution. We have multiple different IT organizations. They don't recognize infrastructure. But when you want to do some automation between different environments, it was nightmare for us. This tool works with this environment. That tool works with this environment. Our platform service is working on something else. So we wanted to unify all of that. So that itself took us more than two years to converge everything. Now we wanted to be on open APIs. If you are moving away from our unified DNS, then a lot of people are asking questions why they actually are moving away from whatever we wrote it. It's already stabilized. Only motivation for that. Actually we wanted to be on open API. So that actually we continue to move that path even though actually migration is going to be painful for us. We wanted to adopt early. So that we are not casting up very, very late in the game. And also some of the use cases that Kunal talked about, they are mission critical use cases. For example, we had some global live balancing and it has its own records as well. Now there are some local lips in the load balances. We want to move to global load balancing. So what we want to do is actually, we are all public internet-facing customers are using it. And if you want to move all these endpoints to a common place, and then you have to immediately delete and then add it to the public load balancing. So if you do ANPTR separately, there will be an outage. And every minute, every second counts for us in terms of revenue. So what is the feature for that? That DNS API is very critical for us. So what we want to do is identify all these mission critical APIs and put it in the community first because it's not only useful for only eBay and PayPal. It will be useful for every other company they are running mission critical applications in the cloud. So if you are running maybe a simple dot-com application, it doesn't matter for you. But if you are really, really running a mission critical application, if you are talking about 4, 9 of your application availability, when you just maintenance, these are all very, very critical for anyone to catch up in the community early. So if you call your application architecture and then what it needs everything into your APIs so that it will be easier for you to manage all those layers later. So don't think that actually, you know, incubated projects, a lot of, you know, many incubated projects are, you know, even no one is incubated at some point, right? So that's why we looked at it. Thank you. So the question is what is the usage rate that we have? So the usage rate for the control plane or the data plane? Okay, so I... So we typically, you know, see, we have, you know, of course in ABN PayPal, you know, it's around, you know, 20-year-old company, and then we have so many automations before even, you know, Wappenstack everything came to our infrastructure, right? And also, you know, we are moving everything from our existing infrastructure into the first version of our cloud into the new version. And if you look at, you know, the existing version where Wappenstack is running, the footprint might look like, you know, around 10,000 hypervisors or whatever. But, you know, the rate of change for that is around, you know, 20,000 people are creating around, you know, 200, 300, you know, VM spare, you know, I would say 10 to 15 minutes. So that many changes are happening across infrastructure. Basically, you know, if you look at, you know, all the developers, right, every day they come in, they create, you know, CI, CI, CD, and then they will be doing a bunch of tests on Hadoop and whatnot, right? They are all creating and deleting VMs and whatnot. So all these records are going through, you know, DNS APIs today. And some of them, most of them are in the preferably that we all talked about, we call it as unified DNS service. And the idea is actually identify each and every zone from that particular infrastructure into the new one. So the one challenge with this actually, you know, designate is a stateful service. And DNS service that we wrote, actually, the DNS master is the source of truth. It was very easier for us. This is completely stateless. You know, we have only, you know, few, you know, states to manage, specifically who is modifying what and then access controls and stuff like that. But designate is completely, you know, the source of truth actually. So even we have a lot of, we are going to have a lot of challenges. Basically, you know, if you are talking about, you know, your MySQL database as a source of truth for, you know, across the region, across the globe, you want to manage your DNS infrastructure. There are a lot of engineering needs to be done. So how is your replication pattern across, you know, your region, sun globe all over the world? So unless otherwise you have your Galera cluster, that is being, you know, scaled across, it's going to be very, very tough to, you know, migrate all your existing, you know, infrastructure into this. And of course we are approaching by, you know, non-musical, mission critical, you know, zones into this and then slowly migrating one by one, by taking time to, you know, identify all these gaps and put it into the community and, you know, migrate slowly whenever we are ready. Because at the end of the day, it doesn't matter whether you use unified DNS or you use designate, our business would not get impacted. That's our, you know, first goal. And we cannot, so business, they don't care what API you use in DNS, but it should not get affected. But at the same time, we want to innovate in the open source as well. So it's a, you know, it's a tough problem. Yeah, so that's the answer to the question? Yeah. Yeah. Yeah, yes, exactly. Yeah, we will be. Yeah, yeah, definitely. So one of our engineers actually worked on that and I will talk to him and then open source. We will make it available, yeah. Yeah, definitely. Yeah, so the problem with that actually, you know, not only for designate the problem in the community, you know, I talked, you know, open stack technical community also sometime back where, you know, introduce more and more gates in the release process. So I don't care if you don't deliver in a couple of features for a release, but I want to put as many test cases as possible, functional or, you know, unit test cases and performance test cases. It is very big cap in the community itself. Say if, you know, Kilo is coming out, how quality it is, right? So can I take it and run it in production? I have to lot of hard work because, you know, every upgrade cycle takes around, you know, at least, you know, one or two months of, you know, effort from every company to make it happen and then just take it and upgrade it in production. So that gap is still not at rest. Of course, you know, we volunteered to provide some infrastructure to the community itself. Of course, we will be working with CICD infrastructure, team and HP also volunteered and we'll be working on that for sure. And of course, you know, it's a journey. It's not like, you know, there's no point in complaining about community. Somebody says, oh, community, oh, that doesn't work. Okay, you are part of the community. Let's do something for it. Right? Yeah. All right. All right. Okay. Okay. Thank you. Thank you.