 So, hello everybody. So I'm Baptiste Jonglet, and I work for Inria, which is a public research institute in France and Here I'm a colleague J.O. and Marie who also work for Inria in our research team So we are part of a team that is called stack and we do research on edge computing in general including OpenStack and other projects So we're going to present our project Keops which is which is a proposal for to to make application work on edge infrastructure So just a warning. This is research. It's a prototype. Don't expect a complete solution that you can deploy We are not we're not being a product or whatever So a bit of context So you all know that the edge is taking off this infrastructure is being deployed and so on but Existing applications, they are they may not run that well on this kind of infrastructure because in highly geo-distributed infrastructure, you have high latency between the different components Application may not expect that and you can have disconnection because the network may not be reliable So these are new problems because you mostly don't have this kind of problem in a data center so There's a new challenge for applications to overcome with these new These new challenges. Yeah, so you could modify the applications So that they would run on such a new infrastructure, but it's quite I mean it's specific to each application and it's quite intrusive. So the approach we want to The approach we take is to try to be generic and not require modification of the application. So just take the application and Do something and then it will hopefully run in an edge environment so To define the problem a bit more in details, let's take an example so we are looking at Service-based microservice-based applications. So just to illustrate we take OpenStack, but you could also do this for Kubernetes or Shalitech or whatever microservice application. So here we have Nova That will talk to Glance because a user wants to boot a VM and you have to fetch the image from Glance And We have two users which are very far away from each other. So this is not very good for locality So of course you have an edge infrastructure. So you will just Replicate your service. Okay, you put two instances of the same service on two different sites and I mean as a basic idea of this works each user will use a local service. So you have low latency It's robust to network partitioning. You can have network voltages. Each instance is independent so This is a basic idea, but of course in some cases you want to collaborate between Between several instances of the service So in this example Again, it's very simple as a user on the right may want to access an image that is stored on the left so Inside the microservices you would have communication between the different instances Okay, so this is one example of collaboration and Marie and Joe will go into more details into what we mean by collaboration So of course there has been some work being done on this this kind of architecture To be able to collaborate So this is just an overview. So on the left you have an approach that involves Service to service collaboration So an example in the open-stack world will be keystone to keystone But this worker specific code in each service, so this is what we want to avoid Then you could have broker based collaboration So in this case you have broker that intermediates between the different services but again, the broker needs to be Specific to each service where you want to to connect to each other And then finally, this is something that we presented at previous open-stack summits You could collaborate through the database So you could either use a distributed database or just replicate database so this this is good because you don't have to change the application is completely transparent but Actually, there are issues because What makes sense in one context for instance on the left if you have a we create a resource it may met It may not make any sense on the right And you could have conflicts like two resources that have the same name But they mean something different in the two sites Like if you talk about an image of file or whatever, it's it may not be consistent between the two sites So we want to show a new approach to solve this problem and as the principles as I said is that You It's what we call local first So we really want instances to be as autonomous as possible really to ensure robustness, but then on demand you can you could act you can add collaboration between the services So it should be generic should work with a awfully any application and Don't don't add your card just keep it as it is So I believe that for tomorrow Thanks, so the As with the principle that Baptiste show you We can so this is basically the same side as before but we abstracted The services because we want the approach to be generic and so work with any kind of services So we can imagine that service a it would be Nova and service be would be glance It was just it was just as before. It's really the same instantiated everywhere and The users will use the the closest site and this this Particular disposition ensure the local first-person principle that that's told you about which Is to always be able to serve local requests at at the minimum Because if there is a partition between the two sites you really need the request to be able to to to to be done executed and this also allows to minimize the communication between DC because you will only talk to your local data center You have an example of a request so like open stack Server create but here it would be service a and with a Sub resource from another site So that is also told you that in order to to leverage the full infrastructure We will probably have to do collaboration between sites So we don't have to have all the resources on every site So for example the first one we envisioned was sharing Which is collaboration where Sub resource is located on another site and you will just get it from this other site the second Collaboration is replication where the user will create replicas of the same resource on different sites and The third one is cross in which resource Spans across different involved sites where you want to where you want your resource to be So as I said sharing is when you have a resource that is on another site and he is using site one and They will need to get a resource from site two. So how do How do they do that is with this kind of request where you can specify a scope of the request and specify Which service will actually execute this request here service a will be on site one service B will be on site two So this provides flexibility For your resources because you don't have to have all the resources on all your site You can have someone one and someone on the other and get them anyway. It's not a problem so this is the main the main Collaborations which help also the other collaboration The second one is application where you can replicate Resource on several sites with just another request that says on which site you want it and We'll do with Sorry, I forgot to talk about that So on the picture you can see the full framework Abstracted jobs is our service mesh. So these Circles represent the interceptions of every message is that comes from that come comes through and it will get Them to care up and if curbs needs to work on something like transfer the site as a request to another site It will do that and it will also store information about the replica so we can and find them later So replication is well for replication you You have two replicas of your resource or basically it's for business first But it also helps to lower latency when you access the resource because you will get the closest to you And it also increase robustness in case of partition you will still be able to get one of the resource The replication as we intended follows an eventual consistency For every replica and it's maintained only through the API because we only use the application API So if you want to do something Finer and it's not in the API of the application you can do it So Joe will talk you about cross. Thank you, Mary So cross is the third Collaboration technique which we are envisioning in our solution. So it creates a Illusion of a single service resource across the involved sites So I'm not sure if you're familiar with a single system image project which happened some years ago So in that we'll be connecting to the multiple Senders and then we'll be spanning a single OS and applications will be deployed on top of that So something similar on the networking and the service rate on the service resources Is similar to what we are proposing with cross so the service or the resource is made available across the geo distributed sites without Creating independent resources at each and every point each and every insights So this particular instance is made available throughout the particular areas particular sites This particular resource at the same time the cross Will be able to manage the resource availability The network partition the local first approach of all the things which we just Baptist toes told about previously Which we are originally envisioned in the particular solution and the cross is based primarily based on two different Principles which is basically aggregation and disability So we need to be able to divide the particular resources into multiple sites such that in later on these resources will act together and then Combine together and give the illusion at each and every involved site that they are from a single pain like a single site Not in a distributed sites So for this the initial proposal was made by Samyantho and all from the stack team And he did some efforts in open stack for this particular project the initial proposal So now I'll be giving it back to Mary So As you saw previously About the request we have a specific domain domain specific language DSL that is irrespective to any platform It can work with both open-stack communities. You just specify the scope And each expression contains the Collaboration operator. So here it will be the comma for sharing and service information Which service will be used location information for this service? So as you can see this is sharing with a comma you mention You want to use service a on site one service beyond site two and here you just will do a replica on VM or deployment on Berlin or Paris and Paris. Sorry. It's not the same And by the way talking about our we also have other types of collaboration that could be needed So you pretty much seen already sharing Replication here as the end person I think it's so you can say sorry say you will be on this site and this site and this site and Cross for cross you have a percentage to to mention every site that would be involved And then you say which is the primary site where your primary resource will be Finally we have the OR operator Which is a basic or And it's just mentioned that you want your resource to be created on site one or if it's not available on Any other site that you list and from this operator we can do the around operator That allows you to specify through so caps sends a bit regularly to each other and So they know at which Latin see they are far apart And so you can specify I want to use site one or if it's not available I want to use another site that is at at Maximum 50 second minute 50 milliseconds of latency so the approach was First made at the first prototype we made was during the open-stack summit in Berlin right here in 2018 At the hackathon and so the team managed to make Only sharing for open-stack. It made sharing work. Actually, there has been some modification in the code for that But not that much and it was presented in the summit in Denver also So if you want I think the videos are available and this Allowed us to pave the way towards care playing and towards everything that was done with Cops And This approach already uses used haproxy to intercept every request and you a code to interpret intercept the code and Okay, so so far Maria Joe presented the basic building blocks which has a different type of collaboration that you may want to use in your To deploy your application and it can be specified to this scope long This couple abstraction But when When trying this on real applications we notice that It can be a bit more complicated than this because the application The services and and the resources especially are not just black box. You can have some Some dependencies some state and so on so We present ways to deal with those dependencies Thank You Baptist So as Baptist mentioned resources not just a black box so an application consists of like a vast heterogeneous microservices and These microservices rely on resources such as a VM a pod service network blah blah blah etc So all these things like how will we it will perform like how can we have or like Like maintain still the dependencies which are having between two resources even if we make this particular resource into a geo distributed scenario so it involves handling dependencies between the microservices and Since these microservices are distributed Dependencies from multiple platforms need to be handled. So what we are observe we are what we have observed is that a generic pattern Across multiple frameworks like OpenStack Kubernetes to solve these dependencies. Can it be achievable? Then we have observed primarily three patterns as of now Which are across the platform and the which can take care of the dependencies between two particular resources the first one is Requires so a requires is a scenario wherein like a VM requires an image to be created So in this particular scenario at some particular point of time especially during the creation or at some particular point of time There requires a dependencies which needs to have a certain other resource to be there in order for that resource to work but it does not require a Attachment of this particular two resources at all point of time The second one is reliance where like It's this particular resource need to have the relationship between Resource a and resource be at all point of time. So for example, if we need a particular in case cubanets Cubanets Pod and a secret relation. So it might require it must be available all the time So in that particular case if it's broken it can affect the part at a particular point of time So this particular reliance means that it requires a strong dependencies at any point of time Then the third one is a composed relation, which is like kind of a hierarchical Relationship where in like there will be a certain abstract layer on top which will control the resources which are there down for example deployment in cubanets which will create multiple pods for that and then A stack in an open stack which will create like other particular resources for it and from a single definition So we have observed some behavioral pattern and all those Relationships between them. So we'll be explaining it based on replication and cross and Mary will be explaining the Behavioral pattern which are observed for a replication scenario Yeah, sorry for the site before IP and image Were reverse, but I'm pretty sure you noticed already So First one is a requirement. So for requirements. It's when you need a resource a sub resource only for creation or update of a like an image for VM In this case we only need to use sharing to create a resource Replica of a on every site And so we just use sharing and if it needs to update we use sharing again in case of reliance where you need the The sub resource at all time it means that In case of network partition the a resource that are on site 2 or site 3 will not be available available anymore So either the user can replicate also the sub resource if you want that's not a problem or Just we don't do anything and we just keep sharing For the replicas and in this case there would be a heartbeat saying to tell the user That's site one is not available anymore And so your the resources a and a on site one and site two so it's actually two and three will not be available anymore because they will be affected by the unavailability of B and so we also need to store information about B In the in the information about B And we need to store the fact that there are resources that relies on it so We can prevent deletion Yeah, we can prevent deletion by warning the user if you delete this this resource will not be available anymore And in case of composition there is actually nothing to do with application We just replicate the first resource everywhere and the application we just do whatever it does all the time But on on every sites locally, so it's pretty simple actually So you want to cross so in the cross-scenario So as we mentioned only single resource will be present at a particular Site and the others will be able to access it and and do the all the things so in this case the Requirement and reliance will be pretty straightforward something similar to the replication in this case and We will be needing to have based on the location and all Location will be needing to have a communication between the particular sites If there is a network partitioning scenario then the framework needs to handle and like create the resource in the local one And then proceed on it The second one is the third one actually is the composition wherein like we extend the sub resource a into different sites So in this example like I have given a simple one. So resource a is creating BCD So in this everything is in the same site so cross also envisions to Distribute these particular resources into other things so B will be there in site one C will be there in site two D will be there in site three for example in this particular case a Still needs to manage it because we don't want to destroy the existing structure of the application a deployment needs to have a Communication between the particular parts in case of Kubernetes. So in that case Particularly cross needs to manage that particular scenario and still maintain the complete integrity even in a geo distributed scenario So we have done a small POC based on our current Current assumptions and theories and which whatever we have formulated so the Requirements which we required was that a generic resource handling across platforms such as open stack kubernetes, etc Decentralized and peer-to-peer architecture wherein like these two can completely peer-to-peer Communication modular and scalable and at the same time it should be able to integrate our collaboration Methods so as discussed our primary target platforms are open stack and kubernetes but like we are envisioning it to be integrate like expanded to multiple platforms and Then we have surveyed multiple existing architectures, but not a suitable Candidate was found for it so We have created a two layer kind of architecture on it So like it is primarily divided into two different sections the key ops core and the key ops glue Where in like the core is the management layer responsible for the deployed application? Metadata and every peer-to-peer aspects and all the communication aspects This particular core is generic to all platforms given as open stack, etc At the same time the second layer is the key ops glue, which is the intermediate layer responsible for Translating the interactions between the core and the respective platforms So glue is independent of a particular platform sorry, blue is dependent on a particular platform and core is independent of a particular platform, so Glue can be developed for any particular platform and this particular key ops concept can be extended to that particular platform at any point of time So in our first Architecture like we came up with a simple solutions for it. We are like just tested out a little bit So we are using the HAProxy for the in the for intercepting the particular resource from any particular CLI we are currently only using we are not using any CLI on our own we as Mary mentioned with the DSL we are able to use the Platform CLI the native platform CLI which is available and it is intercepted by the HAProxy as of now and then this particular request is sent to the Analyzer or translator this section module and this particular module analyze the key ops request and then Breaks it into the particular generic pattern which we want like the collaboration strategy the classification all those things Those data are sent to the core API Where in the process actually happens whatever is required and this core decides what to do there So either if we need to communicate to a different site We will be sending it currently through a rabbit MQ with an AMQP protocol and a P2P a system and We will be communicating with multiple key ops site based on this and the particular data will be sent to a document-based data store Arango DB and when a particular Another side receives the particular request So it is received by the rabbit MQ and core API processes the request And then it sends the data to our angle to the database and then the analyzer translator Transulates the particular request into the native platform as per the requirements. So we are using the For open stack open stack API is for Cuban as Cuban as API So that's why I told like glue is separate for individual. It's separate for platforms. It's in its respective to the platform Then it sends the particular request back to the platform API and processes the data So this is a current building block. So we have just done like very small POC on it and it can change in the future So initial efforts where as I told initial efforts were made for the particular POC sharing replication cross collaboration feasibility is tested and recently developed complete the initial POC for the Replication and cross under Kubernetes. So it is the tier like the proof is there now. So now we need to work on it so What are the takeaways and what is next? so Can we go from cloud to edge without Intrusive change in the business logic that was the primary aim so yes keops demonstrates it and We are open for collaborations under various levels. So as we mentioned before like this is at the initial stage We had some we had some research done and now we are proceeding it into a practical aspect And now it's processing at the same time we are developing more on the theoretical part as well and Multiple development efforts are scheduled with the focus on framework and the collaboration So we are planning on some autonomous loop Based on the Kubernetes autonomous loops at the same time Autonomous site optimization especially for the cross scenario where in like we can actually optimize based on the particular usage Locks or you see statistics from the particular site whichever is higher we can like do that and then better network partitioning handling and then more development on the scope line and the key ops framework So thank you for your attention. So I hope we were able to give some info on what is our work. So you can Conduct us at any point of time on these particular list and this is our public report which we Have done and like we can scan and get the By the QR code and find us there the particular report. So that's it any questions Thank you. Thank you. Yes Many thanks for the presentation. It's really interesting. I'm wondering so I've seen no that the thing and I come from the Kubernetes world So I have several questions and maybe you have already heard them in the past Okay, I see the value of the whole approach of Rubbing resources, but it looks like it's for stateless things No, what happens when you have something that it's stateful Maybe because it's a database or maybe because it's a badly written application that has stayed inside because it has an Internal cache or whatever that you cannot touch Okay, so primarily for a stateless application as you mentioned, it's quite a straightforward thing So our approach is primarily actually focused on a stateful applications So that's how we actually envision key ops because stateless as you told it's Pretty straightforward. So each collaboration mechanisms, which we are envisioning or we are separately working on it So marry is more working on the replication part I'm working more on the cross part and like we are identifying how we can handle the particular stateful scenarios on this case So and this one for example for cross So recently like we are working on a new pattern how we can hide and identify if there's for example a network partitioning scenario and how we can handle our consistency and all those things and improve on it so Key ops is primarily designed for a stateful applications and you're distributing a stateful application Okay, following the same pattern Okay, I imagine that this works super fine for Kind of like cloud native applications No, they have developed been developing that aspect which have a rest API behaving away then what happens? I mean My point is if you already have a cloud native application Why use key ops then I'm not good at this. Good question. So one of the primary aims of key ops was that to migrate a Particular application into a geo distributed scenario now some providers or some framework actually can do it It's there But at the same time there should be a change in the business logic or some additional code efforts needs to be done So key ops envisions to do it just without as you deploy in a normal local Infrastructure with the same effort which is made by anyone will be able to deploy it in a geo distributed scenario With a bit of glue with a bit of glue. Yeah, that key ops will handle it Yeah, yeah, maybe we can talk about it later. So Then I see the whole thing. No, and I make analogies to Kubernetes and then I Look at the at the glue know a key of key ops glue and then I think about the resources of Kubernetes pods and and so on and It feels that by using the pods or by using volumes in Kubernetes I will get what I get with a key ops glue for example and and all the Is that correct in a sense? So is that a good analogy? So glue connects to the key ops API, sorry Kubernetes API key ops glue connects to the key ops API. It's like just a translation between the core and the particular platform because Obviously there needs to be because like open-stack API is not as same as Kubernetes API So there needs to be a connecting mechanism between these two. So that is glue Anything It's not about Seth It seems to me that there should be some or more interest in in that on the on the side of telcos So have you had some connection to telcos ask them to collaborate with you because I mean It's it's the big thing for them, right? And they're looking for okay. What are we doing there? Are we doing edge with open-stack or using Kubernetes or all together or smaller devices in other ways? So they should have much interest in the technology like that Yes and I think orange in France is interested and David Samiento, which was on the cross slide He was paid by Orange, so yeah, there is interest But many are aware with orange. So they are interested in the Okay, okay. Okay. Thank you. Thanks Okay If no further questions, thank you. Just just a second. I will put the slides I think this video is filled I Will put the slides on the comment on YouTube if you need to check on anything at the time Thanks. Thank you