 All right, so up next, we have my Jordan, CEO of Legion, or LCDO for France. So Legion is the company which is basically responsible for being an asterisk. And we should probably all know. Because he's going to tell you all about it. OK. All right, so are we? Yeah. All right, cool. So as I was mentioning, when I proposed this talk for Fosdom, I thought it would take, I could probably fit it into a 30, 45 minute sort of talk, and then I saw it at 20 minutes. So for the most part, I'm going to be trying to give you an impression of techniques today to build a large SIP infrastructure using asterisk in a horizontally scalable fashion, also with other useful tools like Camaleo. And the way I approach things is this is a much longer version of the talk name, which is that I would like to build a large SIP infrastructure using open source tools, but I'm going to emphasize the ability to go hire people who can build services around these tools. And so a large number of things that we're going to be doing is trying to try to simplify asterisk, Camaleo, and the monolithic open source projects in this infrastructure so that I can have other developers come in and build services around them to come help them out and orchestrate them. And in particular, I'm going to try to hire not just C developers, despite the fact that asterisk is written in C. So couple of overarching philosophy goals that I have when I'm trying to build a SIP infrastructure. I want to be able to scale out first. And then worry about scaling up. There's a lot of optimizations that you can do later on in order to improve the performance of individual components. But in particular, I want to be able to spin up new servers because that's just going to allow me to scale faster or easier. I want to have reasonable redundancy as much as humanly possible. I'm not going to stress about being redundant, where it's extremely challenging. We can always come in and improve redundancy in places. But in general, I want to be able to have backups for everything. I want to treat my applications like they're cattle, not pets. If nobody's ever heard this phrase before, when your server dies, it should be like you treat it like your cow died. You have hamburgers. You don't treat it like your pet died, and you have a service for it or a funeral. In particular, I want to actually minimize the necessity of requiring specialized monolith knowledge. This may come as a shock to everybody in the room, but the world is not overflowing with real-time communication developers. We're kind of a rare breed. There's not that many people who know the inner workings of Camalio and Asterisk. And while you can teach people these things, it's much easier to find JavaScript or Python or Perl developers and teach them the basics of real-time communication rather than just requiring right off the bat that you are only going to hire developers like this. Good try. So this is sort of how I view the strengths and weaknesses of the various players that I'm going to build my infrastructure out of. And apologies to all of the Camalio and Asterisk people, including myself in the room, because I give us some meh marks in some places. Camalio scalability, fantastic, right? Scales wonderfully. It's API, it's OK, but it's kind of a solid meh in some places on that. And it requires a lot of specialized knowledge. Asterisk, scalability, kind of a solid meh. Scale's OK, but not that great. And pretty quickly, you need a lot of them. API, we've done a lot of work over the past couple of years to give ourselves a better API for building communications applications. It's one of the biggest areas that we've focused on development in. Required knowledge, though, need a lot of required knowledge in order to be able to build Asterisk applications or to be able to run Asterisk. On the other hand, if I'm building my own services, I'm putting all the burden on to my developers. They scale as well as my developers are able to build things. If they have APIs, it's however good I'm at writing APIs. And generally, I can find people out there who can go build these things. So from a very high level, I want to build an architecture or a system that looks something like this. I have a pool of Camaleos that are performing my registration. They're acting as my registrar. And they're doing the routing to my Astra servers that are acting as my media application servers. I may have another pool of Camaleos that are acting as my proxies to go out to my providers. I have another pool of servers that I'm writing that my developers are writing that are handling all of the application logic. And that is sitting outside of my Asterisk instances. So I'm going to keep my Asterisk instances as simple as humanly possible. They should have a very simple dial plan. And I'm going to push most of the burden of what happens in the call off to the application servers. So we're going to refer back to this a couple of times as we go along. So there's sort of four problems we're going to kind of talk about. And the thing I want to remind everybody is that there's no right answer when you solve these problems. There are absolutely wrong answers, right? You can make bad decisions, but when you make a decision, you're usually just choosing a path that's going to force you to make some trade-offs elsewhere. So we're just going to kind of do some pros and cons on different options in terms of solving these things. So how do phones find each other? How do I handle the SIP registrations? My first goal, I want to use Camaleo for this. It scales better. It handles sharing registration between Camaleo instances in a cluster really well. And Astrus doesn't do either one of those jobs well at all. So I want to get my Camaleo instances to act as the registrar. The other thing I don't want to do, I don't want to send SIP registrations to the Astrus instances because I'm either going to start storing phone location on specific Astrus instances, which is really bad because then they die, and then I have to have a funeral for it because it was a pet. And I stored phone location on it. I don't want to do that. I don't want to send all SIP registrations to all Astrus instances because that doesn't scale well. And when I'm at 500,000 subscribers, I don't want to have 500,000 registrations on every single Astrus instance. It's not a good idea. So how might I go about doing this? Well, in Camaleo, this is a very easy problem. This is a very well-known problem to go solve. Use the location module. If I have a couple of instances of Camaleo in my cluster, I can fork the register requests to the other Camaleo instances. And they can be smart enough to go, oh, I'm just updating my in-memory information. So they're keeping track of all the registrations. In Astrus, I've kind of got two different approaches that I could do. One is that I could create a view on the Camaleo database and look up the location information by the address of record, which is in the PJ SIP stack. We call it dialing by AOR. The second option is I can build what we like to call sidecar services. And these are little tiny microservices that you stand up that expose some information in Camaleo off to Astrus in a way that it can more easily consume, in which case I'm going to be dialing by URI. So if I did with the database route, I would create a view on the Camaleo database that exposed the parts of the location table using the same schema that Astrus needs to consume for its AOR object. And then the rest of the Astrus instances would be looking at their own Astrus database for the rest of the objects. Generally in Astrus, all you're doing for doing that is you're going to take the PSAORs object, map it off to the Camaleo database, such that it actually looks at the PSAORs view that you've constructed there. And then as far as dialing is concerned, you just dial like you normally dial. And when you do that, Astrus is going to go off to the Camaleo database and look up the registered URI that's actually been stored in that view. So I can dial just like I've always been doing it, but I can use Camaleo as the registrar and get the location information from it. The downside of this is this is kind of non-intuitive when you're maintaining a system because now my Astrus instances have a database connection off to a database that it's not really their database. And I had to create a view on it and I had to understand the schema mappings and things. And that can cause some cognitive overload on people when you're trying to maintain things. Another option is we create the sidecar. We create a little tiny microservice that just exposes the location information. And then I go ahead and use that in order to actually get the contact when I go do my dialing. This works great when you're actually doing all the dialing not in the dial plan and you're doing inside of applications that are written on top of asterisk. And you just dial by URI where you create a dial string that's pjsip slash the name of the endpoint slash and then the URI that's their location. And that will actually go ahead and dial to that URI. And if you set up the outbound proxy it'll funnel it out through your Camaleo proxies. The downside of doing this in particular case though is that it's a little bit tricky to do this from the dial plan because you have to go probably use the curl module in order to go get that location. So I wouldn't really recommend it if you're doing in the dial plan, but we're not anyways. We want to build our applications outside in the services that are actually built on top of asterisk in that original diagram. Okay, so very high overview of registration. So let's talk about routing and distribution. How do we get the SIP invite requests to the right media application server? Initially you might think of, well, let's just tie a customer domain to an asterisk instance, right? We're gonna say this asterisk instance is for this customer. On the pro side, that's pretty easy. You put a little bit of routing information in your Camaleos and suddenly you're getting calls on top of specific asterisk instances. The downside is that doesn't scale and it's very inefficient. So let's not do that. Instead, let's just go ahead and round robin and use the Camaleo dispatcher and we'll just start sending all calls to all asterisk instances. That's great, it's very efficient, makes use of resources, it's tolerant to failure. On the downside, we're gonna push all the burden on to asterisk and our application media servers in order to figure out what to do with the calls. So we're gonna make our application people work a lot harder, but that's actually probably the right thing to do if we're gonna go be horizontally scalable. So let's go into the hard problem then. How do we build applications on top of this kind of infrastructure? Because we now have calls that are just showing up to any random asterisk instance and we have to go ahead and figure out what to do with those calls and at the same time, I want to be able to have as many developers as possible building communications applications so that I can get more features to my customers faster. Okay, so there's a couple different things we have to figure out. We can't let any customer information live or be owned by asterisk, that's the first step, okay? So there's usually roughly three things that means we have to not store on the asterisk instance. We can't store endpoints or any specific information on those particular instances. We can't store prompts on the asterisk instances because I can't put every single customer prompt on every single asterisk instance. I can't store my recordings on the asterisk instance. They can record it there, but then I've gotta get that recording off there as fast as possible because the next person who wants to play back that recording may be on a completely different asterisk instance. I may also need to be able to move channels around to other asterisk instances when I realize that I have a call that has ended up in the wrong place. So there are times, for example, if I'm building a conferencing server and Kamaliu's round robin dispatching calls out to all the asterisk instances, I may have to shuffle those channels around to get them all onto the exact same asterisk server for that particular application. The only way you can do this, keep the dial plan simple. Do not put a bunch of custom stuff in your dial plan. Your dial plan needs to be as small as humanly possible. You're not gonna scale a complex dial plan if you, because you can't keep editing it every time you add a new customer. And what's more, you're gonna find the cognitive overload of trying to maintain a complex dial plan will quickly overwhelm you. In one of the systems that we've built like this, our dial plan is six lines, total. Two contexts, three, one, two catchalls in each one. We have two different contexts depending on which pool of Kamaliu proxies they come from. And then we immediately just shove it out to our application servers at that point. Okay, so I need to actually get my, how do I connect my asterisk instances off to my application servers? The first phase that we went with, we decided to just use AGI as a dispatcher. There was a lot of nice things about this because it's relatively simple to get going with it. You can use the HAGI protocol, which actually will then go ahead and do a DNS lookup for a specific type of DNS entry. And we'll then round robin your AGI connections across your application servers. So it's really easy to get going. We can just use AGI to get the calls off into our application servers. And then our application servers can start doing fancy things with ARI selectively as they need to. The downside of this is that you need special DNS entries in order to really make this work. And when you're trying to bring up Aster servers quickly, that means you've got to go update DNS and do some other annoying things. But it works fairly well. A second phase, which is one that we've been thinking about moving to in which a number of other people in the Asterisk community have actually already done some interesting work on, is to go full in with ARI. Use the ARI protocol for the whole thing and you build essentially a fat proxy middleware layer that actually handles the decoupling of Asterisk instances from the application servers themselves. The great thing is that when you pull this off, you're fully decoupled, at which point you can scale your Asterisk servers however much you need to, and you can scale your application servers however much you need to, and whatever the load demands on each, you can keep spinning them up as you need to do. The downside of that is, is you are in for some decent amount of custom development work. If you're interested in that, there's actually something called the Go ARI proxy that a number of people in the community actually have worked on that is working along this path and it's up on GitHub. When you do that, assume you go in whole hog and you decide I want to completely decouple these things, rather than actually having AGI connections going off to the application servers being round robin, you essentially build out of your own message bus and queue. And you're terminating the Asterisk events coming off the web sockets and you push them off onto the message bus and queue and then that message bus queue is responsible for dispatching them off to the particular application server. One of the things that's interesting about this unlike sort of just a generic rabbit and queue, you have to build some sort of concept of the application servers achieving ownership of the channel so that once an application server says, hey, I actually am servicing this channel, it owns that and all the rest of the events start getting dispatched off to that particular application server. All right, so one of the other things we've got to deal with, we have to not put endpoints or specific customer endpoints on specific Asterisk instances. The easiest way to deal with that is just simply to use real time. You map the PJ SIP objects to a real time backend. Please, for the love of God, do use a cache if you do this so that you don't try to pull down 500,000 subscribers when Asterisk naively tries to find out how many different endpoints are out there. And the cache, we actually have just added recently a number of commands to even let you go ahead and prune and clean up the cache manually if you ever decide and you need to go to that level of granularity. Prompts, how do you handle prompts? Have your developers create a small rest service that exposes custom sound files for customers and store the media in S3 or some other blob object storage front-ended by the REST API. Asterisk in Asterisk 14 has the ability to do remote URI playback to playback that media. And so if you issue a playback command using any of the dial plan applications or AGI or through ARI, it will actually pull that media down locally, cache it for some period of time and then play that back for the customer so that you don't actually have to keep pushing specific customer prompts onto specific Asterisk instances. How do you handle recordings? Use ARI whenever possible to manipulate the recordings. What ARI gives you when it's done recording is a stored recording object and that stored recording object exposes rest routes to allow you to pull the binary stream of media off directly from the Asterisk instance. That means you don't have to set up little weird FTP servers or other kinds of hacky things on the Asterisk instances in order to get at the media. When you're done with the recording, pull that media off, store it in another service so that if Asterisk needs it again later, it can do a remote URI playback of the media that was previously recorded on some other Asterisk instance. Last but not least, sometimes you end up in these weird situations in which you've got to move calls around and get them co-located on Asterisk instances. This typically happens in multi-party conferences, call queues, or other really complex applications. Rather than trying to complicate Camaleo by shoving application logic into the Camaleo dial plan, we found we just like to just move the channel. And so if it ends up in the wrong state and the application goes, oh crap, you're supposed to be in this conference, we just do a redirect, which is either gonna issue a 302 or refer and shove it over to the correct Asterisk instance. The trick there is when you do that, the only information the Asterisk instance is gonna have is the request URI that says, I've got this channel, where was it supposed to go? And so we typically embed a little stateful token in the front of the request URI that says, hey, this guy's supposed to be in this conference over here. So when he shows up, don't do the normal, hey, what's this new channel that showed up? Just go ahead and shove them directly into the conference. All right, last general problem we're gonna tackle today. Subscriptions, how do we make blinky lights flash? Again, please don't shove all the subscribers across all Asterisk instances, you're going to probably overload it. Use Camalio, it's really great as a subscription server. Do this for the exact same reason that you use Camalio as a registrar. One of the nice things in Asterisk 14 is we now have the ability to generate state notifications and publish them directly to the Camalio pool. Camalio will then send your notify requests off to the subscribed user agents. Again, handling this in Camalio is dead simple and is one of the standard things that Camalio does, which is great, not a whole lot of work. In Asterisk land, we use a concept called auto hints and auto hints will automatically generate the extension state hints intelligently whenever a device state changes. So rather than having to actually code in all my device state hints into the Asterisk dial plan it will just automatically generate them for you and then you can configure Asterisk to go ahead and publish every time an extension state change happens directly to the Camalio cluster. So I don't, if I have Alice acting on one Astra server as soon as Alice picks up the phone and starts dialing somebody, a hint will automatically be generated for Alice on that Astra server when that channel is there. It will publish it to the Camalio cluster and anybody who subscribed for Alice's state will just automatically get it. And I don't have to do a whole lot of special custom hint work. Okay, that was lightning fast and a whole lot of information. And I apologize. So by the way, the slides are actually already up on the Fosdum site. I already uploaded them to Penta Barf and I put a whole bunch of links onto where the documentation is for a lot of the features that I just rambled off here today. So by all means, go check out the slides and all of the features that I talk about should be documented on the Wiki because I actually had to go refer to it as well to remember how to do some of these things. We've got time for one question. So I have a question about the AGI AI interface where you said the AI layer, you can build a fast proxy in between the asterisk and application layer. Did you consider using service discovery that also exposes like the console that exposes DNS and HTTP and not have to build that intermediate cooling layer? So actually what we've actually looked at console for a number of other things as well. I think one of the problems I glossed over and didn't really talk about at all either is how when you're bringing up a new asterisk instance, how does Camalio know and trust that this thing actually is supposed to be in my private network? So console is actually a really interesting application because I think it can play a large role in a whole lot of that service discovery, authentication, authorization, what are you allowed to do? When we looked at it, I don't think the Go AI Proxy is using it and we aren't using it today for that purpose but it absolutely would fit. So there's certainly a good idea to look into that. All right. Thank you.