 Looks like the last thing in the way for tonight's festivities, so I'll try to make it interesting. At lunchtime, I was having a conversation with someone. I was asking him how this conference is compared to some other conferences had been to like Ruby and stuff and he said that this conference is, there's not as much rah-rah-rah and he was blaming it on the, I guess, the maturity of the language and so on, right? So I thought it's appropriate that we do some rah-rah-rah for Elixir. So let's hear for Elixir. I know I'm just ecstatic about the language. It's a wonderful thing. I'm having a great time programming in it. So I am going to talk to you about about real-time concurrent embedded project in enterprise telephones. Who here knows, who has actually thought about what's involved when you make a telephone call on, let's say, at the office? You really thought about that? You think about that all the time? Yeah, you told me that last night. So you lift up the handset and I'm going to do my best not to kind of bog you down and all the telco lingo here, but you lift up your handset and you hear doll tone. Okay, well messages come into the into the call server. We figured out what telephone it is. Then we actually have to go and talk to a tone generator. Tell what kind of tone we have to switch that to speech path so it actually comes out your ear. You hit a key to start dialing, so you hit a digit and we have to stop the dial tone. I'll get the digit in, stop the dial tone, set a timer, waiting for an interdigit timer timeout. Pressing more digits each time press a digit. We have to figure out if that's an extension or where it's going. So we're translating each digit. Finally, you get to a number. So now you start telling the other phone what's going on. At the same time, you have to give switch in another tone to give ring back to the user. Then you have to start ringing the phone, set a bunch of timing cadences, so on. They answer. Once they answer, you have to remove the ring back tone and then you have to set up a speech path. So you have to actually take the switch that's connecting the two parties and join them together. I'm a little bit in the old TDM, the old time division multiplex domain here. It's slightly different with IP, but all of that stuff's going on, right? And then you compound that with features. So you want to set call forward on your phone. You hit the button. Your display is going to give you some feedback saying enter the digit, the keys on your phone are going to change if it's a modern phone, so on. So just trying to give you a little bit of idea of what's involved in a telephone switch if you weren't, you know, if you're not like Johnny over here who thinks about it all the time. So what I'm going to cover in the next little bit is I'm going to talk about the product that I've built with Alexa. Well, a piece of the product that's built with Alexa. I'm going to go through the architecture to show you how I put this thing together. Talk about some of the challenges I faced and some of the learnings about porting a C application because my application was first written in C and then I ported to Alexa so I can talk a little bit about some of the experiences there. So who am I? Steve Pallin. I'm from Ottawa, A. So Dave was mentioning earlier about his first programming language being basic. The first time I programmed it was actually on a programmable calculator. I programmed Blackjack and on one of the old calculators with the LED displays. And so, you know, when you lost, you had to flip it over upside down and, you know, read and lose, right? You remember doing that, anyone? This was before personal computers. I'd never heard of a personal computer before Vic 20s and so on. So 26 years in telecommunications, various roles. And I'm the R&D lead and a partner in a startup called Dematrotel. So we have been in business for a little over four years and we focus on giving customers with legacy equipment an upgrade path. The company that bought out that part of Nortel isn't really providing their MDing or getting rid of the product. And so we're coming in and offering upgrading to modern features for these customers, millions and millions of lines out there and providing them nice, sexy features, but they can still keep their old handsets, their own telephone. So a lot of savings for them. So like I said, I originally wrote the application in C, then Elixir came along and I thought, oh cool, that'd be really nice because I really don't like writing in C. So I rewrote the complete version except for one more component. There's a piece, a SunRPC library that I wasn't able to find and I didn't feel like writing it. I have since found an Erlang version so my next job once I get this into our alpha site next week is to come back and rewrite that piece and get rid of that little C service that I have running. Why'd I do it? Well, all the reasons that we've been talking about today, maintainability, less code, and it's just a heck of a lot more fun. So it's right now it's in our product verification lab and next week it should be in our alpha site. So I'm going to talk a little bit of product just so you understand kind of some context here. So what did we start with? We started with our product which is a PBX, the core is an open source PBX called Asterix. I know a lot of people have heard about it. It supports, we have it supporting SIP and we also have it supporting some other vendors IP based telephones. So it's really a PC actually even better. So now my toy here doesn't fly and it doesn't thrive so I was kind of disappointed. Actually I'm really disappointed because I follow a model of helicopters for a hobby and so I was just drooling to take that thing for a rip. So it's basically a PC so supporting IP phones is really easy because you just plug them into a switch and connect it up and everything goes great. So like I said we already support the legacy IP PBX phones. On the other hand our customers have this end of life equipment, huge installed base. Basically they've got all this funky hardware, big cabinets of hardware and they have all the old TDM phones with the standard two wire, two pair wires going into all this hardware. Fortunately the main component that drives all that stuff does have an IP interface. So that's what I started with and so what we're after, what we've built is supporting all those phones through that IP interface on those gateways into our IP PBX so that basically there's feature transparency and the phones work exactly the same way. So just to give you an idea of the scaling our target is to support up to 60 of these gateways and about 10,000 phones. I gotta look up here because I don't have my glasses on. Like I said it's the sonar PC. Peace hasn't been done yet but should be done soon and just talking about some of the things that I have to deal with here. You know accepting TCP RUDP links. RUDP is a reliable RUDP protocol that it's kind of proprietary. It's just sequence numbers and stuff retransmits and so on. Line card installation, removal of these things. Just to give you an idea of every gateway has five links. I think four TCP links, no three TCP links and two RUDP links. Every telephone has an RUDP link as well as another RTP, RUDP link. Some of the phone features meant before just to give you an idea. So we gotta detect the phone. They have IDs. They have serial numbers on them. We have to put data on the display. Tones I mentioned before operational parameters like the little lamps on there, drive that stuff. Lots of state machines. The telephony loves state machines. Hence, GenFSM. And we have, sorry. Handled DS, that's alright. I have a copy of GenFSM. It's in my project. I'm okay now. And so DSP resources that we have to manage. And we're off pulling these phones. So if you're plugging a phone in, we actually know it's there. Or if you unplug it, we know it's gone. Okay, so architecture. So basically it's an OTP type architecture. So we have a supervisor. I start off with a supervisor. I start a bunch of singleton gen servers. Then I start up a gateway supervisor. It's responsible for gateway manager signaling. This is where I start the region, one of the several region TCP acceptors. And I got to dig out the glasses. Now I look like an old man. Okay. And DSP links. From there, start up a worker supervisor for every one of these gateways up to 60 gateways that can connect. And it manages, there's line cards in these things. So it manages the line cards. Starts up a device supervisor for every device, every phone that connects. We have to do polling. And actually, sorry, device is supervisor, which then manages all the device supervisors. And it starts up a state machine for the device, starts up a call control state machine, and an RUDP session. So lots of state machines. Performance. Actually, kind of appropriate. Now my performance isn't quite as fancy as the last presentation. So we run this service on the same server as our Asterix server, just because enterprises are pretty cheap in buying equipment, because they have to pay all your pay y'all lots of bucks, right? So to write software. So we need sport up to eight calls per second. It's mainly just the setup and tear down messaging. Approximately 50 messages per call average, let's say 10 bytes per call or 10 bytes per message, and do the housekeeping, the heart beating and phone messages and all that stuff. Definitely, I don't get the same capacity as I did with C. I haven't really ran any fancy benchmarks. I've loaded it up and run top and kind of watch it. It's a little more peaky than the C application. I guess I'm at probably about a 40 to 50% reduction in capacity with this solution. The way I sold it is, Elixir is great too. You know, we can just split things apart, run this over here, this over here. So when we get to the size where we're actually handling 10,000 phones, there'll be very large deployments. We'll be able to put extra hardware in there and kind of split it apart. And that seemed to appease my business partner. So I'm just going to walk through some of the more interesting, well, my view of the more interesting things, parts of this application. So logging, so built the original version, the C version to use syslog. So we wanted to keep with syslog. I actually ported the Erlangs twig to Elixir after I finished porting and I kind of questioned why I ported it, but it was fun. So I guess I just wanted the experience of seeing what it'd be like to actually bring it over. So I ported that and then I think three weeks later, Jose comes out with, hey, we got a brand new great logger. And actually, I say that joking because I knew it was coming. So my next step is to actually try to bring syslog into the new logger. So with all the stuff going on with state machines and so on, I just wasn't happy with your standard three or four levels of debug, severities in logging. So I implemented a category so I can actually fine tune it and I can set up categories in different modules and so on. And then I can turn those categories off and on to really fine tune the logging. Typically, we need a certain level logging in the product. So when something happens, we'll have to go in, turn on the logging, so on. But I will probably turn off the highest level debug logging. So I actually use all macros for my, I've wrapped the logging with my own API. It's macro based. And so I can actually just go in the code and turn off a logging level. And when I compile it, it just compiles out all the code. So from a performance perspective, I saw a big, big increase with turning, actually turning off the call. So configuration and administration. So I had, like I said, I had a C program. We already had a config file and so on. So I had to, I wrote a config file parser, yet another config file parser. We generate config files through a web interface. And one thing I don't have yet is I haven't built in the ability to change those logging levels on the fly. So right now it reads the logging levels at boot. So I have to come back and start kicking processes to reread the config files if they change. The other thing from a kind of configuration, administration, or debugging is I have a fairly elaborate console based commands that I have in there. And I want to get those accessible from the web page. I haven't done that yet. I'll actually talk about that in now. So basically, in my C version I had a, you know, I just had a console program that read a string and parsed it out and did all that stuff for the, for the logging, or for the console commands. With Elixir, I just used IEX. And so it, and basically I just store, put import console into the IEX-EX file. And then I don't have to worry about the namespacing for the commands. So there's probably about 30 or 40 commands in there. So I built it around so it detects the commands with help so that there's a help or a help command to get the help on the commands. I actually implemented it with, with macros. And the other benefit now is it has the full power of the console, right? So you run a, run one of my, my commands in the thing and you're going to sign that to a variable and pipe it and do all fancy stuff. So it's pretty cool that way. It always bothered me when I was working on the C version. If I wanted to, if I wanted to add a new command to the console I had to go up and I had to put a prototype at the top of the file and then I had it in a list and I had to put the name of the, of the, of the procedure and then I had to put the help state, the help strings in there and then I had to go down below and put the body in. It just, it always bothered me, right? So the new version, the Elixir version, basically I have a macro called command and I just called the macro and the first item is the name of the command, a description, your help description and then a list of, a list of parameters and then the body of the, body of the command. So now when I want to go in and out of command it's like, you know, three lines of code. So, so I was pretty excited about that. Of course the help, the help actually knows, knows that. So you don't have to do anything from the help. The help system reads all that. So another challenge I had, coming from a C background and I still struggle whether I'm just too C-oriented or something like that, but I had a whole, a lot of, of defined constants in C and things like defining the, the values that should be in a binary protocol, right? You know, the different commands or all the, all the arguments. And so I was, I was trying to figure out how I do the concept, I guess, like an HRL file in, in Erlang and I wasn't able to figure it out. So I ended up writing a macro to kind of do this so that I could, I could put all my defines in one file and then share, or one module and share them between modules. So module attributes don't work because you can't access them from different, different modules. Now I'm, I'm waiting for someone to actually gonna say, Steve, you didn't have to do it. It was this easy, but I'll buy you a beer if you do. So, okay, so to use, to use this, use this, I, so basically you define your module and, and I actually called it define. And then the name of the, the, the constant and then the value. And then when you want to use it, here's an example where I'm using it inside function. So it's being used as a match. So one of the advantages of this is that, that I can actually use it in matches and so on. So, and and there's the implementation. So macros are cool. I love macros. So that's how I solve the shared module constants. Now I have to say, you know, not every, I didn't use that for everything. So, you know, where I would in C, if I was parsing a binary protocol, and you know, I wanted to name some of the parameters in there. In C, I would create a define with the value of the parameter. And then, you know, when I receive that message, I do a switch on the, on the constant. Right? Those are really ugly, because, you know, I just didn't like that. So in elixir, you know, I have a parser that pulls it in. And if I don't need the value, I actually just turn it into an atom. So then, you know, I'm returning from my parser a tuple with the atom and, and all the data or just the, or just the atom. So the code's a lot cleaner now because I'm looking at, you know, looking at atoms and, and when I'm logging, I'm just doing an inspect on that message. So my inspect comes out as atoms versus in C, when I would, if I'd logged, my log was actually printing out the value of the constant, which was a number and then had to figure out what the name was. So hex numbers, I ended up playing around with ways of logging. So everything I do is in, in hex or all the binary protocols I'm parsing is in hex. So I ended up, first, I tried to override the all the inspect or overrides not right word, but create a protocol or depth, the implementation to kind of all inspect prints out hex, but that was kind of ugly. So I ended up just implementing an inspect hex that I can throw at anything and it just did'll dump out all the integers in hex. One of the patterns I would continually facing is I do a lot of wire shark captures. And so I'll do a wire shark capture. It'll have a nice hex string of the, of the binary data coming across the wire. And then I want to write test case to use that data, right? So, you know, I was copying and I was putting in the editor and then I was doing a search and replace on all the spaces and putting commas in there and inserting OX in front of all the numbers. So because it's coming out like that string there. And so I'd have put OX in front and I thought this is kind of silly after doing it 100 times. So I just wrote a simple seagull that basically allows me. So here it is here. So in the comments, so I just till the H and then the the heck string and it pops out the list of numbers. So so I was kind of handy for my especially for the tests. And there's the implementation. I love how short the implementation is. Now I don't know this this this one this one might get me get me raked a little bit. But so you know, the one nice thing about programming C is the ability to define a structure that overlays your binary protocol. Right? So you know, you have a binary message come in a memory and you just point that point your structure or your structure pointer to that message. And now you can just pull out the fields through dot, right? So, you know, there's an example there of bit fields. And I really didn't have the quote, and I played around with a couple different ways of doing this. And I ended up with I ended up with this. So this is this is actually a test the test case. So so I create a structure, define the fields of the structure. And then I introduced the C structure macros that allow me to find the schema in the Indianness. So here, the the sub data three, it basically the scheme is is to to 16 bit number and an eight bit number. Okay, and then just to show kind of the complexity, the second one the my list records to so this was written with records and then convert the structs. So this one actually the schema is a the first first element called one is actually a list and then the finding of the list and in in the two, it's a list of records and so on. And so this gives me the ability to define a complex message, a fairly large binary complex message with nested fields and so on, and be able to pull those out individually. So and it seemed to work pretty well, I used it for a number of protocols that I had already and see and it was pretty easy to translate this. So we're on Linux, Red Hat, a well sentos to be specific. And so from packaging, packaging and installation, I used EXRM to actually build the build the package, a great tool. So it builds a package of the application with yearling and elixirs runtime. So no need to when I first started doing this, I'm gonna have to write an RPM to install Erlang and write in our RPM to install Lexar and stuff like that. Then I found this thing. Oh, wow, this is great. So, so you just run release and packages all that together. And then I got to think, okay, well, now I got to create an RPM. And I hate creating RPM. It's like, you know, so you take a template spec file, and sorry if people don't know. So RPM is the Red Hat package manager. So it's the package manager that Red Hat based distributions use. So you take the spec file over you and then you change, you know, change all the stuff and then you need to be able to start and stop the service and you need service to run to automatically start. So you've got to create a net script, edit all that all the cut and paste stuff. And you know, three, four hour, five hours later, you have an RPM. So I wrote an RPM generator, it's actually a plug into EXRM. And now if you want to create a pretty new project, mix new yada yada, hit your hit your depths and put in the EXRM RPM and then you only do a depth, depth sketch and and then you just run mix RPM minus minus mix release minus minus RPM and you have an RPM. So and then if you want to customize it, if you want to actually, you know, there's a whole bunch customization reads config file there's a bunch of stuff in there. But if you actually need to change the call flow like the actual procedural stuff, there's a create templates and that just copies the template into a another location, you can edit that and then when you create an RPM, if it finds that the newer version, it'll use that instead of the built in template. So and that were that's been working great for me. So I'm looking forward to building the next 10 projects so that I don't have to spend a time creating RPM testing. So early in the project I was looking for was looking for mocking and I came across Amrita, the person who wrote it here. So a great, a great product. I love it. Actually, I really like it. So it's got you have markings built in. But you know, it has the nesting of tests. So you know, a fact, and then facts in your order, you can use tasks and tasks. Or if you're from the the aspect world, you can use describe and it. So so it has all those great features. But it hasn't been updated for a while. And so I actually my product is still on 13.2. And the only reason I got 13.2 is I had to go in and do the do the changes to get it, I think from 12 up to 13. And so I think it's for me to get up to 14 or ultimately one, I'll probably be going in and doing all the record to structs conversion. So I typically do excellent test coverage if I'm building like platform type libraries. That's nice and easy. My call processing code and all the all the fancy stuff, my test coverage isn't near what what what what it is for for this, the easier stuff. So I probably could spend more time doing trying to do that to that test coverage. But it gets pretty. Sometimes it gets pretty delicate. And the other thing I did is I wrote a simulator to simulate phones. So I, you know, run another Alexa program. And it'll come up and it'll talk to the talk to this app. And say, you know, I've got a whole bunch of phones. And and then I can, you know, test that. And the neat thing about it is I can bend drive sip traffic, I can drive a phone physical phone generator into the call server in the PBX. And then it actually starts telling me phones to ring and answer and so on. So I can actually do a pretty good job of simulating hundreds of phones for performance testing. Okay, so benefits. Supervisors they're awesome. So my C version, guess what happens when I get a bug? And I've got 1000 phones on the system. And who's used to actually picking up their phone and not getting dial tone? Right? No. So phone users of phones and companies that use phones are very particular about their system being up. And so I get, I get a problem, I get a seg fault, boom, all the phones are down. Right? Yeah, I restart the application that comes back up in, you know, phones to register and stuff, maybe depending on how big system is anywhere from 45 seconds to, you know, maybe five minutes. But that's one bug, right? And that's really anywhere any seg fault anywhere. So with Alexa, you know, the best case, I have a bug and I just dropped the message. Okay, if I actually spawned a handler for the message, you know, sometimes I just dump them into into the state machine. In that case, I take them state machine, which takes out one phone. Okay, so and then you know, I guess if I had something in the handling of a gateway, you know, I might take down one gateway and the other 59 stay up. So you know, fault tolerance, it's, it's, it's amazing. Live upgrade. So use this in production. And you know, that's my work, my workflow, right? So I want to change something systems running, edit the code and just are the are the module and poof, I'm up and running. I don't have to take it down, come up, register all the phones and so on. We'll console. I love the I love the the the rebel and multi note scalability. So I'm looking forward to that when we get really big and make a lot of money. And the heck of a lot easier to maintain. So a lot less code and no semaphores. No pointer corruption. No seg faults. Although I've crashed Erlang a few times, but actually the the the viewer, the whatever dumps out the the the Erlang core file, the viewer works pretty well. So it's pretty cool. challenges. So it was peace cake, there was no issues. You know, a week later had it up and running. So I so challenges, well, gotta train people in elixir. So and I also had to sell the fact that Steve you want to do something that there's not really a big resource pool for team buy in. So I had had some challenges with team buy in. And so with respect to the effort to port it and the time to market because we had a C version, we're already out of production with that. But I managed to get through that. I actually learned Elixir functional programming and architecting the solution all same time, about four to five months ago. So I was that's about the most time I actually had to sit down and work through an architecture. I wanted to get the supervisor model right and so on. So it took me it took me weeks actually. And I'm not a patient person. So I like to get in and just you know, start building up and start building up. I did that for a little bit to prototype some stuff. But then I had to step back and so. So that was challenging. excited about the next one now. The language journey elixir that was a little challenging. The the biggest one was records to structs that took me by surprise. But then I started watching the mailing lists more. But unfortunately, fortunately, I really, really over abused the kind of what I call the ability of records, right? So so your custom functions didn't you didn't have to pass it the the actual structure the record it was it was implicit, right? So so converting that there I think there were like 25 or 30 records with usage across even more files. So it took me a few days to do that. That refactor but I like the new one better now. So I'll tell you when I was refactoring it, I wasn't. Yeah, so 25 records. You know, and we definitely need to get more investment in the ecosystem. They're still, you know, we're we're in the early stages. And it's not a complaint. I love being in the early stages, right, being able to get in and and actually mold the language influence language, have some good discussions on and on the test framework that I mentioned before. So what's the summary? Well, the experience has been pretty good. Great community, fun, fun here, too. And looking forward to that's great to be involved with with the pre one dot Oh, that's my first time. Actually, I haven't really been an active member in in different in open source communities before. So getting in and kind of helping influence it and stuff has been pretty cool. I love a lexer. I hate C. Lexer. It's been a pretty modest modest transition from Ruby. Not sure that Oh, that's a summary. Sorry, I was back on the challenges. I built a better product. So a lot better product. I'm really happy with the way the elixir version turned out. And looking forward to one dot Oh, and hey, Jose rocks, man. So good job building this question. I'll start the questions. Yeah, have you used adhesion? Adhesion? Yes. Matter of fact, I want I want to port it. I want an elixir version of adhesion. I have it in a product. Yeah, but what I have, I use it in a product. Okay, okay. So the question is, I use the adhesion adhesion is a Ruby framework, almost like a model view controller type framework for voice applications. So it'll hook into different PBXs and you can create applications that has controllers that type of thing. And there's very well with Ruby on Rails to because you can share active record between your adhesion and your rails. And of course, anytime you build a voice application, you need configuration management, that type of thing, which you can throw in in Rails. So so and and my next project, I would like to try. I haven't done any web development yet with elixir, but I'd love to do and I have a number of projects in Rails already. I'd like to try moving over there. And in that case, I'd like to try to get something in elixir like adhesion. So when you said you couldn't use attributes for to replace your defines, did you try using register attribute because it keeps the attribute permanent so you can access it externally with get attribute? No, I didn't try that actually. Yeah, so one of the docs, I don't remember if it's Dave's book, or if it's on the game started thing, but it says if you want to expose your module variables as constants, use register attribute. I missed that. So when you just use the names, the, I guess, the namespace of the of the module and then whatever the No, you have to I think it's a method on the module module. So you do module, I get attribute that you pass in the module and you pass in the attribute to look up. Okay. And that way you can access it externally because I guess module attributes and in early are always preserved, but they're not any elixir. And so that turns them into Erlang style module attributes. So and then I guess I would also question whether you could put that in a match. I would doubt that would fit in a match. So one of the one of the requirements I had. Oh, okay. One of the requirements I had was yes, so you have a couple options. One of them is to define them as a macro as you did. Yeah. The other one would be you could define a macro that basically inject all the attributes in a place and then you use them as attributes. So for example, you did use constants. And then you could have in your using callback, you could just define all the attributes in there and then they would be available to all modules. And you could use them as attributes. Okay. And and you could register attributes as well. And the idea of registering attributes is that they are going to be stored in in the beam code. So those that would all be at compile time and they would, you know, they would be replaced and so on. And when they are starting the beam code, you can after the module is loaded, you can access them but access them as a function call. So you would need to have another way to access those registered attributes in pattern matches or something like that. Okay. So there would all options and they could all could all work in a way or the other. Now, am I allowed to ask a question or it's a simple one. Why not HRL files in Elixir? Why not? Why not in like the early include files? Why not something like that in Elixir? Yeah, so you can achieve exactly the same thing with macros, right, that you could do with the include files. So maybe we need a better mechanism for defining constants that are reachable from different places. But I don't think we need to have the include files exactly because that's exactly what we got with macros. Okay. Okay. So it just and I get what you're saying. But what I think what we need is something that's fairly easy to do and easy to get to, right? So yeah, yeah, I've got to solve for myself now. But, you know, I guess anyone else wants to do it, they're gonna have to come and get my codes. Okay, another question? I don't follow up. So you're doing like a phone thing. And Erlang was developed for phones. And Joe Armstrong, you know, one of the guys that wrote Erlang and was in the development actually did his thesis in 2003 on a Erickson phone product. That seems really similar to your phone product. Did you read that thesis and see the problems he had doing phone products in the Erlang VM? So I did not read the thesis. I know that Erickson had built a phone product in Erlang. I use that as selling point. But no, I haven't read that. And I so I don't know the challenges he faced. So but it's good. I'm going to go and find out. Yeah, one of the things he said was the they couldn't keep all the calls active. They actually kind of preserve processes offline after the call setup happened. And then like, bring it back to life when the call dare not tear down happened so that they can get higher performance. So now they from what I understand, they built a class five switch, which is a, you know, the switch the service of your home. And they're huge, right? So I'm in the enterprise market and offices are just aren't that big. I'm hoping I wouldn't have the same problem, but I will definitely read that. So a few years ago, I worked on a project, we were reporting something with a fairly hefty amount of use of C bit fields from a Solaris system, all on spark to Intel based Linux. And we discovered that bit fields were not guaranteed by any sort of standard to actually be serialized into given order, if you just dump them straight to memory. And so we a lot of the team members lost a lot of hair over that one. But another thing we ran into was really bad logging facilities available to really make sense of binary data in a useful way. And I've seen some of the stuff that was offered in the documentation for dealing with the bit specific data structures. I was wondering, since you talked about logging, did you find that creating things that gave you detailed information about the bitwise stuff was easy, or was the existing functionality enough to handle what you needed? So so I did do a lot of action on the C version, I did a lot of effort there. So when I was trying to take to look at binary data and and understand it, I actually wrote this was before Alexa. So I wrote a parser, a lot of basically a binary file parser in Ruby, that would take and print it out in some format that I can I could understand. When I actually did the logging system in the C version, then enhance that thing to actually parse the logs I was printing out in the C version to filter them down and print out similar stuff. So I've spent a lot of time working on logging. Actually, the law of my log, I'm still not finished with the Alexa version. It's still not quite as good as the C version. So but there are definitely some benefits like being able to you know, since I parse out a message, and I have a tuple in there to describe the message, and particular things, not tuple an atom, right? Then when I just dump that thing out, I actually get the atom instead of getting a number. And so but there's there's a lot of a lot of work. And then you also have to look at the level of logging, right? So, and then the filtering, right? So if I want, you know, if there's a thousand phone calls going on, right? And I want to look at one phone call. Well, you know, I got to try to figure out how to pick out that one phone call. Thank you. You're welcome. So you talk a little bit about the release process. For me, that's been probably the most difficult part is generating releases in a continuous delivery manner. I was curious, you chose RPMs, and you're using relapse in each of them. So when you install an RPM, does it just run the relapse script to something that's already running? Or? So yeah, so basically, I if if it's already installed, I installed I run the relapse script. If it's not installed, I, I install it and the RPM in the yeah, so I have code in the RPM. And it's actually in the template, right? So that's the default. I've been debating on whether I want to switch between do a delete, like take it down, put the new code in and bring it back up versus doing the relapse. Yeah, the problem with doing the relapse is you have to make sure you've done all your, you know, you've done all your, your migrations, right? So and I'm a little worried about other people coming in. And once the development team gets bigger, right? So I'm thinking I had to put a switch in there. And I might even turn it off by default. So so in the RPM, you contain both the relapse and the full installation. Do you make two releases at release time? No, no, what? So if so, let me think, I can't remember the code now, but so I zip the thing are on on tarot. And then no, so so so I know, I know if it's already running, if it's already there, right? Okay, it's already there. I just kick the relap, right? If it's not there, I put it there and then just start it. Are you doing anything in an environment where potentially you hadn't upgraded yet? Like, if you run that RPM, and you attempt to run the relap, but the version that's running isn't compatible to relap? What do you do with that? Is there I haven't handled that that non sunny day? Yeah, that's the case. Yeah, that's, I'm not looking forward to that either. Yeah, no, yeah. So in that case, what I'd need to do is I'd need to, I need to put those conditions in the spec file, right? So, you know, to look for that error code and then handle it. So, you know, once I come across that, I would modify the template, you know, in the next release. Any more questions? All right. All right, thank you.