 So we're going to go through some main topic questions and then I'll be going down into the audience and picking up questions for the panel, anything that you might have I'll be running down. So first question for each of you. How did you get started with bringing Elixir onto your projects? And did you do that first by getting a bunch of buy-in from the team or did you do prototypes, proof of concepts on your own? Brandon? Sure. So the way that we started bringing Elixir into our organization was at the level of getting buy-in. So starting off you really need to like you can't just be the only person at your organization trying to say like hey let's go use this new thing because it kind of looks like you're trying to chase down a fad or you know you can get that kind of immediate gut reaction to that. So we actually started through just kind of introducing Elixir in some of the like little nice aspects of the language or the VM that it sits on to basically get more people interested so that when we wanted to make our case for it we we had kind of a unified front to be able to make that case. Yeah and in my case it was very similar. We approached the problem from the point of we're going to need to rewrite this tech stack. That was the first agreement that we all reached. It was like things have gotten too complicated. We can't just iterate on this. It's best to tear out chunks and start anew. And once we agreed on that and we were on a PHP stack the question got opened up and like oh are we open to just doing it in a different language if we're less comfortable working in PHP than in something else and that's how the Elixir question got brought up and that's how Elixir got in my case compared to a couple of other languages in terms of what be most suitable for the organization. What did your first prototypes look like when you were when you were getting that started after Viya? Yeah we started off with a auto suggest prototype which is the automatic completion on the search widget which is on every page of our website and I think that was a really important move for us for a couple of reasons. We it's a it was like a real part of the application. It's it gets live production data. It's like a fully fleshed out part of the ecosystem and be none of our internal systems depend on it. Like users interact with it directly so it gets real traffic but there's no dependencies. There's no like weird other like failure cascades that we could cause with picking auto suggest. So that's like what made it an ideal candidate for us to try it out. In our case we had an API that was dealing with basically large sets of data that our customers were requesting so we had essentially a set of test cases that we we wanted to verify that like any requests coming into this one particular endpoint it actually should be transparent to those users whether or not it's getting routed to our Ruby application or our Elixir application. The nice thing about that is through automated testing and a couple of other tricks we were actually able to verify that the results were going to be identical in almost every scenario which allowed us to just slowly introduce just a single endpoint only written in Elixir and Phoenix that was actually routing traffic at you know small percentages at first but we kept ramping those numbers up to see like okay this is going to introduce new errors new scenarios that we have to handle and over time we built that just that one endpoint up to a 100 replacement from our Ruby stack into our Phoenix stack and the users no one had any idea that anything had changed other than their responses were suddenly getting served back a lot faster. So how do you make the pitch when you're going for the buy-in how do you make that pitch to other developers coming from different backgrounds you have no idea what Elixir is it's often they don't and is that different and how is that different when you're pitching to non-developers if you're pitching to management if you're pitching to you know somebody who has authority but these are necessarily working with the code. Yeah so I mean I want to start with the pitching to management part of it that's very I think dependent on what your individual organization structure is and kind of how it works if if your Elixir technology decision makes sense on the back and has a lot to do with like how much like kind of trust and good faith there's been built up in engineering to be able to like make that decision in how much space you're you've got to uh to just like navigate without getting it too much into the nitty gritty details and also with like how you're able to explain or how much risk mitigation you need to be able to explain to managers saying like oh well yeah maybe Elixir is like a really new language so that's a risky proposition but it's built on the Erlang framework and that's been around for I don't know how many years now 30 something 40 something uh yeah so I think those are like the compelling arguments upwards to developers it's well my my winning argument is just I joined the company I didn't know anything about Elixir other than it worked with Erlang and now I'm here so from the developer perspective we actually one way that we found work actually very well was doing lunch and learns so our company was already had already had a concept of brown bags and lunch and learns and things like that where you would just present on something it might be related to something in your main tech stack or it might be something just radically different you know we've done things on GraphQL but didn't actually have any GraphQL in our code base so it was already kind of a nice opportunity to say like hey I'm just gonna show this here and if you like what you see you know maybe maybe reach out to me maybe we'll talk about some side projects that we could explore and over time developers kind of naturally see the value of being able to avoid immutability and you know working with the pipeline operator is basically everyone's favorite feature when they first start off is like oh okay I can just write it like this is great I yes I want to do this everywhere which is why you know there's proposals for a pipe operator in JavaScript now proposals for a pipe operator in Ruby and I'm sure like everything is trying to do it now but then from the manager perspective now that you've kind of got developers excited about something and you want to do that is anyone in here I had to do a lot of communications with upper management sorry okay yeah and you have to use graphs Elixir and Phoenix lend themselves very nicely to graphs whether it's up into the right for you know how many requests you can serve at the same time or if it's your cost graphs where things are going down because maintenance is easier scalability is easier a lot of things are handled in a nice way so both of you we talked about where you started and obviously Elixir has grown on all those projects where there's specific problems or pain points that you guys encountered that your teams encountered as you're going from oh it's the side project that we're starting on to taking over 100% of the API or things like that okay so I would say something probably a lot of people are in here have run into configuration and deployment were kind of the two major ones so just okay we got this into production we want to use environment variables to set different configurations like the pool timeout in pool boy which was especially a headache for us of trying to figure out okay well this is an air laying library that we need to configure we need to configure it at runtime but we're not recompiling this every single time we want to do a deploy so how do we manage that like we want to be able to change these variables on the fly that was I think probably one of the biggest things we ran into and it wasn't a blocker it didn't stop us but it definitely was a little bit of a headache for a while yeah for us we had similar issues which we had a convenient solution to which is we had built out a deployment framework for the front end migration part of our work which we migrated from php front end to node and we were using kubernetes for that and ops work ops team had to put in a lot of work into building that out so that was kind of our solution to the deployment issue but we a couple of the pain points that like remained and we still have some questions about our how do we manage elixir releases versus like earling application releases versus the dockerized approach like how do we balance that because we don't need slash can't use a lot of the power of like the earling like uptime guarantees and but the distributed earling systems when we're in this dockerized world another is the totally blanking the the time when you move from a like big chunky application to the umbrella app structure that's kind of what we're wrestling with right now we've got this like large relatively complicated api back end and we're evaluating like what's the right move there and like how does the umbrella app model mesh with what we're trying to do all right do we have questions about adopting elixir getting elixir on teams from anybody outside right now yeah the question is just uh you mentioned that there are some limitations when you deployed on a doctor architecture you know just curious about the nature of those limitations on yeah so elixir is or sorry earling rather is very opinionated on it's how nodes should be communicating with each other in the distributed earling model and it wants you to have these nodes that are like to manage the connections through the like node supervisor model and with Kubernetes really it's the like the the Kubernetes supervisor or not supervisor the Kubernetes pod manager and scaler is taking care of that aspect of it so it's been tough for us to find like sometimes there's a problem that we see oh we want to be able to broadcast this message to all of our nodes uh we don't have the distributed earling system set up it doesn't really make sense for it to be set up given that we've got the kubernetes architecture so now we're bringing in the third technology component to manage that broadcast as opposed to just keeping it entirely in earling any other questions what is the oh sorry was there somebody hi so you guys mentioned upper management and i'm sort of a mole in upper management here but i came from uh lots of engineering background so i'm actually one of the people in the company and it is trying to push this forward um one of the things that i find challenging is that um when you guys talk about stuff like um it's easy to show like the nice grabs of response times or number of requests per second um a lot of stuff that i hear in the community is like we went to elixir because it was way faster than ruby right or rails and it's really easy to to talk about that comparison i come from a giant company that we use lots of other things that are not slow um or not implicitly pretty slow so it's for me it's harder to make the uh the same sort of comparison to something like java or golang or dot net core or something like that which performed pretty well so i'm curious what your thoughts are on you know trying to convince people like me about how fast phoenix is when there's lots of other things out there that are pretty fast yeah um so if i were if i'm evaluating against something that kind of has like a stigma attached to it of you know how are you going to scale this application after a certain amount of time that becomes kind of an easy comparison point when it's something that is a much more established technology like if you're sitting on the top of an entire uh java stack that has been incredibly optimized and you've really gotten that performance tuned down that actually may not be where i would focus my intent my attention because there are other significant benefits the biggest one in my mind is having that guarantee of immutability and how that affects how much time uh developers and i apologize because i'm kind of putting on my manager hat here but how much time developers are spending working on fixing issues that get introduced and how much of that time pops up how much of that impacts customer experience or stakeholder experience and what kind of like long term costs that that actually introduces versus something where i can actually make some level of guarantees whether it's through what the compiler gives me or what the benefits of an immutable architecture gives me or what the benefits of a functional architecture gives me in terms of you know how much time my developer spending just working on code and adding value and doing better things for our company versus trying to trace down some really weird locking issue or race condition that we have no way uh really to like dive into without taking you know at least one senior resource probably multiple non-senior resources and people like just dedicating themselves for i've had an experience where it took uh two months just to track down one race condition that's a lot of lost time that's a lot of lost effort and it's also a big missed opportunity for things we could have been doing to add value to the company so that cost is actually like double what it is uh yeah i've got a very similar answer i think uh just the if so yeah the context is super important if you're you're gonna have to go into this assuming that you're coming from a place of many architectures because if you're if you're already using one general purpose language c plus plus or or java or go or whatever um or even two like maybe you have python for a lot of your data science stuff for analytics or something or uh or scala or something uh then maybe it doesn't make sense to to use elixir i think the killer feature is yeah the concurrency model is it's done it's solved for you it's baked into the language it's in everything that's in the language you don't need to debug it there's uh it takes a while maybe to learn the idiomatic way of of doing things right in elixir so that is uh that's an educational cost to your company to your developers but yeah historically uh in the many years i had before i was working with elixir the hardest problems to debug are the p-thread gone wild or why am i never entering or always in this critical section um and and so just yeah that like that itself has been great i think there's another benefit to of uh it's simple uh not to say that it's easy but it is simple sometimes it's so simple it's it's hard to like uh to to figure out how you get to do what or how you can do what you need to do with it um but it's a great like learning tool i think uh i've been very surprised with how quickly a lot of the junior engineers at the organization have picked up elixir and um are just like rolling with it and like really learning these like the actor model kind of intuitively uh i just one very quick anecdote um we actually had an elixir app that was running in production that we actually forgot about because nobody had to maintain it it was doing exactly what it needed to do and it was never exceeding like one percent cpu so we literally forgot that it was running in production until we had that database go down for maintenance and started seeing a bunch of error messages pop up in our uh log aggregator thank you very much uh shanti brandon uh that's the end of this panel and we'll move on to your deployment