 Hey, folks Thank you for coming to my talk. I'm super excited to be a tree closure. I'm glad to be here with all of you be virtually though this year and I hope wherever you are watching from you are well and safe and Happy Today, I'll be talking to you about the labor of love Atomic it's taken the last two or so years of my life and I'm super excited to share it with you So maybe quickly about me. My name is Alex the Lou I work as a designer at the absolute bank Bank headquartered in South Africa will operate in the cost Number countries in Africa great place to work. They are look after the team about 160 people Design UX UI designers process engineers service designers and ops team Great bunch of people really into a working with them challenge me every day. Um, but Doesn't really have anything to do why we're here today Turns out after that job when I get home and the full moons out They develop developing me returns and that developer really enjoys closure and Yeah, you could call me a way developer And this is kind of how my my closure journey started It started, you know One job in there at the time it was lecturing and then writing code in in the evening But if we go back to how we got to this point, you know, I must say thought of the JavaScript, you know, and Jeff Jeff Adwords code always Makes me chuckle, you know any application that can be written in JavaScript will eventually be written in JavaScript and That's too true. You know, I I thought very nice You know in my JavaScript obsessed days, I I stumbled across OSJS, which is a UNIX type browser OS that runs in the browser written in purely in JavaScript um on OSJS you can play Wilkenstein um also written in JavaScript and at the time, you know, there was a single um computer board called the tessel Which you know similar to your Raspberry Pi type boards But was entirely written in or you write you you interface with the JavaScript Which is also cool. I think unfortunately discontinued um, but that was pretty neat and maybe you're more Familiar on to the folks folks watching um An nbb basically closure slash babushka that runs in in JavaScript. You know, so JavaScript really did Did take over the world for being honest, right? And at least for me, it was I lived and breathed All the things in my stack the front end the back end And my build tools or all JavaScript The one part that was not JavaScript and was Quite awful to be honest with actually interfacing with database It was always a mission and you had to find this library and it didn't work and this edge case and this bug I just didn't like it. Um I was liberated one day though by sales.js a really cool framework. Um It created a nice wrapper that allowed you to interact with the various databases in a Idiomatic JavaScript fashion and for a long time I actually used that um But when it took away my pain of working with databases The general pain of working in in in JavaScript remained and that's really is Tough agility and maybe it's just a way I write code Um, I mean my front end code is fine that that's what I've been doing when I really like enjoy writing JavaScript with front end But the node piece always felt flaky. It always felt like if you just looked at it the wrong way your back end would come crashing down um And so That's one of the reasons I probably ended up in closure But on the way to closure I discovered A new database that really Reduced the complexity and meant I didn't have to use a framework, you know to interact with my back end and that was Firebase and POS and uh You know both great startups at the time going head to head split the community which one was best um, and I remember in my mind almost Seeing this vision, you know two pop two roads diverge in the woods and I I took the one that's traveled by and that has made all the difference um I mean not given that I'm not biotomic. I mean, I'm sure you can guess it was Firebase that I chose um, and you know At the time, you know, it was At my own preference, but it was luck as well There's Google bought Firebase and really took them to the next level. I've been losing their spirit um Whereas pause was bought by Facebook and taken out back um and so so I really enjoyed this time, you know when I just discovered Firebase and life was easier and I could remove sales and to make my My code much simpler without the framework um But it was still JavaScript, you know on more than one occasion. I overwrote the entire root of my my Firebase db because you know Firebase is basically a decent structure And if you write to the root you can overwrite everything and you lose all your data Have a number of times and it reminded me that I was still living in the land of of JavaScript But soon after I discovered closure It was glorious You know, it just felt robust felt solid. No flakiness at all um and You know the immutability at first really um confused me like how do I program that where all the data is Unchanging um But eventually I the penny dropped um and when the penny dropped then the atomic just Seemed like the obvious choice, you know, uh, what's all the articles that the blogs and the podcasts and the tutorials and um, I was obsessed and eventually You know, I love, you know what? Let me try out this uh Atomic and then I saw the price As low as $1 a day look Look before I got involved in the tech startup scene Um, I was marketing intern for a babysitting company and I can tell you as low as His marketing speak for you will never pay this price Always be higher. I was like nope And you know, what was really the final blow is I saw it run AWS and look looks I'm not the smartest guy But I do know I would know know my way around numbers You know and that AWS pricing calculator I mean Surely the phd to use that thing I was like this Is a surefire way To rack up a bull and just get hits with hundreds of dollars at the end of the month I was like nope not on my watch and so I said Play ball to the atomic And carried on using closure. It was robust. It was stable And I mean, I mean, let's be fair Closures libraries went acting with the various databases, you know, postgres um mongo Made us actually pretty pretty decent. So, you know, at least those are stability of closure in the old way to do Um, but then something interesting interesting happened was about Yeah, two years ago and just before covid um I went to closury gave a talk there about my journey about becoming a coach developer And it was amazing, you know to be in the conference in person I'm in Berlin for the first time We can be passing about closure and You know, it just fired me up. I was ready to get stuck in You know, and then covid came and the lockdowns came And you know what? It was glorious so I'm heavily introverted So being fired up from closure from closure d The lockdown and people not legally not being allowed to visit me I mean Great I could just write code day and day Obviously me still went to Still worked, you know And spent some time in my wife But really all the extra time apart from those things went into to writing writing There's a code and that's where You know this journey started um You know the idea of of being able to have a free and open source The atomic never never left me, you know, there's still something that is in my mind and um, when I came across data script It really felt like You know, this was the thing, you know, I could find a way to make this like the atomic, you know There are no bad ideas if you have the guts And the time you can do it and so when I need to documentation, I can't just listen function and basically it Will fire every time I turn that can hit a script and you could do something with this transaction um And I think it Transactions come in I serialize them store them And I want to rebuild rebuild my my state. I just read from them reverse chronological in chronological order and um There we go To the back sounds like I mean Pretty much the atomic I mean at time goes the initialization time probably just goes through the roof, but You know very simple solution to enter another machine, right? um And so I figured you know this kind of sequential stream of data sounded exactly like a job for Kafka Now, here's the thing folks You know The atomic looked like it could be expensive and aws might be expensive But for sure Kafka was expensive So I did what any reasonable developer would do any logical thing The only logical thing I rebuilt Kafka in five weeks. So I built fast. Basically, uh um the library that's modeled after the Kafka Rapper a pyre In inclusion basically same kind of interface same kind of A pyre that I've accepted ran on firebase and serialized up into firebase And I figured there we go so I hooked it up to The data scooped and transaction come in I'll send it up Send them out and I send them up and then when I want to rebuild my state I would just initialize and I could read them back and there we go I'd find a way to persistently store my data scoop bb Right, you know, but I want to stress this stress this thing, you know, don't want to go out then tell people try this thing out and then it just Comes crashing down. So you know what you want to do build, you know a map you know stored about You know 100 meg of data and about 10 times send it to the firebase and See how it's performed, right? It wasn't too bad to be honest. Um, I did keep on seeing like a A bottleneck, you know at around I think it was two megabytes a second I need to get some more power out of that. So I kept on trying many alternatives, you know, two short bursts of writes long writes um Well easily writing gigs, you know, just going and going and going and going um And then my laptop was really really hot Turns out continuous writing gigs of data um You know with your transaction transaction listener Probably isn't the best thing for more effective my laptop was hot Hit my wallet really hard turned out The free-ness of firebase is free within reason Um, it's that really hurt my wallet I'm probably more so than the atomic ever would Um, but there's an island so you don't have to um Then it got me thinking, you know what there's no need for me to You know start from the bottom I should really stand on the shoulders of giants and that's what I did, you know, I stumbled across data hike um, you know built by the folks at lambda forge um Data hike basically is is Datalog database starts with data Data script as well. Um, and three it's really underpinned by the hitchhiker tree, right? Um, the hitchhiker tree is an exotic data structure It's also cool And and it basically what I was looking for my basically an open source free free version of of the atomic Um, and so the journey began You know clone the repo and start going to the source code Um, because I mean I could learn this logic community if you wanted to know something about database or about a library Just read the code Definitely more to the point than the documentation if there is documentation anyway In that two things stood out Woodstrap because I mean any front end there surely knows the name bootstrap um But that wasn't relevant at this point and then I saw this file stopping. That's like interesting So if this thing can store to a file surely it can store To to file correct and so it's just conserve namespace and that took me to another repository by the same folks Um, and basically it was a t-value store that could run on any kind of back end um And I went through the read be line by line like we all do um and I found this single line right That it could connect to cart's db now in my Task updates that have a lot of database, right and I immediately remembered couch db was jason store similar vibe to firebase I'm sure we could use that to get to get done to firebase and so Luckily, I'd already written a library that connected send data to firebase, you know for firestream and I think let's try it out and so I built a I built a library that really It's a conserved library that writes This key value store or creates a key value store to firebase. Um And uh, the cool thing is in doing so I could now connect Data hike to firebase You know the only thing that was bugging me was, you know firebase has this Limit that the value has to be at least 10 at most 10 makes so I quickly found a way to split the To split the data came more and see the p0s will be p1 p2 and so forth and so They were good to go um We'll conserve ready I could connect to data hike and data would flow in theory from data hike into conserved into my firebase adapter for conserved and into firebase it was the perfect plan and We know what it worked out So let's have a quick look see like Jamie Oliver has prepared something so I can show you what this looks like Uh, so let's just spin up a ripple Over here on the browser you can see that's the firebase through some database. Um at the moment you can see there's nothing in this point. Um But let's first first as first as per usual we include uh See data hike and then we include um data at firebase and now those are included and then um Now so let me talk you through the content very quickly so basically um All data complex are similar, right? So you specify the store which is this piece here And the store has different properties um, so the back end here is firebase And then these properties here are basically the ones that come from the firebase implementation, right so The end of the key basically tells um Data firebase which environment variable to go look Which environment variable contains the um service account credentials This allows you to read and write into into firebase. So in this case it's called fire And we'll be writing to To um firebase with the root data hike reclusion, right as you can see that's where we are here awesome, right? So that's the config now if we try and connect that config What will we get We should get um An exception because as you can see the database doesn't exist. It's now but now we can create the database and There we go. You can see the data has been stored. It's been stored in that same format from the conserved because remember underneath the data hike Conserve is doing the Writing into the store in this case the store is firebase, but you could actually have a conserved store that That writes to postgres, which I also worked on. There's one that writes to redis um And so I think that's the really cool thing about conserved because it basically means you could build Uh data hype or the atomic like database on any kind of of um acid Um database, which is really neat, right? Um, okay back to our our Back to our rebel, right? So we've created our database now we can Connect right when let's keep our connection and then As is the custom once you once you've created, you know a data data log db The first thing is to write your ito schema and that's usually your first transaction And we write up there So name and age are basically the two identities Um or the new or the properties that you have in there Um and see that yellow here means that you know it's updated I mean you'll notice it's the most still no no data. So nothing more is out There's very little data. So you'll see that you know, basically we'll just see this for the most part I mean, this is just a super long string because firebase currently doesn't support um Unreal data. So this is a base 64 representation about the structure Anyway, now we want to transact We're going to add alice bob charlie and some unknown human who is 15 years old We pop those elements in there and see the number of data that have been created again, we saw the update over here and now um We can create a data right State without that and so there we go you know, we're looking for entities um The same entity we're looking for the name and age and showing them So you'll notice the nameless 15 year old is not shown because I'm the constraint of having a name Is not met so I'm the left out. So we only see bob charlie and alice But let's say we wanted all the entities and the ages We could run another query And there we go, right? so That's really that's really, you know, simple as that. I mean it looks like feels like smells like datomic No drama there at all and um As well, you mean you can delete the database So if we delete the database that we see and look disappears from Firebase And now what happens if you try and connect let us see If you connect Doesn't exist, right? So you can see it really is simple to use Very very straightforward and I mean with Firebase um Just a dream so, um We now have phytonc, right? Um The thing I really enjoy about this is that it makes prototyping really easy because it means you can prototype Um, many different businesses on the same same Firebase instance, you know, just separate databases or even just separate routes Um, so I can actually now use Detonement can lock or not the atomic atomic a lot more often Um, I don't really have to think about the cost at least not yet. Maybe someday if I build a business that's as terrible as data atomic enterprise, but That's a problem for another day Pretty mature optimization is the root of all you know Okay, so, you know, I showed you I took you through that. Um And so this is, you know Basically where we are, you know, so add the idea of what Kafka and Firebase at the atomic and Firebase Um, and sometimes I'm actually building a startup around around around this and using both these in production Um, so far so so good collected payments got data in there got paying customers It's really great. Um Which is also really cool because you know when you do open source stuff sometimes in these ideas Not something you'd bet your money on in this case, you know, I bet the money on it and it's working out Um, so that's that's really really fulfilling. You know, the the last two years haven't been erased Um, so you may ask, okay, what's next? Well, uh The folks at uh, you know, lambda for your work on data like server. Um, so at the moment, you know data atomic sit within my application So they like so we can actually have an external server similar to postgres or You know one go whatever Um separate running is on a server on its own um as an instance And then your application actually connects to that so, um So, you know, that's really what that will be the true phyatomic. All right, so, um Over the next 12, maybe not right now. Um, so finishing some work on another Project I'm working on and once that's done. So maybe February next year going to really look on data server and how I can connect that to Firebase Because once we have that, you know, you can really have a deploy to digital ocean and there you have phyatomic running and you can connect um All the applications. There's no drama so It's that thing been my my journey with phyatomic Um, it's been one hell of a ride. I think, you know, in having this really audacious idea to to build, um You know Kafka on Firebase and the atomic on Firebase now I've learned a lot about about The workload to be honest and reliability and writing tests and I think if it belongs for these things, they're probably would be Much worse than I am um, so if you have any crazy ideas folks, you know, let out the crazy and And try them out. You'll learn a lot from the journey. I'd get burnt a few times, but you know that's life um So, yeah, so I mean here's some some some links that I think might be useful, you know the data I create for Eight up the folks are on on discord Very cool bunch of folks. Um It's very welcoming very supportive really helped me on my journey um You can check out conserve as well if you want to see a bit more about what happens Closer to the actual store um, and then some really cool videos, um One of them about Today's log in general by pthilus. It's a really great. There was the number of times that really helped we understand data log databases and then, um The video hitchhiker trees that one is I mean, it's kind of about my pay grade and really not like a computer science type, but Amazing video and really explained it well And hitchhiker trees are just phenomenal. So give that a watch and then Also stop by the lambda project site folks. They're doing really cool things In terms of data databases and making it feel available to the community and improving it. I mean they even got um They got closures together funding think last year. So definitely A great bunch of folks that's that are really driving the community forward. So give them a visit and you know If you see something interesting, you know, get a pull request contribute You know the the thing I really love about the the closure community is that You know, everyone pulls their weight um, so you know Looking at you to pull your weight too um And if there's anything you take away from from this talk, it's this Any application that can be written on Firebase Will eventually be written on Firebase. So Why resist start now? Uh, that's all from me. Um, thank you reclosure for having me. It's been great. Um, and uh, I'll catch you online Cheers folks