 Hi, this is your something party and we are here at the open source summit in Bilbao spending today We have with us Florian Billy the staff data engineer at back market and also made dinner of Delta RS Florian is ready to have you on the show Thank you for having me swap. Let's talk about Delta RS let's talk about the larger project the larger open source project Which is about all about rust about Delta Lake and then we'll talk about your involvement I can introduce a bit what is Delta Lake So Delta Lake is part of the Linux foundations it was fully open source as a Technologies that provides Reliability performance and quality on data lake So it's it's kind of solutions that break down walls between data warehouse and data lake So my project was more about Delta RS Which is an application the software written in rust Integrated into the Delta Lake ecosystem. You are the maintainer of Delta RS talk a bit about When the project was contributed to the Linux foundation How it came to exist? Yeah, it was a long time ago. I think we have like primary needs around how we can integrate To the Delta Lake ecosystem without having python dendings or Ability to access the Delta tables and nothing was there to to like a really light well solution for for us That's why we start to look over different libraries and we found Delta RS, which was created by script In rust and it was really interesting because they have like really a small layout to connect with python and it was Since then that's back market join to and incorporate and to improve the binding To connect more into the python ecosystem with Delta Lake with a simple efficient and reliable Library, so it was one layer with python connected to one layer with rust and Back market joined the effort by improving these python bindings and since then we contribute a lot to the open source and Right after Delta Lake all the ecosystem Was under the next foundation to to drive innovation on At industry wise around that had driven and data ecosystems. I want to talk about rust language Talking about language kind of is a sensitive topic. You know, it also become kind of religious also I want to avoid all of that and also even if you talk about the hey technical advantages technical benefits That also becomes tricky, you know, but since you are involved with of course Delta RS rust And we have been we have seen a lot of you know adoption a lot of developers are moving towards rust As I said, everybody has different reasons for going to a specific language But but in general talk about what are the advantages? What are the benefits that you see modern developers see with the rust that they are moving towards it? Yeah, so it's really good questions. I think according to the last survey of Stack Overflow Rust has been the for the fifth consecutive years the most loved programming language And I think rust has worn the heart of developers And I think the main reason why is because you provide a very efficient and allowable programming language without Promising the features of high-level abstraction API and I think when you are starting to Think about performances or a label system You can think about CC plus plus is kind of low level for performance reasons But you often encoders the same pitfall like memory management data risk collision Segmentation fault all these kind of issues that really complex Complexifier beats the developer experience and we rusts by having this mechanism of ownership or these kind different features you can have the efficiency and Also high-level API abstraction, and I think it's really something nice when you debug We are new product system to have a language that are created for developers And I'm when I'm thinking rust I've been thinking about other tools like cargo wiki Which is a package manager first is really helpful to and convenient if you want to test to To to publish to build your your crate, which is the artifact to build by its rust I think it's really helpful to have this and the compiler is really friendly And it's like a mentor for us because you could help to say hey you have an issue there Hey, maybe you can improve your code base by doing so. So it's really a nice language, and I think right now for my personal usage It's also we are thinking about the energy consumption of the language, and I think rust was the second after see on if efficiency programming across different programming language and I think we we need to take this into consideration when we are building systems to be more Energy aware about the impact in regarding the carbon emission and so on so for me. It's also the best the best compromise between performances reliability high-level abstraction without consuming to not too much resources Let me talk about energy consumption of language a lot of people they don't understand the importance and impact So so talk a bit about When you talk about energy consumption by a certain language What is the scope of that because it's not just you know the the system running on your laptop It's also about the data centers and it goes beyond that also so talk about What do you mean when you say you know that this language is more you can say climate change friendly or you know more eco-friendly Yeah, I think it's you are totally right when you are thinking about Software you have to think about the software life cycle. It means you develop on your local Laptop and it's depending where you are because it's not around building the system But it's running into a specific location It can be in location and where the carbon emission is really low or it can be really high So it depends where you are it's the same process when you will publish or going to the CI your continuous Integration mode that we do build tests in the cloud environment So again, you consume energy and at the end when you publish something and it makes it available for everyone It's depending on where the software is launched So it can be on cloud providers that take Consideration into the carbon emission emitted or not so it's not depending But when you are thinking about the energy conception You think about large use case where your library will be used everywhere by a lot of you of software and they will Trigger download and use every time your code base So if you have a slight way of improve a bit the code base You can have a drastic impact on the energy content conception and right now I feel like we have a lot of power in our hands, you know Not only on for the cost, but we can launch the GPU CPU really easily on different cloud providers So having this kind of mental representation on the impact could help a bit And I think we need to take into consideration that with Active facial intelligence and other topics. You can imagine the workload Required for memory CPU network exchanges and so on. So this is kind of things that I'm on my personal side I am interested in to knowing more of the impact, but I think we need also to take some time and consideration into this kind of Key key key part. It's hard to really predict What can adoption what what what the future looks like for a specific language five years from now because the COVID has taught us, you know And these things change very frequently. But if you look at the trends, you know that what do you see where rust is heading? What kind of adoption you're seeing will be there? Just just talk about that. Yeah, I think rust has been as I said before really a most loved programming language and I see the usage in Particular in my ecosystem in my work around data I think a lot of library was rewritten or improved by Incorporating rest into the Python bindings to improve the performance and the efficiency And I think you have a lot of open source libraries to help you to improve this data processing data ingestion and so on like Apache arrow Which you can the standard format colon oriented in in in rest So you can leverage this kind of library really easily That's why I thinking about rust in the data ecosystem will have a huge impact You can think about web assembly, which is a technology that can run on the browser side and make it compatible with JavaScript So you have a large area of improvements there and the last part will be more on the generative eye with a large language models right now a lot of of Software is built with Python And I think we can have an improvement to integrate rust into this kind of Processing to reduce the inference time to reduce the memory and CPU conception for for this for this model So the AI will have a huge impact in the years to come So I think rust as a as a good part to to take to to be in the in this in this landscape and the last part will be more about Bindings I would say because rust is really efficient to create bridge between the programming language So I'm seeing like a growth of a huge factor of growth to integrate with others to say, okay I want to learn a rust so maybe I create bindings and I start to contribute to rusty actually since you're talking about people who wants to learn and contribute I'm looking for a kind of piece of advice for them, but from different perspective one is that Why they should choose rust, you know, that's once again very very loaded a caution But it goes back to the first point that you made about the advantages of that because sometimes folks who are like in a very early Journey, you know, then they they look at these languages and they like because it's a big investment You know, you have to learn that is number one and then as they do embrace this language What is your advice so not only they they they do the adoption correct But they also become a kind of contributing member to the community as well. Yeah, I saw I saw your point I think the most important part about rust is the learning curve is sometimes pretty hard because you have like a new Ways of thinking I'm thinking about the ownership, which is a key in a rust So you have to learn it So I would say a real welcome man to start small to have a simple use cases where you can learn the language The community of open source in rust is really how some people are really benevolent and if They know if it's pretty hard to to start to contribute to to rust So that's why they are really welcome and benevolent to to share the knowledge And I think it's really important one when you need to to learn something on on why choose rust I don't have a really region things to say you need to trust rust be or instead of choosing go or somewhere else My advice will be after learning a rust. I feel like I am more I Learn a lot more about my code and I code better right now. So I'm still Really, I really loved to to work and to learn a lot of different programming languages to learn Concepts and to improve myself and read rust I learned a lot of things around any development and you change a bit the way and I'm developing right now I will advise to start small take a project is a rust Programming language the online the book is really up full with really well defined and you have all the documentation You can follow the comprehensive rust it was created on GitHub by the Google team on mobile and I'm thinking as well about Tick Manara which is the leader of accelerant learning and it's a good place to learn because they have like a really up full Course and training on rust. So again, I fall like I fall in love with the language But it was really because I started to to see the power and the and the reliability in my past I used quite often see C++ and create binding and all the time It was on debugging on ways on findings The right pointers where it falls the memory management and so on Having rust to take care of it is really great and on top having the mechanism or the the concept of high-level language is really nice because you have the best of both so all High-level API and also good efficiency on on the low-level operation