 Thank you guys for staying this late a very good evening today. We'll be talking about a new database for Android How many of you are using SQL light in your projects? Everyone right so today we'll see some of the alternatives We'll talk about what are the problems with SQL lights will see ORMs and other things and we'll see realm and how it can help you Good evening. I'm Pranay. I work at Intude as a Android developer and been doing Android from four years I also lead the Bangalore Android developer group as one of the organizers So let's start with the joke right so I am sure most of you have heard this So two DBAs my SQL DBAs walks into a no-SQL bar and a minute later They come out why because they cannot find a table and you will not find a table in string So so why do we need a new database all together right? So let's talk about a typical app scenario So you have an app an app will have lots of data Now you want that data to be persist to work offline right you want your app to work offline That means you have to persist the data some way and Only way for now which you can think of is SQL light right that means a lot of tables as your data grows The deep the tables will grow and as you're growing your app you realize that you have duplicate data What do you do in case of duplicate data? You normalize it normalization leads to more tables, right? And as your app is growing and growing you find that there are more many-to-many relationship What do you do then you again create more tables to make to maintain that relationships, right? You have that meta data of many-to-many relationships and now you have this many tables How do you implement in SQL light? This is an actual code of SQL light, right? There is literally a thousand line just boilerplate code to initialize SQL light and everything in your database Lot of strings for single table. So this is just one table and you can see that for each row each column You're defining a string. So let's say if you have more than ten tables your file will grow more than thousand lines Just for defining the table strings tightly couple schemas. So how many of you have used that? SQL light when Update so when you update your schema, you have to take care of all your migrations and that migrations API is not that good And raw SQL queries Those are very very bad So let's take an example, right? So you want to you have couple of tables You want to get data from one table join it from another table. So you use joints, right? So take an example of this query So you want to select two columns from one table join it on another table with the primary and foreign key and then join It with a third table This is typical SQL statement. Now, how do you write in Android or how do you implement this whole thing in SQL light? You create it using a query builder or you write it like this a non-fancy way, right? So I'll say a query you'll write it in select then table names column names and all your data and By any mistake if you forget a single space Right, you'll get an exception and it will feel like breaking your laptops and which happened quite a number of times to me as well So then what's the solution? Let's abstract the problem, right? Why don't we use an ORM? So how many of you are using an ORM? Quite quite few that's so wrong, right? So ORMs are there to help you This are some of the ORMs available for Android then list goes on. I've listed just eight I think right couple of couple of them are by Indian developers sugar or M is by satyan What does ORM gives you? So ORM abstracts all this fancy boilerplate code Which you have in SQL light which you have seen the thousand lines of code It will just abstract away with simple implementation. You can use an ORM and it will simplify all your queries No direct dealing with SQL. So no more select star and everything the one which we have seen It gives you an object mapping. So people come from Java world loves object and what if if you're Table or whole data schema is represented as an object. You will love it So no more querying directly but querying the objects But then if ORMs are so good. Why are not everyone is using it? So there are some problems with ORM which comes Even though they are easy to set up ORM is a layer on the top of SQL light They are inherently slow Most of the ORMs are based on use reflections and The read and write is slow compared to raw SQL and it depends on which kind of ORM you're using There are certain ORMs who have optimized this further using annotation processing and they are pretty far and we'll see some of the benchmarking Even though I said that it's you deal with object ORMs are not entirely SQL free for bulk operations You still need to write queries and Since underlying database is SQL light you get all the problems which are associated with relational databases So what is realm? So realm is a mobile database designed from the ground up for replacing SQL light and code data in iOS It's a object data store. So realm is based on a single principle Zero copy object store. So let's see both the things one by one In realm there is no concept of tables, but everything is an object So let's say if you have a person table in your database schema and realm it will be a personal object similarly Let's say if you want to query Right in typical SQL worlds you do a join But in realm you can treat each object is a node and it can be as easy as graph traversal So you can just say realm.getA.getB and getC simple no joints and other things There's also very interesting concept of zero copy in realm So as I said zero copy object store. So what is zero copy? So how many of you have used cursors? Right, so cursor is precisely based on zero copy principle where when you load When you query the SQL light it gives you a cursor But it doesn't copy the data into the memory it just holds the reference of it and when you say cursor.getString or when you move to that cursor It's actually go and fetches the data from the SQL light realm is based on the same principle So when you do a get on realm object it gives you a pointer to that memory location It's actually not copying that data So if you see this example, I'm not sure if it is visible in the back when you are using an ORM, right? Let's say you are using sugar When you do a read query what sugar will do is it will read all the data from the database and Create a memory it create an object in memory of that data. So you have a replica of data However, when you do the similar things in realm, it will just have a reference It will have a proxy object which is referring to that data It introduced its own problems It will talk about but with this and we'll see the benchmarking the read in realm is very fast And when I say very fast, it's less than a millisecond for thousands of objects Because it's actually not doing anything. It's just pointing you to that location. So let's take an example So typically all the realm objects are standard Java objects, right? So when you query you get the results if you try to compare it's all lazy loading, right? So when you try to compare the objects those are different, but if you try to compare the data, it is safe Right, so it is not actually copying the whole object from into the memory But it's just pointing it and at the time you need it it goes and creates that option So what are some of the features? So realm is designed for mobile from ground up. It is very easy to set up It has a common C++ core. So the entire thing is based on a C++ core That's why it is portable across both the platform. So you can use the same realm data across Android across iOS Only thing you have to take care of the schema should not be different, right? It is a column oriented. The data structure which they are following is B++. It is very very fast They have optimized it for read compared to write and we'll see that in the benchmarking that write Bulk write in realms are slow and the query interface is very very simple You can chain queries together and we'll see that as So enough talk. So how is realm compared to other ORMs and other databases like SQLite, right? So we will see two benchmarking. One is done by realm team and one I have done Write and see how it is compared to the other platforms So this is a benchmarking of realm again SQLite, raw SQLite And as you can see realm is the one which is in the red And yeah and here the higher is better So if you see here the batch write the only place where realm is just as good as SQLite is batch write I even you're writing a lot of objects into the memory because they have not optimized it for write But for other things like simple query a sum or a count realm is really really fast And they have tested it in 1000 objects. Now what I have done is I have created I have taken a benchmarking solution from DB flow a SQL ORM and added realm on that And I have done a simple operation of 100,000 objects read and write If you see here, if you see here the first thing is realm Then there are DB flow, green da, ORM like this are again different kinds of ORM available In case of write realm is comparatively slow than DB flow Because it doesn't optimize for bulk write and take the data in the pinch of salt But if you see read it's just 16 millisecond that too because I am using an event bus It takes time from the way I have published the event and the subscribe That's why you are even seeing a 16 millisecond gap there If you combine this realm is really fast in read but it is somewhere not optimized for bulk write This is bulk write however for simple write it's very very fast again So let's see some code so realm is very simple to add It's just a graded dependency you have to add and it's pretty stable now for android it's 8.6 86.86 just add compile statement and the way in which you get the instance of realm is Just say realm.getInstance anywhere in your app So realm uses a reference counting in that side so you can call getInstance anytime from anywhere You don't have to create a single instance if you want you can use that in your activities And just call close whenever your activity lifecycle is closing And they will take care of rest of the things of releasing them Now what's the method size and this is one of the disadvantage of realm as well So realm is adding 1750 method counts into your app that is not that freaky compare That's large compared to libraries like Picasso but still manageable But it adds a fat size of 730 kb in your apk and specially in India right where the app sizes are important This is one factor which prevents lot of people to use realm But why does it do that as I said it is based on a common C++ score They supply SO files for different platforms so like they will do it for arm64 They will do it for maps and x86 So the install size will be less compared to the size you are shipping the apk But if you want to reduce that size further you can do a multiple apk for different platforms It is little painful process to test it but very simple to configure in your gradle file You can say that generate multiple apk and gradle will automatically take care of picking the correct file That will reduce the size comparatively So let's see how you can create a realm object So just like a normal Java object you just define a pojo The only thing which you need to take care of is extend this pojo with something called as realm object Realm uses annotation processing So the moment you compile the app it will take this whole thing and convert into a realm proxy object And they do their own optimization on the top of it This adds another limitations that you cannot implement multiple interfaces You cannot do your own hash code and other things Your getters and setters should be clean You cannot add additional logic it will strip it out But it gives them very fine control on what your object is doing And very simple schema where you have a normal Java object and you can do anything with it Now realm support different things like primary key, your indexing and stuff like that You can define a field as a primary key and it will become a primary key So how do you write a data? Just define an object, call the setters And then you have to wrap the whole thing into a transaction Because realm guarantees asset properties you have to roll it in You have to make it into a transaction block What you need to do is just call the object, set it Just say copy to realm and just say commit transaction Now copy to realm is required So you have created an object as a Java kind of object You have said new person This object is not pure Java object it's a proxy object for realm So you need to convert that There are multiple ways to convert it Either you use new and then do copy to realm Or you can ask them to create an instance for you Your wish So very simple if you can see just define an object Call the setters and just say commit that's it Realm also gives you an asynchronous block if you don't want to do this So let's say in this case if something happens You have to cancel the transaction on your own If you don't want to write this much boilerplate All you need to do is just give you a sync block Just call inside that execute method just call your setters And it will take care of committing it And in case any exception occurs cancelling the transaction So even reducing your work of manually handling the cancellation So what does realm object supports? As I said it supports your primary key It supports all kind of primitive data types like int long It supports box type like long string Boolean It supports null value for box type data ways Data types and it also supports indexing So for now there are limitations of it but basic things are all covered There is no concept of foreign keys in realm There is a concept of relationship you will see But there is no concept of foreign keys How to query the data as I said earlier All you can do is take the object and say find all So if you see here what you say is realm dot where You define which class because which object you want to query So you say user class you can define certain parameters Where if you want to have certain conditions like the name should be John or name should be Peter you can combine those operations Everything what you used to do in SQL world everything is possible in doing in realm You can batch all the query at once in a fluent interface So all you are saying is realm dot where user dot class Find me all the users whose name are equal to John Or users whose name are equal to Peter and that's it It will give you a list of all those users again This is a zero copy read right so you will get the results Instantly without undue delay and when you try to access it At that time it goes and fetch the object from them So let's see some more examples it supports a lot of Lot of querying types so you can say that you want Age which is greater than eight so assuming that your age is an integer type Right it will do that you can also say that You can also change the queries so if you see on the top You have found that from the dog find all the dogs which has Age more than eight now you get the result you can again query on the result You don't have to go again because it's just a reference so then you are saying that Okay from that query return me all the dogs whose names are Alex Right so you can change queries together and the interface Is very very fluent so what does query data supports So it supports all the conditions like greater than less than It supports modified so let's say if you are finding finding That give me all dogs whose names are john you can say that okay john But ignore the cases so I'm adding a modifier where you can ignore whether it's a capital John or small John or camel case anything it supports logical Operations or an and we have seen that here so you can say or an and here It supports sorting directly so you don't have to implement Your own sort you can say whether you want to sort it based on ascending or descending and can Work over strings in long everything Chaining queries we have seen that you can change the queries it also supports aggregate Operations so you can say that okay if you have an invoicing app let's say you have line Items you can say that okay give me all the line items but give me the sum so You can just call some and it will some all the things and give it the results directly to you So you don't have to do a lot of these things on your own it will directly Supports from the database it supports both async and sync operations So you can create and sink and async This is also very interesting concept and realm for relationships Though there is no concept of primary or there is no concept of foreign keys The realm supports relationships so you can have a realm object inside a realm object Or you can have the same object inside each each other Right though it creates a cyclic dependency but let's say you are Facebook right and you have a database where you want A person object and you want to have friends but friends are also Person right so you can have a list of persons as a friend inside A person object take this example so you Have created a class called as email had some attributes you can create a Class called as contacts and have email as Again an attribute so you are having a realm attribute as an attribute You are having a relationship of email with your contacts Then how do you query this very simple You can create like you can say that find all the contacts where email Dot active is true so you will fetch emails then dot Active means the field which you are accessing and then you can say that what value So the power is amazing right you can combine everything Together you can change the queries you can do whatever you want to do All with very very fast performances so What are some of the other features of realm so realm supports Jason directly so you don't have to convert so in typical What people do is they convert their json through json into An intermediate object and then call setters over all their Orm objects and then supports directly mapping Of json into the database so you can get a json and Call save and it will save all the fields Excluding anything which is not in the scheme right so if you have json has more Elements it can exclude those there is a how many of you love Concept in content providers where it notifies you if you You put a cursor observer content observers right Same things are missing in lot of orm if it doesn't give you a cursor Content provider realm supports something called as notifications where you can just Write for a database change notification and you will automatically be notified Regardless of your data being changed in whichever activities Right so you have two activities you register for a realm listener Right a database change you go to a second activity modify the content You come back you can register for that listener it will notify you that there is a Data change right and then you can update your UI so you don't have to write All those boilerplate codes encryption supports a lot of As like banking needs an encryption support in built on the on The device itself so a typical solution is you use SQL cipher or if you are using SQL light you write your own encryption logic Make before saving you make sure that you are encrypting all the data Before retrieving you make sure you decrypt all the data and the whole operation Is slow realm is supporting encryption and decryption by default In the database built in so you can use a 512 key I think to encrypt it so when you create an instance of realm For the first time you can pass those parameters and it will make sure that the data which is Being stored is encrypted it supports adapters so People who love list views recycler views there is an adapter directly to bind those views To realm and as I said it's a cross platform so It supports the same data can be migrated to iOS as well I don't know the Use cases but yeah it just supports because the core is common And they have a robust third party addons so it's an open source The core is not open source but they have a pretty good Interfaces and exposed so there are people who have built a lot of addons on the top of it Little less for android but a lot of addons for iOS So yeah all the good parts and what was the catch So there are certain limitations of realm you should know about before deciding Whether you want to use that in your project I lied to you it's not entirely a no sequel database it's an object database Each object is a tightly coupled schema so if you are changing An object structure after releasing your app you Have to provide a migration script they have a very good support For migration but still you have to write some code to do that The core is the C++ core which I talked About is currently closed source they have promised to open it under A partial license but still there are speculations on the dates and people have Speculations on when that will happen or whether it will happen or not So people who like open source more might not find it interesting And as I said the concept of zero copy though it Looks great it has some other disadvantages because The objects which are getting it's just a reference it's a mutable It's not a threat safe object so that's one of the problems Then realm objects cannot be passed between two threads So what that means is if you are querying a realm on your main On a background thread you cannot use the results on the main thread Typical use cases is like you query For certain content in the background thread and then you want to tie it with an adapter on the main thread That will not work but you can query the data on the main thread Because it is so fast you will get an ANR There is some support for coming for RX into this I think they have released it yesterday I am not aware of it so far The writes are slower compared to read and it doesn't Okay it doesn't support auto incrementing So if you have primary key concept where you want auto increment the key automatically At the time of inserting the data it doesn't support it but it's fairly easy to write it on your own And of course the size of the apk If you are too worried about the size of the apk being added because it adds 0.8 mb Realm is not for you They have some of the additional things like realm browser It's a mac utility so you can pull the realm files Because realm is just a file which is getting stored on your So by default the directory is contest.getCacheDial So you can use from you can pull the data from there To browse the data and see for any transparency Anything which you want to see. There is also a steto plugin available for that Where you don't have to pull it manually and you can use steto to use this So who else is using realm and is it production ready So realm is backed by ycom There are couple of big apps which are using realm both on android and ios Some of them are dustmash So you can use it on your own So that would be pretty much all for me Thanks Thank you that was pretty detailed You told about the zero copy concept and when objects or the rows are red They are not copied into the memory. So I am assuming the data is saved on the secondary memory Which is private to the app So in this query the references to the physical memory Address of those objects of those rows is taken to you And it goes through them one by one So every time you are trying to get fetch a row it's going to the secondary memory Pulling it and giving it to you Not kind of it's not fetching it. So you have the pointers for that memory Every time you want to display that row when you say Get, the time you say get it goes there and pick the data So I think the only major advantage we will have is in case of Let's say list which is displaying 200 rows instead of fetching all those 200 Getting into the ram and paging through it manually Realm will only get let's say 5 Display 5 and then how you do pagination That will have a significant impact on improvements So if you see right apart from memory, apart from read and write performance The interfaces are very clean compared to sqlite and compared to Lot of ORMs. It also gives you some of the inbuilt methods which are not There in ORMs, not there in sqlite if you want to have like encryption So there are certain advantages on that side, certain advantages of that side And in total it's a good thing to try out in case it suits your needs Where you don't have a tightly coupled schema as well like It is not schema less but still better than SQL Because the size is like quite a disappointment 780kb So if you're not already providing a lot of good things, it's hard to justify the size So as your app grows it becomes tough to replace A lot of things in later point, so thank you Hello So the sqlite advantage was like I can pre-fill the data and ship it For the app, can I do it? Yeah, definitely So you saw that I showed you that realm browser You can modify the realm file and supply it with your data So I can ship it the pre-filled data That was one of the concerns which we had which Sugar ORM or something like that. So one of the things which I've seen with doing those things Like I'm doing it for one of my apps I've seen one plus two phones failing miserably On copying those data base from your asset Folders to back into the memory. So you need to be little caution about it But yeah, they have the support for supplying the realm files because it's just a flat File, so you can supply it I have a question here Have you evaluated any other no sqlite DB for android? Yes, definitely The DB lite, pretty powerful database Entirely no sqlite, it's just a lite version of Cows DB on your phones It makes sense to use it if you're using a Cows DB on your backend It also gives you a nice fancy sync feature where your data can Automatically sync between your servers and your clients Hello So it supports on iOS both Swift and Objective And you can see that more people have used it on iOS The first version of realm which came out was they have very recently Released an android support, initially they launched it with iOS only So they have pretty solid architecture for iOS You can go and read more about it and replace this code data very very Awesomely, that is what I have heard What all encryption mechanism does Rian support like, we use sql cipher for normal sqlite And what all it has to encrypt our data So I have not done the encryption but what I Read from the documentation is they support some kind of encryption where you can pass a 512 Size key and some GPG key which we use For sql cipher, I think so So we need to encrypt our data, we need to put our data and encrypt it You don't have to do anything, so when you call that get instance you have seen right There is a overloaded function there, overloaded method there You can pass the key and it will take Care of encrypting and decrypting it automatically You don't have to write a single line of code, so if you pick that data, you pick that realm file and use it in realm browser You will see all the encrypted data But no one tries to access like the realm table So how easy it is So it is same like sqlite, it will be stored on your apps memory only So nobody else should be allowed, if you have a rooted phone then anyone can pick that file But because the content will be encrypted it might be useless Hello, can I get the json back from the Object store So you can use json with tying it with the object to get that So there is a little tweak which you need to do To work it with json, but you can directly get the json out of it Does this realm handle it or still you have to handle that like It is kind of handle it and it is little complicated there So as I said the realm objects are not thread saves So you cannot share two objects with two threads So on each thread you have to create a new instance of realm So what it will do is it will lock the data for you So if you are trying to insert a data It will lock that and on the same time if you are trying to read it it will create a copy of it So that way they maintain the consistency But that adds up to unnecessary copies of data And creates a little more size, but they try to purge it when they do again a write And there is one more thing like We tried one sample in realm and there is one thing like We are getting some data from the database back on the main thread So like if we are having a lot of database It is not yena, but in the log there is like 31 frames skipping My doubt is like it may create problem while doing animations So is there anything like So the loaders do something on The background thread, but actually it can give you on main thread No, so for now I am not aware of it, I have heard in the morning from Anoop that They started supporting this thing for Rx, but the way in which I am doing this Is in case I know that I have lot of data to fetch I fetch it in a background thread, convert, now loop through that data Put it back into an object which can be shared on the main That even we thought of then, but actually feel like these validation Whatever you have given is not actually true Because there is more boilerplate for GC to clean up So I think it is a very well known Issues for them that it does not work across the thread and very basic example was for ListView where typically you do a get on a background thread And then pass this whole thing to an adapter, an adapter will always be on a main thread It does not work. And as you said like realm has their own Adapter, so does that handle that thing for you Or will they do it on the UI only? They will do it on So see the adapter is just for you to access that realm objects Properly, ultimately you have to pass the results To the adapter, the results you will fetch it on a thread So it depends, if you fetch it on a background thread you will get an exception That realm objects cannot be passed across threads So you said that realm supports Importing from JSON, so in case of relationships Is there some format that is that I must Implement or can I tweak the way in which it forms So only condition for relationship is that anything Which you are adding which is non primitive should extend realm models So if I have added emails in the contacts Email should extend objects, otherwise if you are defining an arbitrary class Objects it will change. But what about let's say there is Person and we have email as a relationship So while importing what should be the Structure so that realm imports the relationship So it should be a recursive nested structure So can I make it work with some other structure for example Let's say on the back end it has a different kind of Different kind of database and they are comfortable sending An idea of person inside an email object instead of a nested thing So can I make it work with that or will it be very slow in insertions I don't know about it but if I take a guess You have to write a lot of code to do that. So basically you cannot directly convert An email object, you need to have an intermediate object with JSON And then copy your data like same So I mean is realm import doing some A lot of optimization while importing from JSON or is it very trivial code Which even if I write will be as efficient as it is But I think they are doing a lot of stuff Practically they will convert the whole thing without you Object So when we read an object They have pointed for only that object for each entry They will have pointer and when we read this property They will get the information from I think they have it for each object not for entry Because each object is represented as in the file some way So we will get it and then So that is just for Doing the 0 copy thing or after play We will get the data and doing some data after that It is a cheat so it is just a 0 copy But if you see taking an example what Aman gave like in case of a list At max you might be displaying 5 or 6 But you might have fetched all 30, 40 in the memory In case of realm you will only fetch that So that way it is a kind of cheat and it is not So that is why I said take the data on the pinch of salt Because you can see that clearly certain functions like batch write DB flow and green doubts clearly bring pretty fast operations So in case of write it is writing the object in the memory So they are not optimized it for write Because they have an intermediate proxy class When you extend the your pojo with a realm object They convert a proxy object use some annotation processing Thing to convert into the format they want So they have an overhead of converting that whole thing into their own objects No self nothing else and Works completely offline Since it provides the Java representation of objects Basically and there is a high tendency that as developers Kind of we go and put a lot of interdependencies like An object including another object and so on Like a kind of a tangle tree or something like that So one thing is about how do is there any Recommendation in terms of the design for the whole thing Because it's a free form and there is a tendency to go To j wire that is one thing and as when the dependence Is increased right how would be the performance of the system Overall So there is no recommended way of how do you design the object It depends on totally on how you want to maintain the relationships And comes to the same question when you design a sequel How do you design a table so totally up to your database Schemas on how you want the schema to be Now it will definitely affect the performance because right it's slow If you have too many relationships into a single Let's say person you have a user you have a address object And couple of things each of them will take the time But the read will be very fast again because it's just having a pointer And only access the data when it's saved Even though you have six or seven different kind of objects inside One single object you are just getting a pointer to those locations And you are not entirely copying the whole thing into the memory So there were questions about thread support in Realm So I have been using it and I can just confirm that since 0.84 They have support for background syncing and background fetching So they have functions like find all a sync So since 0.84 they have this function And they are notified through Realm listeners That they have so you can use background fetching Awesome, thanks Thank you, so if you are more interested in like Diving more deeper than what I have told you into the Realm Like how they have implemented 0 copy and all There is a nice talk by this guy called as Christian He is an Android developer at Realm He has given that talk in advice.com New York this year So just search for his name and you should find it Thanks