 There you go Okay, today. I'm talking about modularity and dependence injection at scale So this is mostly based on the Twitter Android app Like I I'll try to elaborate anything that goes into detail about Android But mostly it's about dependence injection and the things that we've dealt with over the years So to start off This isn't me. This is like nacho and Cesar who's been working on it for the last two years I'm sound. I'm Software engineer at Twitter working on Android mostly focused on performance right now So I work closely with Cesar and nacho for any architectural changes Okay, so to start off. Let's dive in a little bit and like see like what we were doing over the last few years So Twitter is a really old app. So Because there isn't that many audience and sort of guessing let me just show you this is our first commit So as you can see we've been at it for about a decade This is just the Android app and we've started before this happen first initial commit, obviously So during that time a lot of the things were different. I Believe Android Froyo was out. Everybody was using Eclipse for Android. It was an ant build platform Usually everybody had one module. So did we there was a simpler app. It wasn't that many features But as we went forward we extended we you know switch to great Oh, we started dividing up the UI and the business logic and The more we worked on it the more we realized that it provided more benefit. So we started splitting it up even further So we split up The UI from the features itself right like so we went feature wise Then we even started splitting up what we had under the covers for the business logic to be more generic versus anything That was Twitter related We went even further and the started splitting up each one of the features says we As new teams on boarded we were getting more and more engineers working on different features They wanted to maintain their own module. They didn't want to go forward, right? Then we realized the benefits of that and we started splitting up all the lower-level business logic I'm going by this fast just to because it's a brief history, but Stop me if you guys have questions We split up even further we decided to split up our business logic that Twitter related into like subsystems that makes sense and like the infrastructure and Essentially we ended up with an app There was a lot of different modules that support each other to build then to our app at the very top We had the big module which was the app itself and then the features then the infrastructure subsystems and finally the corollaging as all of this was happening we acquired periscope and As we welcome the periscope team, we wanted to give them what we have learned over the years and we wanted to build Help them build their Android app with the same features that we had So let's see what happens. So we started off with the Android app which was this very modular is ab at this point of time and We found that it's actually rather easy at this point right because we could share some of the core logic that was in Twitter related And we can build periscope on top of that we actually took it a slightly further and a team started building basically their sandbox so they can only work on the feature that They want and they don't have to build the entire app. So this Drastically help them with build times right like because you don't have to build every different feature of the app and all of that And it actually it lets you iterate very quickly and like test internally So I know it this was a lot But essentially the core information right here is that we split it up into multiple modules that can be individually tested and all Of that and we essentially built all of this on top of dependence injection Right. So the dependence injection that we used for this was dagger daggers a very common one But before we go down that path, let's just quickly go over why dependence injection really helped us on this, right? So it it's important It helped us we use ability a lot as you saw like that we can use between periscope and Twitter it helped us with testability It helped us with like correctness because we don't have to share like copycode and like could they stick to it And they also helped us a lot with refactoring. We did a lot of refactoring over the years, right? So it's important to have dependency injection for us like and it's actually important for anybody No matter the size of the project that you're working on For us like I mentioned before we use dagger and like it's one of the common ones that's used in the Android world for any mid-sized to large-sized apps and we started using it and we learned a few things along the way right and That's kind of what I want to share today. I was like what we learned and how did we handle some of these things? So in general it helped us a lot with static checks, right? Anything that errors at compile time is way better than seeing it at user level So we want to see something like this more often than like a user complaining to us and even though this annoying It's much better It helps us with code generation it takes care of a lot of the boring code that we're doing right like Here's a walkthrough so we can have Something like this so ignore the code it really doesn't matter, but there's these are modules that depend on each other So each one of them have their own constructor and as the constructor changes we have to update every place that's being used Instead if we could use dependence injection we could have something like this which essentially just Adds this annotation that says hey, this is the constructor that needs to get injected And then we if we change it we don't have to worry about it the generated code kind of takes care of it, right? It also gives us a graph architecture, right? so and this graph architecture allowed us to really scope out and Like figure out like how can we? Split up the object graphs themselves right between the features between all of that One thing that I want to note is that every time I talk about object graph if anybody's familiar with dagger We're basically talking about components and subcomponents There's there's differences between them and I wouldn't go into the details of it, but in general they act similarly So let's see what we had like so I talked a little bit about object graph that we built So in terms of the Android app So we wanted to start at the very top level and like scope out the different graphs that we have So the first one that comes that's the most obvious case is that whole application scope So this is basically singletons that we have in their app That spans across the whole lifetime of the app that as long as it's live following that we have user scope So these are essentially You can think of like like user level databases and like user level logging like stuff like that So these are scoped by the user as a user logs in and logs out and or switches accounts Like we want to go to the correct one from that from there We move forward to like these things that we call retained scopes so in Android what happens is that every screen that you see is essentially an activity and this activity can Be destroyed and recreated as configuration of your phone changes and this could be simple thing as like rotating your screen Right, so we wanted to have some things that are as you're seeing the screen There are things that we need to maintain so we put that in the retained scope and then the Activities themselves. We actually tied them to the view scope So this allowed us to have some some things that can outlive the activities themselves But as long as the screens on view they were Provided and like shared under the retained scope. So This basically maps to these four different scopes that we have throughout our app. So we have application We have user we have retained and we have view Let's dive into a little bit more, right? so if we see Without going full detail, obviously the application scope in the dagger we would define something like this, right? So we This is in Kotlin like we defined that this interface for our object graph. We say this is Application scope and the module that it relates to that will provide the bindings and then from there We basically say that these are the user level Objects that we want to build right the user graph in the user graph. We do the same thing We say this is a user scope and from there we say hey, okay, like from here. We can build anything that's retained scope related, right? so From here like this will be feature level things that we're calling and then same thing in the retained scope as We build it out we have We we build out the view the view graph underneath the covers and then like the view graph is the final one that We have for our four different scopes So these are the four scopes that we had right like the application retained and view scope one thing They will notice that we ended up going with component for the top code for the the top level scope and then subcomponents for the under level scope I mentioned previously there are subtle differences and like one of the reasons for going down this path was essentially the sub graphs need needs access to the the top level graphs right like because it's everything that the application holds even though You're looking at the activity or the view you might need something. That's an application level So we decided to go with subcomponents These are things that a lot of the large companies deal with and they'll build their own set of like Tooling around it to deal with we just decided to use it this way all right, so I've dumped a lot of information about how we're scoping things and all of that This wasn't the main part of it The reason I went through it very quickly is because these are things that you have to deal with as you introduce dagger into your system You have to decide all of these now There are certain things that we ran into and like over time like we decided like we had to like adjust and find things to fix Or not necessarily fixed to basically work around and like get a better at it So I'll start off with a really easy one right so we have a lot of engineers who works on our coast code base and one of the common problems that we have is How dagger like how you instantiate the dagger graph so typically it looks something like this So in this particular case you'll you'll in your application level You're gonna create the main graph right and this particular command is actually the generated code So anybody who's building the application for the very first time Runs into this problem where it doesn't know what this class is because you have to first generate the classes and then the you can You can reference it in like your application So we built a very small helper function and we changed it to something like this right so in this case All we're doing we're just with the helper function that we have for dagger We're just saying hey, this is the application graph that we're generating. Let's look into it Let's see what it looks like so essentially the helper function has the ability to build this graph right and You may be thinking that hey like if you do it this way, it's still fine You're still gonna end up using it so we we use a little bit of reflection I know it's a it's a little costly, but we're doing it only once Right, so we decided to just like use the reflection only in this one method and then the whole graphs presented to us, right? Don't worry about it. This is mostly just checks around reflection and making sure we're doing things, right? The TLDR is essentially like we use a little bit of reflection and we get the graph and once the graph's there We have access to all the objects in the graph Just a note like we will post this so don't worry about pictures it's coming and Shameless plug follows on GitHub. We have a lot of our open source projects right there Anyway, so we basically decided that you know instead of using this generated a reference right like that We have to generate first before we can go forward as new people are onboarding to not have them deal with it We just you know created this little helper. So this was a very easy one and like you really didn't make much of a difference, right? What else so this is this was a big problem for us So we have we've decided on all these graphs and we want to really figure out where should we place them So we started off with like this thought we're like, okay We have this modularized app and we have these scopes that we have defined and what makes sense and we said okay Well, you know it makes sense for the user and application scopes to be at the very low level possible because these are Not really tied to anything and everybody should have access to it Because you from the top you have access to any module that's below you, right? Like you don't have the other way around and similarly for Because the retain graphs and the the retain scope and the view scope is related to the activities It made more sense to be in the feature modules. Let's see how it works. Does it hold up, right? so Let's take an example of tweeting, right? We're gonna tweet out this with an image and you know and like let's walk through what all's involved so essentially to start off our feature is the composer the composer is the one that's gonna take and Let you compose the street so that you can post it So in a very basic sense without diving into a lot of crazy things that's going on We can say that hey at the very least it has an activity Which is the screen it has a view model which defines the logic on the ui logic that we have over there And the delegate is something that we we do which basically Links to the activity so instead of using the activity directly we use this view delegate so that we can unit test that without requiring any of the Android Specifics so it makes sense that hey, these are the feature level components. So it goes to a feature module What next so then we would need the tweet repository because we're basically generating a tweet and we want to put in the repository It makes sense that the treat repository is a user scope, right? Like so as long as the user is logged in in any of the activities you'll get access to it So that this is a subsystem that gets added here and we run into our first problem Which is we wanted the user scope to be at the bottom so that everybody have access to it, but The thing is if you define it that far down you you can't Define the objects at the top right like so all right, but let's move on Let's see what else we run into right so the user scope needs to be at least in the subsystems level Next we have the image right like so we we're adding an image to the street So there's an image uploader now. This is just a Twitter infrastructure thing right like so it's not really tied to the user This is like something that gets kicked off so it's an application scope thing that runs as the image is being uploaded So now it's not really a generic Subsystem so once again, we're in this case where We wanted it to be at the very bottom, but the application scope kind of needs to be a little bit up So one thing that we're noticing is that the application scope needs like to be at the very top, right? Even though like this is something generic that we can have with OK HTTP, which goes at the bottom but the Twitter media image uploader Module that we had is like one level up from that so Let's get even more crazy, right? So if you tweet out in Android, you'll notice that while the tweets being posted There's like a notification that shows up, right? So this notification handler is essentially tied to the application like it's not really related to the activity It's not really happening on the activity. It's just on the app and it's contained there so We want to but we want to put it with the features because it's related to the composer So if we added there what we also ended up doing at that point is like We just move the application scope and the user scope at the very top Because it seems like there's like at any different level like we can have these scopes even though we were really Originally wanting to do it at the bottom Okay, so you know So one thing over here like you know like all the the pentagons right here that I have Are just the different bindings that we were we're having and then like the circles are like the scopes that I've defined So, you know, these are the modules that we talked about so we'll add them right there Right like so so at least that makes sense Like whatever modules we created for the dagger bindings that can stay with the features that that we want, right? So We kind of had to shuffle around a little bit of the ideal location But you know, we can try to deal with it as we go forward. Let's see what else we can do One of the things that we want to look into at this point is like How does how do these graphs then access like from intermediate modules, right? Like so even though originally we said that we wanted one the application scope and the user scope at the bottom And the retain scope and the view scope like further up at the features level But then like, you know now that the application scope and the graphs are defined at the top Which makes sense but we had we run into this case. So imagine this like legacy activity that uploads image I don't know what it does, but let's just assume that it's one of those Now it needs to access the application object, right graph. That's defined at the top to basically get access to this image uploader and That was one of the reasons where we originally was saying that hey it makes sense to have the application scope defined the application graph defined at the very bottom, but We were running into this case. So let's see how we can walk around it Because the module is going to be right there with the feature like as we were talking about So we wrote this another Helper we love helper classes by the way so we wrote this helper class that essentially could be the provider for the application graph and then we Said that instead of doing this where we were storing the application graph At in the application level in like a member variable we instead we change this to basically be Assigning it to the provider. So instead of storing it in the application itself We provide we just store it in the provider and the provider knows how to then get access to the graph The magic part of it is like then we can move the provider all the way to the bottom Like there's no reason for the provider to be at the top even though the graphs get defined at the very top the provider can sit at the very bottom of the of the of the the modules right So in this particular case then like going back to what we had where the legacy application needs access to this media sub graph If if we have the provider then we can just do something like this, right? Like so the provider will give you the graph We know that we are looking at the media sub graph so we can just cast it and access the image uploader Let's clean it up a little bit, right? We can just move it into with a helper method, right? and the helper method can just cast it and Just give you this I'll give you the sub graph We can actually clean it up a little bit more like so you don't even have to think about the provider We can just do it in the graph itself in the graph definition We can write a companion get a getter that essentially does the same thing for you like it will access the provider and like cast it and then The use of it in the legacy module ends up being something very simple You know you want to access the media graph you can just get it and access whatever you need from that graph so This simplicity essentially takes care of this case where we were reaching up and down at this to Basically get down to the lower levels Now we can just reach down like so we can just go to the media sub graph and the media sub graph can reach out To the provider as needed What else did we have? so One thing that you might have noticed from that Example is that the graphs can grow like we were just talking about composers So if we look at just the composer we had at least like the the compose launcher the The notification handler and then like you know with the tweet repositories Whatever related to that and it can keep on growing and that's just like one feature with like a few things at it Right like and then we if we add all the other ones it grows It becomes big So we we thought around and the figure out okay How can we deal with it and one of the thoughts that came about is that we can introduce sub graphs so With sub graphs we can clean it up and we can actually define it So we were already saying that there's the media sub graph But like what if the definitions completely could be used like this way right so Just using a very short version of the application graph. So we had Composer and all of these we can we can just split it up right so we can say hey the composer related It goes to the composer sub graph and we can just basically extend the interface as we need So we can do that for the other ones we can do it for the tweet repository the images and like Whatever granularity that makes sense right and we can build all these sub graphs and make this very simple Object graph definition where you don't have to worry about what all bindings are there right like they're all defined in their own Subgraphs and then like like we were saying before where the media Component modules were right there right, but now we can actually move the sub graphs to that level right and We can actually put them right where they're defined So wherever the bindings are we can put the graphs right next to it Kind of like helps us get rid of this very big graph and distribute it to with all the features that we have all right Question comes up immediately from that is what about testing right? we have all these sub graphs the sub graphs knows how to pull things on their own and The sub graphs is looking at the provider directly. So how do I actually? Take advantage of the testability of dependent in injection and like you replace it So what we did was we added this little thing for the provider So the provider we added a map where it can basically Store some of these overrides that you have and an ability to overwrite it and then in the getter We essentially look at it and see if there was any overrides if there is any overrides will return you that or we can Just return you the real implementation that was already there. So then like you know, that's taking care of now We as we started using this more and more we ran into this problem of decoupling features So I want to dive through it a little bit more and give you a little more detail Because this was a doozy So let me use a example of the app navigation So essentially if we have a very simple Twitter app imagine it has only four modules Imagine it only has the main activity. It has the ability to show tweet details It can do a compose and they have DM for direct messaging, right? Now you can get a tweet link via a DM and if you tap on it you want to be able to open it So now in Android how activities work is that you for To launch that screen you need the activity to create an intent around it and then you basically launch that intent So for in this particular case the DM activity will have to know about the tweet detail activity So that it can launch the intent related to that and then you can go from DM to treat activity However, there's a feature where you can share a tweet through DMs, right? So then the tweet detail activity also has to know about the DM activity So this means that we have this circular dependency and you know that obviously doesn't work So we thought about it and we basically ended up introducing Subsystem called navigation and the net so this is how the navigation subsystem essentially works So we created these intent factories essentially for each one of the features, right? We call them activity args They are related to the features directly and then we moved to these activity args into the navigation subsystem So that way like every activity as you need to do all you need to have access to is to the navigation subsystem And you kind of like take care of this circular dependency Let's dive into a little bit on like what the activity args look like so I mentioned before that they are basically intent factories So essentially the interface looks something simple like this And from there what we want to do is essentially just generate the intent that's required to launch the activity One of the immediate bonuses that we got out of it is we actually had type safety of the arguments because they're just like Bundles that's defined in Android So that was a big bonus, but like it's kind of like something that we just got out of it now it as I mentioned it creates the internet activity, right and then For all of these different ones. So now we have these activities and all these like intent factories spread all around We wanted a easy way to collect them all together So we built the service discovery the service discovery what it does is essentially uses the multi bindings feature for dagger and Pulls in all of these intent factories that we've defined. Let's dive into a little bit. So Even smaller example here But essentially like we have these intent factories defined, right? And we want to basically have them in the object graph somewhere so that we can easily reference to it So we use multi bindings from dagger, which will just essentially collect all of these definitions because they're defined and then put it in for us Dagger can put it in as a set or a map We've decided to use a map and essentially what we said is that for each one of these activity arguments Like map it the values the activity arguments and the key is that the activity class So then we have access to the activity class and we can generate the intent Let's dive into like what we did so taking composer module as an example So we added this tag into map, which comes from dagger Which basically says that pull this binding into a map and then we said use activity arguments as a key and then the value would be the activity class itself This essentially generates this map right that we store in there another helper function Called activity starter. So this isn't a navigation subsystem. So this map is then later accessed So when we want to create something we have this map and we Have this helper method now that can kick out the activity So all you need to know is that this is the activity argument that's required Now Android requires context for a lot of these things. So we take in context for this case But then all it does essentially is that it generates the intent like we said, right and nothing more to it Essentially getting rid of the circular dependency that we had and now we essentially have this module like thanks to dagger We have this map and we can actually relate the different intent factories to the activities themselves All right So we learned a lot about dagger using dagger over the times and over the years And as more and more engineers were onboarding and more and more features that we were writing We started seeing some of these even like starts training So I want to go into it even more details that we ran into So we used it for initializers like is the same features like with the service discovery We used it for all the initializers We used it for like notification handlers and we also use them for deep link handlers like that way We just spread it out wherever they're related, right and like this particular thing like we basically ended up building this Thing right that's on top of dagger so that we can like help Distribute the graph like into the modules themselves, right? So the subgraphs however ran into a problem So we found out that the subgraphs themselves are not atomic So let me let me explain So we have this graph definition that I mentioned right So there's a top level graph and then all the subgraphs to find in the modules that they have But then to use dagger we essentially have to actually define a module next to everything Which defines where the bindings are So you to build a graph at the very top level you have to point it to the modules And then you also have to point it to the subgraphs because of the definition that we have and Then it gets worse because we share code with periscope And we would like to keep on doing it when you have a new definition on that's a shared module You essentially have to define both of them in both the apps and this will just keep on growing right like now We have sandboxes that everybody's using so everywhere you have to remember to do both of them So what can we do? So to start off like so using this broadcast example So that's a subgraph definition which says this is the binding that we're having and then the dagger definition Which actually provides the binding So we started by basically annotating that hey, this is our subgraph So we know what a subgraph is and then we're saying that hey This is the scope for this subgraph and from there we said hey We can actually say that hey This is what's tied to the subgraph right like this is the module that you need for the subgraph and then For the the big graph at the very top Instead of saying that hey, this is this like component with all these modules We can essentially say that hey this part can be replaced with that. This is an object graph That's all like that's all we need to know right we can do the same thing for the user level stuff Because it's a subcomponent, but essentially that's also an object graph. That's defining everything So the big difference between the subgraphs like you said before was just pieces of the object graph the object graph is like the full definition so this annotations then like we basically send it through the annotation processor that we wrote and it can crunch and Generate the dagger compatible components and subcomponents So we define the subgraphs and like just let it generate whatever is dagger This is like fully dagger compatible Right because we are definitely using dagger under the covers It's just like like simplifying the one of the things we ran into What else? Let's see. So subgraphs are not encapsulated right now, right? Let me let me explain that With this example, right? So what I mean is that when you have all these different subgraphs Like you actually don't know how they relate to each other You don't know who needs what it's only when you add it to the object graph that you realize that oh, I have everything that I mean right What we would like is that at a sub grapple at a sub graph level We know that it actually has all the bindings that it requires so what we did is that you know, we took this definition of the modules and like of sorry of the subgraph and We added the dependencies so we know that this subgraphs has other subgraph different dependencies and then You know instead of having this issue when that compile time at the very top level in the graph We say hey, there's a binding missing we can actually get the error at a very lower level So we know that this particular subsystem is actually missing a binding and then fixing that we'll just fix the entire graph right and The bonus from that is that now subgraphs are fully documented right like we have from this definition alone We have what what are the modules that's related to the subgraph? We know what are the dependencies of the subgraphs and we also know what bindings that's being provided So looking at a subgraph, you know what all it means What else we ran into so we ran into the case that subgraphs are not reusable And you're gonna say South I thought that's the whole process of doing it. That's why you're doing it But like there's a little bit that what I mean is a little bit nuanced So we can use this network subgraph right for example in any app that we want But then when Periscope came along they said hey, you know, this doesn't work for us we need to use a special cache and Since it's defined in the subgraph. There's no way for us to reuse it without copying the whole thing So we thought about it and we are like this is a big fail What if we think about it in terms of just what we would do with a class, right? So if you if you were a class, we would say that hey, you this is not a final class It's an open class so you can extend it and then we could say that hey The cache is also an open function and you can overwrite it then for Periscope you can actually Like override that binding even just by extending this and just overriding the special cache right like just saying that it replaced the cache with this special cache and then something like so Would have worked and so we basically Added annotations that basically references the same idea right so we're saying that this particular subgraph is open Which means if you can extend it and then we're saying that this particular bindings are open Which means you can replace it We didn't do it for all we we left the power so that in some cases as you're building the module You know that you cannot replace it so you don't want to do that for everything, right? You want to limit it so we built it in instead of having everything be overwritten. We we allowed it to be defined Right and then like when the periscope comes in it can define its special cache and like said that hey This is a subgraph. This is the override So we can reuse the override annotation and say that it binds to the cash, right? final thing that we ran into is that Subgraphs don't support inversion of dependencies. So this is this was very tricky As we started doing more and more things into the subgraph We ran into this case where we wanted to define this database subgraph and the database subgraph in general like you know We we have this definition of the schema of like what this database holds but a database subgraph is more like an abstract thought right like we have multiple databases and each database will have its Own schema and we want to invert this dependency so that we can Provide the schema for different Databases, but also have like the modules themselves like anything that uses databases to get advantage of this subgraph so just like before Just like open before we added abstract annotations So the abstract annotation says that you cannot instantiate this graph you can extend it you can build something like this for Twitter for example like we and You it if it completes all the bindings then it's a complete graph, right? And it's not going to be an abstract and then you can invert it and you can say that this is the schema that the Twitter Database requires and everything else is provided from the database And so that was all the subgraph stuff and finally for the graphs we ran into this big one is that they are hard to maintain So just like dagger where you have to define all the modules at the top now You have to define all the subgraphs. So something Like this happens even though we are putting all the bindings in their own little graphs, you still have to define the graphs and It gets really insane when you have a lot of features if you haven't seen it trust us we've run into this and so what we decided is like all of these Information is available to us. So can we build an annotation processor that can crunch it, right? and then essentially does automatic discovery and Essentially, all you have to do is say, hey, this is there's this object graph It can find the subgraphs that's required and pull it in So essentially, you know at runtime at when you're writing code You don't have the types available. So you don't really know but at compile time These are all there because of the automatic discovery Right and we did more from there, right? So once we solve that we added support for Factories like in general like just first-class support. We had it improved our build times a lot because of the distribution of the graphs and We are working on an ID integration right now This annotation processor that we created we call it site and You know, it has all these features that I've been talking about for the past 30 or so minutes And we are planning to open source it this year Okay. Thank you very much