 Hello and welcome everybody. Thanks for making it through this day to the last round of talks Yeah, so my name is Andrii I work at the company called Celera one I Would estimate my Python experience around five years so far, but this is the first time I'm giving this talk So I also hope for some feedback after you from you up afterwards in the end and Yeah, let's start so first of all I will give Some words about the company you who we are what we are doing then introduce the architecture of our platform in a course green level Give some information on how we use Python in Pyramid in general and our software Describe in detail to our analytics subsystem and we finish with some overview on a general development process in our company So first of all who we are companies called Celera one for short We are calling ourselves see one the company is relatively young. It was established in 2011. It's based in Berlin the company is Quite small. I would say for now. We're around 25 people But it's already quite international because we're coming from nine different countries and the main product of the company is the platform for doing the paid content content recommendation real-time decisions on Content access for users and of course analytics. We are also developing our own programming language It's called COPL as you might correctly guess that stands for Celera one programming language It's functional language and strongly typed The main customers of our company are big media and publishing companies in Europe so try to represent the Infrastructure of our software in a layered level. It's somewhat hard because in reality they are quite often interconnected, but This is how it looks and we go from bottom to top So first of all the first and maybe the hard Layer of our system. It's in memory What we call engine and engine is a custom solution implemented in C++. This is a no SQL in memory database It's a bit special. It's not only our robust storage. It also provides some business logic So engines are usually coming in pair where one is the master and the second is the replica and they are connecting to each other using zero MQ and this is the point actually where all the real time real timeness of our system happens and It stores data in the form of events and streams and indices and the typical use case would be The real-time user segmentation for example when request comes in we can define Already to which user group user belongs and this is not trivial task because it usually Each user action brings him actually to the different groups So basically each action can take him to different groups and range is quite fast in this case He can compute the user group membership just within a couple of milliseconds and provide the result The next layer would be analytics system This is our scheduling application Written in Django and a set of workers Django was chosen of course for its admin panel and workers what they are actually do they connect to the engines collect systematically some analytics metrics and statistics data and Store it for later usage by the upper level and the upper level is where we Actually use pyro meet finally this is the level of restful API and this is somewhat a gluing layer Because it's used for integration of all third-party customer systems into our platform So basically it exposes API Which are then used by the customer systems like SAP or and so on and so forth to interact with our system It's pyro meet application served by US giant could be served as one big melodic application or running in several US processes and the topmost layer layer. It's What we call communication and proxy It's Implemented in open rusty framework. Basically. It's a bundle of engine X and Lua code We also wrote our own extensions in Lua and because it's super robust super fast well, yeah, let's be honest that Python sometimes could be slow and Open rusty was super fast. So part of API is also implemented in this layer. For example, the endpoints for event collections This is our the most frequently triggered API where we get like 10,000 requests per second. For example Yeah, it's implemented in this part. Also, it's together with the engine is responsible for making these real-time decisions for example on content access and It's also responsible for requests forwarding to different sub applications if they are running in the separate US giant processes so before installing our software on customer side, we are usually doing some assessment and Some what we sometimes face challenges and the biggest challenge is for example at the our biggest customer We face that we need to serve at least like 10,000 requests per second for this We kind of tweaked our system And yeah, depending on the customer workload, which is expected in SS The setup can come in different ways. The typical most typical ways is when we have two front ends and two back ends because I mean engines The biggest cluster so far It's up to five front ends which are running our Python applications. Also serving just creep and open wrist applications And the back end could contain up to nine engine pairs So totally 18 machines 64 gigabytes from each and the data would be Some part of the data is sharded all over the cluster some part of the data is copied for availability reasons and This makes a store billions events in memory providing super fast access to this data and It's giving us possibility to serve around 10,000 requests per second We also use three Sorry to Mongo replica sets the first one will be used as a storage for the application data for Python applications and the second one is the persistence layer used by the engine internally. So the logic is that engine keeps data for sliding window 30 days and then starts to back up this data in the persistence layer for Availability reasons So how does the Python software stack look like so first of all, this is US GI as a web server running in usually emperor mode Then the pyramid as a web application server Then we are using some plugins together with Pyramid notably This is a colander and corny's colander is the library used for data serialization the serialization We are using Jason, but it is also suitable for parsing for example XML Some basic validation of the incoming data could be also used implemented in colander and then the corny's is a plugin from Mozilla It actually simplifies our developers life to implement the restful services It's also quite useful because it's integrated with Sphinx and helps to generate the documentation Then we wrote couple of wrappers on top of requests library Because we are interacting with the engine over HTTP and we have just some classes which wrap up requests to interact with our engine then Pyramid itself is built on top of the soap component architecture and we are also reusing these Components in our code to implement so-called team plate points. I will talk about this in a moment Then the build system is build out And as I also mentioned already we're using Django for the web application of our workers management And the robot framework is used for testing so Hopefully, this is readable This is an example of some hello world application which is using the Pyramid called corny's and colander and I'm going to explain step-by-step, but it's on this slide. So first of all We are defining the data schemas. They describe the parameters that handlers with the later expect and Would it be the query string parameters or Request path parameters or incoming data payload parameters They could be parsed and tweeted as a specified type So the first schema would be used in the get handler. It specifies one parameter called username It should search for these Parameter in the Query string and treat it as a string parameter and we are saying that this parameter if it's missing It could be discarded from payload. So basically that means that after trying to access this Parameter in our handler Well, it would be missing. We should keep in mind this and Yeah, going to second schema, which is used in a post handler We are describing some basic JSON structure consisting of three fields Where each of those fields should be also treated like a string We are saying that the they should be found in the request body And at this point we can already use some basic validation for example, we say that the field message should be from five to twenty characters length and The full field should be one of the valid values bar or bus Providing these cornice are quite good in tricks will with colander providing this information during this data deserialization These basic validators will be already checked and the suitable error message is generated and propagated To the client to the requester. So you don't actually need to treat these special cases in your handlers The cornice plugin would do this automatically for you Then for some more custom validation, for example, you need some dependencies between the fields in the incoming payloads You can do custom callable validators and pass them later to the Cornice I'll talk about this in a moment then We define finally we define our Rest service it would be called hello service and available at the path. Hello Then we Decorate our handlers get and post handler respectfully with the created service and we pass schemas and if we have custom callable validators We also pass them so at this point we defined Handlers for get and post if our requester account calls for example put again the cornice handles this by its own and the error message would be generated like 405 method not allowed So yeah, I would say that this quite simplifies your life And especially if you keep in mind that if you're doing the pyramid application for each of the handlers You need to during application config include its path You need to add this line for each of the handlers that you plan to write Instead of doing this you only need to include cornice during application boot time and just define services like shown I think it's much simpler Okay, and This would be an example of robot framework test. So we define the Two end points for getting post and now yeah robot framework. It's a keyword based test suit for Mostly integration testing because as I mentioned earlier our business logic is somewhat split between Python application and the engine itself That's why we are mostly doing integration tests so User can combine it so their own keywords to implement more complex keywords and stuff So our tests would put up The MongoDBs the USJ applications the engines on the background and this exact test case would then test if our Response to the put method is what it is as expected. So just Showing you that Running this test Passes yeah, at this point the engines are started locally on my machine Then the test is executed and it has passed then it's generating and nicely looking Report we can where you can see the locks which what happened during your test if there were any failures in our case Everything is great So we are happy to go Okay Let's continue Yeah, also We're implementing our application in the way that we this distribute the Logic of application into different sub modules so that we are making sure that Different features that could be put on to the cast output it to the customer So we have like for example SSO module the CP integration module the analytics subsystem and on depending on the customer demands we're developing and Shipping these modules to the customer So they could be all served as one monolith application or each of them's running In the emperor mode and served separately in a separate USJ application Also, one of the challenges is how to keep because customer base is quite big we have round eight customers some of Upcoming customers, so we need to keep our code similar, but also we need to Provide custom solutions for our customers because the demands could be different Their systems could be different and the best example, maybe is the SAP it's quite inflexible It's quite slow sometimes and for those we need for to for example sometimes develop some custom code And these custom code would be then placed in a separate package And we are trying to keep our generic code base as generic as possible and for this we are implementing so-called team plate methods in our code Team plate points and then the custom hooks which implement this custom logic would be overwriting the generic behavior in the runtime and Thus we are able to deliver the custom solution to our customers the example of such case as I mentioned earlier is a SAP integration and this is an example of real existing API for importing SAP catalog So we are reusing the soap interface in this case We are defining the interface called custom catalog transformer So the whole idea that it has to transform method which would take the catalog in the format of which customer defines and do some transformation and Transform it into the internally acceptable format and Yeah story then We have a generic implementation Which actually is doing nothing. Well, it's called default transformer. It lives in a generic custom code base It's doing nothing. It just assumes that the incoming payload is already in the internally acceptable format Then it's during the application boot time. It's registered by calling register utility and In the meantime in the cost customer specific code base in the custom package we're defining the catalog transformer called sophisticated transform which actually does some magical transformation whatever and Brings the catalog payload into internally acceptable format and then in the customer code this would override By register and this utility also during runtime and then including these custom component into the generic code base The behavior will be already tailored to the customer. This brings a benefit that the API's Handlers they all stay the same so they don't change and you don't have to split your API between different packages They all still live in the generic code base but it still gives you a possibility to implement custom solutions for your customer needs and Yeah, time to speak about our analytics subsystem So essentially schematically it would look like this We've got engine pairs the Data which we are going to collect the analytics data. It's started between the engines So we need to query each single of those then merge this data and Store it for later usage. So workers they connect to the engines periodically query the data do this pre-aggregation and cash it for later usage in the MongoDB then The metrics API will also call it analytics API This is the pyramid application Which would then later read this data and according to the incoming requests from our single page JavaScript application Which then should later the analytics data it would filter this data additionally using Mongo aggregation framework produce the result and based on this data, then nice graphs and charts would be drawn and And as I talked already, we are using Django. This is Scheduling application so it manages workers. It's possible to see if there are any failing tasks If the tasks should be restarted, how does the whole execution process looks like? Yeah So at this point I was trying to do some showcase of our analytics. So basically this is our demo system and the graph stands for page impressions and it's Possible to see this time span for example of one week using the data different time resolutions time resolution usually means how real-time the metric is So this view shows currently a time span of one week with the resolution of five minutes then we can see the resolution of One hour and even more cause growing resolution of one day So the time span stays the same, but the total numbers represent different time resolutions Okay, I'm sorry This is how our Django admin panel looks like here is an overview on the completed tasks failed tasks you can disable metrics collection for time if there was any deployment Happening on stuff like this and on the right side. This is the configuration of the metric job itself So on the left side on the left bar you can see This is would be the time resolution for which we collect the data and this column means how real-time the Metrics collection should be Okay, and for the end this is the last slide Given overview on the typical development process in our company. So there is a developer He makes his changes commits them to the code review tool that we are using Garrett then the code gets reviewed after some time the changes are merged to the git and Jenkins keeps overview on the git repositories and After the code is merged it starts all the different tests when your tests are green We're actually always trying to keep that our master branch is ready for being bumpers and then released So if the tests are green and all okay, you can then Bump the version of your package then it gets packaged into an egg put into internally hosted Python egg server then the documentation would be built and Yeah, it would be ready for release and when the release time comes We have a wrapper package So all the versions would be included by the build out both from internally hosted Egg server and also from pipey that would be combined into the dep or PM package developing depending on the customer operating system and then ops guys are doing their magic putting our installation on the servers For availability reasons. We're using do it usually doing in the halfway So first we upgrade one half of the cluster and then the second one So this brings virtually no downtime and it's not visible to the end users who are using these systems Okay, so thank you for your attention. Thank you for coming to this talk today and questions So I understand you use both Django and pyramids Can you clarify? What exactly does Django and what exactly pyramid so maybe you can share some experiences or what is better for which use case What are the strong sides weak sides? I? Have some quite some experience with Django. I think it's really nice framework But mostly I think that everybody loves it because it has its magical built-in admin panel and Django is only internally used for us. It's not visible to anyone. It's just for us to oversee how our workers are doing and If there are any failed us that we need to restart if there are any problems, so it's only like an internal tool and The pyramid it's more flexible. It's used to implement the Restful API as I described and showed the examples and this is what actually is visible to our customer systems so that If they have some legacy SSO systems and they want to connect to us they would be using our pyramid API Thank you for the talk. Actually, that was one of the questions. I wanted to ask so thank you But I have another one What is your development effort now at the moment? Is it on the analytics part or is it on the? the data scaling Deployment on larger scale if you had to know any more customers would it be easy to do? Sorry, can you repeat the question once again? So the development effort you have at the moment Is it on scaling the existing system or is it on coming with the new analytics new algorithms? Yeah, so I would say that well, we have two teams one is C++ team which develops the engine and I would say that most of the computation efforts are there and that part and In the Python team, we're mostly working on bringing the different metrics data. So we need to Do different aggregations to optimize it Usually this is actually the layer where we consume probably the most of the memory So it's quite memory intense always. We are trying to Use different techniques for now the Mongo aggregation is doing fine but the workload is distributed somehow between bringing new features which are demanded by the customer and implementing more different kinds of analytics and Views I mean like the charts that would be shown to the customer Because those are then which are used by the business analysts and based on this data They are doing some decisions which can impact like the income and stuff like this more questions Right in that case. Thank you again