 So, welcome everybody, thanks for being here and nothing with us talk, we really appreciate that and we are going to, Carlos and me, we are going to give this talk, it's called Max, Real Time Messaging and Activity Stream Engine, I'm Victor, I'm a senior Python developer and IT architect at UPCNIT, I'm a Plong Foundation member and I'm Plong Core developer since 2010 and I'm also an author of a book that is called Plong 3 Interests which was published in 2010 by Backbook. Hello and I'm Carlos, I'm a Python and a JavaScript developer and I'm working with Python for the last eight years and just because this project also occasionally with Erlang and also I'm the kind of guy everyone asks for regular expressions and with Victor we've been doing websites with Plong in the UPCNIT that is a private company that is owned by the UPCNIT, Barcelona Tech and we have more than 400 sites in Plong running in the university and the last four years we also make some projects with Pyramid. Okay, so against all odds we wanted to do a demo and I don't think we are able to do that because the Wi-Fi is a little bit crappy but however we tried to do that so let's see what happens. We wanted to do that because of course an image is worth why 1,000 words and if it's not working right now maybe we're going to all the show, come on. Okay, maybe we're giving up on that and we're almost there. So we basically wanted to show you the stream which is the basic UI implementation. So we have the stream that is the central part of here which as I said is the UI implementation of Max in which we can post things like this. So hello everybody. And then the posts get registered and right now Carlos has also sent me a real time message which I can see there like in a real time notification and now if I go to that view then I see that Carlos has already sent me that message and I can reply to him like this. So this is basically what we've done. These are the two main parts, main features of Max and we just wanted to show you. So let's back to the presentation. So a little bit of history about Max. We've initially designed Max being the fact that we wanted to implement some kind of social internet which is a concept that is nowadays is very trendy for the university itself. And we initially designed that key feature that the internet should have which is the streaming engine and after that we also added the messaging feature as well. And today Max is used by more than 30,000 students and 8,000 university staff and it's integrated mainly on the online campus of the university and also in the institutional collaboration tools used by the university staff. Basically what is Max, so you already seen it, it's an engine. You've seen the UI but basically it's a RESTful API. We almost have 100 endpoints and it has these two main features. It's a multi-source user and application activity stream because not only it's capable of receiving activity from users but also from applications and we can impersonate applications by talking in the mouth of the users too also and we can do that kind of things. And then we have the asynchronous messaging and conversations feature. We've already seen it and it's our GPL license. So how we came to that concept, to the concept that it used in Max. We had a lot of forums in the university, in the online campus and in the internet also and we wanted to modernize the user experience for that forum so we thought that maybe we can reuse some of the concepts of the forums in special from topics because if you are a user you are interested in special topics but you always have to go to the website and check that topics and we wanted that the information came from a modern point of view to the user. So we basically introduced the concept of a context. So in Max we basically is the way that we... So in Max we have contexts which are mapped directly to topics, to the old topics which are the main features that they have, they are related to a unique URI so we can map it to virtually any page even if it's not related to Max directly but in this context we can aggregate posts, posts that have social features like commenting or likes or the favorite feature. And we added also the concept of subscription so I can subscribe myself or someone can subscribe me to a context and this person can give me... can grant me some permissions that it allows me to interact me with the context in different ways. So we have the context that is mapped to the topics so as a user I can subscribe myself to all the contexts that I wanted. So what are the other features of the context? As I said they are identified by unique URIs and we can assign permissions per context which are basically now we have these eight permissions which allows us to model different kinds of contexts based on that permissions and that gives us almost 6,000 kinds of contexts which is a pretty useless fact however. And we can also overwrite the permissions of the user in a granular way because we are able to grant or revoke specific permissions per user for allowing users to do more than the context does. So what are the real life examples for that? For example we have now these two scenarios where we have the community sites and the online campus and the community sites is kind of as I said the social internet at the university where we have a lot of communities which are directly mapped to one context which can have any of these like institutional events or traditional news and in the online campus we have all the subjects that are mapped to each one to a context. In the community site we decided to have these three community types just to show you what we can do and we have the open communities where everyone can join and can live at will. We have the closed communities where the owners should invite me in order to join but I can leave also whatever I want and the institutional community site where the site admin decides who subscribes but no one can leave. This is the example of the institutional news or the institutional events where I want that everyone be subscribed but nobody leaves. As a summary of features we have the activity stream, you already seen it, it stores activity from user applications and we have that social feature too like comment slides, favorites and we also have support for attaching images and files. The conversations feature is almost the same because in kind of way are sharing some of the basic features of the activity stream now later Carlos will show you and we basically allows us to have one-to-one to best conversations and group conversations as well and it supports also images and files. Then we have the JavaScript UI widget which you already seen and it's the reference implementation of the UI attacking the API you can use it as example for implementing your own UI if you wish. So we also have the notification engine for the real-time features that it's platform-specific push notifications for Apple and Google and also these allow us to have kind of internal notifications too which we are exploring right now that we want to be able to send notifications for example to the desktop, to the users not related with the real-time messaging but also for other information that the user could have. We also have an external source aggregation we are currently monitoring Twitter, hashtags in Twitter which we can push it inside the stream also. Also a key feature that we are able to deploy Max on Premiere or whatever we want and this address some security concerns that we could have if we are using a more popular tool like let's say WhatsApp or iMessage or things like that and if you are worried about data privacy and ownership we provide a way to have like a corporate WhatsApp or corporate iMessage so it's also a good feature and this is the summary of them. Okay I will try to explain you a little bit some of the components that we will to make this possible. We have kind of three different kind of components on the left you can see the components that are the ones that have a user interface the ones that the user really uses and sees which are the mobile apps and the integrations that we have done with other systems like Plone and Moodle and the JavaScript widget that it can literally be put anywhere. In the center in red there are the main software that we developed to do this. We will go through some details about it and at the right some of the well-known software products that we used as the backend side of Max. To start what we call OCDs is an implementation of the server implementation. This is built on top of Pyramid. This is a very simple implementation. We only support what is called resource owner creations flow because as we only are using UALF from trusted systems we don't have the need of the more complex flows of OALF for identifying and giving permission over resources. As the systems where we are using this are trusted by the final user it's that system that provides the credentials to obtain the tokens and do the validations. We are using MongoDB both in OCDs and in the API for storage and one of the reasons that we developed our own system is because we know that we will have some clients that have specific user directory systems or single sign-on systems and we wanted some flexibility to adapt to those clients so we made OCDs pluggable so which client can get its own plugin to adapt to its features but we have a base LDAP implementation and we offer also an LDAP service to our clients if they not already have a directory service. Max is the API part. It's also built on top of Pyramid why we use Pyramid. We were using Pyramid for the last four years we know pretty well what features it has and it fitted well with what we wanted to achieve with this API. I will not go in deep about all the endpoints that we developed on Max only a few of the implementation details that we used. For example, on the routing of all the endpoints that exist in Max we have a centralized place in the code when we define all the URLs of all the endpoints and this is used by Pyramid to actually define the routing and we use it to generate automatic documentation on each implementation of each endpoint and we also use these traverses it's a combined way of routing using URL dispatch for reaching the URL and traversal to retrieve the object that is the object referenced by this endpoint so once we enter in the definition in the actual code of that endpoint we already have all the initialization, authorization and authentication done and so the real code of each endpoint is very simple and very short. We also use the feature of Pyramid that is called TWINS TWINS is actually a decorator but is managed by Pyramid so you can define a lot of TWINS and you can define in which order the TWINS are executed and basically you can perform tasks before and after a request enters the system so you can modify the request that has arrived and modify the response that will be shown to the user. As an example, one of the TWINS that we have implemented is a check that is done on a special value on a header that we are using because we thought that maybe in some point in time we would reach a point that we will have no option that makes some breaking change in the app so all the mobile applications that we have distributed across all the client devices will be broken so the mobile devices and the API server agreed in this value that it's just an integer number that we are incrementing over time and each request is checked on this number so if we have to make a breaking change we just increment that number and then mobile apps know that a change has been made and that probably a new version of the application is ready and it tells the user to upgrade the app so we have a control about this scenario but we try not to make that kind of changes it's just in case we have no other option and the last thing about Max is how we implemented the exception handling we wanted a unified way to show the errors to both the final user that is using the API and a way to catch possible handled exceptions and bugs that occur in the system so what we've done is that every known exception that we have in our code is rised as a custom exception and then this is handled by Pyramid catching the exception and rendering a customized JSON message for each kind of error and for known handled exceptions we record the trace back of the error we record the information that was in the request and we built a little user interface to be able to inspect that debug information and quickly act on possible bugs without having to deal on contacting the user trying to reproduce the error, etc. On the real-time messaging part we have used RabbitMQ this is really the third attempt that we made while we were exploring different possibilities first we tried with Pyramid and Socket.io then we tried with Node.js and Socket.io too and finally we discovered one of the plugins that come with RabbitMQ that is a storm with WebSockets plugin I don't know if anyone had heard about it what we can achieve with this and also with unique routing features that has RabbitMQ is that we can directly map a client that connects it to the system to the internal Kiwi and Exchange system of RabbitMQ and also we have developed an Erlang plugin for RabbitMQ to connect with our authentication This is a diagram of what we've implemented to wrote the messages One feature that we are very proud is that the structure of this design holds the security of where users can read and write from So each of the exchanges that you see here are connected by some bindings that are maintained by the API calls So anytime a subscription is made the correct bindings for each user are populated and so a message can only go through the system if that bindings exist So the security is implicit in the design of the bindings We also developed a kind of protocol message that is what messages inside RabbitMQ contain and that message is designed to be packet and unpacked to use the minimum size possible So we have a kind of specification to unpack these messages based on static values that we know So we can go from this human-readable format to this nerd-readable format So it's not unreadable but it's easily debuggable we have to plug in and see some of these messages And finally, this is the MaxVani component that is the Kiwi consumer It's a multi-process Kiwi consumer so we can start many consumers for each of the types we have developed and it's in charge of taking the messages doing the appropriate tasks for each message to fill in the APIs and so on And that MaxVani component uses what we call MaxClient is a Python wrapper for the REST API that wraps all the functionality from the API and this is what is used from MaxVani to call the API I will skip this We also developed a special version of the MaxClient that is the UISKI MaxClient which is a way to from the client execute all the codebase from Max using a special fake UISKI server so we can make a request in the same way that is done by the real API but without using the real servers And at last, the Twitter external aggregation is already explained by Victor So I want to show you now the current integrations you already seen in the demo which is the site of communities which is this social internet We have the Moodle integration which we call the Ulearn Campus and also the iOS and Android apps So this is, you already saw it the social internet with the key the JavaScript which is in the middle This is the Moodle site or one of them because we have a lot of themes also and with the integration of the stream here which allows the students and the teachers to have a very tight communication even with the stream and with the conversation part These are the iOS apps And this is the Android one So they basically have the same functionalities And potential integrations almost virtually whatever you are thinking of we are also thinking in put this stream also near P's or in, I don't know any HTTP web app that you are thinking of because of the JavaScript you can put the JavaScript with a very little effort and make it work with Max anywhere and we are already exploring that as I said with maybe putting in our ERP or something like that We have a huge list of to-dos because there is so much to do We wanted to add social integrations like follow, like a Twitter-ish feature or the share, like the Facebook one or the G-plus one We definitely have to finish and polish the documentation Maybe explore the realization of each literal service and make it a real microservice Exploring also some other kinds of cash like Redis We also wanted to port it to Python 3 because we are using 2.7 right now and maybe use also a sync.io and improve our API by adding some JSON LD or hyperlink technologies there and of course adding some kind of end-to-end encryption which is a very tough one but nevertheless it's on the list So we are here also because we also want to explore if we can build a community around Max We are very, let's say, happy and proud of what we've done and we know that there is a lot of things of improvement We also definitely want to explore what if we can do a community around it and please contact us if you are interested or you find it interesting and public requests are welcome These are the resources where you can go and take a look at it The first one is the documentation one and the other ones are the repos of Max So thank you Thank you very much Do you have any questions from the audience? You said you tried several technologies besides Rabbit and Q when you were choosing it I'm interested in knowing which ones and especially if you took a look at MQP 1.0 as a protocol to use Can you repeat the final question? I'm interested in knowing what technologies you guys looked at instead of Rabbit and Q and I'm especially interested in knowing if you guys took a look at A and QP 1.0 as a protocol to use rather than Rabbit and Q with 0.9 I think I already said it We used a pyramid with the first two approaches without thinking in a QE broker inside the system but we realised that we couldn't achieve a real time with a little load with a system like Rabbit and Q in behind Rabbit and Q is using EAM MQ but we are using some of the extensions that Rabbit and Q has done over the standard protocol So we are a little tight to this to these features More questions? No more Let's thank the speakers again then