 Hello, I would like to remind you to ask question at the end of the presentation Our next presenter works as a messaging architect in the Deutsche Börse His name is Jakob Schultz Please invite him Hello everyone This talk is In the IOT stream But I hope you won't be too disappointed that my background is actually not IOT but finance I work for Deutsche Börse that's one of the world's leading exchange market operators. We run several stock and derivative exchanges We have our own clearing house. We run services with and settlement and custody and In the last years I Spend the time building interfaces based on the AMQP protocol And that's not only interfaces which connect our own applications, but also interfaces which Connect us with our customers who write their own applications. I Think that's not that usual and that's why I would like to share the experience we made What we learned and then discuss it with you Before I start with the first slide Let me ask you how many of you know what message-oriented middle areas and how many of you know AMQP? Okay, that's quite a lot a lot more than I expected I have to say I'm glad to see that So let's start In finance the IT is not really that different from the other areas We have usually some server which We run and then we have some customers who run their own applications which are communicating with our server and Do some transactions exchange information in the past We did it usually in the way that we had some libraries which we provided to the customers as As the API and the customers use them in their own applications and The protocol between the library and the server was usually something proprietary not really well documented not really opened so Can someone see what's wrong with this approach? It's basically with the library API you as a service provider are intruding into the space which does not belong to you you ship some library Which has to be installed by your customers, so they usually don't have really free choice of operating system They don't have free choice of hardware. They often have no check note choice of programming language or choice of programming style and Since it's your software running with your customers Then you need to provide a lot of support when the customer calls and says look the library is not working for me We have to deal with The facts which operating system is using which patches did he installed? And that can be very complicated, especially when it's not your computers. You have no access to these computers so Lately we are switching more towards protocol based API. So what does change with the protocol? API the API is not anymore the library as it used to be but the API is the protocol and the library and the client application is full responsibility of the customer so as I as you can see from the picture the separation between the service provider and the client is now much more clear and This may sound on the first look like if you offload more work onto your customers, we wouldn't have to write the library Tested supporting on their own, but If you choose a good protocol for the API then maybe they don't have to do it because maybe there are many different libraries available on the market and they just take one of them they choose the Operating system programming language, whatever they want and they just develop it accordingly and Because no software is running on the customer side, which you provided that Frease you from a lot of the support activity because it's a customer software and the customer knows the software because they wrote it and They can support it themselves Yeah, I would eventually go to get to it at the end in reality. It's of course not that There's no support activity which you have to do so at the end. It's your customers and if you want to keep them happy then Okay, sorry to repeat the question. So the question was Whether it happened that the customers Call to us and complain that their library is buggy and complain that the protocol is wrong and not working so this definitely happens and If you want to have happy customers, then you have to deal with it somehow but it happens a lot less than when we really shipped the libraries to the customers and Later during the slides I will get into more details what to do so that these situations don't happen Too often at the end if they reach out to you then yes, you have to work together. It differs a lot customer to customer Some customers don't call us at all. Some call us very often with let's say Simple questions which they might be answer able to answer themselves. So I mentioned that with a good protocol This protocol API might not necessarily mean that you are just uploading work on to your customers So what makes a good protocol? There are many many different technical things which You can use to define a good protocol. You can talk about performance throughput latency reliability What was important for us and what's important for the protocol APIs is the openness of the protocol and That's why we have chosen a MQP as the protocol of our choice a MQP has quite a lot of history. It was created as a kind of Common project between IT companies and financial companies, but and that's why this talk is in the IOT stream Today, it's very valid also for the Internet of Things There are some older versions of the protocol in particular 0 9 and 0 9 1 which is basically a synoning for a bit MQ it has a Quite a huge community if you search on github for a MQP I'm pretty sure most of the results will be for a bit MQ But it's still basically one product one community The same applies to the 0 10 version of the protocol, which is actually where we started Some years ago The 0 10 is implemented in the Apache Cupid project eventually sold as redhead merge messaging by redhead, but again, also it has a good community It's still just one community and that's what's different with the MQP 1.0 and If you start today, that's what I would recommend to you because with a MQP 1.0 You don't have any more one community, but you have much bigger ecosystem. It's not only Apache or redhead who implements a MQP 1.0, but already today you have working products from Microsoft IBM, but also from some smaller companies. So you have much bigger choice and if we talk about openness it's Approved as ISO standard and you probably can do Much better in terms of open standards than having an ISO standard but just by choosing the protocol That doesn't really create the protocol API is because the protocol is just mean how to do it But it's not the API itself So what do you need to do to specify the protocol API? You need to clarify network security you need to specify how to send or receive messages and Many more things including documentation and support So let's start the network AMQP is usually running over TCP. So you can basically use the public internet infrastructure and If you would use it for Internet of Things then that would be probably your choice in Deutsche Berse We are using usually VPNs and lease lines, which give us more performance security and stability But they can get also expensive and they are not very suitable for everyone Also, if you want the API to be used directly from web browsers You should definitely consider WebSocket support The binding between AMQP and WebSockets is slowly getting more and more final in the Oasis committees which are standardizing it and That allows you to write applications which talk AMQP directly in the browsers using JavaScript a bit with the network related is encryption AMQP supports both TLS over AMQP as well as AMQP over TLS the second one Seems to be the usual choice So you open the normal TCP connection on top of it you establish the TLS channel and only then you open the AMQP connection I would say security encryption today. It's something what should be considered as mandatory but Which versions of SSL or TLS protocols should you support which? Cypher suits lately there were a lot of different issues with some of the versions some security problems when you start from scratch, it's very easy to Select only the the versions which are still considered secure which are the latest Versions and that's very easy, but over the time if you are running the API for several years It gets more and more complicated. So our oldest API I think it's running over six years now and some of the clients who were brought on the beginning by our customers are still running there and they are actually still working and every time when you update the TLS configuration and Decide that you will not support some older versions you risk That some of these old clients which are maybe running on some operating systems, which are not really updated Since years might not be able to connect. So it's always a bit of trade-off between the Security and between the compatibility with the customers Depending on your data the security should probably win at the end But it's important that you don't stop supporting SSL 3.0 just from scratch from Tomorrow, but that you discuss it with your customers and give them some time to test and prepare their applications once the TLS channel is established the AMQP connection can be opened and The client can do the authentication AMQP supports Sussell that's a simple Authentication and security layer which gives you many different options You can start with something as simple as username password, but you can use Kerberos or as we do you can use TLS client authentication, which is done based on the SSL certificates You should be aware that the more unusual mechanisms you select to use The higher there is the probability that some of the clients around there will not really support it and it will not work with your API Basically username and password is supported by every client, but with the other mechanisms It gets more and more complicated In Deutsche Börse we are using the SSL client authentication That basically means that the identity of the user is taken out from the TLS channel and is established based on the certificate Used when opening the connection. So basically the username is usually constructed based on the subject of the certificate Well, but that's still not everything you can use different ways how to authenticate the client the Usual way on the normal internet is to use the public certification authorities that basically means that You don't say Exactly which clients you trust, but you leave this to a certification authority which you trust that can lead to a lot of different complications mainly because as I said the username is the subject of the certificate and Your view on who the user is is Usually not the same view as very sign has so you your customer may be under some customer ID or You gave him some username. For example, we have something what we call member ID, which is a five Character code which car identifies every customer But that's not something what's known to the certification authority. So they will not issue a certificate which is based on this Identification which we are using so that's not that great option on The other end of the spectrum are self-signed certificates Which is actually what we decided to use and what we are still using the self-signed certificates have the advantage that You tell your customers generate a self-signed certificate and provide us in a secure way with the public key And because they are generating the certificate and signing in on their own if you Tell them use this as the subject in the certificate They will simply do it and you have in the subject exactly the username which you need but that still doesn't solve The main problem because it brings together another issues one if you say self-signed certificates for a lot of people that means unsecure which Maybe is not necessarily true but You want your customers to use your API so You either have to convince them or we have to do something else other problem with the self-signed certificates is Then it was one of the things which was not really supported in all the different clients So during the years we spend a lot of effort Either on our own to implement the support in the different clients or to push companies like redhead to implement this in their clients and Last but not least the problem is not only in the clients. The problem is also in the messaging brokers because currently we are still using the Merge messaging broker, which is based on Apache Cupid and That has a great support for using the self-signed certificates, but if you decide to use active MQ There the support is not that great anymore. I'm not sure you can really do it as easily as with merge messaging So I wonder is there some third way which might work better and To be honest if I would be starting some new API today, I would probably try To use private certification authority where you run your own certification authority Your customers don't sign their certificates on their own But they send them to you so that you can sign them. You still get the advantages of the self-signed certificates That you can define the user names, but It's not public CAs so it still doesn't always Sound as something what should be trusted especially if you are dealing with some banks and financial institutions In other point maybe in the internet of things Using public CA can get probably very expensive if you have separate certificate for every client So that was the authentication Authorization it's something what Should be on the API as well I won't talk too much detail about authorization You should use the principle of least privilege. So You should give every user only those rights which it really needs but I would talk more about Different resource limits because that's what's missing in many of the Messaging servers and that's what can make your life More complicated or more easier depending on the support Imagine that you are running some service You have some customers connecting and some hacker takes down your server and the customers cannot use it anymore that's a Bad situation, but it's even much worse if It's some developers somewhere on the other end of the world who is writing a new application It's developing it against your production system then starts half-finished application and Goes for lunch The application is maybe creating more and more temporary queues until your server runs out of memory and dies and That's something where the limits can help. So if the Messaging server supports it. You should definitely set all the different limits for maximum number of connections maximum number of sessions maximum number of Temporary queues maximum message size maximum message rate everything what's possible Should be limited now with all the basic stuff in place the clients can start sending or receiving Messages so the entry points are basically the structure of your queues or topics which you are using to deliver them We kind of learned several rules which Help us make better interfaces if we follow them or maybe the other way around By not following these rules. We made a bad experiences The structure of the queues and topics is usually very hard to change Once you have hundreds of clients running somewhere and connecting to it So that's something what you should very carefully think about Before you really have the first release of your API and let the customers connect to it Another thing with which we made a good experience is that Every user should have its own namespace in the In the queues and in the topic so maybe start every queue name with the name of the customer because that helps Very much when you are analyzing some problems on or when your operation teams are running the API Yeah so So the question was whether with user whether it's meant the actual customer and user or the project so here I Mean definitely the end user But you can interpret it in your own way who the end user is so most of our Interfaces the end user is the company which we are talking with and not really the Individual the person which is Using the application which is connected to the API Together with the namespaces Another very useful you rule is that you should definitely not use any an anonymous Object so something like 10 Q 1 2 3 4 5 6 Doesn't help you when you are Operating the system or when you are analyzing some issues in the logs because you have no idea who does this queue belong to who created it and so on and Another useful rule is that you should always Create rather more queues than less because in the clients which the customers will write It's usually much easier to write more receivers to receive from 10 different queues Then to receive from single queue only 10% of the messages may be based on some binary payload Which is inside the message You have to be aware that not every Question is has a yes no answer. So the API is don't always work In a simpler request response way, that's what looks nice in the simple examples, but very often you need to create some workflows which Include multiple different requests and responses or maybe which include even multiple parties So one customer requests something the server technologies the request then asks another customer better He approves this transaction and only when this other customer approves the transaction the transaction is actually completed and entered into the system and All these workflows need to be properly documented Yeah So the question was whether some metadata can be Associated with the queue to identify the user I would say that's Not really part of the MQP protocol. So within the protocol, you of course know which user created the queue But that's not always obvious from the software and the way how you can do it. That's then a bit specific To the implementation. So the Apache Cupid or Apache active MQ. I think they have something called queue policies which allow you to The temporary queues be automatically named using some patterns, which then might help in these situations So that the random names are not created like this, but that they follow some Schema. Yes. Hey, so the question was whether there is some limit. How many queues can be created? I Would say again, that's specific to the implementation which you are using So on the Apache Cupid, it's definitely no problem to have thousands and thousands of queues It of course a bit depends How many customers you have or how many connected clients? Let's say that our interfaces have usually hundreds of customers if you for example implement Smart toothbrush and sell it to millions of people and all these toothbrushes send some Messages to you then you probably cannot afford to open many many queues Then in that case, maybe you can help with something else Most of the implementation supports some sort of filtering when reading from the queues, but The filtering is easy against some headers or some application properties But the filtering is very hard against The message payload which might be some binary data or some complicated XML So by using some application properties to distinguish the different types of messages It's much easier to use only single queue, but for the clients to use the filters On the messaging server to get only selected messages. Yeah, okay, so I will try to repeat the question I hope I will get it right So the question was whether using the namespaces with many different users and with for example five character codes Identifying the users whether it's just doesn't create more cows So Lot of the problems are At least in our case a lot of the problems are not really discovered by our teams We are operating the API, but a lot of the problems are customer calls the customer support and says look this is not working for me and Then when you have really the names in the queues It's much easier for you to find the queues used by this customer and to see are they full of messages? Are the queues and they is someone connected to the queue might be Let's say that in our business. They are not that many potential customers So maybe that's a bit easier for us. So let's talk for a moment about the message payload AMQP as its own type system or at least each version has kind of its own type system Which it's great which is great and supports a lot of stuff, but It's unique for AMQP. So You can use this type system to encode the message bodies, but then You might not be really free to decide, okay Let's use also MQTT as the alternative protocol or let's use web sphere MQ as a Alternative for connectivity because these you would basically need to transform these payloads into something else because some MQTT client will not be able to decode the AMQP payload luckily you can very easily use text or Some binary data as a payload in the AMQP messages So you can choose whether you want to use XML or JSON or whether you want to use for example something like Google protocol buffers or B You should be aware that Human human readable formats might be intriguing because it's easier to analyze all the problems, but AMQP itself has no built-in Compression currently so if you use something like XML, which is very Verbal then you will get the big messages and you need a lot of bandwidth. So maybe using some binary formats Will be better for you It's also when you are transferring a lot of numbers, which is what we usually do decoding these numbers from the text formats into some floats It's quite Intensive on the resources on the hardware resources. That's again a lot easier if you have binary encodings and There's a lot more then Just these things We talked a bit about it on the beginning. How does it work when the libraries are not good enough? The first problem which you will run into is that the documentation of the different open source libraries Is not good enough. There are usually many different code examples, but they are simply too primitive So if they contain some authentication, it's always username password based authentication They are often just showing how to send one message, but they don't show you how Send or receive thousand messages in very fast pace Yeah, also, they usually contain just some hello world content and not the real binary data Which you want to send over the protocol. So at the end you will always need to create some programming documentation and Prepare some code examples so that your customers know how to use the different clients and Over the time you will definitely see Many negative use cases so things which some of your customers did and which didn't work really well and That's another thing which is very useful for documentation purposes Unfortunately, I'm Saying here that the open source projects usually lack proper documentation and examples and I have to say we wrote our own documentation and examples, but didn't really manage to commit it back to the projects which Is Unfortunate and I should definitely find some time for it Another problem which you may run into is the compatibility. So on paper, it's everything perfect You have a client from vendor a which supports a mqp 1.0. We have a server from vendor B which supports mqp 1.0 and The standard is is a standard so what can go wrong? I Was actually positively surprised that we really have Clients which were not updated for the last six years Which are still able to connect against the latest versions of the messaging broker I on the beginning expected that the compatibility would be much worse But it's still great if you can find some time to test all the different clients Make sure that they work somehow at least with some basic tests against your API is that they can authenticate that they passed authorization and then you can create some Compatibility lists so that your clients know What's compatible and what's not yeah so the question was whether we still have to support a mqp 0 10 and Answer is yes We are still running Most of our installations on the merge messaging software which supports both 010 and 1.0 and we are basically now in the process of Getting the customers to the mqp 1.0 So that eventually once everyone is there we can kind of cut off the support for old protocol and maybe to move to some newer applications That's That's one of the advantages of the merge messaging or Apache QP project that It provides you a clear migration path To the 1.0 protocol you don't have to say okay tomorrow I will stop supporting a mqp 10 and Start supporting only a mqp 1.0. Okay, so the question was whether we have Separate versions of the 1.0 or 010 APIs or Whether we have some obstruction layer to make Basically the same thing work with both at the same time Yes, so in the Apache QP there are much messaging broker There are simply some cues or an a mqp exchanges which are coming from the 010 world and Once you enable the 1.0 protocol you can Have access to the same queue so we didn't really change the queues or the exchanges and the topics at all We basically just let the users access them over the new protocol. So At the end, it's just one API for us Yes, the broker does all the magic it translates the messages from 1.0 to 010 and vice versa Another thing which is useful for the compatibility is To test the upcoming releases It's always easier if you manage to find out that the new release of the Cupid proton client doesn't work with your interface and maybe get it fixed before the release because once it's released then You will get into a problem that your customers download the latest version start developing some application then they found out that it's not running and That can be avoided if you test the upcoming releases when they are in beta or release candidates question Yes So the question was whether we have some set up where the customers can participate in the testing we have something what we call a simulation environment which is Basically identical to our production environment. It's operated by the same team and And that's what the customers or their vendors who are developing the applications for them can use as As a sandbox for testing so when we do some upgrade we always do it in this simulation environment And only a few weeks later. We do it in the production to give everyone time to adjust for the end maybe as a inspiration One of the interfaces which we have is called you're excreting fix a male interface and it connects One of our subsidiaries You're excreting with its customers So the clients on the other end connecting to us are all the different major banks funds who Are using our exchanges? As I said the it supports the zero ten and one point zero protocols Additionally to that we have as in connection alternative IBM web serum q because When we started with it some of our customers who have a lot of web serum q installations Insisting that we have to use web serum q on the beginning. It was a lot of them, but during the time most of them found out that the a mqp is very easy to use and that It's not as unreliable unstable as they were afraid on the beginning and I think we have only single digit number of customers who are actually using the web serum q connectivity and all the documentation to these interfaces is Public so if you want you can have a look at it. There are some programming guides With the code examples in different languages and so on and that's it So if you have any further questions, yeah in the The a mqp zero ten Just to repeat the question the question was that are filtering of the messages is done on the server side on the client side On the zero point ten protocol. There was no support for filtering on the server. So Usually If there was some filtering then it was on the client side That's much better on the one point. Oh version where the filtering is support is kind of building to the protocol So I think all of the one point. Oh implementations When creating a receiver from the queue you can basically specify your filter and Then the broker automatically gives you only the message is matching the filter it then depends a bit on the product which Exact filters are supported or not. It's usually quite easy if you want to filter based on some application headers but It gets much more complicated if you want to filter based on some binary payload or even XML payload That's all time we had for this presentation. Thank you So if anyone has any further if anyone has any further questions, I will be around so you can ask And those who ask the questions they can pick up the scars here