 Hi, my name is Klebert Sokonik, I work for Red Hat. I'm on the ActiveMQ PMC. I'm one of the commuters on ActiveMQ and Artemis. I have been pretty much involved with messaging for the last, I don't know, like the last 10 years. So I have been working with a lot of messaging products on Red Hat, beyond the hype of doing HornetQ at the time, like refactoring J-BOSS messaging as HornetQ. And later on, I have been involved with Artemis, which is pretty much a continuation of. It's not much a continuation. That's what I'm speaking in a few minutes. So when I submitted the talk, I wrote because Artemis 2.0, it's already 2.1 right now. So the stream of the release is being pretty active. And there is a lot of stuff going on on Artemis. I'm going to be speaking about Orangine and how Artemis started, doing a features and doing an overall of everything. We have a short audience. Actually, I'm competing with Kafka right now next door. So it's not really fair for me. Like, a Kafka has a lot more people are interested in that. So this is a message broker. And as Adrian said in the previous presentation, message brokers becoming more commoditized. So it's actually a pretty hard space to compete on. It's like a really complicated to get it right. And it's taking me a lot of time to time in QE and quality engineering and running tests and doing a lot of stuff on that. So Artemis, I don't like much of the show in hands, but since we have a short audience. And by the way, if you have any questions, don't wait until the end. I consider myself a bad presenter. So if you guys have questions, you'll actually be more interactive, be easier. Artemis is a message broker. It has the original horned queue. So it has a lot of, it's used to be pretty much a JMS broker. It's a little beyond that now. It's messages are streamed to be in memory. Like, we always favor like a fast delivery and with guarantees of delivery. So messages are acknowledgment and guaranteed delivery. And we have a lot of, we have like a pretty strong paging system, like when you are running out of consumers or your consumers are running behind. Horned queue started in October 2014. Horned queue actually has its pretty old history already. So it's starting Jboss MQ. Like I have, we had a really heavy refactoring to Jboss Messaging. Jboss Messaging was having a lot of JDBC dependencies, like a lot of, like a JMS and was really heavy on another penis with the application server. So Horned queue, we did a lot more stuff into making it standalone. But still the main use case for Horned queue was to Wildfly and Jboss 7. And more recently with Artemis, it became more like a standalone product. And it's actually the foundation for Jboss MQ 7. We, Red Hat, just did the announcement a few weeks ago. And I'm going to be speaking about the open source version and what can you do with Artemis. Even though this is a foundation for MQ, I have seen lots of users like using just the bits from the open source streaming. So even the open source version is pretty mature in the production ready. I have seen customers doing like a billions of messages a day with large farms. And it's a pretty stable product, I would say, right now. So the reason for moving from Horned queue to ActiveMQ was, first of all, Red Hat contributed for ActiveMQ and was also doing Horned queue. And also inside ActiveMQ, we had a super project called Apollo because a lot of what we needed to bring ActiveMQ beyond on performance would require a new infrastructure, like a lot of NIO code. And it was not very easy to develop ActiveMQ into that. So they tried to up Apollo. And Apollo didn't really succeed. And then we started with Artemis. And we have had a lot of streaming on the community, and since we started, we have over 3,000 commits since we joined as Apache ActiveMQ. One of the good things about that is we could get features from both ActiveMQ. So now we have open-wire support on Artemis. You have also a very strong XA. One of the weakness on ActiveMQ was known for was it's XA, like if you need a transaction with an application server, Artemis has a really good support for that. We also have a West GI and things that came from both sides. And as a foundation for JBoss AMQ, we had improved a lot the AMQP support. So it's really high performance. I could run actually right now into my laptop a QPdCPP client and generally 70,000 messages, non-persistent messages a second on the laptop, even on MQP. And this benchmark here is called Quiver. It was developed from one of my co-workers, Justin Ross. And you could use that to measure performance on different protocols. And any of you guys are using Artemis or ActiveMQ here? Like how many of you guys are using JMS? ActiveMQ. What version have you played? With Artemis. So I'm going to speak a little bit more about why we became with Artemis instead of changing ActiveMQ. We had some talks about what would be required to make ActiveMQ in IO and more reactive. And it wasn't so easy to change the code. And so as I said, like a HornetQ was donated to ActiveMQ. And we had this idea that you could do asynchronous responses right into the server. So whenever you get a client talking to the server, things will go through what you call internally an operation context so that the server never blocks a thread to help the client to serve the client. So that's actually the main difference. And why we wrote Artemis from the, why we brought Artemis to the ActiveMQ community. So it doesn't mean different, actually. Like say if you are writing to the journal, like I say, sending a message on ActiveMQ and you write to the journal, you will have a thread blocking on ActiveMQ. On Artemis, you fire, forget. And you basically, on this class that you call a cooperation context, we just set up a runnable. And whenever the IO or the replication is done, it will send back to the client. The client will think that we are blocking because you need, like, guarantees that the message was sent and was accepted by the broker before it came to block the client. So the client will think the server is blocking, but we won't have never a thread waiting to be blocked on the server. So it's our, I think. Is it still preserved that you don't say something about it? Yeah, so from the client point of view, nothing changed. But on the server, it changed a lot because it's actually interesting because when we came up with that, like a doing development, the server was not scaling to what we needed. And just by replacing a blocking thread to this kind of thing made the server scale and be a lot faster. And it was a few years ago, but that's when we could get, like, a spec GMS numbers really high. And we actually broke the benchmarks nobody used ever since because it came so fast that nobody ever did a GMS after that. With Chio, MQP became so well maintained that for a while it was something like it was there, but it wasn't really the main focus on development. But like I think it's Artemis 2.0, MQP is becoming really strong. It still supports the Hornet Q and Artemis clients. It's actually still the fastest. So if you need just like a Java client sending messages, GMS, or even like a Decore API, it's still the fastest way. It's just saying like we can send like maybe like a 200,000 messages a second easily on Java. But if you need something like for different clients, it's like MQP will provide you. It's too bad I don't have a laser point, but MQP will provide lots of clients for that. We also have MQTP stomp and open wire for legacy support with active MQ5 clients. And the last three years, we have been doing a lot of testing on this to make sure it's stable and working. And I can see a lot of going on in the community for that. Like a lot of discussions on it. So we're talking about the producer side. When we did the, yeah, I'm trying to, I understand the question, but I'm trying to repeat the question. So you're asking if using a synchronous on the survey that improved the performance on the client. Yeah, when you use no blocking, we are not. So I'm talking about, so the question is when you use no blocking, if that improved the performance on the client, especially from the producer side. It didn't improve the performance if you had a single producer, because you would have like a single thread on the server. But for the general case where you have like many producers, it scaled quite a lot. Because you could like say if you have like 10 producers signed into the server. So we don't flush on every request. So we have on the operation context and we have something that's called time buffers. We wait a few nanoseconds before we flush our rights to the disk. So if you have like a 10 producer signed into the server, we will listen for all the 10 producers before we do a flush to the disk. So you reuse the right time between multiple producers. And for doing that kind of NIO, it improved really well. So up to the point that if you had one producer signing, when you send like a persistent message, especially if you are requesting to sync on disk, so you cannot sync faster than the disk is capable of syncing, right? So like for having two producers, it would still have the same rate because we are using the syncing for multiple producers. And that's when we were able to scale a lot more. That was one of the benefits of doing NIO on Artemis for being able to scale for multiple producers or even consumers when they are acknowledging rights, so you have to write to the disk. But also the contact switch on the server was minimized. And also the memory on using a lot of threads was required like we had cases during development even before releasing that if you didn't have kind of like a context in place, we would run out of memory quite easily depending on the number of clients. So you still have like limitations from the server and everything, like everything else, right? So but we could use a lot of more hardware by doing the kind of trick. So if that remains the same with the consumer, I think it, not I think it certainly does because the consumers will also be doing acknowledgment, right? So the acknowledgment will also have to translate into a right on the disk, right? And you have like a multiple consumers acknowledging. If the question really depends on how you're consuming. So if you are consuming, I forgot the name of the term in GMS, like when you consume like a no-acc, where you just acknowledgement lazily, it's probably not much a difference because it's just right. But the fact that we had like fast storage, like it improved the acknowledgement time from the server to the clients. So it definitely improved, not as directly as you can see in the producer, but it did, it certainly did. So for MQP clients, we have QP GMS. This is, this light here is actually old. It's already supporting GMS 2.0 or 2.1 now. You have like a C++ for and Python. So actually do need to read that. But like one thing I recently did some testing is even on Azure or MQP.net, it works quite nicely, like even with AAR, with.net. Recently on Artemis 2.0, that was one of the latest change that we did before releasing Artemis 2.0 was changing how messages are treated internally up until recently a message would be converted. Like I say, if you're dealing with MQP on Artemis, up until recently we would convert the streaming from the message from the MQP message into a core message because it was a core broker had some protocol, it's a message format before. And recently we changed that to be totally agnostic. So if we're sending MQP message, the MQP message will stay in its format the entire life cycle. So there is no conversions when you send it to the client who send MQP server will keep it as an MQP message. And when delivers to the consumer, it's just like transferring the bytes to the consumer. So that means that Artemis now is more like a pure MQP broker, actually agnostic. We didn't do that to MQTT yet, but MQTT actually doesn't require a lot of transformation before transforming between a core message and MQTT. So it doesn't really impact a lot on performance, but it's something that we can do pretty soon. And this kind of thing, there is another sub project inside a party called Cupid, I don't know if you guys have seen Cupid. You can use libraries from Cupid, the dispatch broker, I have a slide about it where you can do interesting bridges in your data center for networking. And having this as a pure MQP or pure core, like depending on how you send the message, it really made a lot of differences. Another limitation that we had for between 1x and 2o, this was actually something that we inherited from Hornet Q, it's like we had a prefix on every destination. So like whenever you created, especially if you were dealing with JMS, like a Q would be like a JMS.Q, whatever name you choose, and or JMS topic, whatever name you choose. Now we don't have a prefix anymore, but we have like a different types of addresses so we can have like any cast, a multicast. It was still being internally mapped to either, you could map in JMS terms between topics and Qs, but it's just a more generic term. You can do a few tricks now like determining how many maximum consumers you can have. Those are a few interesting features that we have on Artemis right now because of the Addressee model. These two changes actually knew the new Addressee model and Artemis being able to deal directly with the MQP is actually what drove us to split the 1x into 2x. If it wasn't for those changes, it would be just like another major 1, 7, 1, 8 release. So when it did that, we had to jump for it too. In the serving component terms, we have like clients from different protocols. We have our own internal client, which is a core client. The networking, we are dealing with Nettie. Nettie actually started trusting and working at Red Hat at the time that they were writing Nettie. So like Hornet Q was actually the project where Nettie started. It became independently after that and actually they left Red Hat and they're doing a lot of higher flights now, but it's really tight coupled with, like it was built around Nettie and around the persistence, the way we do it with persistence. We have a protocol manager connected directly with the networking and to the broker and we can plug protocols quite easily. It's quite easy to develop a new protocol, especially after now that we internally can do special messages in their protocol format. Just going a little bit more in detail on how the context switching works inside Artemis. On the application context that we have, whenever you add some information to the journal, we also add what we call the, it's not actually a runable, it's actually IO task and it could be doing into three types of journals that we have. One is native Libio. Libio is like a native API from, actually it's quite at the kernel level. Libio, it's like when you write something to the journal, you're writing something into it's mapped directly from the memory directly to the kernel. It's quite fast and it's used a lot of less CPU, it's quite low on CPU. We also have like a regular Java NIO for like a regular files. It's still fast, it can use a little bit more CPU because of the memory copy between the buffer and the kernel when you write the message and recently on two zero and two one, we added a map or memory mapped file. And performance in numbers are going quite fast or a lot, lot faster. Even faster than what we did with Hornet Q. What happens with the talk of them actually making it crashous. The IO task, what will happen to the IO task if the server crashes? Up to this point is the same as if you had like a thread blocking, right? So it doesn't change much from that point of view. So if you had a thread blocking on the server waiting for the IO to happen before it responds back to the client, it doesn't change much. So if the server crashes, those are your tasks will be gone, right? So and the client will receive a signal that the server crashed and it will then perform a HA action will connect to a backup or to try to reconnect to the server as soon as the server is restarted. So when the, so just help me reproduce your question. If the server, in a traditional way, it will be blocking. Yeah. Yeah, so. It's the same because it's still the same as if you had a blocking or non-blocking. So I'm never telling the client that the message was synced into, it was actually synced. So it's the same semantic, but instead of having a thread waiting, I have like a data structure somewhere with a callback for as soon as the data has written to the disk. From this point of view is exactly the same semantic, doesn't change anything. So if you are looking into, so it's the same thing if you think what will happen to the thread, if the server crashed, the thread will go away. What will happen to the IO task if the server crashed? It will go away. But if what could happen then if the server crashed is like whenever the information is persisted, then you confirm back to the client. But then you have the same semantics if you are blocking or not because then you could write to the disk and you could have a crash at the time that you're gonna send the information back to the client. Now the client has to do some duplication check, XA or to make sure that the information was stored there before. It doesn't, like the fact that we are doing this synchronous doesn't change absolutely nothing. Thank you. Okay. Is it tunable how often flushes? If it's tunable how often flushes? Yes, it is. When you, I'm planning to do a little demo. I'm not going to do much, but it should show what is done. Is there any consequences for this, is it to this in terms of, you know? If there's any consequences you order, how? Or if there's a broken crash. No, there is no consequences for ordering. It pretty much happens in order. With Lib, if there is any consequences you order in, it happens in order because the IOTask is actually guaranteed you order. It's never answering a question. Doesn't asynchronously pull the lock? No, no, it goes to a list. It doesn't pull off out of the list. There is one internal thing with Lib Ior because when you write, we're using Lib Ior. Lib Ior, it's sending a signal. So there is a library in Linux called Lib Ior, but everything that's done is just sending a signal to the kernel. It's really shallow, it doesn't do much. And the kernel may decide to write out of order. But then on the structure of the journal, on the structure, I have to guarantee that the whole set of the transaction is there before I can reuse the information. Actually, if you look at the journal internal format, we have ads and deletes in a file. As soon as the file is full, we actually pre-allocate a file. The default size is 10 megs. You can configure that to whatever size you have. As soon as that's full, you have records and file to add updates and then you have a delete, right? So as soon as you delete the whole file, the file will be ready to be reclaimed. And when you reclaim a file, we don't delete the file. We just set a header with an ID instead of filling up with zeros. Then the whole file gets back to the reuse. So that's way we can do a very fast writing on the files. There are a few other internal things that I didn't put in the picture, but as part of those ads, you have a transaction ID for each record. Even when you're not sending transactions, the broker may write with an internal transaction because you don't want to have a message and not have the ad to the queue eventually. Because you write, store the message, then you add a reference to the queue. And then sometimes we do internal transactions even though they are not supposed to the client. And then because of the kernel and Libra, you decided to write in different order, we want to make sure that everything is there before we actually read, okay? Does the journal ever, if you have unacknowledged messages? If the journal ever gets compacted, there is a situation where the journal gets compacted with those ads and the leads. Eventually, you could have what we call a linked list effect, add, delete, add, delete, and like if you have one record hanging at the beginning, it will hold a whole list of files. So what we do is, so if the file gets too big, we call a compacting and with the broker running, we copy to a new set of files and start fresh on that. So yeah. So it does eventually get compacted? Yeah. It doesn't work the way the active queue does, where if you have unacknowledged messages, that whole neural intent will still stick around. Okay, I didn't know that. Yeah, so the question is if it's different in the active queue, yeah, it's different. Yeah, in active queue, you have one unacknowledged message in there. It will keep that 10 tags on disk until it expires or you... Yeah, it's different. We have one, right now we have one journal for the entire broker. So like all the messages are stored in the single journal. Unless you're paging, like I'm gonna speak about paging in a bit. So the... I actually saw a joke once on internet saying like a write-only devices that's never read. So it's actually as the broker is running, it's write-only and never read from the journal. So it's built for fast recovery in case of the broker crashing. This is actually why you want a message persisting in a messaging broker, just to recover from disk in case of crash. So for that, the message will stay, it's not, we never read from the journal. So unless you restart the broker, right? And it's fast writing. It's different from paging. Paging was actually reading on Kafka. This was written many years ago but Kafka has like the partitions, right? So we call page files. So you write to a page file, you keep growing a page file in case you are writing beyond the capacity that you configured. Then you start using these partitions or page files to go beyond what would be capable of the memory. We do reading cache. In that case, when I read something from paging, I read the whole file. So like I said, if I'm reading 10 megs of files, then the whole file will be in memory. I think I'd keep up to two page in memory that's configurable as well depends on how many consumers you have, right? If you don't use transactions, this works really well. It's, there is no limit for that. If you do however is using transaction and paging, there is a limit because transactions still need to be stored in the journal. It's, I mean, you can go really far with transactions, but if you're sending one transaction for every message, you're still using a lot of data structure in memory for keeping up with the transaction. So if you plan to use messaging as a database or to stream a lot of stuff, be careful with transactions and paging. So basically a transaction will be required to stay in the journal. I think this slide here where it's talked about it, right? So when the client sends something to the context, the context will take care of like a writing to the disk. It will also write, the context will also be responsible to talk to the replica. In case you're using HA and replicas, the replica, whenever you start a new replica, it will catch up with the server. It will copy everything to the server, to the new replica, to the backup, and it will then start doing a synchronous write. So you write to the disk, you write to the replica, but you don't need to write to wait a sync here. You just need to guarantee that it was accepted by the backup. And it's on a lot of tests that I have done, it has a minimal delay impact on the master. It will add some increase a little bit of latency, but it's not, it doesn't affect much on the throughput. It's really well behaving at this point. Does it not acknowledge the client to allow that? It doesn't acknowledge, no, it does not acknowledge the client to propagate the replica. Yes, we don't send confirmation back to the client to it was written to the replica. However, I don't need to wait a sync on the disk for the replica. I just need to write sync for the disk here. Because it only has to be a memory on the replica. On the replica, it will be written to the disk, but it will be on the memory for the, I mean, it could be on the cache for the disk. Yeah, it could be in memory already. Yeah, because whenever the server crashed, you have to close the file before you activate the server so you don't need to wait for a sync on the replica. That would be too much going on if it was the case. For our high availability, we have replications, as I said. We have some split brain protection based on PINX. We have a component that is called like a network health check. You can specify a list of PINX addresses in case you want to configure what to do in case you pull up the cable of the replica, right? ActMQ5 actually, well, the replica that you had, like if you lost the connectivity with the live, the live would stop itself, right? But what to do if the replica crashed and not the server? Then you lost both nodes, right? So what we do is to differentiate between if the replica crashed or if the live crashed, we can check in a given AP just if the network is still available before you can actually enable or activate the backup, activate the replica. And the other way, it's the best, it's using more than three nodes using a quorum. Right now we do our own quorum. We have been talking about using ZooKeeper in the short term future, but right now it's still using our own quorum and we're voting for that. Another option that we have is like a based on shared storage. Right? That'd be a switch over, like... If I would keep the internal code... Yeah, if that's the end of the simpler setups. Yeah, I'm not... First, let's hold on to... I'm talking as a developer. I wouldn't drop the code. I would add as an option to use ZooKeeper, not looking to remove the code for the quorum. We may improve the code when I remove it. So I can see the way I look the code whenever I'm refactoring to use ZooKeeper, I'm pretty sure that we'll find different ways and it will keep the same semantics and what you have now. ZooKeeper is pretty good for big setups, but for a point, it can be a lot better. Yeah. Okay. So we have, with Artemis 2 now, we have two ways of clustering. One is the internal clustering based on bridges or what we would call a network of brokers, not TMQ5. But with Artemis 2 and the way we're dealing with MQP now, that there is one interesting project in Cupid called dispatch router. This is actually, if you guys look up on the documentation around JBoY's MQ, it's what it's called Interconnect on MQ. It's actually the same thing. Basically, whenever you don't connect a client directly to the broker, you have clients connecting to the router using MQP protocol and you could have like a pretty extensive pictures on this, right? You could have like a many routers. You can do actually quite complex setups with this. It's thinking about like a big networking and supporting like millions of connections on IoT scenarios. That's the kind of thing that this would go after, right? So like a Cupid's patch would actually improve quite a lot. The next slide, actually this is my last slide, after this I'm gonna do some running a broker. It's actually quite recently we have been using, we have been using NetE for forever. And we now using the food buffers for everything internally. Like when you write to replication, when you write to the disk, when you write to paging. And that's decreased quite a lot of the garbage collection pressure. And the next step that we are doing now is like when we have the message, the message will also have a body on the server. That body will also be using a food buffer. I'm sure that will improve quite a lot of things. It's actually what I'm working now it's on this step right now. There are the things that we have to improve. Like we're still looking to do admin console and that we have one on internal for Red Hat but we need to bring one for actually even active and the Q5 needs a refacing on the admin console and the website of course. This would be a bit challenging for me because I have to look at the screen as I do this. When you download Artemis, you have this little Bing executor here where you can create a broker. So if I'm gonna call Artemis create, I'm gonna create a broker at my presentation, work my presentation. What's happening here is asking me a user, a password, if I allow an anonymous access, pretty much I already created a broker. But what you see here is auto tuning the journal. So it's saying that my disk can do 25 writes per millisecond. This is actually calculating the most optimal value for the time flush for the disk. So the timeout for the journal timeout here will be 40,000 nanoseconds. So that means the broker will issue a write to the disk every 40,000 nanoseconds. So if I go to my presentation, I can start the broker and there are a few tools on the CLI, one that I liked a lot as the producer. So it's saying like 1,000 messages in 648 milliseconds. If I do again, it probably will probably be like a 500 milliseconds for this much. And if I do consumer, it received like all this 1,000 messages. This is actually pretty fast and it's still guaranteeing, like a guaranteeing, making guarantees about the acknowledgement sign. And on the ATC, you see the broker configurations. If you want to like to change, say if you ever change the hardware, you could change like the time writes and do a bunch of stuff here. And the last thing I wanted to show on this crash demonstration, there are lots of examples here and all of them are runnable. So say if I go to topic, if I just type maven verify, it will run the broker and run server and do everything. So all these examples are runnable and you can use them to debug or making sure you understand the broker. So just to conclude, Artemis is very rich in features. It makes a lot easier in the client. So it's not just JMS. And I hope you guys enjoy. Could get some information to you guys. Any more questions? I was curious about the, is Artemis gonna get to that point? Is that how it's a little less feature rich than the active view over it now? Yes, about the question, about the question is if we have an a console, we don't have a console right now, but we had some missteps with the community about like the way that we were going to do a console. And now we, since we deliver two zero to one, we can dedicate some time into that for, it is on the world map, it is on the world map actually. Well, I'm not sure if it's on the public road map, but it's, I'm definitely, as a developer, I'm looking myself and doing that. It's just like it's so much being done on, just refactoring an MVP was a lot of working to get that. What, how difficult is immigration for Artemis? If you're not using, I think the only thing that's stopping some people from moving is what they call a virtual topics that you don't need with JMS too, because you can have different ways of creating consumers. But if you're not using anything special like that, it's pretty easy, just JMS makes that easy. Like even with support, even the active view clients with open wire clients. So it's not very difficult at all. Data porting, there is an export that was just released from ActivityQ5 that can import an Artemis. So I mean, thank you guys, I have to, cheers. Thank you. Thank you. Thank you.