 Hi, my name is Karsten Schnitter. I work as a development expert at SAP Business Technology Platform. My team provides logging services for applications running on that platform. Today, I want to share our experience with Fluentd when providing logging for Cloud Foundry applications. Let's have a short look how logging in Cloud Foundry works. An application can write its logs to standard out and standard error. Those logs are collected by the Cloud Foundry LogoGator system running in the container runtime. It can forward the application logs as well as the access logs coming from the Go router via syslog. Any component that receives such a syslog stream is called a syslog train. I want today to show you how to set up Fluentd as such a syslog train. That means I will explain how to register a syslog train in Cloud Foundry in general, and then how to configure Fluentd as such a syslog endpoint, how to pass the application logs, and finally, how to pass the access logs from the Go router. So let's start with configuration of a syslog train. We assume that Fluentd is already running, and we know the URL to Fluentd. What we now need is a service binding to forward the logs. That can be a service binding to a user-provided service, which we will use here, or any other managed service. So the steps for user-provided service are first to register and create a user-provided service, and then bind our application to that service instance. So let's have a look at an example. We say Fluentd is running at the URL fluentd.example.com. And we also have an application running on Cloud Foundry by the name of MyApp. We will create a service with the name Fluentd logs. For the forwarding, we can choose between three protocols. The first one we will use is plain syslog, and there are more secure options with TLS support for syslog or syslog over HTTPS. So let's use the CFCLI commands for this thing. So first, we create a user-provided service with the name Fluentd minus logs, and then with the minus L switch, we give it the syslog train URL that points to our running Fluentd instance. Then we bind our app, MyApp, to the newly created service instance, Fluentd logs. And that's all there is. Word of caution, you need to secure the Fluentd instance you are binding here against unwanted access. So what do we see at the Fluentd site? Incoming messages will have the following format, and you will get them by configuring source type syslog. The message format is RFC5424, and the transport will usually be TCP, depending on the Cloud Foundry configuration, it can also be UDP. So let's have a look at the parts of this message. The first is the priority that is default to info by Cloud Foundry. And the next thing is the ingestion timestamp. That means the timestamp when the Cloud Foundry logger data received the message. And then there's information on the origin of the message. So this is the org name, the space name, and the app name. Unfortunately, all three kinds of information are a little bit unhelpful if you really want to distinguish the log messages. The first really useful thing is the app ID. I use the placeholder here for the UID that is usually there. And the next thing is the source type. I will explain about the source type later on. So all of this you get just from the syslog configuration to the lab. But as you can see, there are more parts. The first part is the so-called structured syslog data. And this contains all the CF metadata to attribute the logs to its source. I will explain how to pass this part in the next slide. And the final part is the log message itself. And depending on the source type, there can be different formats in the log message. So it can be the application log messages. It can be the assess logs from the go router, or it can be runtime events from Cloud Foundry. And I will explain that one as well. So let's have a look on how to pass the metadata. So the extra data field will look at something like what you see below. This contains a lot of source identification. So it will tell you the app, the message originated from the space the app was deployed in, and the org the space belongs to. And also, if the application is running in multiple instances, you can distinguish the running instances by the instance ID also contained in this metadata. And then there is more information. And this is information about the Cloud Foundry runtime. So you will get to know what the ego cell deployment was running on, what the IP of the cell was, and such things. So to pass the extra data as below, we apply a three-step process. The first is a pre-processing that just removes the brackets at the beginning at the end of the content. Then we pass the metadata fields. And for this, we use the labeled tab-severated value filter because then we don't need to know how the fields are called. And finally, there's a post-processing step. You will notice that Cloud Foundry quotes all the values and that one we want to remove. So in configuration, this looks as follows. Removing the brackets is a small Ruby code snippet. Then you see the configuration for the LTSV parsing rule. So also very basic. And finally, in the post-processing step, when we remove the quotes, this again is done with a small Ruby expression. But here you need to know the name of the fields. So this is no longer dynamic. But still you specify all the fields you want to have. So you don't need to take off just the ones that are interesting to you. So now we have the metadata. We know where the message came from. Let's see for the message itself. So there are different message formats depending on the source time. The first part, everything that starts with app slash, these are messages that come from the application itself. It's jobs or the sidecast that are deployed by the user, by the application developer. And this is user-generated format. And here is where you put your custom rules depending on the application you're looking at. Then there is RTR, that stands for Go Router. So these are the access logs. This is plain text. And it's basically an extended Apache log format. And I will show how to pass this with the general RECX pattern. And finally, there are a lot of source tabs like API, staging, log regator, SSH and cell. And these are runtime events from Cloud Foundry. They are in plain text and we leave them just as is. So if you have these kinds of different formats, there are two approaches you can take. The first one is to use the multi-format parser plugin and that's the one I will showcase. And the second one is to really switch the parsing by the source time. So how would this configuration look like? So here you see the multi-format parser in action. For the application, I assume it would write JSON format. So that's the first part. The second part is the Go Router logs. I left the expression how to you because it's rather lengthy and I will show it on the next slide. And the last one is just the format none for all the then runtime events. Okay, now how do we pass the router logs? Of course, this is an SS log. So it has a lot of fields. The first fields are fixed in format and they're very similar to Apache logs. So you get for every request, the host, the user, the method and the protocol and also requests and response sizes and such things. And then there is an additional optional fields and they depend on the Go Router configuration. So they may differ from different Cloud Foundry installations. So there might be HTTP headers, for instance, for tracing like Zipkin or Open Tracing, there might be response times or again some metadata about the application. So if you want to pass such a router log, the expression that I will give now needs to take care of this. So again, we can use two steps. The first is to pass the fixed format. You find the expression down below. So you see it is rather lengthy but it's just taken from the Apache 2 filter and extended a little bit to cover more fields and also to take care of the quotes that Cloud Foundry once again emits. And once you have this, you'll notice at the very end, there is an additional data field and that will contain all the optional fields. And this is the second step. You can pass the optional fields just as we pass the CF metadata. It's basically again key value pairs. But this is now a very Cloud Foundry configuration-specific thing. So this needs to be tailored to your landscape. So summarizing all of this, Cloud Foundry can forward lots of applications via SysLog trains. Fluent D can easily receive those messages as a SysLog train. And if you want to pass the incoming messages, you will require an advanced rule set to first pass the SysLog path and then take care of the metadata and the log messages and the three kind of formats of log messages you will get. So I hope this presentation is interesting to you and thank you for your attention.