おはようございます。私のプレゼンテーションに感謝します。始めましょう。私はまさきまつした。NTTコミュニケーションのオープンスタックR&Dチームのメンバーです。日本で大きなテレコンコンパニーです。ネットワークスポンサーのインターネットアクセスです。オープンスタックはキロサイコルを開始しました。リパティサイコルは16パッチをトロップ、オスロロー、コンフィッグをコミュニケーションに読めました。私はコミュニケーションについてコメントをしています。まず、ロックコレクターを読みましょう。ロックコレクターはユニファイトロッキングレイヤーで読みましょう。ユニファイトロッキングレイヤーはコミュニケーションについてコメントをしています。マネットフロッキングレイヤーはアドホックスクリプトをマネットフロッキングレイヤーに読みましょう。コメントを読みましょう。ロックコレクターです。ロックコレクターは3パッチを読みましょう。インプット、フィルター、ココロを説明しないようにエンジンでドームの間に説明を進めているのですインプットの説明はコレンティングデータに適用するこのフィギュアはコナッツプラグインがログファイスを読みますサイスログプラグインはオープンスタックプロセスをしていますフィッタープラグインはそして、多くのフィルターを追加することができます。このフィルターの出発プラグインを各デスネーションに送り、フレンディはローグコレクターの一つです。このコレクターはC Rubyを作り、Googleのフレンディとしても知らないでしょう。フレンディはクバネティスのスタンダードローグコレクターです。フレンディはTREASURE DETAINCで保管しています。このプラグインはアンライテックスインフラストラクチャーのサービスです。ローグスタッシュはローグコレクターのインフリメンテーションです。このプラグインはJ Rubyで、イラスティックコールで保管しています。このプラグインはイラスティックサーチを作り、この2つのローグコレクターはとても似ています。このプラグインはとても似ています。今日、このプラグインはとても似ています。このプラグインはアウトラインです。まず、この2つのローグコレクターはコンフィグレーション、プラグイン、パフォーマンス、トランスポットプロットコールです。次に、フレンディとローグスタッシュはフレンディとローグスタッシュをインフリメントしています。まず、コンフィグレーションの1つのプラグインです。フレンディとローグスタッシュは違うサイズでコンフィグレーションを使用しています。フレンディとローグスタッシュはタッグで、ローグスタッシュはタッグで、このフィグレーションはノバAPIローグスタッシュとオープンスタッグとノバ、ノバAPIローグスタッシュはタッグで、オープンスタッグとノバ。ノバAPIローグスタッシュはインフラッシュdbです。これをこの部分にしています。ノバAPIローグスタッシュはイラスティックスタッシュです。ここはインフリグレーションのフレンディです。不安定です。この頃、フレンディがノバAPI.xのコンフィグレーションをNOVAAPI.LOCK and the inputs will be tagged as OpenStack.NOVA.Every input will be tagged with tag attribute like this.Then, tagged inputs are in much section.In this case, the input in the previous slide will be matched at the first sectionbecause it's tagged as NOVESTAT.NOVA.You can also use wildcards like the second section.As a whole, NOVA related tags will be routed to Elasticsearchat the first section, and all other OpenStack related logs will be routed to InfluxDB.If you want to output logs to multiple destinations, you can use copy program.In this example, FriendD sends same logs simultaneously to two destinationsInfluxDB and Elasticsearch.Let's take a look at LogStash configuration.Differently from FriendD, there is no tags in LogStash.All inputs will be aggregated and scattered to all outputs.This figure shows that inputs from NOVA API and CINDER API are aggregated and scattered to outputs.This is the configuration corresponding to the previous slide.Let me compare FriendD and LogStash in configuration.In order to compare, let's assume that we have three inputs corresponding one to oneto each output separately.In this situation, FriendD can route different streams separately in simple tag matching.On the other hand, you have to split logs to do the same thing with LogStashbecause all inputs are aggregated in LogStash.You need to define conditions like this to split aggregated logs.As you can see, these conditions might be complicated.For the second case, let's assume that we want to aggregate logs from all inputsand send aggregated logs to several destinations.In this case, LogStash can be configured quite simplyfor unconditional scattering.To make the equivalent configuration with FriendD, you have to use copy plugin.Let me summarize this part.FriendD and LogStash have different styles of configurations.Generally speaking, FriendD is suited to handle multiple streams separately.By contrast, LogStash is good at handling logs in gather scatter style.Next topic is about plugins.Briefly, they both provide number of plugins.There are more than 300 plugins for FriendDand more than 200 plugins for LogStash.Popular plugins are bundled with LogStashSo you will not need another plugins in many cases.These bundled plugins are maintained by LogStash project.That's a great point of LogStash.FriendD is packaged only with minimal plugins.So you will need to install some plugins.Most FriendD plugins are developed by individuals.So some plugins may not be maintained well.Plugins can be installed easily by one commandwith both FriendD and LogStash.Next thing is performance.To be honest, I can say which one is fasterbecause that depends on circumstances.What I can say is that both FriendD and LogStash are more than enough faster to use for OpenStack.They can handle more than 10,000 logs per second.Then filters greatly affect performance.It's not a good idea to apply heavy filters on log collectors.Heavy workloads should be performed on data processing structures.Next, I have heard many times that CRuby is slowbecause it's a GVL and only one thread can run at the same time.That is not true, especially for IO bound loads.This diagram shows two threads attempt to perform IO operations at the same time.When thread1 is executing Ruby code, thread2 has to wait GVL.However, GVL is released just before actual read or write in this section.Then thread2 acquires GVL to execute Ruby code.Actual IO operations can be performed in parallel.Let me move on next topic, transport protocol.FriendD and LogStash have their own transport protocol.FriendD has forward protocol and LogStash has Lamborgac protocol.They both support failure detection and fallback.Here is an example of active standby configuration in LogStash.They are primary node and secondary node.In this configuration, secondary node is used only when primary node fails.On the other hand, FriendD runs in active active mode by default.In this case, Desk1 and Desk2 are configured as destinations.SourceFriendD instance will send logs to Desk1 and Desk2 simultaneously.Outputs are equally balanced.Please note that it's client-side load balancing.You can configure active standby mode by standby attribute.When you specify standby attribute on the server, it is used as secondary node.In this case, as a node named secondary, we've used when primary node fails.FriendD has some additional features.You can configure weighted load balancing.This configuration means that 60% of outputs are sent to Desk1and 40% are sent to Desk2.If it's Desk1 or Desk2 fails, of course, all outputs are sent to the active server.The another feature is at least one semantics.When require arc response is specified,SourceFriendD weights acknowledgement from the destination.SourceFriendD instance attempts to retransmit until acknowledgement is received.This feature enables more reliable output, but it may affect performance.Let me summarize.Both transport protocols are capable of high availability in active standby mode.However, as we have seen, FriendD's forward protocol has some great features.Active active mode and at least one semantics and weighted load balancing.Next thing is forwarders.Folders are lightweight implementation in order to tailor log files and send logs to log collectors.Folders have only limited features and not plugable, but they use less memory.They are built as one binary.This means less dependency and easy installation.These two folders have little different behavior when multiple destinations are specified.FriendFolder will send duplicated log to all servers.On the other hand, LogStashFolder will send only one server marked as active.I'd like to move on the main topic, how to integrate log collectors with OpenStack.I will introduce three ways to send logs to log collectors.First, you can tailor log files by LocaFriendD or LogStash.In this way, you have to pass many form of log files.The second way is AssistLog.AssistLog is installed by default in most Linux distributions, so you don't need to install agents to servers.The third way is DirectOutput from OsloLog.OsloLog is a logging library used by OpenStack components.We can receive logs without any passing in this way.For the first way, we can tailor log files.This is a logging architecture with localFriendD on each OpenStack nodesand two aggregators to receive logs from them.You can configure high availability in active standby or active active mode, as we have seen.As you know, there are many many log files related to OpenStack.It's troublesome to configure to handle all log files, so wildcards are useful to specify log files for components.Here is an example of tailing log files related to NOVA.Another problem is passing.You have to pass many forms of logs for analysis.Here is an example of passing NOVA APIs log.It's a regular expression here, so this example is a configuration only for NOVA API.log.The second way is RCSlog.In this way, OpenStack components will send logs to localRCSlog.Then localRCSlog will send logs to aggregators by CISlog protocol in TCP or UDP.Logging.conf is a file used to configure logging in detail.Here is an example of logging.conf to send logs to RCSlog in JSON format.Output to RCSlog is configured in the handler section.CISlogHandler is specified here, and CISlogFacility is set to local1 here.In Formatter section, this JSONFormatter will format logs in JSON.It's an example output from JSONFormatter.The CISlog facility is useful to distinguish where does the log come from.In CISlog, you can assign user-specified facilities as from local0 to local7.Logs are tags like CISlog.local0 in friendly.For example, you might assign these facilities, as you can see.Here is an example of RCSlog.conf at OpenStack nodes.CISlog supports high availability in Active Standby mode.Both Flentys and Logstarch have bundled CISlog plugin.There is a difference between them in using protocol.In Flentys CISlog plugin, you need to choose either TCP or UDP,and UDP is used by default.If you want to receive logs reliably, you can specify to use TCP with protocol type attribute.On the other hand, Logstarch will listen on same port with both TCP and UDP.To pass logs in JSON format,you can set the format or codec option like this.The third way is direct output from Oslo log by using FlentHandler.FlentHandler sends logs directly from OpenStack components by forward protocol.Logstarch also can receive forward protocol from FlentHandler.Currently, FlentHandler can send logs only to one host.That's why I introduced localFlentty in this figure.This localFlentty is responsible for buffering and load balancing.Here is the login.conf to use FlentHandler.The FlentHandler is included in FlentLogger package.To handle logs with FlentHandler, logs must be type of dictionary in Python.The FlentHarmatter provides logs as dictionary.It is proposed by us as a blueprint for Oslo log.By using FlentHarmatter,Flentty and Logstarch can receive logs without any passing.This is an example output from FlentHarmatter.You can get logs in this form without any passing.Finally, let me summarize this presentation.In the aspect of configuration,Flentty is suited to handle logs separately.Meanwhile, Logstarch is suited to handle logs aggregately.To achieve highly available log collection,both Flentty and Logstarch are capable of failure detection and fallback.However, Flentty is some great features.Fly-side load balancing,at least one semantics,and weighted load balancing.The last topic was integration with OpenStack.FlentHarmatter enables to get logs as type of dictionary without any passing.We need more reviews for our blueprint,FlentHarmatter of Oslo log.So, that's it.Thank you for your time.We are running our booth over there.I hope you will enjoy robot racing over WebRTC in our booth.Thank you.