 Okay, and in this talk and I will talk about friendly and that is a log collector middleware and Listen in Ruby Okay, and I am and satoshi Tagomori and as known as and Tagomoris And that is my and Twitter and GitHub account name and and I am and walking in and line cooperation And line cooperation is your company and is serving a messaging application and line Please let me ask in two questions first Who knows Ryan? Whoa Fantastic, thank you. Thank does it's very good very good news for my coworkers and one more questions One more question who use this line. Oh Not so many people use this line, but thank you. Thank you very much. And I Will tell to our marketer That's and you should work hard in Singapore. So, okay Okay, and line at today and line has her 470 million users or more and and we are handling and 10 billion messages or more 10 billion or more messages are In a par day So and we have and many various services and applications on line platform That is why that that is why we must handle and many kind of metrics and and many various kind of logs and many huge amount of logs and that is why and I am a user and Committer of friendly project One more question and who knows friendly Oh Some people knows friendly fantastic, but and many of people does not know friendly That is why I am talking now here about Freddie Okay, and friendly is an open source data collector to simplify a log management That is from and friendly offshore website before friendly and we must handle logs and power for each Sources and destinations and we we are we must read and logs by tail and tail commands and Process and these logs by any scripts and post metrics and to grasp those and and the other side and we we Execute and secure SCP by Chrome and and copy a lot of log files into storage systems And we must consider an error handling and buffering More and we must and handle and many and pairs of sources and destinations and Lootings and and Amazon API keys and and the log formats. That's very very complex Processes and Situation becomes in chaos Okay, and friendly so this program and and our situation is very controllable and Friendly and also does and formatting and buffering and retries and routines Okay, actually friendly is friendly is open source data collector written in Ruby and lands on syrupy and Unix like OSS and like in Linux or Mac OSM or BSE or and the others and Friendly is with an error handlings and routines in its core and friendly has a plugin systems and for as an input plugins and also plugins and buffer plugins and Friendly has and many built-in plugins And friendly is distributed on ruby gems dot orgs. So we can install and friendly is by simply and gem install friendly and Frontiers plugins also distributed on ruby gems dot org. So so we can install and plugins and by same way RPM and dev devs and binary archive packages are also available by Treasured data that is that is a company hosting Friendly's developing development Okay, why friendly? That is there. There is very big reasons. Friendly is always so cute That is very very important and There are many logos and mascot characters in especially and big data software world Like he yes. Yes. He's so cute Yeah Anyway, why friendly? Friendly has many good points and and I will talk about these points and for from now at first This is friendly event friendly event is very simple on Friendly event is and built on three elements and tag and time and record Friendly's tag and is used for rooting and and tag is a special actually dots separated a string and Time time is an unique time. So I am 64 64 bit integer and Time is for time is for matted for our local times and local formats and record record is actually JSON objects and and also and hash object with string keys in Ruby world and record record record is Does not need any schema. So we can put any data into record Next and friendly this is friendly's configuration and Friendly this friend is configuration syntax is as same as an Apache Apache web server and It's have sections and source sections For input plugins and match sections with a tag patterns are for output plugins and we can like and any number of the source sections and match sections in and one friendly configurations and Friendly also have Ruby DSL configuration syntax This is this configuration configuration example and expresses Just same as and previous Apache configuration And and also of course we can like any Ruby code in this configuration to extract loops or any animators Okay, and Next and tag-based routing who friendly does tag-based routing and friendly Can have and any number of input plugins and input plugins and meets or events into the friendly core and friendly core and mix These events are once and then Friendly core roots its event these events by tag patterns to and each output plugins Output plugins are processed these events and by these their own configurations and Friendly output plugin can Amit events into friendly core. So and this this style and friendly output output plugin performs and like filters okay, and Friendly has and already friendly already have and many public plugins and 300 or more plugins and we can install these plugins over Ruby gems.org by gem commands Okay, and next friendly patterns and and we let's see and what friendly can do This is the first and most simple pattern friendly leads Lines from file and pass pass it into Friendly events and then friendly formats these events and light by lines into files This is very simple and By changing and output plugins we can light and events to any storage systems or External and middlewares like MongoDB My scale elastic such how do phdfs Amazon S3 Amazon Redshift or and Google BigQuery And many more plugins exist. So and we can do we can light and Events into many more and middlewares and external storage systems and next pattern Friendly can receive events data from other friendly node and Friendly can forward the events data in to another friendly node over TCP over TCP IP and friendly can also a friendly can also receive events from friend's logger client library of and each programming languages over TCP IP and friendly forward plugin and have load balancing and Active standby features. So just it's very useful for and friend make friendly clusters and Friendly the secure forward plugin and can provide And forwarding over and internet and and over SSL with authentications and Okay, friendly can connect with other middleware like a c-thrug a patch through Facebook scribe or a patch cuffer and And friendly can input and events from these middlewares and friendly can output Events into this middleware Okay, next and friendly a can copy events. So and Friendly writes our event into Hadoop HFS and At the same time and friendly can forward and that's event into another friend T Okay, and friendly can count events by string regular expression patterns. So and friendly and Can reports pattern regular expression pattern one Events with friend a regular expression pattern one 60 and second is 20 and also By these string regular expression pattern and also Friendly can count events by numeric value but numeric value ranges And friendly can Aggregate numeric values value numeric values like and then outputs and maximum value and minimum value average average and and submissions and For example nine percentiles this plugin is very useful for Http servers and response times and and or more use cases and Friendly has and many other various inputs like and this set this that is a is a about and the Linux performance later or In SQL SQL plugin does SQL select statement and Handle its statements results as an friend the input friend these friend these input events and more well and friendly has an execute plugin and execute plug in calls execute an external and any other external command and Handle handle its output as friend these inputs What? Actually, we can do anything with these plugins And also friendly has and many various plugins output plugins like and notification to not notify on IRC Like this like this. This is a 100% of and response code is and and 400 is that is very bad situation and and same as hip-hip chat or or Friendly render a numeric values on graph tools and Also and friendly out execute plug in a can do anything by an external external commands and One more and one more on the last pattern and friends the output plug in can emit events into friends equal So that is just works as filter. That is this is one example friendly execute execute filter plug in it can and use an external command with filter with filter as filter like and Unix Unix command command pipelines So this is a perfectly in stream processing and filter So and friendly friend these input events are processed processed as and stream processing and and Outputs friendly can output and these results into any other storage systems or and other nodes and when we are using a Stream processing RPC server Norikra Norikra is a string Norikra provides in stream processing with SQL and friendly Norikra plug in friendly output Norikra output plug in sends events into Norikra and Norikra process these events by SQL and then and friendly fetch these in SQL results as friend these events and and send these events into another another nodes and another systems and This is their piece that and friendly does error handling and retries for all these plugins. This is very useful For these reasons Friendly can resolve these these other programs of these chaos situations into and this controllable situations Okay, and friend these versions Friendly but friend these versions friend these Latest version is in version 0.10 dot 50 and this buzzing is released on middle middle of this month and friend these and 0.10 dot X versions are stable versions and and latest versions and many more and many minor features and a feature updates and bug fixes are also included and and new features as for version one is Also included friendly version one is planned as the first major a major release major release and planned Someday in this year and friendly version one is completely compatible with this current this latest versions and Also have a new additional features and on these and roadmaps For example on new configuration syntax and and new plug-in backends or demo process management and and much core CPU supports Okay, and friendly a current friendly runs on sylvie and only Unix like or systems, but and friendly on jrb is under development and our developer is and trying trying to Support and cool IO and for jrb Trying to fix and cool IO to support jrb and jrb is a bit friendly is based on cool IO that is event events driven programming library So this this fix if when this fix We will be completed and we can we can start to and support we can start to support jrb on for friendly and Also friendly on windows is under development and Windows branches exist on friend is reportedly and 95% are what workings but and not complete and Some developer is working for and friendly on windows Okay, and use case in dying and and this is Nitix data for the view and and we are using a friendly cluster for to and Collect log data and and the roots and and doing stream stream processing And friendly cluster out We have and two major and friendly cluster and one is a delivery one is for delivery and stream map processing and and and second is for an aggregation and stream reduce processing and in detail on we have and fall friendly cluster and Deriva cluster does Deriva cluster and store logs Deriva cluster collects logs from many servers and Store and these rocks into an archive storage and and the copy Copy and load load bouncing and send and next cluster Next cluster worker cluster worker cluster does and pass these rocks and store and these rocks into an Hadoop HDFS and also forward on these rocks into and stream processing clusters and And watcher cluster watcher cluster does and and does and processing for monitoring and not notifications and and then and we can receive a lot a lot and notifications and for for too many error responses and and other situations and Also, and we can we have an the set and complex event processing cluster This is a general-purpose stream processing and and our application engineers Lights SQL and put into this no regular cluster. So a no regular cluster process and friend events from friend V and and each application And friendly and store these results into and each application metrics and storages Now we have an Man many of our servers and for Frenzy, but that's not a big problem. So and we are an hundred and five point five or more beer events and pa par day and and in at big time and and and 150 or more events per seconds. So and friendly can Affinity have an enough performance enough high performance for these events. So and Enough and stable so and friendly does not fail in this Two years. So and we can we are very happy to using friendly This is wrap up and and friendly is very Flexible systems and and we have and many and plugins and we can light any plugins easily easily. So Friendly is very I believe that and the friend T is very good software and friend T and can Can make you happy if you are trying to collect and any many kind of drugs or Try to start and stream processing and in and very easy way Thank you Thank you Satoshi. Do we have any questions for him about flu and D or line or anything? Okay, and please and ask me question and in very slow English Hi And and the question is about and false tolerance or And not a failure or any other problems. So, yeah Torrance Okay, and Friendly friendly hub and No, never force torrents for no driver force torrents and friendly hub and fire buffer for this problem and friendly and friendly have and many types of buffer plugins and fire but and with firebuffer and friendly and lights these events into fire as buffer and that is and that's and causes a performance penalty, but once events Litten in firebuffer and friendly does not loss Unless disc is broken so and in friend these Crustler ever forced for friend this and the cluster level for torrents and we can and copy events and Forward into another node. So and and we can also use load bouncing and active standby. So And if and we Deplicate these events and lights and two different and presses place presses So and one place One node if one node and fails, but and events are also saved but and friendly hub and no Acknowledged systems. So and friendly yes and friendly hub friendly does acknowledges by an only TCP and and handshakes and and acts so and we We are not we have no plan to Agno to support acknowledgments for and cluster level acknowledgments and that is that's that's brings and very big and performance penalties or We are not planning planning and for acknowledgments for and performance reason Okay Thank you. Any other questions Well, we we use fluent in production and and it's changed our lives It's like a second. You know that the diagram, which actually could you go back to the diagram? Dependency that mean the yeah, that one. Yeah. Yeah Yeah, yeah, that kind of summed up our life Yes, yes in our environment and that is Was real Yes, Randy. So our problem any reason why if you go back Why Python and talk because only handles the easy thing. That's my little joke. So Thank you Satoshi. Thank you. Thank you very much