 And welcome to the CNTF End User Lounge where we explore how cloud-native technologies are being adopted by end user organizations across different industries. The CNTF End User Community consists of more than 145 vendor-neutral companies that use open-source software to deliver their products. My name is Cheryl Hung and I'm the VP of Ecosystem at CNTF. So in these live streams, we're very excited to bring along some of our end user members to showcase how their organizations navigate cloud-native and use it to build and distribute their services and products. And we are on every Thursday, sorry, every fourth Thursday at 9am PT. This is an official live stream of the CNTF and as such is subject to the CNTF Code of Conduct. Please don't add anything to the chat or ask any questions that would be in violation of that code of conduct. Just be respectful towards all of your fellow participants and towards the presenters. If you have any questions for us, then we will be monitoring them throughout the stream. Just ask your questions through the live stream chat. So this week, we have peritus.ai here with us. So before we dive into some questions, could you please introduce yourself? Thanks Cheryl. Thanks for having me here. It's pretty exciting because I watched the two sessions that were there in the last two months. And one was from CERN, other from Twitter. I have interacted with both organizations in a past life from an open-source perspective. So this is awesome to have a startup on this conference. Thank you. Yeah, I'm Santosh Srinivasan. I am the co-founder and head of engineering of Peritus.ai. So we are building out knowledge networks to help developer self-service where they work and this knowledge comes from experts they trust. All of our platform is built on cloud-native technologies. So this is a great forum for us to discuss this. Fantastic. I know that Santosh is your co-founder at peritus.ai. So maybe you can tell us a bit about the story of how you started this company. Why was this an interesting problem for you? It's a fantastic question. So just a little bit of a background about my own career. I started my career in Bell Labs Development Center. Later I went to grad school and landed up at Yahoo where we were working on what we used to call as large-scale data processing and that eventually became big data. I got lucky and I was introduced to the Hadoop ecosystem by a good friend of mine, Arun Murthy, one of the lead engineers on Hadoop. And that journey began in 2008, about 13 and a half years ago into open source. So in that journey, we built Hadoop at Yahoo for Yahoo. It was not commercial. It was not meant to be used by other people as is, but it was meant to foster innovation. And what we noticed was with open source, if things work, people pick it up and the adoption is really tremendous. I noticed two or three different flavors in open source where you can have viral adoption. One was the Yahoo flavor where we built something at rapid scale, massive scale, hundreds of millions of dollars. It was somewhat enterprise ready, but within Yahoo. And the other one is something like Mongo or Elastic where it's so easy to use and people gravitate towards it because they can use it and get to go without having to talk to many other people. The Hadoop ecosystem became so massive that entities started commercializing it. And I ended up in one of those entities called Cloud Era. So I went from being an end user, contributor, committer to a governance member in Apache to participating in a commercial activity at Cloud Era. And at Cloud Era, what I noticed being on the other side is open source is easy if you're a developer, but if your skill levels are below that of a developer, it's extremely challenging to use. And that's where the enterprise companies were forming. And this is the genesis of Peritus. How do you take technology and make it consumable by a large set of people who may not have the same level of skills as the creators of those projects? And that's why we are building out this knowledge network to help people get to that as quickly as they can. That's awesome. I mean, it's definitely something I agree with that if you are a developer, open source is pretty great. It's pretty fantastic. If you're not, there's a very high barrier to entry. So I'm excited to see how Peritus is going to help kind of lower that barrier to entry. Let's talk a little bit about your company, your team as well. So you're the VP of engineering. Is that correct? That is correct. Yeah. Yeah. So what are the responsibilities of you and your team now? Yeah. So one of the things about being a co-founder is my official title is co-founder of VP of engineering. I do a ton of things and sometimes people call you as the chief janitor as well. So you have to do anything that relates to the company. But my primary responsibility is to build our technology and product. So a significant amount of our engineering resources are built towards using the technology to rapidly build features in the product. So one chunk of the engineering org is for infrastructure because without infrastructure, we cannot build our product quickly. And when I first started the company, and since now, it's been about three and a half years, the challenge was I came from the big data scale out technology. And we are trying to try those technologies. And this is when Kubernetes was sort of taking off and it was not quite mature. So we tried a few things and finally settled on Kubernetes about two years ago. And we invested a lot of energy into doing both the scale out and also giving developers flexibility to spin up things on demand to do their own innovation. So that's one part of the organization, which is infrastructure. And we try to use multiple cloud providers because some of our customers tend to say, hey, just have to work on one or the other. And we wanted to be careful here because if you choose any cloud native technology hosted by the provider, you're kind of stuck in some sense. It's awesome. I would say it's the value is tremendous. But if someone else wants you to be on a different cloud provider, it's challenging. So we decided, okay, everything is going to be containerized. Everything is going to have REST APIs. And we'll use Kubernetes and then make sure it complies with whatever the cloud provider hosts. So it's easy to migrate from one to the other on demand. So we use a bunch of technologies from CNCF. The prominent ones are already popular, Kubernetes, Helm, use Keda, the cert manager, Prometheus. We are looking into using Falco for security purposes. And a bunch of other open source technologies, Mongo, Elastic, Lucene-based stuff. The list goes on and on. We are heavy into open source. Well, of course, I'm pleased that you use cloud native within your infrastructure. It would be quite funny if you didn't. And yeah, I'm also quite fascinated to hear that you look to the few different things before settling on Kubernetes. Is that right? That's right. Can you share some of how you thought and why you made that decision? It's just experience bias. So when I left the cloud data, we were trying to launch a cloud initiative to spin up clusters on demand to do scale out and Elastic stuff. So when we came in and we were saying, okay, let me first set the context of what's the problem we are trying to solve. We are trying to build out a knowledge network. So knowledge resides in multiple forums, documentations, KB articles, forum conversations, GitHub issues, GitHub PR comments, all of those things. So now we want this from everywhere in one central location. We have a knowledge graph which supports what we finally end up doing is provide recommendations to developers as they are working. So we have a machine learning team of a few engineers who are rapidly doing experimentation to try out the quality of these recommendations. We have the infrastructure team trying to help spin up, scale down and maintain stuff. And then we have the data crawling ingestion and organization team. And then finally, you have a full stack UI UX team trying to productize the underlying technology. So these are the four different components to the organization. So coming from a big data technology, I said, okay, crawling is a solved problem. So let's just spin up things. Processing is interesting because it depends what you want to process and how. So since I came from the Hadoop background, we tried a few things there. It worked, but it was slow. So then we were a little bit stuck about, hey, how do you really make the scale out and not be stuck to a hosted service from a given cloud provider because migration becomes a problem. That's when we landed on Kubernetes. And it was challenging. The reason it was challenging is we are highly skilled engineers, people who have been at bigger companies with the same background. And getting to use technology is one thing, but if things don't work, trying to find out why, fix them as soon as possible without waiting on the community to provide a fix. And I've been on the other side providing the technology. Now I was consuming technology from open source while trying to have very fast product iteration. So that's how we landed on Kubernetes. And that journey began roughly two and a half years ago. And what does your infrastructure look like now? You mentioned you have multiple, you use different clouds. Can you give us an idea of the size of your clusters? Yeah, so we do things on demand. So what we have done is for production usage, we have a hot, hot setup. In case things go wrong, we can rapidly switch between two clusters and they're running their own namespaces and Kubernetes. We have somewhat of a staging environment that we prep before we go live on production. So we want to do testing and stuff. And for developers, they can spin anything on demand. So we've integrated right from the time you do a commit, your PR gets merged. You can get it deployed on the cloud almost within a matter of minutes into your own developer namespace. And you can create as many developer namespaces as possible because you're doing rapid prototyping. So let's take an example of you're doing a feature development. And feature development cuts across all of these four different teams, infrastructure, machine learning, data ingestion, and UI UX. So you want this subset of engineers working on a feature to work on a feature branch. And you should be able to deploy to the namespace for this feature branch from each commit to your own developer namespace. And when you think it's reasonably stable, you go through your PR review process, merge the branch to your main branch. And then the next release, we try to do one or two releases per week. It goes to staging. And then we promote it to production. So this whole life cycle has been built on the public cloud infrastructure. With some caveats we trained to use, let's say Jenkins. We didn't pick any of the build infrastructure in the cloud. Because sometimes when you want to do a build or builds happen within a minute, incremental builds. And most of these cloud native infrastructure, they don't give you that. They can go up to two minutes, not up to a minute. So we went for developer efficiency and speed as the first order for building infrastructure. Oh, very cool. It sounds like you have a pretty smooth pipeline then for the very nice developer experience, I could say. You mentioned some of the challenges as well earlier that you faced. So right now, what would you say are your biggest challenges when you're building infrastructure or you're managing deploying applications? So you brought up pipelines. A lot of the work is the workflow orchestration between different components, which are all microservices. So when we first started building pipelines, I have a lot of experience building pipelines right from a Yahoo days to cloud error days. And I built many execution engines to do that. So we picked some of those to start with. I'll just name a few for the audience here. We picked Airflow. I was the development manager for Uzi, which is incredibly difficult to use. Airflow was very, very simple to use, but it lacked a bunch of features. And we are trying to make a workflow engine work in Kubernetes. We looked at Kubeflow. Kubeflow was like very, very nascent when we picked it up. And we couldn't quite close the feature gap for us to be off to the races. A challenge we had when talking internally was understanding the mission on the vision of this open source project while you're trying to pick the technology is a challenge. If you end up picking the wrong one, you'll have to pay the migration cost later on. That's still within the company. The bigger challenge is if you hire someone, they may not even be familiar with the technology that you picked, even though it's open source. At the higher order stack, you have these angular versus react battles going on. So I have more of a core backend technologies, not a front end UI person. So as you go to the backend, these are the fundamental pieces you're building on. You can't change them quickly compared to the higher layers of the stack. So when you're thinking about this workflow thing, we tried a few. We tried Airflow. It worked initially, but it wouldn't work for us. Then we really explored to see, hey, have you built a Kubernetes integration that we can contribute to? It was not there. Then we looked at Kubeflow. It was too nascent for us to pick up. Then we said, okay, begrudgingly, we agreed, let's build something in-house knowing that at some point in time, we'll give this up or migrate to something which is more standard. So we have something built internally, which looks exactly like any other workflow engine, just that it works for us. And in the next two quarters or a year, we'll start migrating away from these. We are looking at MLflow and Kubeflow for the machine learning parts. Argo. Argo is something that we are very interested in because the activity in Argo started picking up last year. If you look at the commit history of Argo, I think the first official commit on CNCF GitHub repos is in 2017. The project itself was around from 2015-2016. So if we had known that Argo was active, that would have been a more natural choice for us to pick when we made our choices back in 2018-2019. Always very hard to know how things will turn out in the future, though, two or three years down the line. Yeah. Interesting to hear how you've had to make those decisions up once, how you've chosen to make them. Well, I had a lead come to me and say, don't build anything in-house. And the reason he said is we'll train people on a bunch of proprietary technologies and that skill is not portable. And when we hire smart engineers, we'll spend like a month or two training them on our technology. That's not useful to us either. He pretty much said it's a loose, loose proposition for us. Yeah, I mean, that's totally fair. I've heard this as well from other companies that one of the benefits of open source is that you don't need to spend time training or retraining people when they come in or out. Let's take a slightly different angle now, because I'd love to hear from both your experience and from what you've seen in the knowledge base that you're building out. I'd love to talk about some trends and predictions for cloud native. So just to start with, over the last, let's say, three, four, five years that cloud native has been around, what are some of the trends that you've seen? Yeah, so we actually did a deep analysis of CNCF. So I have that maybe we can talk offline about it, but so here are some interesting things I found that you may already know, right? So I looked at the graduated and incubating projects. Although we have used some sandbox projects, I just narrowed the focus to something which is well known. So there are 36 projects in that list. Just as a quick tidbit for one project emissary ingress, there was no reference to any of their activity on the main landing page for CNCF. So including that, I mean, there's no reference to any GitHub link community page or anything. Everything else is very nicely set up on the CNCF web page. I know where the GitHub repo is, the dev stats, and where the community is if they do have a community. So interesting stats. So now that you asked is 70% of them have a Twitter account. It's omnichannel. So they have to have a presence online. Two thirds of them are on Stack Overflow. Two thirds. Just north of 60% are on Slack, or they support a public Slack channel. And then after that it starts going down. Some of them have Google groups. Why is this important? For an open source project to be successful, you need a thriving community. You need people who can help answer questions so that both developers and end users can rapidly use, adopt and build out features and technologies. So what this tells me is if you take out Kubernetes and gRPC, they over dominate the activities going on in CNCF. So we actually did analysis saying with and without Kubernetes. Then we found out we have to also do with and without Kubernetes and gRPC. So what it told me is most of the action is happening on GitHub issues. Most of it. We also analyzed the Stack Overflow traffic for these communities. It's anemic. So they have overall a challenge of building out communities to rapidly evolve and move from sandbox to incubating to a graduated project. If you're mostly on GitHub issues, it means it's a developer-centric community. The end user community is somewhat missing in action. That's so interesting to hear actually because I assume that everything would be close to 100% on Slack and on Twitter. Actually, that surprises me that those numbers are quite low. Well, not low, but they're not 100%. Yeah. So if you really think about people who are using these technologies are complex. Let's first state that this is not easy technology. So people who are building it are very sharp, very smart. I'm talking about the after building out the technology or in the process of building out how do you get it off the ground? So you need to have an omnichannel presence. So most of them have that in place, but they have to be where the developers are. They have to be where the end users are. And for that, most of them use the technology on their computers. You're probably not doing something on your Kubernetes cluster on your mobile phone or iPad. Probably not, no. And so most of the action tends to happen on Slack because that's where you want to have a question. And there is this metric known as developer satisfaction. I ask a question, how many hours before I go and make my own choice before somebody has to respond? So if I don't get a reasonable response within my working day, and if you say, let's say I post a question at 4 p.m., then maybe the next day morning, I am going to do something on my own. So 24-hour window is kind of the sweet spot. So I am on several of the CNC of Slack channels. So I see this activity, it's very high activity, and these public Slack channels have a shortcoming because Slack doesn't give you free versions with infinite capacity. At 10,000 messages, you age out. People tend to ask the same question over and over again on Slack. So there's a gap here. So people who are building out the product, they are fatigued by trying to answer the same question. They have to dedicate time to answer questions on Slack. And you have to set aside time. So that's a challenge. You can't expect people to be doing product development and support and community building simultaneously. So that's an open problem. And I think we are working on a solution there. I'm happy to chat on how we can offer it completely free to all of CNCF. Oh, that would be interesting. Yeah, we can chat about this offline, as you say. Do you have any predictions that you think are going to come up in the next, well, maybe through the rest of 2021 and into next year? Always dangerous, I know, to make predictions. I'm not going to hold you to them. Especially when things are recorded. Just between you and me, nobody else. Well, I can give you an abstract notion of what will help make these predictions. And maybe then we can use that to say what might happen. I said at the beginning, what kind of open source technologies tend to succeed? Technologies that have been proven within an enterprise company that are being open sourced. And technologies that are easy to use. So these are two key factors that is just on the product side. And the winners are actually going to be people who build out communities where you can ask a question, get a response as an end user or as a developer. In the larger scheme of things, the half life of a project is dependent on the community. I've been in Apache where many, many projects have been shut down. They move into Apache attic. And that's a sad state of affairs. One really massive project is Apache Mesos. Very cool project. I've seen that they most have talked to one of their founders. It's a really impressive technology, but unfortunately they're considering moving into an attic, right? That's very sad to see. Very complex, highly useful project. So you have to build out your community and you have to build it out so that it sustains beyond the initial founders involvement and when it gets incubated or when it graduates. So based on these three things, maturity, ease of use, community, you can predict which of the open source projects in CNCF are going to be successful. Interesting to hear that. And also I very much agree with you that community building is much harder than it seems. It takes a lot of time, takes a lot of persistence, but it's the only way for the project to outlive the original maintainers. So what do you think is the most surprising thing that you've seen so far? Now that Perotista AI has been around for a little while, anything surprising that you've seen? So I've been on a lot of the Apache side of things in terms of governance and all of those things. And I am relatively new to CNCF, the whole model. I recently reviewed the due diligence document for Argo, which is trying to move from incubating to graduated. I was very happy to see the level of rigor that goes into declaring a project to be ready for the next stage. I was really impressed with the level of rigor. On the mechanics of how the decision is made, it's so public, everybody can see the decision making. That's impressive because transparency means people trust the process. And the last part I find somewhat funny, but also I found I had a good feeling is everything was on git. For a developer, that's awesome. So for me, CNCF, and the last part I forgot to mention is open source is also all about licensing. I think 100% of CNCF projects are Apache software licensed 2.0. We definitely strongly recommend Apache. So that's very friendly, both for people contributing and when you import the technology in, you're fine in terms of a legality of using it. There are certain open source licenses where you have to think 100 times before picking it up because it triggers these things of, hey, if you're using it, you have to open source everything around it and it's too complex. Yeah, definitely. Open source licensing is before I joined CNCF, I did not have a clue how complex it can get. What more question for you about trends and predictions in cloud native? Is there anything that you're personally excited by and you think more people should know about? Security. I'm not from security, but the number of times you get asked about security when you go beyond proving value to getting adoption within enterprises. The first question you get asked is, are you compliant with X, Y and Z? So security projects, and I mentioned that we are considering using Falco. So those are things that I think people may not consider it as the first of the second project to think about when they're building out because you're just trying to prove value. So Kubernetes and security and cloud native and security will be an area which will get very hard soon. It'll get very hard. Can you tell us a little bit more about why you're looking at Falco? Yeah, so when we built out our products on cloud, we went to some enterprise customers and they asked, they would give us this huge checklist of things we had to pass before they would let us in. And some of these questions are like, have you had a third-party consultant certify that your product is secure? Have you done some kind of pentest? Have you done SOC2? A whole bunch of these things. So some of these are compliance things. You do it at that point in time. It doesn't mean you're doing it every day, every minute, every second. So you have to be doing that. It's somewhat in the realm of observability and monitoring from a security aspect. So raw metrics and monitoring infrastructure engineers are awesome for debugging and things like that. Security is life and death for some companies. The recent ransomware that happened with this pipeline company, which carries oil, physical pipelines on the east coast of the US, that was alarming. First and foremost, I didn't even know this company existed. And the CEO of the company said this, we wanted to be nameless throughout our existence. We never wanted to be in the news. And they probably supply nearly 50% of oil to eastern United States, 45% to 50%. So when you build out these technologies, if they get hacked, then there is no recourse other than having to pay ransom and then trying to actively patch holes. And it's an adversarial game. You patch one, something else opens up. So for us, what it meant is you had to get ahead of it. Some of the cloud vendors have really good technology that supports it. They do port scanning. They do anomaly detection. But you also go deeper into your own use of the products and try to find out what else is happening. So in that context, we are looking into using Falco. Yeah, very, very interesting. I've forgotten about the oil pipeline story, but yeah, definitely something to be concerned about. Let's go talk a different topic again. You recently, or Perotus, I recently joined the CNTF and user community just a few months ago. So how did you hear about it? What got you interested to join? Two main reasons. One, we are using the technology piece by piece while not getting involved in the community. So one of our technical advisors said, hey, you guys have to be in CNTF. So look at all these other things that are being built. And what we are trying to build as a knowledge network, you have to do it for a domain, which is as it stands today, taking off or in common parlance hot. And you need to be aware of what's happening. So our technical advisor said, here is the community where you need to be there. Thanks to him, we got involved. We started building out our technology as a subset of that. Then we broadened the lens saying, what else are we missing? That's how Falco and other things came into the picture. And then we said, oh, if you have to build out a knowledge network, you really need to understand these technologies from an abstract perspective of what's happening, what are the gaps. So that was the second reason we got involved. Very cool. And what do you hope to achieve by being part of the community? Yeah. So our foremost objective was to understand and build, which is the main motivation. The outcome that we're hoping for is if we can give out a roadmap or a guideline to new projects and existing projects to say, here is a path to success. And we can help you get there faster, sooner, with little to no friction. That will be awesome. We want to provide the intelligence for CNCF based communities to do their job better. Developers can focus on building product and users can go on consumption. And we can take care of, let's say the community intelligence part at no cost to CNCF. That will be awesome. So we want to provide the community intelligence to CNCF. So that will be an amazing outcome for us. Super, super cool. I mean, I really look forward to seeing how we can collaborate a little bit further. I'm going to wrap up now, but is there anything else that you want to share with our audience before I wrap up? Oh, no, this is thanks for having me on this channel and it is phenomenal. I mean, I continue to be amazed by how well CNCF is run. It's kudos to all of you because it's a sea change from having been part of an earlier open source community. Now I've been on the outside looking in. So I hope to participate further and, you know, spend more time on the community side of things. I'm very happy to hear that. And it is a pleasure to have you in the community as well. So thank you so much today. And thank you everyone. Thank you to our audience for joining our latest episode of the Cloud Native End User Lounge. Again, we love having other end users to come and join us. So we really hope that you've enjoyed the interaction and heard about a little bit of something as well. We're going to bring the latest Cloud Native End User stories on the 4th Thursday of the month at 9 a.m. PT. And also don't forget to join us for KubeCon Cloud Native Con North America, which for the first time in a year and a half will be in person as well as virtual, which I'm very excited by. So if you're coming in person, it's going to be in Los Angeles. It'll be October 12th to the 15th. And you can hear the latest from the Cloud Native community. Also, if you would like to showcase your use of Cloud Native Tours as an end user, go to cntf.io slash end user to join the end user community. Thank you so much. And we will see you again next time.