 Hey there. Today we have Abe, is that correctly, did I correctly pronounce your name Abe? Yeah. That's right. From DataWire and we're going to talk about something really cool that you really want to check out if you haven't yet. And that is telepresence, a tool that every Kubernetes developer should really know and use. So let's start with a bit about yourself. So a bit about your background. Why are you helping Kubernetes developers? Sure. Well, I started out consulting in the finance industry in the Greater New York area and I built lots of these sort of internal systems that were all about having pipelines of data flow and analysis and moving large amounts of data around in a smart way. And these systems got more complicated and I mean, you know, Wall Street does a lot of really complicated computation on the trading desks has been for 15 years or whatever. But I noticed that it became more and more difficult to keep these complicated systems running together. I guess ultimately I noticed the sort of natural tendency towards breaking up the big monolithic pieces of software into smaller pieces. And when I caught this pattern, I started researching it myself and then opportunity struck. And my friend from college who is a DataWire CTO approached me and said, hey, we're starting this firm and we're looking to help developers with microservices and it all just sort of aligned with what I had been experiencing and it made sense for me to jump on board. Yeah, so I think we've been focusing on helping developers be more productive and make their lives easier and more natural. And that's always, as someone who's done the development side of things for a long time, just for other projects, it really appeals to me our approach of trying to make that process, the process that I think of as my process, smooth and easy, even in a complicated microservices sort of space, is the whole thing just appeals to me as the way to go. Right. And for the ones who haven't experienced it yet, what is it like to develop applications on top of communities and microservices? Well, so Kubernetes is, there's a lot to say there. Kubernetes is, well, okay, my CTL likes to liken it to a helicopter. It's a really powerful device. It can do a lot, but it's very, very complicated. And until you get the hang of it, it's very dangerous in a sense. I mean, essentially, there's a lot of stuff and Kubernetes tries to do a lot of things to make sure Kubernetes tries to do a lot of things to make it so that your workflow is smooth. But if you don't understand what's going on, you'll be surprised again and again, oh, Kubernetes did that. I didn't understand why it did that. And so, the, basically, it boils down to understanding what Kubernetes does. And then you have a great deal of power to build complicated systems and not have to think about all the details. Once you have reasonable expectations for what's going to happen, then you can work with those, and Kubernetes becomes an incredible accelerator towards keeping things running and getting new things up and all of the little issues that crop up. So, I guess as a developer, in the modern world, we're being tasked with having our service online and making sure it stays online, right? So, the traditional split of developers writing code and then ops keeping it running, that's sort of going away. And by allowing developers to have the agency to make that possible. It's not just about dumping responsibility on the developer, it's also about giving the developer agency to make that happen. There's a lot of complicated new stuff that developers have to do. And Kubernetes makes that a lot simpler by giving you the tools to, let's say, declaratively treat the state of your deployment of your system as a thing that you can say, this is how I want stuff to be, and Kubernetes sort of makes it happen. And once you understand the details of what Kubernetes will do to make the world a certain way, once you get that internalized, then Kubernetes is fantastically useful and makes things very easy. And until that point, you can run into all sorts of trouble. If I understand that correctly, you're arguing that it's quite a learning curve and with your tooling trying to help developers to be faster and more productive, or is that a fair characterization? Yes, there's a high learning curve, but it's associated with a lot of power. The productivity gains can be fantastic once your team has, let's say, internalized a reasonable workflow for working with Kubernetes. I mean, when you get started, you're talking about all of these new concepts, services and deployments and pods and so on and so forth. And it sort of feels like you're mucking around with lots of little YAML files. And it's not always initially clear like, should we just cargo cult this YAML file and just change the bits we need to change and push that out? Or should we take the time to understand it all? And I think over time, as you work with this stuff, you get a handle on what's really important. Most of the time, you care about deployments and making sure your services point to the right place. And then the other stuff only comes up when you have to do nitty-gritty details or you're changing the shape of your individual services in some way. And then you start writing templates for your YAML and you get a set of, let's say, constructs that you can reuse and you understand what they do in Kubernetes for you. And it all becomes very smooth and quick over time. And I mean, we're building tooling around this stuff too. Right. Coming back to tooling. So telepresence is special. How would you, you know, elevate your page for telepresence? What does it do? How does it help? Well, telepresence lets you place your, let's say, your development laptop, whatever it is you're sitting in front of coding, effectively place that in the Kubernetes cluster in a virtual sense, such that you can run your code in your development environment with your debugger or however you want to do it. But it will act as if it's running in the Kubernetes cluster. And this is a powerful way for you to develop your service or debug your service without having to deal with the standard deployment cycle associated with Kubernetes. So perhaps I should talk about that a bit, the standard development cycle. Okay. Let us dive into that a little bit more, yeah. Okay. So the normal process with setting up a new deployment or a new service in Kubernetes is you have your piece of code and you need to put it into a docker image. You bake a docker image for this piece of code. And you push that up into a repository so that the, so that Kubernetes can then pull that down into the cluster. And then you need to update your YAML files associated with that deployment so that Kubernetes knows to get the latest version. And then Kubernetes goes and does its thing, which basically means bringing up the new version, shutting down the old version, if the new version comes up, if it doesn't fall down. And this whole process is not necessarily slow or long, but there is definitely a process there. And if you start out doing it all manually, docker build, edit your YAML file, coop control, apply, et cetera, it can take a while. And we've built a tool called Forge that automates a lot of that so that you can, I mean, you can imagine writing a script that does these commands. Forge is sort of a very smart, powerful script that it takes care of writing those scripts for you, which doesn't try to take over your process or whatever, but in any case. So Forge is a way to speed up that cycle. But ultimately there is a cycle there. There is some amount of latency between you make your code change and you have it running in the cluster. So telepresence lets you sidestep all of that. With telepresence, as soon as you can run your code on your own machine, you can run it in Kubernetes. It is literally one and the same thing. So if you're writing, say you're writing code in Kotlin, and you have your IDE running, which I guess would be IntelliJ. Right. You have your Kotlin code in IntelliJ, and you hit the compile key, sorry, the run key, and now your code is running. With telepresence, you can make that be in the cluster right away. You don't have to deal with the bake cycle, the push-up to the repository, and then pull down from Kubernetes and all that good stuff. If you're using a language like Python or Ruby where there's not even a compile step, for example, if you're writing your application in Flask or in Express, you can literally edit your code, and the running application will update itself and immediately show you the newest results. With telepresence, that's how quickly your service development can go. You can have almost an interactive feel with developing your service because telepresence makes your local development environment appear like it's in the cluster. And there are essentially like two directors, right? Like you want to call other microservices that run in the cluster already, and you want to be able to be called like you as your service from other services that run in the cluster. IntelliPresence supports both ways. That's right. So consider your organization has a Kubernetes cluster and an application that is running in Kubernetes, and you're writing a brand new service. So in that stage one of writing the service, all you really care about is being able to run your code but access the services that are already there, access maybe cloud systems like Amazon RDS or Google BigTable, something like that, that your application might want to use. And so telepresence takes care of that for you by effectively putting your machine in the Kubernetes cluster in a sort of outbound direction. Now over time, you develop your service, you deploy it and it's running and you find a bug or you discover that you need to add a new feature. But now that your service is out there, other things are going to be accessing you as well, right? This is not a brand new service anymore. This is something that's in production. So now you need both directions of telepresence's proxying feature. So when you run telepresence and you swap that deployment out and it's now running on your local machine, other services that are out there will hit the Kubernetes service and that will go to the telepresence pod that will come down to your local machine. So effectively, all of the services that are in your cluster that are accessing your service will continue to do so even as that service is running on your laptop. So it's really a first class citizen. It has FQDN. DNS works as normal. It can just access it from other services. You as a developer don't really see a difference. And at some point in time, you're going to switch over to the traditional model and building the image or what's your recommendation? What's your flow there? Well, ultimately, you don't want to keep running your application on your development box. So once you're satisfied that things appear to be correct as you want them, you've done your Git commit and your push and so forth, you'll do your forge deploy or whatever your deployment system is. And so you'll bake that Docker image. You'll put it into the repository. You now have this immutable blob that represents the current state of your service at this particular version. And you will deploy it into staging and then ultimately into production. And it will run in Kubernetes. So telepresence is not there to be part of your production system. I mean, sure, if you have this crazy the world is burning situation, maybe you'll use telepresence to get into production and do some very, very temporary short-term nasty things. But in general, telepresence is there to help you as a developer, build your system, improve your service, debug your service, basically make your development process be quick and interactive as it would be for something that wasn't a service running in a Kubernetes cluster. Got it. So in a sense, it's the proxy is the new SSH into the production server. You can at any point in time access it and do crazy things, but you essentially should only be doing it for development and maybe troubleshooting, but not really. So once you're satisfied, you kind of like bake the image and then really deploy it through other means. And I think it's fair to say that, sorry, go ahead. Sorry, go ahead. Okay. Okay. I was going to say, so with Kubernetes lets you do things like canary deployments and things like that, splitting traffic between multiple versions, especially if you use service mesh tools like Istio or Envoy sidecars or things of that nature, let you do some advanced routing stuff where you could set things up such that most of the users of your service get the version that's running in Kubernetes, but a small fraction of them get the version that's running on your laptop through telepresence. So it's very reasonable to try to take a running system and debug it using telepresence by stepping through your code in your debugger, setting breakpoints and things like that. As long as you understand that some portion of your traffic is going to be very, very much slowed down when you do this, because obviously when you stop at a breakpoint, you're stopped and you're not returning an answer. But I mean, in a staging system or with the right type of redirection setup, for example, you can give your test user a special cookie or a special HTTP header and set up your routing system like Envoy can do this to only send to the test version of a particular deployment if you have that header set, something along those lines. So you can carefully control what traffic you're debugging and then use telepresence to get that debugging info right in front of you in your own IDE. What I was about to say or ask you is it fair to say that telepresence is not only useful if you have a real life cluster somewhere in the cloud or on-premises, whatever, but even if you run MiniCube or MiniShift on your local machine, it's still kind of shortcut this building image, deploying it, pulling it. And so you still benefit from that even if your cluster is on your machine. Absolutely. So with something like MiniCube or MiniShift, you have the option to Docker build using the Docker that's running in the MiniCube. So you can avoid the one step. You can avoid pushing out to the repository so that Kubernetes can then pull it down. So you reduce the latency a little bit, but that said, it's still far more interactive. I mean again, the ability to be able to literally edit your code and hit save and then try to access the service and see immediately the new results is just, it's beyond going through that process. And I think using telepresence and developing in MiniCube is a great thing to do if you're on a plane and you don't have good connectivity or if you're starting out with Kubernetes and you want to understand what's going on and really see what happens if you crank up too many things in the cluster or start killing things willy-nilly or whatever. It's a great exploration tool, but it has its own drawback. MiniCube doesn't work quite the same way as Kubernetes does in cloud providers or likely how it would be provisioned on-prem. So yeah, there's a bit of a trade-off there, but telepresence is certainly still very useful in that case. Talking about, we had a couple of use cases now and I think everyone gets it that if you're serious about developing on Kubernetes, then you need telepresence to really be very agile and having a high developing velocity there. Can you talk a bit about limitations or stuff that people should be aware of that managing expectations that they know this won't work or you have to make sure that certain Kubernetes version or whatever the requirements or limitations might be? Sure. Well, let's talk about how telepresence works because that's ultimately where the limitations come in. Telepresence is trying to do something that isn't natural for Kubernetes. Kubernetes typically controls its hardware, its nodes, and controls the pods or the various sets of Docker containers that are running inside it. And so it's very well designed for that purpose. What telepresence does is deploy a pod into Kubernetes that then pretends to be the service that you're running on your own machine. So some aspects of this work perfectly. The pod that's running in Kubernetes has access to all of the configuration, the config maps, and the secrets, the environment variables, and the file system. And telepresence can copy those over to your local machine without any trouble. Other aspects of things are a little more complicated. So there are three ways that telepresence works right now. If your code naturally runs in a Docker container, then you want to use the container method. And what that means is telepresence will manipulate your Docker container running on your local machine to set up its networking such that it appears to be running inside Kubernetes. But I think the more common case is that your code is running directly on your local machine, like you're using your debugger, and you hit the go button, or you're running your express, or your flask service. And so for that, we have two methods. There's inject TCP, which uses overriding the linked libraries that go with your executables to intercept the network traffic and the DNS traffic. And then there's a VPN, the proxy VPN. So what we're trying to do there is set up a VPN-like system where all network traffic is captured, and then the right subset of it is sent to the Kubernetes cluster. So in either case, we have limitations. With the library-based injection of code, you run into issues where if your library is completely statically linked, sorry, if your executable is completely statically linked, for example, most Go code, then you cannot inject a library into it because it's not trying to load any libraries. And certain commands, especially operating system commands like Ping, they don't follow the normal path of network connectivity. So those things won't work. But, I mean, if you're developing an application, this is not a major limitation most of the time. Now Go is rough because lots of people love Go for writing microservices. It's a fantastic language for that. And so you're sort of stuck using not the preferred tool chain using GCC Go or having it use the C library approach to doing networking rather than the highly optimized Go specific stuff. But for most other things, if you're talking about C code, Java code, Python, Ruby, Node, etc., you're pretty good. We run into a few other issues specifically on Mac OS because of system integrity protection. And that basically says this injection of code through the library mechanism doesn't work for any binary that's under system integrity protection. And so, yes, your code is not going to be under that. But if you happen to run your code through a shell script, the shell is protected. And so that injection gets lost. And by the time we start executing your code, the injection is already gone. So, again, there's ways around this and telepresence sort of tries to work around this. But if you have a hard coded slash bin slash SH or slash user slash bin slash N in your script, you're stuck. So, these things are sort of unavoidable. Apple works very hard to protect its users. And it's a trade off like many other things. So, what I hear essentially is that, sorry. Now, the other approach is the VPN approach. And that avoids messing around with your binaries and instead operates at the system level. So, it captures all the traffic on your laptop, including your web browser and so forth that you might be doing other stuff with. Now, it selectively only forwards the packets that are relevant by looking at their destination IP addresses. So, it has to figure out Kubernetes is over here in this space, maybe it's 10.0 slash 16 or whatever block of addresses are apparently pointing to Kubernetes. And then it captures and forwards just those. Generally speaking, that works great. The one limitation right now is that you can only run one VPN at a time. So, if you need to use a VPN for your own purposes, you can't run telepresence in this mode at the same time. And similarly, because the proxying is all one piece, as in you're replacing one deployment in the cluster with your own, with telepresence, you can't, with VPN mode, you can't run telepresence more than once simultaneously. So, you can't proxy two different deployments simultaneously. Whereas with the other mechanism you can. Now, this is something I'm working towards improving in the future. Right. We come back to that, to future steps. But let's stay for a moment just to wrap up, essentially that part. So, essentially what I hear is pretty much all languages, you have no problem. I'm sorry, I lost you for a few moments. I might have had a blip in my networking. Could you say that last part again, please? Yeah, essentially wrapping up this part in terms of languages. All the languages are supported. And for some languages or systems, like essentially Go, static queuing, you need to do a bit of some workarounds, apply some workarounds, or macOS, which has certain other security requirements. But there are always workarounds that make it possible for everyone to actually benefit from telepresence. So, there is no, like you cannot use this language just for certain languages. You need to use some workarounds, essentially, or certain systems. And one thing before we get to the future part, in terms of, since we are at security, quite often we see issues with, especially since we have RBEC now and vanilla, when it is 1.6. Are there, do you see any issues around people with using RBEC and or is Elinux, so any kind of security related access control or whatever, otherwise settings? Well, you bring up two very tough topics, actually. So, with regards to SELinux, we've noticed that it's definitely possible for SELinux to basically keep telepresence from working entirely. And I'll be honest, I've largely sidestepped that issue for the moment. I'm testing on a sort of a default Fibora install and on a default Ubuntu install. And I haven't run into issues there specifically. And I'm looking towards users getting back to me and filing bugs or whatever the case may be, when they run into issues, so I can address them one at a time. Because SELinux is a huge topic, and I will fully admit I do not have my brain wrapped around that just yet. So, we'll see how that goes. But so far, at least with the standard configurations, it hasn't been an issue. With the role-based access control, I think what we've seen so far is users are using telepresence with their staging setups or with their mini-coups or things of that nature, smaller, sorry, non-production clusters, where perhaps we've not run into this yet. So, there's definitely experiments I have in mind. I want to see what happens and how telepresence can fail. I think the thing we have going for us here is that as long as the developer has the ability to deploy the service, the code that they are working on, they're going to have the same capability to deploy what telepresence does. Telepresence is not messing around with the cluster as a whole. It is only working within that pod. So, I believe we should be fine, but I have to admit I have not pressed against it yet. Right. So, I did not intend to put you on the spot here. It's really just really this expectation management that people know. If they would need to disable SELENX, that's fair. Or if they, oh, be careful. You need to have these and these elevated permissions for the RBAC settings or whatever. I think since there is, let's be honest, there is no alternative to telepresence. It's the only tool in that space. So, it's just fair to say, if there is something people should be aware of, be aware of. But as far as I understand, more testing is needed. So, we need the community to get back to you and say, hey, I am running into this issue. But so far, essentially, you haven't run into bigger issues there. Exactly. So far, I haven't run into it. And I look forward to having specific use cases that need solving that I will happily look into and try to, I mean, let's be honest, oh, you have to disable SELENX is not a great answer. It's never a great answer. So, I'm hoping that we can come up with maybe a more pointed solution if problems arise. So, yeah. And is there like, you know, there are like managed environments like Google Container Engine or OpenShift online where maybe a user doesn't really like even have the privileges, whatever. Is there anything that you need to be aware of? Does it need anything and the elevated privileges in the cluster to run? Or can I just use it with any kind of cluster? You should be fine with any kind of cluster as long as you have the ability to deploy that, well, basically, if you can deploy your code, if you can create a deployment that runs your code using your image, then you should be able to use telepresence to deploy its code. Now, telepresence's Docker images are in, I think we're using, goodness, are we using Kway? I actually forgot. They're in the public cloud. So, as long as you can pull that down, you're fine. If you're firewalled off, it may get slightly more complicated. You'll have to replicate the telepresence Docker image into your private repository or whatever. But no one has brought that up yet. So, I'm curious to see if we run into that. That's actually a good point. So, if you're behind the firewall where you can't pull from whatever public registry you're using, you can still benefit from it. You just need to take these images and serve them via your own porn site on premises or whatever registrations still work. Okay. Awesome. Let's move on to the future. What's not necessarily only telepresence? You mentioned a few things in terms of improving and definitely you're seeking feedback and more use cases from the community and testing. But where is that whole thing heading? What's next for telepresence or other developers support tools in a data wire? Well, on the telepresence front, we started out with telepresence by addressing particular use cases. For example, I want to develop a brand new microservice and I want to be able to talk to the cluster, talk to the services in the cluster. So, for that, we had to make sure that the local instance of your service that's running on your machine can talk to the cluster. We need to proxy connections from your machine to the cluster. And we solved that. And then there was another use case of we need to be able to access services that are out there. We need to be able to access the secrets and the configuration that's out there. And so that is where we brought in this proxying of stuff from the cluster down to your machine. And we sort of wrote these things as the use cases came up and it works great for those use cases. But we've discovered that people want to do more. And at this point, it's starting to make sense to sort of separate out the pieces of what telepresence does and let users access those pieces individually. So, there's kind of three pieces really. There is network from your laptop to the cluster. And that's the sort of thing where I feel like someone who works on a Kubernetes application could kind of leave that running all the time. There's really little reason not to have that running unless your cluster has services that have names that conflict with other things you use regularly. Why not just be constantly connected to the cluster using this telepresence channel? Then there's bringing stuff from the cluster down to your machine. Now, at the moment, if you're using inject TCP, you can do that with multiple deployments. But if you're using the VPN approach, you can't do that. So, we're talking about how we could possibly separate those two pieces so that telepresence can put multiple deployments out there and collect connections and bring them all down to your local machine independently of the portion of the VPN that's responsible for pushing data up to the cluster. And so, that would allow you to use the VPN methodology but replace multiple deployments. The third piece is perhaps the simplest one where you need to have a deployment in the cluster to access configuration and secrets. So, again, we run into this situation where we want to be able to do multiple deployments but we don't want to run into this issue of having to use all sorts of workarounds for Go binaries and static stuff and this and that. So, I'm trying to brainstorm up with the team how we can separate these three things still offer a reasonable user experience. We don't want this crazy, complicated command line. So, we're working on it and we hope to end up in a place where, as a Kubernetes developer, you can reasonably swap out as many deployments as you want to have them all running on your own machine if you need to. You can redirect services to point to your machine rather than having to create multiple deployments and then you can access all of the usual file system type stuff and environment type stuff that you can get in Kubernetes. But we're still a ways out from that. So, telepresence is open source and I assume you're open to patches or someone? Absolutely. So, we are on telepresence.io is our website. Our documentation is there and we're on GitHub so it's data wire slash telepresence and it's linked from telepresence.io and we are very happy to receive issues and pull requests are of course always welcome and we're particularly interested in hearing about how you use telepresence because, as I said, that is how we develop the feature set that's already there. We want to know what you're trying to do so we can help you do it. Is there somewhere like a road mapping section or something where people can see what you plan for the new future or anything like that? So, I believe we have, no, I take that back. I don't believe we have a roadmap on the site at the moment. So, to be clear, I took over development of telepresence from the original developer a few weeks ago and I'm basically running down his set of suggested changes because he's brilliant and he made telepresence and it's amazing. And I'm trying to do the things that he suggested first because he's more aware of the community than I am. But as that process winds down, I agree it makes sense for us to start writing down what we want to do next and having that up there and requesting feedback and comments. Awesome. And how do people reach you via Twitter or how do people best reach you? The email is probably the best way to go. Okay, Arc3 at datowire.io. Yep. One last question around the future, whatever, of telepresence or other tools. Is there anything in the, you know, in a SIG, in a special interest group or wider community where you, you know, want to submit it or to bring it in a standardized form or do you want to keep it in your space? Are there any plans or just out of interest? Let me see if, let me answer and you tell me whether I understood the question. So, Datawire is involved in the Cloud Native Compute Foundation and we are, regularly we present at Kubernetes. The community meetups, KubeConf and things like that coming up. We have some stuff going on. And if, you know, if you can help us talk about telepresence elsewhere, we'd love to talk about it. And as well as Forge and all the other great stuff we're doing, Ambassador. So, we'd love to get the word out. As far as development of telepresence goes, again, we develop openly in GitHub and we are happy to hear from the community and go where you need us to go. So, I guess if there's a Kubernetes Meetup or Cloud Native Meetup somewhere out there and people would love to, you know, have you speak there, that might be possible if the distance works out. Absolutely. Oh, we also hang out on Kubernetes Slack in the SIG apps channel. And, well, not just there, but there specifically telepresence does come up from time to time. And we try to answer questions there. Perfect. That's a great place for people to, yeah, reach out to you and start starting questioning and giving feedback. Awesome. Hey, thanks a lot for your time. That was great. And, yeah, let's hope that, you know, seen it's available for OpenShift and I'm a big, big user and fan myself that this, the word spreads and then you get even more feedback and more users. Yeah, thanks a lot for your time. Thank you very much.