 Hi, I'm Yao. I'm a software engineer at Bloomberg data science platform team. Yeah, and this is Yifan. I'm also a software engineer on the data science platform team at Bloomberg. So okay, so Bloomberg had a quarter in the New York City. It's a global finance, medium, and technology company, and we have dedicated AI teams and they convert the results in the Bloomberg terminal products. So in the earlier of this year, we just announced the availability of AI-powered earning costs summary on the Bloomberg terminals. So our data scientists adore the digital notebook because they can explore their dataset and produce their models in an interactive way. However, it will also bring security vulnerabilities such as unauthorized access and data breaches. So in this talk, we're going to share our experience about securing a Jupyter notebook using e-steel OPA and also network, Calico network policies. So first, let me give you a quickly introduction of our data science platform at Bloomberg. So we build our data science platform on top of Kubernetes. It offers data exploration like Jupyter notebook and Spark, model training like PyTorch and TensorFlow, and the model serving using K-Serve. So our notebook clusters are integrated with several data storage within Bloomberg. So like we show here, for example, we have three compatible object storage and Hadoop. So for those data storage or data sources, we have in-house credential management to secure the production data access. In this talk, however, we are not talking about those credential management. Instead, we're going to focus on the secured ingress and e-grass. So in order to understand why security matters here, so let's imagine a very simple use case. So we have Yifan. He is from the data science platform team. He creates a Jupyter notebook on our Jupyter notebook cluster and in the team's namespace called DSP. And he wants to use this notebook to access some production data storage. So such a very simple use case, right? But it will incur multiple security issues. So first, how do we know that Yifan is really who claims to be? So it is an authentication problem. And then, secondly, it's Yifan allowed to access the Jupyter notebook URL and used the notebook created in the DSP namespace and running on our notebook cluster. So that is an authorization question. And third, is this notebook allowed to access the production data object storage? This is more about e-grass control. So now I'm going to turn it over to Yifan. He will tell the storage about our secure ingress control. Thank you, Yifan. So how do you address these security issues? So the first is Yifan really who he claims to be. This is an authentication problem. So our solution is that we build a single sign-on system based on OAuth 2 and we deploy an open ID connect, which is OIDC for short. We deploy such a proxy that handles the SSO callback and it verifies that the token is valid and then routes the request to the next component. So when a user opens the URL of a notebook, the request goes into an Istail gateway that is deployed in the Istail system namespace. This Istail gateway is connected to an Istail virtual service that is deployed in the DSP tenant namespace. This virtual service is again connected to a cluster IP service that is in another namespace called Network Proxy. And this cluster IP service will route the request to a pod called Network Proxy. This Network Proxy will first check whether the request has a JWP token in the request authorization header and if the request does not have such a valid token, it will redirect the user to the external SSO provider's login page where the user needs to enter its login credentials and typically just username and password. When the user is successfully logged in, the SSO provider will send a JWP token back to the Network Proxy pod and the Network Proxy pod will verify its validity and if it's valid, it will set this JWP token in the request authorization header. After that, the Network Proxy pod will send a request back to another cluster IP service that is also in the DSP namespace. This cluster IP service job is to expose the actual Jupyter Notebook pod, but instead of going directly into the Notebook container, there's another Istail sidecar container that also lies in the Jupyter Notebook pod and it captures this request. The Istail sidecar tries to authenticate this request by looking for an Istail request authentication that is also configured in the Istail system and this isn't thought during the initialization of this cluster and in this request authentication, we specify that the request needs to have a bearer prefix and it must be issued by the issuer called .blumberg.com and if the requests have both met, fulfills both these requirements, then we think that this request is successfully authenticated. So now we have addressed the first issue. The second one is if I'm allowed to access the Notebook and this is an authorization problem and to answer this question, we configured authorization policies in the principle action and resource format and we store it in a custom policy store. Principle basically means who is performing the action and typically just a user or a role account and action means what kind of action is the principle allowed to perform. For example, it can be read or write and resource means what resource is the principle allowed to perform the action on and typically it's just a Jupyter Notebook in our case. So in the bottom left corner, you can see an example of such a policy. The principle is set to LDAP group data science platform which represents the entire DSP team and action is hard coded to access. Resource is set to such a format that the DSP star. So Notebook cluster is the name of the cluster where the Notebook is running. DSP is the name space where the Notebook is launched and star basically means every resource in this name space and we store such a policy in the policy store. And besides the policies, we also leverage several functionalities provided by still an open policy agent to help us build this authorization system. And to further take a look at this system, we have an OPA bundler which periodically pulls the policies from the policy store. And the OPA bundler will pack all these policies into a huge JSON file together with a regal file which tells how to authorize a request against the preconfigured policies. And the OPA bundler will put the things together inside a tar file and serve them as an OPA bundle. And we also have an OPA server which periodically pulls the bundles from the OPA bundler. And this is a detailed explanation of how the OPA bundle looks like. So in this data.json file on the right side, we can see that basically it just consists of JSON items. And one item looks like this. It's a key value pair. And the key is the LDAP user ethon which represents ethon the user. And the value is another key value pair. The key is access, which is the same as the action we configured in policy. And the resource is also the same as the resource in the policy cluster, no cluster namespace dsp star. So it's worth noticing that the OPA bundler expands the LDAP group data science platform into a list of LDAP users and put them in this JSON file. And here is what the regal code looks like. What it does is quite simple. So first it imports the data.json file we've just seen into this regal. And then it tries to extract the principle action and resource headers from the incoming request. And we'll cover how these headers are set in later. And finally it tries to find the policy in the data.json file that can match the given principle action and resource. And if there is such a match, it will set the allow field to true and basically means that the request is successfully authorized. So let's move back to this diagram. We have talked about how the Esto-SciCar works in the authentication process. And in authorization, it is also a key component. So here we add a WebAssembly plugin to this Esto-SciCar using an Esto-Wasn't Plugin resource. What this WebAssembly plugin does is that it takes an HTTP request as input. And as we know that the request has an authorization header, which is set by the network proxy pod. And this authorization header has a JWT token in its value. And the token contains some basic information about the user. For example, the username. And the WebAssembly plugin will extract this username from the unmartial JSON payload and set it in the output request principle header. And in the incoming request header, we also have such an authority header, which is equal to the URL that is being requested. And this is set by the browser. So in the URL, we have such information as the name of the notebook, which is JUP-EFAN. And we also have DSP, which is the name space of where the notebook is running. And the name of the cluster, which is notebook-cluster. So the WebAssembly plugin is then able to reconstruct the resource identifier from the input request. And so here, as we can see here, the resource is cluster, notebook-cluster, namespace, DSP, JUP-EFAN. And the action field is hard-coded to access. So let's move back to this diagram. When a request comes into the SDL sidecar and finishes the authentication process, the SDL actually has no idea about the OPA server or how to use this OPA server for authorization. So here comes our Envoy filter. And Envoy filter allows you to customize the Envoy configuration generated by SDL Pilot. And in our case, we use such an Envoy filter to tell SDL that you need to delegate the authorization to an external service, which is the OPA server. So the incoming request will be routed to this OPA server by the SDL sidecar. And inside the OPA server, it runs the regal code we just seen and tries to find a match, a policy that can match the given request. So here, we do have such a match. And so the OPA server will say, yes, this request is successfully authorized and return the decision back to the SDL sidecar container. Once the SDL sidecar receives such a positive decision, it regard the request as successfully authorized and will redirect the request to the actual notebook container in this pod. So the user is finally able to see the notebook open in the browser. So if we need to put all the things into this bigger picture, when a request comes into the SDL sidecar and finishes the authentication process, it will go through the WebAssembly plug-in to get the additional headers patched. And then the SDL sidecar container will route the request to the OPA server. And if the OPA server says, yes, this request is authorized, the SDL sidecar will route the request to the different notebook container. And the user is now able to open their notebook. So that's all for authorization. And let me hand over back to Yao to cover the egress control part. OK, so finally, we come to the last question. Is notebook to be able to access our production data storage? So actually our notebook cluster running in production is actually in the production network. So it is by default to access all the production data set. So some data set is fine. So user, they can use notebook to interactively access those data set. However, some data sets are very restrictive. Even the teams and the users, they are allowed or they have full access to those data set. They are not allowed to access those data set interactively, say, using notebook. So in order to protect those very sensitive production data, we decide to use the cataclysm network policy to control our egress. So here I just show a very simple example. So in this corner, we have the global network set. So here you will see there is an annotation. It's our machine group annotation. So basically this global network set just represents all the machines that runs the object storage services. So we also have a CIDR resolver. You can think it's a long running service. So this CIDR resolver just watches the network set, and it tries to parse and understand this machine group annotation, and then auto-populates the machine IPs into this network set. And then we also have the global network policy. So the cataclysm network policy is actually, you can think it's a group of Kubernetes network policy. You can define the ingress and the egress control. So here I just show the example of egress. Basically the first rule we define here is to allow the outbound traffic to the object storage. And by the end, the port is also default 443. And then at the end, we put the default deny, which means we will ban all the outbound traffic except the access to this object storage. So yeah, those policy and network set are saved in the cataclysm XED. And then we have the cataclysm Qube controller. It synchronizes all the policies and the network set, and sync that to the cataclysm node. So the cataclysm node is actually a demon set running on all the Kubernetes worker node. And it tries to understand the network policy and translate that into the host machine IP tables. And thus it can achieve the egress control. So let's put the last piece in this huge diagram. So we have defined our network policy and network set. So they are saving XED. And the cataclysm node, they just translate those rules into the IP table rules. And we achieve the egress control. So you may also find that that is our L4 level. And how about L7 egress control? So we are planning to use the cataclysm network policy plus E-Steel to achieve L7 control, but it's still like working progress. So yeah, that's basically all we have. So in summary, we have SSL based on OAuth 2 for authentication. And we have a custom policy store. And it's still an OPA for authorization system. And we have a protocol to build our L4 egress control. And that's it for today's presentation. And thank you all for your interest in this topic. And thank you all for attending. Yeah. We're still minutes for questions. Yeah. Sorry there. Yeah, we have a few minutes for questions, folks. If you've got any, go ahead and raise your hand. And we'll come around with the mic. Hi. Nice presentation. Do you use Jupyter app for notebook serving or just plain node? We use Jupyter server. Now Jupyter hub. OK. Can I inquire that how much cluster do you allocate for this old tool combination? Because I think it will suffer the compute resource very heavy. Oh, so yeah. So for our use case, we have dedicated notebook cluster. So basically on that notebook cluster, we only allow user to run the notebook. So you mean the compute resource competing issue? Do you mean the GPU things or like the what kind of? Yeah. Network and like memory and CPU. Yeah, GPU is fun. So basically our strategy is like we are a multi-tenant platform. So each team, they have their allocated resource. So they just allow to use the budget they submit every year. So they are now allowed to use more than those kind of memory when they start a notebook. Yeah. Good. Thank you. So thank you for the presentation. Very interesting. So in Jupyter hub, it is heavily customizable. And there is a way to add an authenticator. We connected with LDAP, for instance, and doing all the authentication and authorization directly in the Jupyter hub. Why did you choose to use Istio and all the web assembly and OPA and not just the Jupyter hub authenticator? Is there a reason or? Yeah, there's another reason. We build this whole framework because I think it's more like a standard authentication and authorization problem on the cloud. So those flow, or you think this big picture, will also be applied to our platform UI like the Spark history and TensorFlow board. So that is why we designed the whole thing. And it will be applied on multiple different UI application on our platform. Yeah, I know like Jupyter hub, they also provide this like rich features. But yeah, it's like we share the whole framework. Yeah. Hello, yeah. So thank you for sharing. Just a question about the egress part for there for. Any reason to choose Kotl code? In my understanding, CDM may be a little bit simpler in the use case. Yeah, that's another interesting. So yeah, this is like the technical decision because we use the Calico for our CNI. So it's like we build up of the container network things on top of that. And then yeah, then if we use the Calico CNI and we use the BGP and all those things together, then the network policy will be a very natural choice for us for the egress control. Although I know L7 is not here, but we can still rely on Istio to help us achieve that. I know like CDM is another option. I mean, in the future we can try things in that direction. Yeah. Change it, right? Unless you have a need to change it. Yeah, then that will be a huge change for our platform and then yeah. Any other questions? Please give them a round of applause for Great job.