 Hello, my name is Orivalo, I am a NetBSD developer, where I work mainly on providing ways to script the NetBSD kernel with Lua. Today I present a special use case on scripting the NetBSD packet filter, the NPF. I would like to start with that thought, that supports that we can split complex programs written C in a core part and in a configuration part using a scripting language. That's the idea we want to use here. I will first introduce the concept of scriptable operating system and the use case on scripting a packet filter. Next I will show an example on spec in the SSH version. Now you present some issues on scripting operating system, motivate why choose Lua as the scripting language for the kernel. Then I will show our kernel scripting environment in NetBSD. Then I present Lua data library for binding the system memory to the Lua script. Then I present the NPF Lua, the bind between the Lua interpreter and NPF. And finally I will show our conclusions. So what is a scriptable operating system? A scriptable operating system is the combination of extensible operating system with scripting language. It is the application of a scripting language to write extensions in an operating system. The motivation to have an operating system scriptable is to give more flexibility to the operating system, helping users to meet new requirements to configure the kernel subsystems. Also to allow non-kernel developers to change the behavior of the kernel. And also we can prototype new features using scripting language. We could have the operating system scriptable both in its user space and in its kernel. But the key idea here is to script the operating system kernel, which we call kernel scripting. It is the halfway between using just kernel parameters and actual kernel modules. Also the halfway between using domain-specific language and system program language. There are two ways to provide scripting to a kernel. We can extend scripting language. In that way the scripts will treat the kernel as a library. They will invoke the operating system kernel and can embed the scripting language. In that way the scripts will treat the kernel as a framework. The kernel will invoke the scripts to perform the extension. We have some use case for scripting the kernel. By embedding we could have a packet filter done in using scripts. It's the main purpose of this presentation. But we could also have device drivers implemented or partially implemented using scripting language. We could have process schedulers invoking scheduler routines to decide which process should run. By extending we could have protocols implemented in the kernel using scripting language. For example we could have HTTP server using Lua. The motivation to have a packet filter scriptable is to deep inspection the packets, the network packets. We could provide traffic in shape or intrusion detection and prevention. We could also have new features implemented in the packet filter like for locking or new protocols. Suppose we have an SSH server which has a vulnerable version. We can use Lua script to inspect the application layer to identify which version is running on this server before it communicates its version to the client and then drops the communication. Here is a script designed to do that job. It just uses the Lua string library to match the version of the SSH protocol. If the version is the vulnerable version it drops the packet. Otherwise it lets the packet go through the network. We also have some issues on developing a scriptable operation system. We need to ensure that the scripts will not compromise the overall integrity of the operation system. We need to ensure correctness. The script should not incur harm to the operation system. We should guarantee the isolation between the scripts. One script extension should not modify another one. You should preserve the liveliness of the system. Also, to make sense, a scriptable operation system has to provide ease of development for the extensions. It must provide ways for users to implement useful things using the script language. Run efficiently reasonable amount of time to do not compromise the system operation. We can achieve correctness by sandboxing the scripting language. We can remove features that could cause harm to the system. We can have automatic memory management to prevent scripts from dereference, new pointers or leak memory. We can have only one thread for each extension to prevent deadlocks. We can have protected calls to let the system fail safe when the script performs incorrectly. We can use the same strategy of kernel modules allowing only privileged users to load scripts. We can have isolation by having fully isolated execution states on the scripting language. Each station propose could have its own execution state. We can achieve liveliness by capping the number of executed instructions of the script. The ease of development could be achieved by the nature of the scripting language. They are very high-level language, usually dynamically typed. We can also provide domain-specific APIs to users for users' writing extensions. We can achieve effectiveness and efficiency by providing proper binds. Binds are the interface between the scripts and the kernel. If you want to extend a specific subsystem, we have to provide new binds to connect the kernel and the script. We can have both binds to the embed use case that I showed before and for the extend use case. Proper binds can also help us to achieve the other issues on developing a scriptable operation system. We have proper binds that can deliver to users' scripts, domain-specific APIs, and we can perform verifications on the bind site. But it is also the most difficult task on scripting the operation system. For example, in our case on NetBSD, we have a kernel environment to script the kernel, but we lack bindings to extend the subsystems using scripts. So why choose Lua as the scripting language for a kernel? Lua was designed to be an extensible extension language. It can be both embeddable and extended by the host application. Also, it is just a sea library, an ISO sea library. Lua is almost free-standing. It's really easy to port Lua. It has no dependence on the operating system in its kernel. Lua has a very small footprint. It has only 240 kilobytes on current NetBSD or AMD64. Lua is proving fast. It performed very well in many independent benchmark tests. And it is MIT-licensed. Lua also has safety features like automatic memory management, protected call, and which is really important here, fully isolated states. We can have separated states for each purpose of extension and can also cap the number of the executed instructions. And why not choose something else like Python or Perl? And the main answer is the size. Both have around megabytes of size. Python has two per one, which is the same magnetic kernel. And also, both have OS-dependent code, and they are hard to embed. They are mainly used extending the language to write the final application using the scripting language, not otherwise. So in NetBSD, we have our kernel scripting environment. I started to work on a kernel scripting environment in 2008 with Lunatic for Linux. Then I ported it to NetBSD in a Google Summer of Code project. And then Mark Baumer developed a new infrastructure around the Lua port for the NetBSD. And in this year, I started to work on NPF Lua, this use case. Lua 4 is composed by the kernel embedded Lua, which is the Lua interpreter ported to the kernel. The main difference of this version of the Lua interpreter is that it has no float-pointing numbers. It has only integers. We also have, in Lua 4, a user interface, the Lua control program to load scripts into the kernel. And a kernel programming interface to allow the kernel developers to prepare their subsistence to be scriptable using Lua, which is defined in cs slash Lua dot 8. So when a kernel developer wants to script his subsystem, what would he need to do? First, she'll need to write a binding to connect the Lua interpreter and her subsystem. This bind could be in the way of extending, providing APIs for writing extensions, or in the way of embedding, calling a script in a specific key point. For example, in NPF, we use the embedding way. We call the user script when a package arrives in the interface. So once the developer has extended his subsystem to be scriptable with Lua, the user can load the script using the Lua control user interface. Then, in that case of NPF, the kernel subsystem can call the script to perform some action. Lua data is a binding for connecting the system memory and the Lua scripts. It is a regular Lua library that could run both on kernel and user space, and it binds the memory by pushing memory blocks represented by a pointer and a size or an Mbuff structure, which is used to represent the packets in the kernel. It performs boundary verification on each access on the packet, and it has support for packet data using declarative layouts. Lua data also has support for bit fields and for string fields and conversion. We can convert data objects to Lua strings. Then we can, for example, apply pattern matching to the packets. And we also have support for engine as conversion. In the SSH version, for example, we use Lua data to convert the packet into a Lua string. It's the only point we have data copied in this example. When we push packets to the NPF Lua using Lua data, we perform no copy on that packet, on the Mbuff structure. But when we want to access it as a string, as a Lua string object, we need to copy that. We could, for example, delimit segments to copy just a part of the packet. But in that case, we copy the whole packet, then we apply that pattern. I highlighted two points where we are actually using Lua data in this example to convert the Mbuff to a Lua string. We can also have declarative data using Lua data. We define a special Lua table using offsets and size. In that case, we are representing the RTP header for the site. If a packet holding the 8263 encode should pass or not through the network. Then we just have to define an offset for that field and size. Then we apply the layout to the data object. Finally, we can access that portion of data on the packet using the Lua table notation. NPF is the NetBSD packet filter. It supports layers 3 and 4 inspection, stateful filing, IPv4 and v6. It has a special feature that is important here. It is extensible. It supports rule procedures to be applied once we have matched one rule. In that example of the SSH protocol, we can have NPF rule, regular firewall rule to match the connection on SSH port. Then apply our script to perform GP inspection. NPF Lua is the binding between the NPF subsystem and the Lua scripting language. We have a kernel module and a parse module to interpret the NPF configuration. So we can define a procedure in NPF that we call a special function named filter on the user script. In that example, we can apply that script on all traffic passing in the interface. Once we have the NPF configured by using the Lua script, we can load the user script using Lua control program. It is an ongoing project yet. We are under development. We have some issues that we are working in right now, like having actual Lua rules instead of using rule procedures, because rule procedures cannot be used to match the packets. We have to match it before having the firewall rule applied. It then applies our script using actual Lua rules, binding it inside the NPF. We can use it on the regular order of evaluation of the firewall. We have some adjustments we have to do in buffer handling. We have to fail safe when the pull-down API fails. We want to have support for no continuous strings using the Lua L buffer API for not having to ensure the continuous of the Mbuff. We could copy just the piece on the chain of the Mbuff. We also want to have support for packet mangling, modify the packets. Lua data has support for modifying the packets, but we are not communicating the NPF already. We want to have support for automatically scripting loading in the NPF configuration instead of having an additional step to load the scripts and predefined layouts like IP, TCP, UDP to... Users don't have to write their own layouts for such regular use. We want to have a Lua network library for defining address, for example, and rule editing, editing using... to modify the rules present on the NPF configuration in runtime. And also, we want to have NPF configuration entirely written in Lua from the user space. We have used a full-fledged and general-proposed language for packet filtering where we can pattern matching using hash table, for example, which allows us to do deep inspection on packets. Our SSH version, for example, had no measurable overhead on a 100 Mbps virtual network interface. We achieved 96 Mbps with or without using scripting. This example has only 12 lines of Lua code. Attesting it is simplicity. And Lua data is a generic binding that could be used for other extensions purpose such as device drivers or implement network protocols. Here are some reference. You can look after that. The source code is not in the tree already, but the patch can be reached in the site. And Lua data is also in the GitHub, and it can use other systems on user space. So if you have any questions. When you're doing deep packet inspection, what happens if the data that you want to examine is not sent in a single packet? What happens if a string is split between two packets? Can you hold a packet for later processes? You can hold information for later. You can, for example, use hash table to hold information, hold the state on the script. But if you will hold the entire packet, I think it's not a good idea, but you can. One will not hurt, but... So you're taking wire data and you're converting it into a string in Lua. What are you doing to sanitize the data you receive? I did not sanitize any data. So that means that if there is a hole in Lua, I can basically get control of your firewall? No. I really think not, but you can safely put in Lua a string from C with something different to the characters. Lua socket, for example, the bind library, use it in main place, use the Lua string for that to represent binary data. I don't think it is a role, but we can talk about it after that. Any other question? Hi. Have you thought of making some enhancements in Lua so that you can skip the copying of the packet? Yeah. Sorry. I missed the... The meat of the centers. Well, have you thought of changing the way you handle packets in Lua? You said that you made copies of the packet. Can we skip that? For using the Lua string library, I think we cannot, because Lua has its own representation of strings and it obligates you to copy that if you want to use the Lua string library. But you can't have your own string library to don't copy the data. But you need to implement features that you want, such as pattern matching. I would like to know how you debug these Lua scripts. How we debug? Yes. Yeah. We can print in the log interface, in the system log interface, and you can use the Lua debug library. It is also ported for the kernel. I have a question myself. How do you deal with the memory model, the memory management model used by Lua compared to what you do in the kernel itself, because the memory is... You have more constraints and all that. Yeah. Lua, in fact, don't have a memory allocator in its core. It allows the host program to deliver its own memory allocator. So it defines a prototype that must respect to... It's such a realloc that it could be called to a lock memory and free memory. And we just have to implement our locator and give that to Lua. I don't know if... Okay. Any other question? So in the S-Sage version example, you said that there is no measurable performance overhead with and without the Lua script. And how is this possible since with the Lua script, you have an additional memory copy? Yeah. It's limited by the network, not by the copy we are making. We are making... The other... The latency of the network is bigger than the copy we have done. So is this on a... Are the Lua scripts ran on a per packet basis or on a per stream basis? Like if I'm trying to detect whether a user is downloading an ISO, is it going to, you know, make a string that is 4 gigs, 4.7 gigs in size in memory so it can run that Lua script on that stream? No, you have on the packets. You... You can save some states as a target before, but it's not streaming. In BF, is this stateful firewall? Does it... Can you set it to process the first packet of a state, for example, the TCP connection? If I can inspect the first packet. Yes. You can write a regular NPF rule to match that packet and apply the Lua script for that after you match that packet using the NPF. And every packet, you can apply a TCP layout to inspect the fields of the TCP and also you can have... You can use bit fields like that on TCP or IP. No more questions? Thanks. Since you mentioned originally that this feature is meant to be secure and safe and data isolation and sandboxing, can you support multiple individual instances of the Lua interpreter? Multiple instances of Lua interpreter? Yes. Yes, we support. But is it my to-do list to have it on NPF? Okay. By now, we have only one Lua state for NPF, but in the kernel, we have multiple Lua states for... specific subsystems. Yes, thank you. Right, here. No more questions? Here is my information if you want to talk to me later. Please be welcome.