 Hi, I'm Kohei Tokunaga from Entity Cooperation and I'm a maintainer of BuildKit and Revia of Continuity and today I'm going to talk about BuildG, an interactive debugger for Dockerfile. So first, let's get through the summary of this talk. BuildG is an interactive debugger for Dockerfile and it provides interactive prompt for debugging Dockerfile with breakpoint and it also supports running a container on an arbitrary instruction and deeply inspecting it and BuildG also integrated to IDEs like VS Code, NeoVim and Emacs and IDEs GUI-based debugging feature can be used for debugging Dockerfile and this integration is achieved on DAP which is debug adapter protocol supported by these IDEs. So first of all, debugging Dockerfile is hard. It's even more true if you write a large and complex Dockerfile and it's almost impossible to write a correct Dockerfile without cycles of debugging and fixing and for inspecting the internal state of Dockerfile, you might modify it and insert like a run LAS, run cat, but modifying Dockerfile sometimes can cause cache miss and it requires running the Dockerfile again and this costs extra time and so the lack of tooling in Dockerfile and also makes debugging harder interactive tools even shell cannot run, cannot be used on Dockerfile and debugging tools might be unavailable on lightweight images like busybox or scratch and these issues sometimes make debugging a 5-minute problem take hours and one of the root cause of these difficulties of debugging of Dockerfile is a lack of interactive operation Dockerfile is typically executed as no interactive and atomic job and tools like Dockerbuild process the Dockerfile and the schedules or instructions and then they are executed all at once without providing interactive operations for inspecting the Dockerfile So how can we interactively inspect Dockerfile for making debugging Dockerfile a lot easier? so here build-a-g comes in and build-a-g is an interactive debugger for Dockerfile and It supports two types of UI for debugging Dockerfile and the CLI provides an interactive prompt for debugging Dockerfile and GUI provides a rich UI for inspecting Dockerfile line by line word visually and this is integrated to IDE's like VSCode, Emax and Avim and more so you can use your favorite IDE for debugging Dockerfile and It supports rootless execution. Build-a-g is implemented based on build-kit and levelages it features like faster build by aggressive cache usage So first let me introduce the CLI of build-a-g and it provides an interactive debugger prompt and this provides commands useful debugging and they include ones for controlling execution and inspecting Dockerfile instructions at Cetura And this figure shows an example of usage of the prompt build-a-g debug command starts a build debugger prompt and You can debug this Dockerfile using commands like break and continue and From the next slide, I will introduce some debugging commands provided by build-a-g Breakpoint allows us to stop or break the build at the specific line of Dockerfile. On the stop to instruction We can request further commands for deeply inspect the build's internal state This command example shows that we set the breakpoint at line 5 of Dockerfile and then resumes the build and The build will run until it reaches to line 5 And as shown in the example build-a-g shows the Dockerfile contents with marking the line where we stop In this case line 5 is marked And you can step by line by line using a next command in this example When your build breaks at line 2, you can use the next command Then the build resumes and runs until line 3 And sometimes you want to launch a shell on an instruction to inspect the root file system in the Execution environment. Here you can use exec command This allows you to launch a shell or other processes My including shell on an arbitrary instruction and in this example, we launch a shell on an instruction then it installs figlet on the root file system then run it and Of course instead of running figlet and you can do an arbitrary operation like running ls command cat command or debug asset etc so Yeah, build-a-g allows to use an arbitrary debugger image when launching processes on an instruction In build lightweight images like a busy box or something are sometimes used and This is really great for minimizing the result image size, but sometimes lacks tools for debugging the build so instead of The base image of the stage you can use your own favorite image for inspecting the instruction And that is allows you launching like a shells on an arbitrary instruction even on scratch-paced stage And in this example, we specify Ubuntu as a debugger image and the build uses busybox as a base image But we can use ubuntu instead when we debug it and the original original busybox root file system is mounted at the slash debug root directly, which is configurable and so you can still inspect the content through that directory and Build-a-g cache is a result of each instruction this helps speeding up the second time debugging as the Docker file development sometimes require multiple cycles of debugging and fixing and Yeah, reusing the cache can help speeding up the iteration as discussed later build-a-g Implemented based on build-a-g it so this cache functionality level edges are build-a-g it internally, of course And as shown in this example build-a-g print the log message When the cache hits? Okay, so how does build-a-g work? So build-a-g is based on build-a-g it as I mentioned and a build-a-g it is an image builder developed and the mobile project If you use like a docker build a newer than 1809 and enable you enable Like a docker building is equal to one environment valuable You always use build-a-g it and it supports features for efficient builds including a concurrent multi-stage build and a remote cache secret months multi arc root rest, etc So build-a-g embeds build-a-g it so you don't need to run build-a-g it on the node Build-a-g also Added some patches to build-a-g it for allowing a hooking hooking instructions Hey, so So how does build-a-g to one instructions? This will be yeah, this will need to be know To understand the how build-a-g works So build-a-g it interprets docker file as LLB and LLB is the intermediate representation of docker file So as described in read-a-me of build-a-g it LLB is to docker file what LLVM IR is to see When building a docker file build-a-g it converts it the docker file into LLB and the LLB is a LLB LLB forms the dependency graph of build operations Called OP and such operations or OP includes like a file OP for file operations like a copy instruction and the easy copy for executing commands like a run instruction and In this slide I describe an example of conversion from docker file to LLB and Docker file contains three stages here first stage starts from BZBox base image and create a file named foo and the second stage Starts from Alpine base image and creates a file named bar The third or final stage starts from scratch and copies foo and bar files from the above two stages Into the final stage and that this docker file is Converted to LLB as shown in the right graph and I just shows dependency or dependencies of each LLB operations like a exact copy and the file OP and Each operation corresponds to the instruction in the source docker file And when build kit builds it and it schedules and executes these operations and as concurrently as possible So, yeah, so here how is a debug functionality implemented in Builder G? so Builder G hooks each OP execution in build kit Builder G patches build kit to enable this and each LLB operation Contains the location information of the corresponding instruction in the source docker file So Builder G can use this information for mapping each operation into the source docker file So when I use a request a debug operation on the line in docker file Builder G converts it to the LLB level operation using the location mapping information Now when you launch a shell on a docker file instruction Builder G launches a container which reproduces reproduces The execution environment and root file system and the build kit supports a feature to run a container And this is provided through an API called new container Yeah, this is very great build kit feature recently added and the Builder G level edge is that API for creating a Container based on the hooked LLB operation Builder G passes the result root file system of the hooked LLB operation into the new container API and at the same time Builder G gets that OP execution environment information from LLB and passes it to the new container API as well and Builder G supports root of rootless execution and so non-root user can run Builder G In terms of a build kit it supports rootless using a rootless kit So we can run build kit in a new user name space created and configured by rootless kit and Inside that name space rootless build kit can run containers for running the build and Builder G takes the same approach with with a transparent UX When build G run by a non-root user it automatically runs rootless kit and run to build G It's safe in a new user name space created by rootless kit Okay, so yeah, I think I think Yeah, it will be more Easier to understand if you I show some demo about this feature. So, okay, let me show a demo about Debugging of Dockerfile. Okay, so I believe we can see see it and Here I want to containerize these are very very simple go run application and named say hello and Dispring just a hello war to standard out and I saved these file as a say hello.go and and I wrote a Dockerfile And this Dockerfile is also simple to build the application that creates a runable containerized application The first stage The build the first stage builds this application using go build command and the second stage Containerize this application on scratch image But but as discussed later at this Dockerfile contains something wrong. So I will show Builder G features for debugging this Dockerfile. Okay, so Here I use very very simple Dockerfile But maybe you sometimes have more more one more complex Dockerfile So okay, so first of all, let's build it. Okay, so yeah, we'll do it self succeeded. Okay, so Here I want to run it Something hello world should be printed here, but it fails. Why and Yeah, the error message says that there is no say hello binary no such file or directory What's happening? So let's debug it using build G So you can yeah launch the debug session using build G debug command. This is very yeah easy to use and K the build starts and Build the proceed until yeah, okay the output says that the build is now stops at line two So you can show the source Dockerfile using list command And according to the output of list the go build go build happens at line four as highlighted here So, okay Let's deeply inspect this step first and check if his ago binary is correctly created and Here we can use break command and To set the breakpoint and they use you can use the continue command to continue the build Okay here Yeah, as you can see here, we use the break and continue so We are at line four So let's check if this stage has a hero binary and you can larger shell on the rotifier system on this stage using exact command as shown in this This command and you can and yeah here according to the list to command the The final stage of build copies a Say hello binary from thrash out directory over the first stage. So here we need to create the binary at the Thrash out directory. So first let's inspect this thrash out directory Okay, but search out directory. There is nothing so it looks something wrong happening So where is the say her binary? So here we have the view full view of the root file system So we can directly find that file on the root file system Using fine. So here we ignore thrash proc directory, which we are not interested in here So, okay, we find the say hello binary at thrash go. Hmm. This is a default working directory of golden image The but the binary was graded here because we didn't specify the output destination output destination to the go build command actually so let's see Yeah, there is no output destinations is specified in this docker file So so what we need to fix is just specifying the binary destination in this docker file Like this, okay, so Okay, we fixed the docker file so go back to the terminal and Reload this docker file again and inspect the final stage Okay, here reload the command to reload the change the change the docker file so you can ensure that change is effective Okay, so let's proceed the build until line for again and ensure that the say hello binary is successfully created under thrash out directory and you can directly exact That binary specified by specifying the path. Okay, it works So we've all done finish the build to G session. Okay, then the build to this docker file again Okay, and Okay, we'll succeed it. So let's run it again Okay, so yeah here. I'm finally and we can successfully created that if successfully debug this docker file and yeah It now works successfully Okay, we done the Okay, we done the demo so second Let me introduce the GUI based debugging of docker file on build G. Okay, maybe many of many of you interested in this topic and So build G provides not only CLI but also intuitive GUI for debugging docker file and the GUI is integrated to integrated to IDs like VS code in Max and NLVM and And Will do the newer than version 0.3 support of GUI base debugging of docker files this is integrated to ID is like VS code in Max and NLVM and As similar as CLI and build G GU will do this GUI mod also provides basic debugging feature of docker file as listed here and so So I will introduce some How these features look like so we use a VS code here, but the GUI looks similar also on other IDs as well When you start the debugger IDE looks like this picture. There are some panels on each on the Each one describes the execution information of the docker file that on the central panel and you can see the debugging docker file and the Debug console shows the progress information of the docker file And in docker file the line is highlighted with yellow is the current break position of the docker file in execution And then you can set a break point using F9 key on the target line And this picture shows that there is a break point at line 6 marked with a red field circle and You can resume the build using continue button on the toolbar were using F5 And there are some Features on this GUI because you can also step line by line step set over step over button on the toolbar or F10 key allows this when you push Push that bottom build regimes until the next line and as and when the build stops again The position of the yellow bar is also updated to the current break position And you can launch a shell on an arbitrary instruction on the debugger console console repel and you can use exec command and This launch is a shell on the instruction of the current position And then you can access to the shell via terminal panel And then you can also bring your own debugger image by configuring launch.json config file And that this helps you further inspection of breaking the instruction So how the integration is implemented on the build g? First of all, this integration is achieved leveraging debug adapter protocol or DAP supported by IDEs and that that is a standard protocol between IDE and the debugger and that this is proposed by Microsoft and this API allows Navitory debugger to integrate to IDEs GUI and IDEs including VS code emacs and a Veeam and provide rich GUI for debugging programs and use DAP to control the corresponding debugger. And so from debugger's perspective the debugger can integrate to the variety of IDEs by just implementing DAP protocol based on its debugger feature And I will known debugger's Debalages DAP to integrate with IDEs like Python Node.js and C-Shap And the build g implements a DAP for integrating with IDEs so users can Debug Dockerfile using the rich GUI provided by IDEs So currently build g is known to work on VS code emacs and a Veeam But it should work on other IDEs that support DAP and as shown in the figure build g talks with IDEs via DAP and build g translate DAP operations into build g debug operations So note that you need to you still need to install build g on your host and to use this feature So on VS code VS code build g extension allows debugging Dockerfile I previously referred to the VS code build g repo for more details about the configuration and On emacs DAP mode can be used to debug Dockerfile and to enable Dockerfile debugging You need to configure DAP mode And as a detailed configuration of DAP mode is also available on the build g repo as well and on NEOVIM NVM DAP enables Lookerfile debugger and like other IDEs to enable lookerfile debugging you need to configure NVM DAP And as a configuration is also available on the build g repo as well Okay, so before entering to the related works Let's do the demo again up on the VS code Okay here So we use the exactly same example on VS code here. I want to Okay, yeah, I want to Containerize this simple go application name say hello that the Prince just hello world as discussed previous demo and Yeah, I saved this file and say hello then this is a Dockerfile I wrote and this is also the same as I used in the previous demo and this Dockerfile Beals the application on the first stage and the second stage containerize these say hello binary into onto the scratch base image But again this contains something wrong. So let's debug it using build g on VS code. So first of all Let's build this Dockerfile and the build starts. Okay build succeeded. So then run it Okay, um build succeeded, but run failed again and the error message says that there is no say hello binary again This is exactly the same. Okay, and so what's happening and this debug it using build g and On Dockerfile type F5 to launch the debugger Okay debugger started here and Okay, yellow bus shows that we are at line 2 and The go the go build happens at line 4 as I highlighted here So let's deeply inspect this step and check if the go binary is correctly created then you can set a break point just by clicking the panel and The continue button stops the build. Okay So we are at line 4 And let's check if the stage has say hello binary and You can launch a shell on the root file system of this stage using exec command in ripple. Okay as join the debug console and Then the shell Will launch at the okay the shell. Okay launched on the terminal Then again, you can start the say hello binary using a find On the shell, but before doing that, let's take the there is nothing in the slash out directly Yeah, there's nothing so we need say hello binary here, but there is nothing so Let's start say hello binary on this root file system using final command again Okay, so Yet and we found say hello binary at the slush slash go So this is a default working directory of the golden image. So, uh, yeah The root cause of this bug in this docker file is that we didn't specify the output destination of go build So what do we need to fix here is? Yeah, like that. We need we need dash or option and to this go build. Okay, so Um Back to the debug console then we start this Debug again Okay, we start bottom can easily restart this build and And yeah inspect line for again Okay, we are at line two. So continue it. Okay. We are at line four So then ensure that the say hello binary is successfully created in the root in the out directly So we can again directly execute that binary on repo Okay, her reward is printed. So it works. So, okay. We've all done. So finish the build g session Then okay, ensure that everything is fixed So disconnect the debugger and on terminal. Yeah, as usual we can build this docker file and run it again docker build and Yeah, let's build this oops Yeah, I build this context again Okay, we'll just succeeded and then run it again Okay, it works and everything has been fixed. So Okay, so yeah, so there are some works in community and to support interactive debugging of docker file as well so I will introduce some related works here and Okay Now the CTL is docker compatible CLI of continuity And this is developed as a sub project of continuity and another CTL supports build g build g based interactive debugger since version 0.20 and you can use it via another CTL build a debug some command and As showing the code example this launch is a debugger prompt for the specified docker file context Then it enables the interactive debugging same as build g and the docker build x is a docker CLI plug-in for build kit and Since version 0.9 provides a monitor mode As showing the slide a monitor model does the launching processes like shell on the build result so this allows you inspecting the build result quickly and We are still walking in progress towards support for the complete debugger with breakpoints So please try it and any kinds of feedbacks are very very welcome Okay, so build g is actually still in a narrow stage of development. So there are some works to be done for maturing it We will also continue to work on the integration of debugger feature with tools in community Including builders like build kit and desktop tools, runtimes, IDEs, etc So finally and this is a summary of this talk Build g is an interactive debugger of docker file and it provides interactive prompt for debugging docker file with a Breakpoint and it also supports running a container on the laboratory instruction and deeply inspecting it And the build g is also integrated to IDEs like vscode, neo-vm, and emacs So IDEs GUI based debugging feature can be used for debugging docker file This integration is achieved on DAP debug adapter protocol supported by these IDEs So yeah, I'm pleased to write it and any kinds of feedbacks are very very welcome. Thanks So I think we have a five minute or something. So if you have any questions Please ask here and we can use this mic And any questions? Yes, please So do you necessarily need to docker runtime for this or would it also work with for example portman? So You mean Do we need a container runtime on the host to use build g? Yeah, and does it need to be the normal docker daemon or can it also be some alternative like portman Yeah, actually you can run build g without docker You don't need docker on the host, but you need you need runc which is a very very low level Container runtime and build g depends on it So but if you have docker on your host you should also have runc Installed on your host so build g should work on your host, but yeah But you find if you find any issues. Yeah, feedbacks are very welcome. Okay. Thank you. So anything else We have a virtual question Okay Can I use build g with Padman in rootless mode? Context in some cases docker with a daemon is disabled Especially in corporate environments when security is tight So Okay, so the question is Yeah, can we use can we use build g on the root on the host with rootless portman? Okay. Yeah, and actually build g doesn't depends on the container Like a container engine like a portman or docker. So you don't need a docker or portman installed on your host But instead you need runc and if you care about the rootless Yeah, build g supports a rootless mode based on the rootless key So if you are not allowed root privilege on your host You can also run it in a rootless mode as shown in this slide. Do you have any other questions? Okay, I think there's nothing so yeah, please try it and the feedbacks are always very welcome. Thank you