 Hello everybody. My name is Felix Becker. I work for Source Graph on the TypeScript support, and maybe you've seen my GitHub handle on some GitHub issue, if you're active in the TypeScript community on GitHub, because I love TypeScript. TypeScript is one of the greatest programming languages I've ever used, and it seems especially recently a lot of people agree. This is the Stack Overflow Developer Survey from last year, where TypeScript made it under the most loved languages. This is the NPM Developer Survey from last year, where now 46 percent of all NPM users stayed there already using TypeScript. So, I got one and one makes TypeScript so great. I think there are many things that make TypeScript great, but the one thing that comes up over and over again is the tooling. How it makes the developer more productive, not as like a side effect of having static types, but as the first class goal and focus of the entire language. You can see this in the design decisions and in the design goals that the TypeScript team has set out, not striving for that 100 percent correct type system, but always for the biggest developer productivity gain. When I picked up TypeScript for the first time, I was absolutely blown away by how much faster I could write JavaScript code. Getting autocompletion on symbols, jumping the definition on fields, finding references to functions, getting compile errors and warnings and even automatic fixes for them, even more advanced refactorings like renaming symbols, or converting an entire promise chain to an async function. That's a very recent feature. What's great is that all of these features don't come from your IDE or editor, but they're open source and built by the TypeScript team themselves, and shipped within the TypeScript NPM package. So when you NPM install TypeScript, you don't just get the compiler, you actually get language service APIs that provide these features, and a standalone executable called TS server that editors can run to provide these features. That TS server is not just a one-shot CLI command, like you may know from ago has a bunch of these. But it's a long-running RPC server that the editor spawns, and it then sets up file watchers to continuously compile your project in memory. When the user wants to invoke something like go to definition, the editor sends a very simple JSON message to that server, telling it, hey, can you find me the definition of this symbol at this line at this offset in this file. TS server resolves that and then sends a JSON message back over standard out. That contains where the definition was found and at which line and at which offset. What's great is that TS server makes it possible that no matter what editor you're using whether that's VS Code or Sublime or Atom or Emacs or VIM, you get all these features and you have the exact same experience when you're working on your TypeScript code locally. So in this talk, I want to take it a step further, and I want to ask what if I could get all these features everywhere outside of my local Dev environment. For example, on the millions of repositories on GitHub or even while reviewing code in pull requests. What if as a library author, I could find references to a symbol that I have in my library and find other repositories that use this and maybe figure out, oh, can I safely deprecate this function? What if as a user of a library, I could jump to the definition of a symbol inside another library and get taken to that source repository to understand the implementation of it, or maybe I catch a bug and can immediately do a pull request. What if I could apply these cool refactorings like converting a promise chain, not just in my local project that I've checked out, but across hundreds of repositories with a single click online. The good news is we can do that. First step we need to do for that is, instead of running TS server locally on the user's machine, we run it in containers. When you open a file in your browser on GitHub or any other code hose or sourcegraph.com, we connect to that server, not through standard IO, but instead through a web socket. Of course, it's not just that simple because given that TS server is running in an isolated container, the files of the project don't exist on the same disk as they do when you're running locally. But instead they exist on the code hose and they're shown to the user that way. So we needed some way to tell TS server that instead of looking for the file on the file system, it should get that project somewhere out of band. TS server itself, the protocol unfortunately does not support this, but a very similar protocol called the language server protocol, may have heard of it, or short LSP does. LSP was created by the Visual Studio Code team to standardize a protocol that provides these code intelligence features, basically in the same way that TS server does, but language agnostic. I'm pretty sure LSP was heavily inspired by TS server because if you squint your eyes, they look exactly the same. If you don't though, there are a few improvements in LSP. LSP is language agnostic. It uses the JSON RPC standard, which makes it a little bit easier to build on existing libraries for clients and servers. It improved request cancellation, and most importantly, it uses your eyes instead of the file paths, which you can see here. In most cases, these are actually URLs, because they tell the server how to locate the resource like a code file. If it's a file colon URI, the server would know it's on the file system, I'm going to read that from disk. If it's something custom like unsafe file colon, it would get that content from the client as that file is not safe yet, and we can teach the server a new capability, which is in the case of an HTTP URL, it should fetch that resource through HTTP. When we send the server our first message to initialize it, we tell it that this project is not located on disk, but instead it's located in a GitHub repository, and that it should fetch this from this HTTP endpoint. The server can do a bunch of things with these HTTP URLs. This is an API we built, but it's really just very plain HTTP that anyone could support. All it needs is when you do a get on some file in the repository, it should return that as plain content. You can do a head request to figure out if a file does not exist and check the 404 status code. You can do a get on a directory too, and just get a very simple plain text listing of the contents. You can even send an accept header to instead of getting a plain list, get the contents of a directory or the root in an archive format, like tar or zip. So with access to the files of the project, we can service request from the TS server running in a container to the user accessing a GitHub repository in a browser, and the user can do things like go to definition right in their browser. What's cool about this URL is that it doesn't matter really which endpoint hosts this, even though this is hosted on typescript.sourcegraph.com, you could give it any root URL of any service that supports these basic HTTP features, and it would be able to analyze that project. And since we can construct these URLs even for arbitrary revisions, we can even do this on pull requests on GitHub, where the red section of the diff always talks to a server instance for the merge-based revision, and the green and white sections of the diff always talk to a server instance of the head commit of the diff. What this does not solve yet, though, is code intelligence across repositories. We gave TS server access to the files, but it doesn't have any access or knowledge of the dependencies of the project. For example, if you're calling a function and then on the return value, you're calling another method and you want to jump to definition on that, we would need to know what is the return value of that first function so we can resolve that. And typescript knows these through type declaration files for the .NET crowd. They're basically like symbol files, kind of, and they contain typescript function and method stub definitions that only contain the signatures but not any of the implementation, which makes them a lot more compact and faster to parse. And these type declaration files are shipped inside normal NPM packages. So what we do is whenever we detect that a definition could not be found or a tooltip seems insufficient, we just install dependencies from NPM. And TS server is smart enough to detect that new files were added to node modules and consider these now available types. And we can even be smart, and since we're only doing static analysis and don't need runtime dependencies, we can only install packages that actually contain typescript declaration files, makes things faster. So now with dependencies available to TS server, you can trigger a go to definition on a symbol that comes from a dependency. And TS server will return the definition of that dependency inside node modules. The problem is this only gives us the location inside node modules in the type declaration file. And the user who's accessing the repo in the browser can actually view that because node modules is generally not checked into the repository. So to give the browser a URL that is viewable by the user, we need to map this file path to where it's actually defined in the source repository. So we need to figure out where's the source of this package. And luckily, the package.json contains everything we need for that. Almost all package.json have a repository field, that point to the repository. They contain a git head field, which is undocumented, but it exists, and npm publish adds it when you publish a package. And it's the head commit of the repository at the point where publish happened. And then there's the directory field in case the package is in the monorepo and it's not in the root. Which is an RFC to npm that was ratified just a couple weeks ago. So this is a very recent feature, but it's already being adapted by very popular monorepo packages. So we have the repository, the revision, the directory of the package. But the file still points to the type declaration file, which does not exist in the source repository. And this in the past is where we had to bail and do some hacky heuristic and just try to find that symbol somewhere with a fuzzy symbol match. But TypeScript 3.3 had a really cool new feature called declaration maps. And TypeScript always had support for source maps for the compiled JavaScript, which are basically just JSON files that contain mappings from ranges in that JavaScript file to the ranges in source files, which allows you to step through the TypeScript code with a debugger instead of having to step through the compiled JavaScript code. And declaration files are basically the same thing, but they're instead of mapping the JavaScript file, they map these signature stubs in the declaration files to the positions in the actual source. And we can read that.d.ts.map file to map the declaration file to an actual source file and a new position. And with that, we have everything to construct a full URL, the repository, revision, directory, file path, and the new position, which means you can even do this go to definition and jump to a completely different repository right in your browser. Now even more interesting for a library maintainer, in my opinion, is to do the inverse, which is answering the question who in the universe is using this symbol? Let's say I have a library and it contains a function. I'm interested in how that function is being used or whether it's used at all. So that's what we call cross repository find references. And let's say we want to find references for a symbol in a file. The steps are basically the exact inverse of what we did for global definition. We find the closest package.json to figure out what package does this file belong to. And then we need to figure out what other repositories or packages depend on this package. And luckily, NPM has an API for that and it's documented on Stack Overflow. So it's a raw couch DB query that you send to some legacy endpoint and you get back a very weird JSON structure that eventually reveals the information we need. So we can use this to get all the dependents of the packages and then for each package use the repository fields again to figure out which repositories these belong to that we should search. So then for each of these dependents we don't actually know yet whether these actually referenced a very symbol we were looking for references for. So for each dependent we create a new server connection and we ask the server for references to the URI of the definition. And the server recognizes that this URI it should get references for is not in the project that was initialized. It's an out of workspace URI. And it knows that it needs to find this not inside the source code but somewhere in Node modules. So it basically runs the inverse steps of go to definition. It finds the closest package JSON by walking up the URL and trying out different locations to figure out the package name. Then it can find this package inside Node modules and it searches for .d.ts.map files that point to this source file that we're looking for. And once we have that we can use that declaration map to map the position in the source file to a position in the declaration file. So it's like a reverse map. Now we're going from source to declaration files because this declaration file actually exists in Node modules, this doesn't. And once we have this position we can send that position to TS server as for references. And since we do that for a lot of repositories we then accumulate all the results and stream them to a client. So I can say find references on a symbol and get streamed references in completely different repositories than this one. Unfortunately it's not always sunshine and rainbows when developing this. One of the biggest pains with this is that TS server is very focused on this editor use case with very high level APIs that provide just the features that you commonly want in editors. Like go to definition, find references, find symbols. But it also means that it's kind of like a black box that only provides this high level API. And it does a lot of things asynchronously like compiling in the background and handling file watch events. So there's often no way to know when TS server is actually done with compiling something or when it has started to consider new files I just added. And that can be problematic when you're doing global find references where you initialize and then straight send another request because there's no way to know whether TS server is already finished with compiling everything. And until then it will just return incomplete results with no indication. So that would be kind of a wish for the TypeScript team to allow more hooks into figuring out when TS server is done with something. And a similar problem with Yarnit NPM given that it's a CLI tool, there isn't really any way to efficiently hook into the installation process. And for example, filter out any packages that don't contain type declarations. There's some light at the end of the tunnel though. Yarn 2.0 plugins should allow something like this where you can hook into the install process. And Yarn 2.0 is basically gonna be completely different in Yarn 1.0 in that it's not targeting the CLI use case first but it's gonna be API first and CLI second. So that's very exciting for programmatic use cases. And then another problem and big future potential is in ELSIF. Going through each dependent for global references can take a lot of time and can be very slow. It's fine for when you just want a small set of examples of how your function is used. But if you want like an exhaustive list of all the references to assemble, for example, because you need to refactor them then it can really take a long time. And that's where ELSIF comes in which is a very recent proposal by Microsoft. It's called the language server index format. And it's dumped by language servers. And it's basically like a cache. So you can answer LSP requests from that index without having to spawn and initialize a language server and it having to analyze the project. So that would speed up this whole fine references thing drastically. And last words, everything I just shown is live in production and you can try it out on sourcegraph.com. If you wanna try it out on GitHub or another code host you can install the source graph browser extension which is free and I should have put a QR code on there. And also everything I've shown is open source. Sourcegraph itself is open source. The TypeScript support is open source. You can fork it, contribute, look at the code. Hack around with it. Thank you for listening and you can ask me anything. No question, yes? Oh sorry, can you speak up a bit? Yeah, so in the case of private repos you need to somehow provide authorization to the language server to access that repo. One way to do that is actually to put the access token right into the root URL which GitHub supports, sourcegraph supports that too. So the public typescript.sourcegraph.com instance of the TS server can actually also be used to point it to private instances of source graph or GitHub enterprise and download private repositories. And you can also set up a private source graph instance that holds all your private repositories too. Free to try out. Any other questions? Sorry? Can you have enterprise one the same way then? Practice tokens? Yeah, any other questions?