 Hey, so I'm Tim O'Shea. I'm from Virginia Tech. I'm just a researcher there, and I wanted to talk a bit about the possibilities for tensor signal processing and then how that bleeds into machine learning as well. So just to set the background, I'm sure a lot of you've been watching this, but what's been going on in machine learning and the resurgence of neural networks has been pretty awesome to watch. This has primarily been in the computer vision area. So there's a lot of people in the self-driving car area, as well as facial and object recognition areas, and natural language processing, where we've made lots and lots of advances in the recent time in driving down error rates on translation and object recognition and all these things. So neural networks from a computational standpoint are really interesting because they have this fairly well-defined compute structure that constitutes a network. So if you look at the calculations for a neural network, you have this kind of, it's really basically a big multiply accumulate for each layer. So you have these inputs that are multiplied with a series of weights that are summed and then run through some non-linearity like a sigmoid function, and that's basically one layer in a neural network. Now, if you look at these so-called convolutional layers, you no longer have a separate weight on each input, but you have these weights that are tied in a way such that when you slide a patch across an input, you replicate the same weights at different shifts. So that's really the only difference. Then if you look at some of the bigger models, so this is from like a year and a half or ago, one of Google's vision networks for doing object recognition. You think of that transfer function for a layer, that's one box in this network. So we now have this network which is 40 or 50 layers. Some of the ones this year were over 100 layers deep in the complexity that they're doing to do this object recognition. So there's been a really, really strong software ecosystem to support machine learning for computer vision, and it's really starting to centralize around this tensor programming model. This is really interesting because you're building these big tensor graph expressions, and then as a big abstract data flow representation, and then you map it down to hardware to run. So two of these architectures are Fiano and TensorFlow. Fiano was around first and pioneered this, and then Google secretly rewrote it internally, and then eventually released it to the world, and now they're both very popular. But the whole idea here is, these are basically libraries that allow you to write big NumPy expressions essentially, where you have many operations that are forming, a data flow graph, and then take it and the idea is you build this graph expression, and then you go through the back end, and you go out to any one of a number of targets. So right now, we go to basically GCC and CUDA as the back end, but there's a lot of plans underway at Google to do this thing called XLA, which is like their next generation tensor back end, and they're doing a lot of work to sync this up with LLVM so they can actually, you just use LLVM abstract representation to pass it in. Then there's a lot of rumors about potential support on the QDSPs that Qualcomm builds for base bands, as well as some of the Xilinx chips. There's been a lot of talking about Xilinx potentially supporting this as a co-generator. So it's really cool. We may have finally a language that we can target to a lot of different platforms from this way to express things. So this is used heavily for neural networks, but it's not really neural network specific. So if you look at this as basically a simple expression, we start by defining these variables like A and B, and we can define operations on them like adding or multiplying, and then eventually we go into our session and we evaluate these expressions. So this is basically building up your big, in this case, a very small kind of data flow graph, and then this is going to be compiling and running your graph. So you can build much bigger things in this. So I talked a little bit before about GR TensorFlow, and the whole idea is really at the core of this, this is just kind of like a kernel work call. So we can wedge it right into a radio block and use it to offload these tensor operations from within a work function. So this is something added a couple of months ago, and the whole idea here is, can we take a sync interpolator of a signal and offload that onto GPU or onto whatever, using this kind of tensor language. So this is basically the definition of a sync interpolation. So you're going through here and up at the top, you're defining the graph through this series of castings, tiling of the input, doing these gather operations, which are basically index lookups into vectors, and then doing a reduce sum, which is basically just a big multiply accumulate. So once we've kind of issued these commands, we've built this kind of dependence graph for the operation. Now whenever we run it, we could change the back end to either the CPU or the GPU or some of the targets that are probably coming soon. So that's just an example. I think there's not many people out there using this for pure signal processing like this yet, but I think there's a huge potential for it. The tensor languages are pretty expressive for almost any of the commas problems we have, and the kernel compilers and optimizers are really quite good. So this is kind of a pretty exciting way to offload onto GPUs and things. All right. So let's switch over to the machine learning side. So that was kind of basic signal processing, but I guess my research now is really centered around, how do we take these machine learning ideas and apply them to radio technology and radio communications? So these are just a couple of the questions you might ask, can we apply it in these ways? So really, I've been looking at, can you learn entirely new communication schemes, or can we just learn algorithms to put into our current waveforms as different subsystems? Then in a sensing scenario, you may want to know what's around you in your interference environment like Tom was talking about this morning where you may have EMI emissions from your power supply or whatever. You may want your system to be aware of all these emitters around you when it makes decisions about setting up the communications link, and then just how to optimize existing systems. So these are some of the high-level goals. So this is some of the earlier work from this year, and I think we saw a little bit in the GR Inspector talk, which was awesome. So the whole idea here is, for a long, long time, when we looked at doing modulation recognition, which is a classical communication sensing task, the approach was generally to do this expert feature approach where you extract a bunch of high-order moments or high-order cumulants, along with maybe statistics on the signal and the phase and the envelope. Here we're trying to go to a whole different route of just let's instead, let's go to feature learning on the raw input signal and just put the INQ in and let it learn a whole new set of its own features. It's funny, I mean, if you look at what's happened in the image, the computer vision domain, this is exactly what has happened. So until five or 10 years ago, there was a whole discipline of feature engineering in vision, which has been largely replaced now with these end-to-end learned features. So I think a lot of this is going to happen in the comms and the radio space as well. So this is really showing that for modulation recognition, we can do pretty much the same thing. So we start, there's a dataset that's up on the website, and this is basically just using Gini radio to generate a whole bunch of synthetic signals through a whole bunch of synthetic channels that use the dynamic fading model, and then this is what we use for that experiment. So this is just essentially, this is basically just a small convolutional neural network that we're using to do this classification. If you look at performance, there's a bunch of the solid ones here, are different variations of a feature learning approach, which take in the raw samples, and the dotted ones are basically different classifiers using these expert features like higher-order moments to classify. So here you can see in the best case for both, we get three or four dB sensitivity improvement with these methods, which is pretty exciting. Three or four dB is a lot in some worlds. So, and then, yeah. So, all right. So one of the other cool things you can do is, if you take, for instance, if we look at this network, our output is 11 values, right? So these correspond to one per modulation class. You typically use like a one-hot encoding for a classifier. If you go back one layer, you have this feature map of activations between each layer. And generally, this is reducing in size as you go through the network to become a more and more concise representation of the information. So if we go back one layer and we just take these activations here, we actually now have basically a compressed representation of the signals from the input in this feature map space. And so what's really cool is we can start now using that compressed representation to just kind of visualize where are these signals in this kind of abstract modulation space, right? And so now we can start to cluster them and we have this basis now that is relatively good at separating a bunch of different signal types. And so this is kind of interesting because if a new modulation pops up, if these features generalize, we hope we'll get a new cluster over here or something that's discernible so that we can start to recognize things online without having to train it for every explicit case ahead of time. All right. So then this goes to some of the work that we did this week at HackFest. And so how many times, I guess raise your hand if you've ever written a flow graph that's like a usurp source through a head to a vector sync, right? I feel like, okay, we've got a handful. So sometimes it feels silly to use Gini radio when all you want to do is just grab a chunk of samples off a radio. And so that's the whole point of this NumPy UHD wrapper. And so this is just like a super, the thinnest interface you could have to say, tune my radio somewhere and grab some samples from it. And so super simple. And so what we're doing here is, this makes it really quick to just take little looks all over the spectrum and generate spectrograms for them. So if we do that, we can start to get all these samples of different bands that are all around us. And so this is just a sampling of a bunch of spectrograms across common bands. So if we use this in a ConvNet, we can also build a classifier for the different bands. This is not a very good one at the moment, but that's just because the slides were a bit late. But so it's not necessarily that exciting to classify which band you're on, because we have allocation schemes. This isn't really that useful. But the thing that's really interesting about it is, we learn these sets of features for each band that make that band what it is. So the LTE downlink band is filled with LTE signals that have all the properties and features of an LTE signal in the downlink. And so our network, as it's learning this, it starts to learn all these features within that comprise that band. And so what we can do is we can now take any random spectrogram that comes in like these two, and we can look at where in that input spectrogram, where there are activations that led to thinking it was that class. So we can say, what in this image makes us think it's LTE uplink? And so if you do that, you can look at and get this localization mask that's used in imagery for object segmentation. We can do the same thing in spectrum. So we can now say, well, once we've trained these features, we can take an arbitrary spectrograms and really identify what's going on where and start labeling things. So then the last little application thing here, this is the thing, if you look at machine learning and comms, this is what I'm I think most excited about right now. And the whole idea here is, if you think about a comm system at its most basic form, it's really like you have some bits somewhere, and at some of the location, you just want a good estimate for those bits, right? That's kind of all it is. You're just taking bits in and trying to reconstruct them somewhere else. And so that's pretty much what an autoencoder does in machine learning, is it takes some input, it goes through a couple of different representations of it, and it tries to reconstruct the input. And so if we cast the basic communications problem as an autoencoder, we have this case where we go through some input, we form some new encoding of those input bits, we go through some channel impairment, which we don't control, this is kind of defined by a wireless environment. And then we go through this decoder representation that somehow transforms the samples we receive back into bits. And really, that's the whole process we want to optimize. So we can treat this as just an optimization of a reconstruction process, where we have this layer in the middle that introduces this randomness that mirrors what a wireless channel does. And so what's really cool here is, we can let this thing learn now some completely arbitrary physical layer. We don't need PSK or any of these things anymore. And we can actually start getting reasonably good capacities out of these encodings. So we can now just learn a radio for a specific radio environment without any expert knowledge of the past 70 years of signal processing, which is pretty crazy. So if we do that, we can start plotting basically BER curves. And these are some old ones that aren't as clean. But you can see that in some cases, we can outperform the performance you would get on a modulation without any error correction, which is really cool because it means the encoders and decoders that we're learning can actually learn a little bit of error correction as well. So you're basically learning something that does better than just a normal modulation scheme. So there's a, I'll mention it later. So this is just an example of a couple of different runs and what the resulting learned bases look like in this case. Yeah, and there's many, many things that affect this. So there's right now still a million tuning dimensions that you can look at while training these things and optimizing all over all of those is still a big challenge right now. And right now the other big challenges are this, this works for very small code words. But if you try to scale this to like 10, 24 bits or something that LTE might use in an LDPC code, it's much harder to scale it. So that's one of the things we're looking at now is, how do you scale this up to like more real situations and over the air? All right. And then lastly, so when we learn that in Coder and Decoder, the Decoder has to learn how to translate from some channel representation to the bits, right? Now, if you do that just naively, we get some process that takes a long time to converge because it's having to learn how to estimate the channel, somehow apply those corrections, and then somehow go back to a bit information encoding. So there's this notion from vision of attention models. And the whole idea of these attention models is that instead of learning some arbitrary transform like how to equalize, you can simply break the network into learned parameter estimation, kind of an expert transform like equalization, and then our learned Decoder. And so in doing this, you basically tell the system, well, okay, this is how you would apply equalizer taps, or this is how you'd apply frequency correction if you knew how far off it was. And turns out if you do that, this converges like much, much faster, orders of magnitude faster here, and we get significantly better performance as well in the auto encoder case. So it's like we're helping the system along to learn the basic physics of the problem to reduce the complexity that the machine learning has to fit. And so this is exciting that this works. And so on top of that, there's this whole problem of we have a whole discipline of estimation and detection. And it's interesting to just look at, can we treat these estimation problems as just machine learning regression problems? And so we started looking at that. And if you look at, this is comparing the map estimator for center frequency estimation with basically just a neural net train to do regression to generate estimates for the same thing. And if we do that, and this is under a Rayleigh fading channel. So if we do that, we get a slightly tighter distribution from the regression process. And so this speaks to like, why do you wanna do this? We have a lot of great analytic models for these, but one of the problems with them is there, many of them are derived under much more simplistic channel assumptions. And so if we can do this with real data and regression, then we can start to model all kinds of non-linear effects and all kinds of much more complex channel models than we can in our analytic forms. And so it's really kind of exciting about where our estimation processes can go in the future with this approach. Right, so ML is affecting a lot of fields in computer very quickly right now, which is pretty exciting. And the area of applying this to the RF radio domain is super young. There's really not much work out there yet at all. And so I think if you're a student right now looking for kind of topics to work on, this is like, you know, super ripe. And I wouldn't point you anywhere else. But obviously I'm a little biased. So there's also, I have a journal article that'll be appearing on Archive Monday. And it goes into this autoencoder and the mod rec process much more rigorously. So if you're interested in a more thorough explanation, definitely go and check that out Monday. And there'll be a link on radio ML.org. So I'm trying to start kind of turning radio ML.org into somewhat of a community for people interested in this. So, you know, if you would like to contribute tools or data sets or articles or examples or anything to radio ML and start contributing, like please let me know because I would love to have your work there. And yeah, I think that's about it. Fanny. Questions? Questions. Yep. So you mentioned about using TensorFlow for signal processing. So how much, so you gave an example of a signal interpreter, how much work do you know that's been out there for what happened to you? Yeah, so there's not much out there at all for like radio signal processing yet. I think almost all the work has been done in computer vision. And so there's a lot of tools to do kind of computer vision operations, but very little of that has been applied to radio yet. And I think that's kind of one of the niceties is that the computer vision guys are, you know, paving a lot of the roads for, you know, figuring out the compilers and all the back end and support issues so we can use a lot of the same infrastructure in radio without having to reinvent it. I think Google said they moved like their machine learning for all of their services over to TensorFlow. And they have like a 30 or 40 person team just supporting the TensorFlow product line now. So there's a lot of, you know, stuff we can just take for granted at this point. So maybe it was for you, like the divinity knowledge of real-time computation requirements, latencies, you know, what kind of, you showed comparisons up here at a performance, you don't worry about like actual runtime performance. Right, yeah. So I haven't, there's not been a whole lot done on latency yet, but on computational performance for the moderate example, I did a op counting comparison to the kind of state of the art kind of classic algorithms and we, so the initial network was something like nine times more operations than the classic method, right? So, but it turns out then when we go in and we start doing certain things within the network, like we use one by one comb filtering on the output of several convalesers, which is a trick and one by one filtering sounds very silly and I think to most electrical engineers because you know, why would you do that? But the point of a one by one filter is actually that it's combining in the number of output channels. So you have actually N channels of 2D image coming in and M channels of 2D image going out. So you can use one by one filtering to reduce like that number of channels down. And so by playing tricks like that, we were able to drive the complexity down like maybe 50X or so. And so we're now a factor of about four lower in op counts than the kind of classic approach in that application. So I think like, you know, there's a lot of things that make it attractive, right? So all these multi-core architectures, you know, this is a very parallel thing, right? You can divide up the tensors in width and in depth onto different cores. And one of the other really cool things is there was a paper looking at how much representation precision do you need for these networks to work? And, because right now this is all done in single point single precision floating point, but they showed that you really only need four to eight bits of precision throughout the whole network for this to work equally well. And so there's another potential, you know, huge savings in complexity. So yeah, I think ultimately for comms and for sensing, this is potentially drastically lower complexity than our current approaches. Yep. Yeah, so I can't do it justice here. If you'll read the paper on Monday, so the question was about coding and whether we compared to Shannon and so forth. And yes, so the baselines here are, you know, I didn't wanna go and put everything that's gonna be released Monday yet, but if you go through the paper that's coming out Monday, we have some very, very good baselines where we're looking at comparisons to, you know, PSKs and DPSKs with Hamming code and, you know, we're basically right up against capacity. And so we're much more robust there in showing, you know, the capacity that you can achieve with this. And you can come very, very close to, you know, the channel limit. Switch over. Okay. All right, the next speaker. Ah.