 Thank you very much. I assume you can all hear me. Okay, so this session is a beginner session We're gonna walk through the design of a an audio application using micro python and the pie board So we'll be constructing a continuous listen repeat audio application Something like you might find in a toy you talk to it and it talks back to you This is for beginners. So I'm not going to assume anything in terms of programming and stuff like that We're using obviously micro python, but the pie board and its audio skin So micro python Unfortunately, there is another track at the moment which is introducing you to micro python So most of the experts are probably there But it's a lean implementation of python 3 optimized for micro controllers, but with numerous modules for hardware control The board itself is really quite small. I've I've got a copy here But luckily I've brought a bigger version So it's basically just a micro controller with a few additional bits and pieces like LEDs and switches It's very small in terms of capabilities and in terms of RAM and processing power But it does come with really cool bits of hardware and in python All you have to do is import pyb and you've got access to pretty much all the hardware quite simply Christine spindler is running a poster session in the Sponsors Hall so she can tell you a lot more about the hardware and its capabilities We also need an audio skin the ability to record and play back the board itself doesn't come with a microphone or a loudspeaker but You can basically attach an audio skin. I've got one here on an aluminium casing that gives you basically a Microphone a loudspeaker and the ability to add your own microphone if you want to But be prepared The audio skin comes in bits. So in order to use it you're going to need good soldering skills and a very good soldering iron What we're going to be doing is recording and playing back speech So I suppose we need to understand a little bit about what speech is I have a Spectrogram a frequency domain recording of the utterance six here and basically you've got an axis which is showing you frequency so there are your high frequency parts of your six and This axis is effectively time in terms of frames, which are about 10 milliseconds long So that's 350 milliseconds worth of speech if you want to record high quality speech You need to sample a very high frequency because human speech goes up to 14 kilohertz But over the phone you basically are restricted to something like three to four kilohertz because they discovered that's all you need to intelligently understand what's being spoken So how do we go from analog to digital? Well, we're going to use an ADC of course Which is on the board and we have to consider a few items and that is effectively the sampling frequency sampling frequency And there are basic standards here Nyquist Shannon tells us that in order to capture an analog signal accurately We have to sample at twice the speed of the maximum frequency Here we've got and if we don't what we've got here is sampling at one point five times the speed and you can see the sample points Basically give you an indication. It's actually a lower frequency and that's called aliasing. So you have to sample quite quickly the second consideration of course is The resolution how many bits you're going to store each individual sample and the pie board gives you devices which can Record at 8 bits or 12 bits a bear in mind that a compact disc the original formats were 16 bits at 44 point 1 kilohertz 44.1 kilohertz is an odd number, but they're very good reason for it. So you need to understand The frequency is going to be capturing the capture resolution and how much time you need to record speech So how do we record on a pie board? Well, we basically create an ADC object And then we can use one of two read methods We can call read timed or just read So we just import after we've imported my pie board we create our ADC object It's on a particular pin on your board. So you just connect it to pin x22 and once you've done that you can Hand your reading you're the process of capturing samples over to Micropython and it will effectively read a Block of data for you at a frequency you dictate so you can set up a buffer and say capture 6,000 samples per second that's okay But you've got no control over the real-time aspect of this it captures and you have to wait till it's finished So building a device that you want to be continuous And be interactive that's not the best thing to use so you in my particular case I'm forced to read the ADC samples manually and It's my responsibility then to record them at a designated rate So we can capture how do we replay well, that's using a digital to analog converter and Again, there's one of those on the board. It's quite simple to set up There are two parts to the DAC You need to set the volume and there's a potentiometer on the I squared C bus and then once you've set the volume You just provide the DAC with your data buffer and it will play it back out for you. So setting the volume is quite simple as Connecting to address 46 on the I squared C bus and once you've done that You can create a DAC object You define the bit resolution that you want to play back at you have a choice of 8 or 12 here And then you can use its right timed method provide a buffer provide a timer a built-in object in in micro python and Tell it how to play the audio in this particular case normal means just play the buffer till it's finished and then stopped I think they have a circular option So you can fill a buffer with a repeating pattern and just get it to play continuously So I have mechanisms to record and play back, how do I know that it's working? Sufficiently quickly it has to be real time because I have to capture the audio at a designated rate So I'm going to rely on hardware components to do this But when I I'm also going to have software to do the actual signal processing. I say signal processing there There's not a lot you can do on the pie board at this stage But there are two ways to do it. You can either use an oscilloscope Attach it to a pin on the board and you can you oops You can use a pie board pin and give it a symbolic name And you can just set that pin high or low and your oscilloscope you can see You can measure the Duration of the code that that that you're surrounded the pin Control with alternatively you can use timer objects. So you can create a timer to Calculate to counter one one microsecond rate and just read it so The advantage of the oscilloscope is it gives me the Opportunity to put a little picture up. So here's a screen grab of a The oscilloscope on the board that I've been using and this low period here This trough is effectively the duration of the capture function This is the method that is running at 8 kilohertz in this particular case. It was recording done at 6 kilohertz So this is all the time it takes to read from the ADC and put that sample into memory and do some crude Calculations with it as you can see it actually takes quite a bit of time 104 microseconds But you can do clever things with it with the pin control and discover that using a timer costs you 20 microseconds And actually doing the read itself is a relatively expensive 50 microseconds So that gives you a clear indication of how fast you could actually sample this at you're not going to get 20 kilohertz Because it's going to take you 50 microseconds just to get the the value into your board So initial setup is basically I need a buffer to write my data into I need a function to collect the data And I need a play play mechanism So a crude diagram here. Here's my capture function. Its responsibility is just to read from the ADC It's connected to a timer and I simply Create a timer and attach a callback function. So here I could have a sample frequency of 8 kilohertz So this function will get called at 8,000 times a second It simply does a read puts it in a buffer when that's finished my play function goes and provides that data to the DAC So I have some recordings that I've made Earlier, I wish I'd said something a little more impressive than one two three in Italian, but Hopefully this will work. I'll just play back an issue an initial recording at 8 kilohertz So that's uno due a three, but you'll you'll notice it's very very noisy And that's that's at 12 bits 8 kilohertz And when you analyze the noise you get really quite what disturbed and disappointed that of your To 4,000 samples 300 or so are just noise. It's a very noisy signal on this board for a number of reasons But I'm losing quite a bit of information Just in the noise and it's unpleasant unpleasant to to listen to so there are quick ways In order to reduce reduce the noise and the simple one I put together is something that just searches for periods of silence in the recorded speech and just sets them to zero So to do that I Basically record into my speech buffer as I call it and on a second pass. I go through this speech buffer knowing Roughly where zero is again, this has to adapt This is not a constant depending on your hardware zero might not be in the middle And I go through the buffer looking for periods with zero and mark their positions And then that gives me the opportunity to do two things I can then adjust my estimate for zero because it might have changed But I can also go now go through the buffer and set all of the noise to the noisy silence to to real silence and our playback if we hear the original and Then we apply this simple algorithm Yep, so a lot cleaner There's still noise in the speech signal and you can't simply just take away those bits because The speech in the noise is still valuable So I had recordings at a killers at 12 bits. So I decided well, there is so much noise So how about recording at 8 bits rather than 12? If I play that So 8 bits gives me an added advantage and I can store more speech on the board. I'm very limited in the Java heap I've probably got about a hundred k bytes Minus whatever the operating system and the Python modules take away from me So I can only record very short periods of speech, but at 8 8 8 bits I can get about 8 to 10 seconds and and there you'll probably heard this a lot of problems with 6 and 7 say and set a and that's probably because I'm recording at 8 kilohertz. So we are we're right on the edge of Bandwidth here. So a lot of the high frequency stuff just gets lost or disturbed. I Also in so in order to compensate for that I decided well How about increasing the capture rate from 8 kilohertz to 10 kilohertz? So I'll hear the 8 again and then I move up to 10 kilohertz Hopefully the say and set a will be a little better So I've made some improvements why can't I sample at 44.1 kilohertz like I have with the compact disks Well as we've seen it it takes me too long to read from the ADC and also the power of the device Sorry the memory of the device just won't let me capture much data at 44 kilohertz even if I could sample at that rate. I Wanted some application refinements So it's okay. Just recording and playing back for the purpose of this was to impress my nephew Look, here's a little device. You could put in your cuddly toy talk to it and it talks back to you So it has to have some sort of Automatic Wrong button Automatic speech detection so you don't have to press a button for it to listen So and I wanted to be continuously listening I wanted to be able to record to the SD card this device gives you the ability to plug in a fast SD card I wanted to play with the LEDs to give you some indication that it's listening. It's recording It's playing back and I wanted to be able the very least to disable the device by pressing the user button So it stopped listening So how do we can how do we interact with the user button? Well, there's a there are two switches a reset button on the board which reboots it obviously and a user switch And these are brought out as little brass buttons on your aluminum on your aluminium casing So in order to use a button all I have to do is provide a handler function the work I want to do when the buttons pressed I need a switch object Which represents the switch and then I just attach the handler via a callback method. So there's my work in this particular case I'm just setting a control variable, which just prevents the recording from taking place. I Create a switch object from the pie board module and then I invoke the callback method providing my Function so every time I press that button that function runs. That's quite straightforward Driving the LEDs. Well, that's pretty cool. We like things that light up and flash So it's got four LEDs and four colors So you can create much like the switch you can create an LED object and you can switch them on off toggle them and so on so again pie board LED 2 for for the green LED and I can switch it on I can switch it off I can toggle it LED 4 supports an intensity method which allows you to provide 256 levels of brightness they all support the intensity call But only the blue LED allows you to do the cool throbbing type of effect Automatic speech detection Okay, so this requires a little bit of thought the pie board doesn't give you this this is all part of my application domain code so Basically, I want to capture some speech continuously and Only start recording when I think something's being spoken. So this requires obviously the ability to determine whether there's speech or not and That's relatively Simple in this implementation So I have I have two buffers. I have my original Speech buffer that I'm recording to but I have a very small buffer that's treated like a circular Memory so as the capture function is running it's in two modes when it's listening for speech and to it's then recording So while it's listening it's writing to this circular buffer are doing some very crude analysis on the signal level to determine whether it's silence or Noise and because I've done some analysis of the noise I can understand roughly at what level these samples start to look like speech so once it's discovered that their speech it then switches to Straight record into this main buffer and then and then stops Writing to an SD card is useful for storing data in recordings The device does have some flash on board, but that's very very small and extremely slow so you have to insert insert an SD card and Discovering an SD card is my approach is really quite crude some Micro Python guru might say oh no, you don't want to do that. So I basically just look for SD in the system path if there's an SD there's an SD device and then I just do my Basic Python open right close so finally putting it all together I Have sketched this Application diagram the green parts are basically the MicroPath and hardware services the blue parts are the software modules I've provided and the red is basically just data So when we when we initialize the board the capture playback main loop Essentially set controls the capture function with some control variables So it will tell it right start listening if I've not pressed the please stop listening hold button the capture function then essentially Listen for speech it's capturing all the time once it's detected speech it writes into this long buffer It conveniently lights the amber LED for me to tell me it's recording when once that is done it then Unlights that LED the and sets a control variable So at least my outer capture loop now understands a recording has been made So it can set the blue Playback but playback LED and does one of a number of one of a number of things now Remember that I have a little bit of speech the speech detection buffer So I need to attach that to my original buffer because in my crew DAC method I can only give it the address of a buffer and play it and also that is circular so we don't know where the start of it is So the copy function unrolls this untangles this buffer puts it in here into there I then Once the copies done I then run my crude clever attenuation method which adjusts that buffer it adjusts the value of zero for the next capture etc etc and then Calls the play function once that's done it optionally dumps to the SD card if it exists and then the whole thing starts all over again and That effectively is the entire application the code is Is all Python and I've published it on to onto github So all I've got to say now is thank you very much for listening. I hope it's been useful You've got my contact details there and the address of the source code And I assume that effect. I'll probably publish these slides up to that address later on If I just play Thank you Alan for that wonderful talk So he ought to automate it away my job. That's not good Okay, yeah, thank you. I hope there are some questions are there, okay Hi, so you have a very small budget in terms of time to process Does that mean there are certain things that you shouldn't do like probably import modules How about calling functions and in general is there any way to make sure that no matter what pass your code takes? It would still be done within the budget to allocate it. Yeah indeed, so The overriding rule is inside your capture function you have to do as the absolute minimum that's required And my capture function essentially just calls the ADC Checks the sample value and puts it in the buffer all the processing that you would ever want to that do that signal I can't do in the buffer because I've measured the time the You're in a callback function apart from other interrupts that are going on the board you You probably are fairly safe in assuming you've got control of the CPU here your Python on metal So apart from other interrupts that are occurring you probably don't have a lot to do you could Yes, I don't know what would happen if you over if your capture function lasted too long You could probably detect that by setting some sort of like lock variable The one thing you can't do in a callback is you can't create any Python objects And you can't do floating point processing in Python because that in itself creates Python objects So you really are at the bottom level of code just moving bits and bytes around memory But that's why I had to put in that the pin function and the timers to to make sure there is some jitter You can see that when the board is running, but as long as you're within quite a percentage point of your A loud period you're probably quite safe Thank you. That was fascinating and you mentioned your nephew. I think at the at the beginning and have you used this with Children or young people in an educational context and if so, how did it work out? Not yet. The problem that the pie board has is that when I pull a cable there's no built-in battery backup So it has to be powered by something So I'm waiting for them to produce a battery module so I could put this into You know the the stuff toy to allow them to do that sort of thing. I mean these things are Incredibly popular now. They're all over the place. You know, you squeeze you toy you press that press the hand and talk to it But no, I haven't I haven't got back to him. Yeah, I've just released version one. So Thank you Some questions over there Have you done a testing with clear signals so you can tell which part of the noise that we're hearing comes from the recording And which one comes from the playback because usually those those Speakers are very low quality So No, I haven't but the the pie board the pie module does come with schematics and diagrams so one of the things you can do is Obviously not use their built-in microphone and there's an ability to provide your own signal that the path to the Analog components here is very short But that is the next step is to provide a reference signal on here to see where the noise is coming from It's not unusual to find noise in devices like this. They're not exactly professional quality Analog components. They need proper grounding and earth planes to separate them from the digital noise But no, that's the next step is to provide some sort of reference signal in at this point Okay, so first here Thanks for the talk. It seems like a really good way of not only getting into embedded but also Learning Python because I'm quite new to Python. So I'm going to play around with this I've got two quick questions. Hopefully firstly coming from an embedded background. Can you Do the ADC in the background? So for example in your in your code You were waiting for the ADC to get back to you and then you're buffering that data Can you just pull that and then have an interrupt come back to you? Or for example, use a double buffered approach like you would do with old audio Sorry and the question was do you know if there's anywhere in Rimini we can buy these whilst we're here You've got one here No That's a good point. The the the ADC read method obviously MicroPython's open source so we could go in there and have a look what's going on But the read time is is not very useful because you don't know where it is and you it's real time So at the end of the day you have to do whatever you want Once every 8,000 times a second if you can't guarantee that then there's a maximum length of time that you can record So the only way the only what I've done is is the absolute minimum Which is read from the ADC and essentially put it into memory, but I am I am checking the signal level I could I could connect two boards together Maybe connect this to something more powerful like a Raspberry Pi and do a lot more signal processing because you can some really clever things when you turn your continuous time domain signal Into a frequency domain because then you can walk into speech recognition and actually understand what people are saying So but no you really have you're very limited in what you can do But yeah, not as cheap as the pi zero and I I don't know whether MicroPython runs on the pi zero I would guess that it does but the beauty of this board is that it's got so much Hardware ready and you just have a microphone and a loud speaker Okay Yeah, so actually my question Related to what you said just now I wonder what's the advantage of using MicroPython on on pi board versus using one of the you know Bigel bone Raspberry Pi one of these boards that are more powerful and allow you to use the full blown Python You know and all the hardware that you can really work with so the main the main reason I looked at this device is its physical size It comes with headers, but you can buy the board without any headers So if you're going to attach this to a smaller component like a toy its size is important and and also it's the power consumption I think this is only drawing a Fraction, you know at tens of milliamps whereas the Raspberry Pi just consumes too much tower. It's too bulky So it's a trade-off between power In CPU and battery consumption and and and things like that So I mean this is about half the size of a Pi zero it's got but it's it's loaded with kind of quite a bit of hardware to play with So the reason why it's not cheap. I think it's 35 euros or something So it's not terribly cheap compared to the Pi zero, but it's got some really cool Hardware features and it's it's Python on metal. So That's cool and it makes noises Okay, so maybe I have a last question. So when is Alexa running on this thing and I can well, okay Yes, indeed. Well, as I say the problem is getting from the Continuous domain signal to the frequency domain. So you need to get to Something like that once you once you're able to get the data into that you can do your own speech recognition And that's I've done that in my past life, but Alexa they have an API so you can give it data So yeah, just needs time. Okay, so let's thank him again thank you and you