 Just always saying the same thing. Just this is just checking my microphone See if it's working Oh, okay, so we have this file It's a open BCI While that we open liking Was it 10? one minute in Python And this one it looks like There's obviously There's obviously nothing On what's a child to three and the rest? Mean they might be something There yeah, when you have things like that It's not the it's called it's not child script. So I can't zoom in and out a range Over there quickly. No, there's nothing there is there just a flat line So we can get rid of it and just look at the was in zero and one There's essentially no avad data in this particular file. So it's a text file. It has a header a five The first five lines as header. So I printed it out here Their file can record the eight channels sampling rate of 250 the board being used open BCI a Board Wi-Fi something It doesn't really matter. Let's do range to close this windows. Have another one open Right, so that's that's what we have Okay, yeah, it's the major first thing that comes to Mind is major DC shift. So I Would say yes, we can filter it out quickly But yeah, there's something wrong In the recording so it should be Usable essentially it's meant to be then Sampling rate We can ask Can we correct the X access to show time instead of set number of samples It can do it. We have GPT for open as well, but I don't think we need it. That's pretty Straight forward Yes sampling rate. So yeah, I don't want it to be taken. So so the reason I'm doing it There's a general. We might actually have a more general tool Ideally in JavaScript in JavaScript that you can load Your data and we'll display it for you do some sort of analysis You know mainly looking at the signal quality amplitudes Things like that. So this that we just Open it quickly Python yes in theory Open BCI have their own Python library But I don't want to use it because I want to make the tool generic so it can take any data so we just want to use a NumPy muttplotlib pandas if necessary currently not using pandas Using muttplotlib to plot can actually compare it to pandas But ideally we want to take it to JavaScript. So it will be a stand alone Tool so I don't know if we should continue this or not Anyone has any questions so interior. Yeah file like this. So yeah major DC shift is Some major artifacts Interestingly enough that the DC shift continues after the artifacts. So I suspect there's just something really wrong with this file So in truth would be interesting to see so yeah, so when you record you need the signal to settle So if you're trying to catch a baseline ideally we need to record for long enough to see what happens After longer periods of time, there's something really wrong if it didn't settle for 10 minutes. They could be a Major a Shift the DC shift so indicating that something is wrong with the recording Isn't really wrong The electrodes might need to be reattached or things like that Yes, we can Do the time as well Comment this out so we have the Actual time of the recording on the x-axis the sampling rate of So currently we have the number of samples And here it is Should show us time in seconds and yes, this is 10 minutes so Yeah, if the baseline didn't settle in 10 minutes It's no good In this case, yeah, if you can make the recording better. There's a DC drift That yes, we can filter Now if Copilot can Just do it Yeah, we'll leave the high frequencies for now and this drift is obviously low a Frequency components. Okay, so it's taking the mean That's one Do it would it you mixed up with the access as well can try it out I'll do it before over here somewhere. So we're changing the data We're plotting the data as well We're trying to convert string to float So first those meant to be float the numbers like here What a filtering libraries. Yeah using a high pass filter Sci-pi that looks more like it. Yeah order five. It's fine This is returning the What are they called The coordinates of the filter It's the coordinates of the filter. It's not actually being applied And this is actually using the filter Returning my order six not necessarily order five What is it trying to do didn't win this Cut if it's not there. Um five is default, but then we're using six. Okay, it doesn't matter five six. It should be the same It should not see much different Get the field filter coefficients So I'll be using the function getting A and B Then we should be using the butter high pass filter function Filter the data data Wait a sec. Ah, that's just turning it into numpy I into a float and applying the band pass filter. That's okay Unconvert string to float type fix it for us There is actually The air is actually elsewhere We're just trying to filter that signal, but Yeah, even without filtering You can tell there's something really wrong with it. There is a major dc shift So we converting columns to floats All right, because of the timestamp Right, it's mixing because I have pandas It's mixing in pandas. Do I Actually have to do it Yeah, it will be It will be a mix of stuff And can it go There somewhere It's not a daytime string Yeah, the time column You're Going into a wormhole It's not 22 is it the timestamp Number of channels sample rate odd sample index the timestamp Ah, it's the last Two timestamps. Why? One regular one formative It's a bit odd It's a year and they can get rid of it. We don't actually Yeah, we don't need the timestamps Let me tell it timestamps Yeah, I can just use the sampling rate It's through writing the whole code, right? Yes, we have first lines as the header the rest is the data a channel names That's standard filtering We need sampling rate twice. We can fix it later right, so we head Copilot writing the whole code Would it also I'm getting the same pretty sure shed gpt somehow always does Better This code in give it the give it the error It will just Most of the time It's able to sort me out Let's get into this The area you're encountering is because there's a string in your data array that can't be converted into a float Specifically, it's trying to convert a timestamp the 8th of February 2024 30 now is 22 minutes and 58 seconds point 184 And obviously that's not going to work as a float This issue probably arises because the data you're trying to process includes both numerical EEG data and non-numerical information like timestamps When you use data equals NP array data one D type equals float it attempts to convert everything in your data array to floats Which is causing a problem To fix this you need to make sure you're only attempting to convert the actual numerical EEG data to floats Not the timestamps or other non-numerical data Here's how you can do it When reading the data lines skip the non-numerical columns like timestamps Then convert the remaining EEG data to a lump array of floats Let's modify the code to address this This approach skips the first column assuming it's the timestamp and the only process is the actual EEG data Give this a try and it should resolve the area you're facing Can we give it the sample? This is the sample of the data set Uh, which uh columns am I ignoring the first column is uh sample index And it seems like the last two columns are the timestamps Sorry the last three columns So yeah, the last column is the string that we can convert or just ignore A how to modify the code Yep Got it. Thanks for sharing a sample of your data set Based on this it looks like your data set has multiple columns with different types of data Specifically you have sample index a numerical index for each sample EXG channels 0 to 7. These are your primary EEG data channels Excel channels 0 to 2 accelerometer data, which might not be relevant for EEG analysis Other several other columns with unspecified data Analog channels 0 to 2 additional channels possibly for other types of sensors Timestamps two columns at the end for timestamps For EEG analysis, you're probably most interested in the EXG channel data The accelerometer analog and other channels along with the timestamps can be ignored for now Here's how you can modify your code to focus on just the EXG channels When loading the data only select the columns corresponding to the EXG channels Ensure all other columns like timestamps and accelerometer data are ignored Here's the updated code This code snippet will extract only the columns related to the EXG channels Which are typically the ones of interest for EEG data analysis It skips the first column sample index the excel channels the other columns the analog channels and the timestamp columns Instead of this line, we need it after the bypass filter After these two functions Right, so that's the filter bit Looks like the filter is ringing The other good thing about GPT-4 is that it can just pop in the output Looks like the filter is ringing Also can we create a chart with two panels One for the original unfiltered signal and the filtered waveform below How to change the code to do it Yep, I think that filtering might be a bit too aggressive The plot you're seeing indicates that there's a significant transient response Often called ringing, which can happen when filtering data Especially with a high order filter and a very low cutoff frequency This is likely an artifact introduced by the filter itself And not actually present in your original data To create a chart with two panels One for the original unfiltered signal and one for the filtered waveform below You'll need to plot subplots in Matpoblib You can do this by using the subplot function Here's a modified version of your code to do that Generating the code This code will generate a figure with two subplots The top one will display the original unfiltered data And the bottom one will show the filtered data You will also see titles for each subplot to differentiate between them As for the filtering issue You could consider using a lower order for the filter Which would reduce the ringing Another option is to apply a zero phase filter using SCIBI Signal, filth-filth instead of L-filter Which avoids phase shift and can reduce transients Here's how you could modify your butter underscore high pass underscore filter function to use filth-filth Try these adjustments and see if the resulting plots are more to what you're expecting Right, well I'll have trouble integrating all that Can you either generate the whole code without any placeholders Or give me prompts for github-copilot To change the code, whichever one you prefer Sure, let me provide you with the complete code This will read in your data, apply a filter And then plot both the original and filtered signals in two separate subplots Yeah, so it's giving me the whole code I mean we can close github-copilot for now Yeah, the github-copilot, the github-copilot One's a copilot, one's a GPT Giving you this, the window is not scaling properly Make sure you replace path underscore to underscore your underscore file dot txt With the actual path to your EEG data file This script will Read your openBCI text file Isolate the EEG channel data Apply a zero-phase high-pass filter to the EEG data Plot both the original and filtered EEG data in two subplots Regarding getting help on github, you can search for repositories related to EEG signal processing or openBCI You could use search terms like EEG signal processing github Okay, okay, okay, okay That's okay Yeah, we don't actually want to use the openBCI Python library Well, because we want to make this thing as generic as possible And potentially eventually have a tool on the website To allow you to upload your own data But ideally I would like to try making it in JavaScript And yes, JavaScript has some AI as well What's the AI doing? It's coding, coding This is better This interface is still where people are screaming at me for using GPT-4 like this But that's actually better It can hold longer context I mean github copart was really good at So we generated this code now from scratch It's opening this EEG file That someone recorded 10 minutes of quick and dirty EEG Actually might close it I don't think there's any So there's a question of if I record on battery and filter the signal So that's already filtered the signal Ah, yeah, the filter is a bit too aggressive So we're currently trying to make it less aggressive So we can reduce the order of the filter We'll make it less aggressive because it's currently ringing And we're also now plotting two separate subplots One with the original data and one with the filtered data Regarding the battery question I don't know, I don't think it's a battery problem There is a DC shift It's horrible It's like And it's going over like the whole 10 minutes Let's see if this code works Then I can actually create quickly a copy My gith is not currently working properly So I'm just using the old good version control Just naming the files Renaming the files It's stuffed up the fault and name Most of the time dealing with this weird thing So I'm using Ubuntu, right? Visual Studio Code I think there's something wrong Because sometimes when I hit the CTRL E shortcut It just literally puts an E in the code I'm pretty sure it's... I'm pretty confident I'm pressing that CTRL button Right, so that's... I don't know if... The person who sent me the data is watching or not Yeah, so the ringing is okay Now, probably this filter it actually removes way too much And it's ringing That's the original data There's a major DC drift So I don't know what's up with those applications It doesn't really matter because... Oh wait, it actually might matter We had to... Can we before filtering Would you recommend doing the trending first Before applying the filter Because the amplitude is this At this crazy minus 15,000 Minus 2,000 Also, can you explain DC drifts Why they are happening And how to potentially deal with them In this particular case, the DC drift is happening Of an entire 10 minutes of recording And I shared the recording as well The raw and the filtered The filter is just ringing And that the trending will help the trending The trending, it won't get it Come on, let's turn off that microphone The trending The trending, you should know what it is It's common Before applying the filter Because the amplitude is this crazy Minus 2,000 Can you explain Baseline Explain Baseline DC drifts So I don't think they are battery-related If that's what the question is I'm pretty sure the battery won't change much But you can try Yes, that's some basic I think it's to do with how the electrodes are being placed So essentially, you need to monitor Ideally, before you start recording, you monitor impedance You monitor how well the electrode is connected to your head Also, could you mention monitoring electrode impedance When recording EEG And how it's important or not Yep, so I'll close there See what it says The thing about, where is my... Oops, try another voice The trending before filtering can indeed be a good idea Especially when you observe a slow drift in your data That isn't related to the actual signal of interest This drift is often referred to as baseline wonder Or DC drift It can be caused by a number of factors including Electrode skin contact Imperfect contact between the electrodes and the skin Can introduce low frequency noise into the EEG signal Body movement Even slide movements can cause changes in the electrical properties Of the skin and the electrode interface, leading to drifts Environmental factors Temperature changes and equipment instability can cause baseline drifts Physiological factors Breathing, sweating and heartbeat can all impact the baseline level of the EEG signal The trending removes these low frequency trends Without affecting the higher frequency components of the EEG signal That usually represent the neural activity In practice, the trending can be as simple as subtracting a moving average Or fitting a polynomial to the data and then subtracting it For EEG data, a simple linear trend is often sufficient Which can be done using skippy, signal, the trend Skippy Here's how you could add the trending to your code Yeah, so try To The trend first So this will just Make, turn the values Move them to around zero That's what, that's all It does So we have any From Sipy, signal and port The trend as well Not currently being used And the drift I don't like coding Restructure this code The messages in LinkedIn live are just horrible They disappear on me So there were like seven comments But I can only see The last one I don't see them in the chat at all Yeah, LinkedIn Was streaming This, yeah Twitch is really quiet And YouTube Quiet as well And I might keep this one short Let's finish this quickly Let's focus on that It's actually listening to what's said The trending before filtering can indeed be a good idea Especially when you observe a slow drift in your data That isn't related to the actual signal of interest This drift is often referred to as baseline wonder Or DC drift It can be caused by a number of factors including Electrode skin contact Imperfect contact between the electrodes And the skin can introduce low frequency noise Into the EEG signal Yeah, so I don't know why But most consumer grade devices Don't even, don't even bother Recording it, it's absolutely important To record your impedance You know what the It tells you what's the quality Between the electrodes And skin connection It might be not sitting well Might need to reattach the electrode Yeah, depends Generally if the electrode is not well attached It will start seeing more mains So whatever, if you're 60 Hz If you're in the United States 50 Hz, if you're in Australia I'm not sure about other countries The body movement yet that Signal there must be motion And the interesting thing about it Is it's not recovering very well You can see after the movement Whatever it was It takes some time for it to recover That could be like a disconnection Or something Something is wrong there Yeah, it does probably just Movement motion Anyway, sorry for that For mentioning it was just a very bad file Would be interesting to see what happens Like after a longer period of time If it stabilizes or not Because that's where you will be getting Your baseline recordings from If there's no more questions I don't think we're so normally We're doing other things on this channel Some just organizational general things What is it we are trying to do For the reason about section Probably have to rewrite it But yes, we're trying to democratize Biomedical data Make it available for all It doesn't say so but Sorry for the ads Yes, the website is monetized So go check it out You will be learning And helping the project to be sustainable But mainly I'm after comments So if you tried any of the tools Please do let me know There's another one There's a bunch in the works If you're watching the live streams In the videos, you will know Quality tools Not sure if that's a relevant ad I mean, it's targeting me I don't remember searching for any Quality tools The one tool that we haven't Actually published So yeah, there's a lot of this No good type of You can actually go to that URL It will show you what it was If it has a HTML What is this CG music There's no link So it's essentially like an unlisted If you copy that Go to this side Hopefully it will start my Ah right, that's the CG Sorry, wrong one Actually forgot what it's called So currently we're doing the Cardio Quest bot It's essentially an AI A machine learning thing That is playing this game So hopefully soon it will be Available for you to play this It's currently working pretty well I'm still testing it That's what it looks like Close that Yeah, that's what it looks like So it's the same as this Cardio game Which you have to play So you have to click on Normally ECG And do not click on a normal one To get your points And try to get There's no leaderboard But I was able to get to 1,000 So please do let me know If you're able to How much did you score That is playing the game So it has an abnormality measure And yes, we can just split it Into below above 0.5 Something reasonable I think it's already doing it Sorry, it's not 0.5 It used to be 0.5 So 0.3 is more normal I think normal Should give me 0.1 Yeah, 0.1 is normal So in theory anything like above 0.2 Should be considered abnormal And Then you Measure its performance So eventually we'll have Your score And the bot score on the side For you to see Which one is doing better Or worse So hopefully That will Be Available Sometime soon But go check out the website There's many interesting things That you might like Or dislike Please do, if you dislike it Do let me know How you think I should improve it Because yes, it's Work in progress No, so that's why this site is called Bionic Chaos But it seems to work pretty well Now The Cardio Quest bot At the machine learning Bot that is playing this game It's a fuzzy logic Based The parameters we're currently looking at This frequency amplitude And the number Of positive and negative peaks Which is probably The most important in the ECG Then we have those Eight rules That essentially make the decision So the example is If negative peaks If there are too many negative peaks Or Too few The abnormality is likely So it's giving you like 0.5 Whatever value And If the frequency is Normal Then it's giving you Unlikely And these rules are all Combined together So ideally there will be some sort of It's actually really hard to do To review this thing It's same, it's meant to be explainable Explainable AI You can look at Combinations of two rules So currently we have Essentially three, four Parameters So we have a frequency Number of positive peaks, negative peaks Amplitude And the abnormality Output, which is again For the sake of this game Because it needs to make a yes-no Decision Most of the numbers Actually between 3.1 We can make it differently So we can do 3.2 Or 0.15 Because we get a 0.14 Normal Then the abnormality is unlikely So the answer will be Normal And anything else For possible or likely will be abnormal So yeah, we can Do it that way But we might just Continue testing this For a while Until we make it available As say next week So let me know If there's any questions Check The questions At the moment By the way, I know someone might ask If the Bot is potentially cheating If it has some You know, inside that you Might not have So no, the bot is only seeing The waveform And not seeing anything else So it doesn't have any labels Or anything like that So it's the same As You playing the game You just seeing the waveform The bot is the Same In terms of improving Over time, well It's performance already Like through the roof So it's already Does Extremely well So we might have to Make the task harder By implementing more Waveforms So currently we only have Five or six Yeah, we have two variations Of the QRS No P wave, no S wave, no T wave So we can More of these examples Then the task will become Harder and then we'll see How well it performs In terms of Over time there is only Yeah, some of the Fresh shots, the normalization of the Fresh shots is becoming Better Over time, yes We'll look Into it next week Keep it sweet and short Bye