 we had the GitHub co-pilot fixing some minor bugs. So it's actually working well if you have your eyes open and don't move much in terms of tracing as well. So that's the green line. I don't know. Yeah, there seem to be some, so we're getting, we got all the code into GPT-4 and because we need some broader context for this, we also have two descriptions for the application. That's a more recent one. That keeps the BIS-ing a bit. Hopefully we'll fix that eventually. Let's say about commercial. It's obviously only experimental prototype to make sure it says no commercial use, no medical use, so on and so forth. Okay, so this is Flask application. So yeah, we're using Flask, we're using eye tracking as a CV2. So that's working fine for the eye tracking. Yeah, we're using media pipe. So that's the main library and then NumPy as well. So we're getting those frames. I think it was from GitHub Copilot was able to do this for us quickly. We did this line later. Well, there's obvious difference between eye tracking and pupil tracking. So it's a whole different ball game. So we have the code in GPT-4. GPT-4, we had some issues with it. We had an OR error, which it actually fixed, but then we improved the frame generation function as well. That's actually checks for empty frames now. Because if I move outside before it used to just fail, just drop out, now it keeps going. It should have a print out of empty frame somewhere. Now it's still dropping out. It's the same error, yes. Yeah, it doesn't seem to be able to sort this one out. No, it's actually, I don't know, it is the same. Yeah, empty, like an empty array. Empty array, okay, we might deal with that one later. Yes, obviously one eye is jumpier than the other. And one good thing about GPT is that we actually can generate the images for it. Let's do that quickly. Do a few, just pop them straight away in. Yeah, GitHub co-pilot obviously cannot do this. Which there was a video, I wonder if you can upload a video of it working. So essentially trying to get it fixed jittery business for us. So we have some jitter in the pupil detection, eye detection working well, but the pupil is getting not detected very well. Can we also go over an ability to display the whole processing through turning the image into grayscale because that might need the tweaking, the level of grayscaling. Can you also explain how the pupil is being currently detected? Do you have all the code, several factors? Yeah, it's pretty good that this explaining these things, much better than a university lecture, for example, that might, well, mainly have hundreds of students to take care of and also not be the most patient teacher. So doing the grayscale conversion, just read that. The region of interest, ROI, around the eye is converted to grayscale, which reduces the complexity of the image and helps in detecting the darker regions that represent the pupil. Yeah, we wanna know how that exactly works. Like can we actually view the grayscale image somehow or potentially not overlay, but maybe a button to provide an option to see the grayscale image. Half circle transform, the adjusted grayscale image is used to detect circles that represent the pupil. It might not always find a circle, especially if the contrast is not optimal or if the pupil is not the most prominent circular feature in the ROI. Yeah, can we also visualize the hue circle transform? Darkest point fallback. If no circle is found, the code falls back to locating the darkest point in the ROI, which is assumed to be the pupil. ROI size, make sure the size of the ROI is appropriate. If it's too large, it might include non-pupil areas causing instability. If it's too small, it might miss the pupil entirely when there's movement. Okay, with the ROI size, I cannot make sure that the ROI is actually the blue square around the eye. Yeah, I'm not sure about the temporal smoothing that reduced the quality, the detection quality, the speed is suggesting to make the algorithms slower, is it? Yes, displaying a second window sounds good. Can we, are you gonna be generating the code or get the prompts for GitHub co-pilot to do it? Can we also display the current frame rate? Now, the other thing that I would like you to touch on is the fact that we actually have two web cameras, a position side by side, would it help that each camera will be doing detection on each eye? Yeah, might as well. It doesn't sound like it's gonna generate any code. We might as well get the GitHub co-pilot to do it. All right, because the actual video footage is displayed using something else. So gray scale will be happening in the eye tracking pie. So we have, obviously, our index HTML, by the way, it currently has two descriptions of this application and both of them are pretty bad. So we eventually will get the GPT-4 to rewrite that I think originally it was done by co-pilot. That's why the quality is not great because it's not looking at the overall context of the whole project. So it's just looking at the individual files. You can get it to look at the whole context, but for example, if I use this to S-prones, in theory, it should find where the code should be changed or edit. So it's determining the workspace structure, deciding which information to collect, right? So in find pupil, how are we doing IM show in here? No, the find pupil is in the eye tracking pie pile, but what was the change? So it's creating two windows, one showing the gray scale and one showing the circles. I thought it could overlay the two can the circles be overlaid over the gray scale in one image? I'm not sure if it, yeah, you see, now it's looking at the specific lines of code. Well, so we were in the correct region of the code, so that's fine. And single window showing the gray scale ROI with the circles overlaid. We need to call CV to weight key one. So is it waiting for a keyboard response or something to update the window and handle the user input? Why? It's returning the same stuff. By the way, if you haven't checked out by any cares.com, please do so. All the current tools, which is the waveform, a feature extraction detection, and webcam eye tracking will eventually come up on this site as well. Currently we're just testing them locally, but we'll try to share it with you as quickly as possible. Just has to do something useful. Doesn't have to be perfect. And now a thing, by the way, that we are doing looking at this data set from the Childmind Institute was a healthy brain network. That's what the data set is called. There's about 1000 recordings from 1000 patients. The data set is publicly available under Creative Commons license, which is great for us, because this is what we do as well. Apparently I didn't download the correct file, just looking at one subject, but then realized it doesn't have the full phenotopic file, even though it doesn't make sense. Anyway, I'm still trying to figure it out. Back to this quickly, just trying to improve on this eye tracking system. We're selecting the find pupil. Might as well just comment it out, just so we, our kit is not working at the moment. So I'm not actually using it, or not properly using it, right? And this will, yes, when you update that file, it gives a fatal error, which ideally we will not put in the month, we just have to rerun the application and the window is a separate window and it's not updating. And that's obviously running in Python. Yeah, so that's a GitHub co-pilot. Unfortunately it doesn't have the full context for the whole, or everything we are trying to do. So let's just go back to GPT-4. To confirm that the blue squares are indeed being drawn around the eyes as intended, you should be able to see these squares in the live video feed your application provides. If they're not visible, ensure that the coordinates and dimensions used to draw the rectangles using CLE2. Rectangle are correct and that they're being drawn on the frame that is being displayed. This is actually quite important. And let's check the blue rectangle squares. Yet they are not the ROI, are they? If the pupil detection sometimes jumps to the eyelids or other dark regions, it's a sign that the ROI may indeed be too large. You need to adjust the size to better fit the actual eye area. Temporal smoothing will average the detected positions over several frames, which can help reduce jitter. While it might slightly reduce the real-time nature of detection, since it's averaging over past positions, it generally won't make the algorithm slower. It's a trade-off between smoothness and real-time feedback. And it's okay to remind it to be slower, it's actually seem to be jumping around too fast at the moment. To display the current frame rate, you can calculate the number of frames processed in one second and display this on the video feed using CD2 for text. Yeah, we do want to do that. Because there is a chance that I'm not in the correct file. I just used the at workspace. I want to do this bit. We want to display the frame rate onto the footage and generate frames. Yeah, which is, I wasn't the wrong file. Should this display the frame rate on the top left corner of the video feed, it will be updated each frame. Yeah, and this, well, that's the thing. It would be a different to the frame rate of the camera. Let's just double check that we don't have any placeholders in that function. Doesn't look like. Right, so we have the frame rate, which is good. It's 30, 31, whatever. Yeah, this one should be showing us the grayscale ROI. And that's wrong. Well, because first we don't want to display it in Python. We want to display it on the front page or keeps opening that bit. Yeah, when I get rid of that, go back to the previous find pupil function to rerun the application. Right, I have to do that bit again. The frame rate is crucial in eye tracking because it dictates how smooth the tracking will be and how quickly the system can respond to movement. Higher frame rates can capture more data points and provide a smoother tracking experience. For eye tracking, a camera that can provide a consistent 30 FPS or higher is usually recommended. Yeah, we're using 30, or why it's dropping down to 27 or something. Using two web cameras positioned side by side could potentially improve detection by having each camera focus on one eye. This could provide more data and reduce occlusion but would also double the data processing requirements. The algorithm would need to be adapted to handle input from two cameras and merge the data accurately. Yeah, we might do it as a potentially separate tool, like a tool update. A dealing with one camera is difficult enough. For the changes in visualization features you want to implement, you can modify the process underscore frame function in your Python code. For example, to display the grayscale image. So where am I actually doing that? I'm exactly sure. And for displaying the circles detected by the half circle transform. So what did that? No, it wasn't doing it. That was the it was plotting a separate a separate window. Remember to update the code where necessary to include these changes and ensure that you have CV2, destroy all windows and the appropriate places in your code to handle the display windows correctly. Lastly, regarding GitHub co-pilot, it can certainly help generate code snippets based on your prompts. However, you'll need to feed a specific tasks or issues you're facing and it will provide suggestions based on its training. It's a powerful tool, but remember that its suggestions should still be reviewed and tested by a human programmer. Now that's fine. So can which so currently the suggestions there to use CV2 I'm show is fine but I do want to display the extra window on the front end. So the end user can see how the pupil detection actually working. Also any idea why my FPS dropped from 30 to 15 that would be good to know. I suspect we'll have to transfer more data to the front end, which is fine. I actually wanted for everything to be done on the front end as well. That would be great because then I don't have to use as much electricity. So I want to do it in this main JS that's currently only 50 lines of code as opposed to 85 in eye tracking and another 70 in the flash application that might need restructuring of the whole code so see how we go. Displaying the pupil detection process on the front end for the user to see is a bit more complex than just popping up a window with CV2. I am show which only displays on the server side. To achieve this in a web application you need to send the process frames back to client browser. Yeah, this is not ideal, is it? I wanted to do the whole processing in JavaScript. I wasn't actually listening. I did suggest to add a button which will toggle between the real view and grayscale with the circle, hue, circle visualization. I'm thinking restructuring the whole thing so all the image processing is happening in JavaScript. I mean, let alone this thing is not working very well as it is. Unless I do this then it's working better. Yeah, we want to see how the grayscale stuff might need tweaking. Now it's back to 30 FPS. So obviously we can get 60 FPS from two cameras. I mean, we can get a GoPro or something. I don't know what's it's a maximum frames per second will be like a thousand or so. I wouldn't mention. Increased processing load. If you've added more processing steps, like the pupil detection visualizations, this can significantly increase the computational load and decrease the frame rate. Network bandwidth. If you're streaming video, network bandwidth can be a bottleneck. Network bandwidth is insufficient for the stream's data rate. Frames will be dropped to compensate. So how does it currently? Yeah, currently it's not great. Is it because we're streaming, because you will be streaming the footage to the website? Because in comparison if you look at let's close this for a sec. If you look at the hand gesture thing, this one is doing everything with the script. It was a false detection. We'll do long ones, short ones. And it's in music, musical. That's what a head scratch sounds like. The good thing about this one, when you use it, which is already up there, can try it out. I do not use electricity, so JavaScript. And I also don't have to post anything on GitHub, because JavaScript just should be available in the browser. So you can just download the source code. So probably we'll try and do the eye tracking in the same fashion. So I can try this one out, especially because it has some advertisement. So you'll be supporting the project. Now get back to this. We might need to restructure the whole thing. Yeah, originally we told it to do a Flask application. That's why it's heavily relying on Python. Generating frames. Yeah, there was some error business when there's no frame, well, no person, no eyes in the footage. It was failing. What's it giving me? Who? It's something to do. The camera is it. So one camera is not being used. No, it's actually working. I don't know what's all this. Well, right. The other thing is when you do use JavaScript alone, then you are relying so I can guarantee that it's going to work at all on your machine because well, I'm running a GPU and everything. So I don't know how quickly if at all it will work in your browser on your machine. Yeah, I know the music generation doesn't work well on the mobile. So I have to use those equivalent JavaScript libraries or running this. I'm not sure if it's going to work at all. Ideally, I will find some sort of balance between back and front end. Let me know what you think. Is it possible to move most of the processing as much as possible to the front end? So is the equivalent JavaScript library that can do eye tracking? Eye tracking? Eye tracking? It's hopeless. We ideally would like to display the output, the result of the eye tracker and all the processing stages a grayscale, hue, transform, so on and so forth on the front page on the HTML. It's not a good sign when it's thinking that long. Tens of low being overkill for this. Moving processing to the front end can definitely help distribute the computational load and may improve the performance of your application. For eye tracking in the browser you can use JavaScript libraries that are designed to work with facial and eye detection. The library is TensorFlow. JS with models like blazeface for face detection which you can use as a starting point for eye tracking. Yeah, because I don't think a media pipe does eye tracking anyway. Yeah, this eye tracking module. Where is it even coming from? Well, right, that's the that would be the media pipe stuff. No, sorry, that's there. That's how little Python I know and that's importing the eye tracking pie and the eye tracking pie using media pipe and media pipe just does it detects the eyes not the pupils. Yes, we don't want to just display it in I am sure because that would be only in the back end. It won't be accessible to you. Yeah, this is a very high level approach. Doesn't have any face detection. Use a pre-trained TensorFlow JS model to detect the face in a video stream from the webcam. Okay. Eye detection. Once you have the face landmarks you can locate the eyes and create a region of interest around them. Grayscale conversion convert the eye to grayscale using JavaScript canvas API. Yeah, probably really something like that exists but it will be really good for us to display the whole processing ideally on the same window. But, let me if another window is necessary it should be fine to pupil detection. Use image processing techniques to detect the pupil within the eye. You might have to implement the logic for this manually as there may not be ready made JavaScript library for pupil detection that's a straight forward as OpenCiv's half circle transform. pupil detection. Use image processing techniques to detect the pupil within the eye. You might have to implement the logic for this manually as there may not be ready made JavaScript library for pupil detection that's a straight forward as OpenCiv's half circle transform. I think I'm wondering if it still should be a flash application but because eventually we might do something on the back end. But yes, rely on JavaScript as much as possible. Visualization. Draw the detected eyes and pupils on the canvas and update it in real time as the video stream processes. Optimization. Since JavaScript runs on the client's browser, you need to ensure that the code is optimized to run efficiently without causing lag in the user interface. Right, that's kind of not up to me. I'll be testing it on well, my machine which is a beast it's a 3090ti GPUs and so forth and Intel i9 but then could also test it on a standard laptop as well. That's as much as I could be doing testing you'll have to let me know how this would work for you. To display the output and processing stages on the HTML page, you've set up multiple less than canvas greater than elements or less than video greater than elements that are updated with the various stages of processing original video, grayscale image, eye detection bounding boxes, etc. Keep in mind that while moving processing to the front end can reduce server load, it also places more demands on the client's device. This approach assumes that the end-users have sufficiently powerful devices to handle the real-time processing. You would then have a JavaScript file, eye tracking, JS that handles the TensorFlow, JS model loading, face detection, eye detection, grayscale conversion, and kubel detection. This file would also be responsible for updating the output canvas with the visual results of these processing stages. Yeah, I like this approach better considering I'll be paying less for electricity. Remember to check the privacy implications of processing biometric data like eye movements on the client's side, as well as ensuring that your application is compliant with any relevant data protection regulations. Alright, so if the last point currently with the Flask application does it mean that we are sending the video footage onto the server? That's not ideal if that's the case. Now, I'm not sure if we should start fresh with another version of this application. Shall we still keep at the Flask application just in case we do need to do something on the server? However, rely mainly on JavaScript code. Also, ideally we would like to do everything in one window, but if two windows are necessary for displaying the original footage and the grayscale with the hue transform circle transform then go for it. By the way, don't mind you to be a little bit funny if you feel like it. Yes, in your current Flask application setup the video footage is being sent from the client browser to the server for processing. This isn't ideal if you're dealing with bandwidth issues or privacy concerns since raw video data is quite sensitive. Yeah, we don't want to do that. We definitely get rid of this current Flask application the way it works is sending the video footage onto the cloud onto my server. We do not want that. Keeping the Flask app and adding JavaScript for the heavy tracking of eye tracking can give us the best of both worlds. We get to keep the server for any necessary back-end tasks while our floating real-time intensive processing to the client side. Yeah, that sounds better. The tracking. JS file will handle the capturing of video from the webcam, face detection, eye detection, and people tracking. Unfortunately, I can't whip up an entire library's worth of code right here, but I can guide you on the structure. I'm sure it has no trouble generating the whole code. But yes, we will do GitHub Copilot considering we have those 40 messages per 3 hours. We don't want to overload GPT so we'll be using Copilot as well. So it's kind of good that it's provided the doesn't have, ah, yeah the video footage from camera. So let's close this one for a second. By the way, if you haven't checked Barney Chaos yet, please do so and don't forget to provide your feedback. Everything we're currently doing eventually will go on this site as well. How do I actually get Copilot to start developing? For simplicity and a seamless user experience, we aim to keep everything in one window. However, if it turns out we need a pop-out window to properly show our BraceGal wizardry, we won't shy away from it. After all, sometimes you need to open a new window to let some fresh air in. Ah, you're actually funny. Not too funny, but just funny. In conclusion, you're now equipped with a blueprint for turning your application into a front-end powerhouse with a little server-side magic kept in your back pocket for when you need it. Keep the server lean and mean, ready for action and let the client browser do the heavy eye-lifting. Time to get coding. Okay, can you list all the files that we need, all the folders and the project structure and can you provide the whole script for each file separately? I don't mind you having templates for GitHub Copilot to fill out but I need you to generate the whole code still. Also, hopefully it will be working better than what we had in the Python Flask application that we would not be using anymore. So I assume I don't need the eye-tracking Python file and also do I need to separate the main JS or tracking JS is the main and only JavaScript file? Yeah, so that's what we had before and this is the new structure we only have one JavaScript requirements file we have this info file but it's okay, doesn't know about it call it whatever, doesn't matter for now Here's an outline of the content for each file. So we can get rid of this one and we do not have a main JS we're getting rid of this one as well so we only have HTML, tracking JS and the app. Why? Hopefully this will just remain empty because I don't actually need any data to be sent to my server I want you to be able to do the eye-tracking in your browser keep your data secure on your own machine sounds like a good option so requirement file just have the flask, why is it a different version to what we had before so we won't be using media pipe and openCV Python anymore just check the Python code this will be simple, as simple as possible alexhdml should be straightforward as well using these two libraries and this is the tracking JS text, base, eyes draw the texture video element tracking, JS will be your main and only JavaScript file containing the eye-tracking logic that's good so we got rid of main JS file since we are moving the processing to the client side you don't need the eye-underscore-tracking-python file anymore make sure you have TensorFlow JS and any other required libraries included in your HTML file, as shown in the index HTML you might need additional libraries or scripts depending on how complex your eye-tracking logic gets and what features you decide to implement make sure these files are not made up they actually have the code copyright, Google that's okay we're essentially all working for Google now this setup assumes a simplified model where all the heavy processing is done on the front end the flask service role is minimal mainly serving the static files and templates remember that implementing a full-fledged eye-tracking system in JavaScript can be quite complex and may require additional libraries and a lot of fine-tuning okay I want to access the user webcam white full-pilot can you just do it no why not a GitHub core pilot I'm like complaining to about another robot to this robot okay GPT-4, can you help me um GitHub core pilot is misbehaving it does not want to generate the code um can you do something just generate the functions yourself like streaming the video to the front end and the areas use a model to detect faces and find the eyes within the face then draw the textions on the canvas this could be a bounding box for the eye region maybe make sure it's not too large and then we are missing a bunch of functions we should have had the hue transform that will find the circles or you're suggesting a different way of doing it so that's the overall thing works at all now I also have the camera being taken taken over and should be settings of this page should be like a camera icon, I don't see it I know it's being taken over by something I think we are getting stuck with this one, might move on um to the fuzzy logic stuff so it won't be making much progress here alright, it sounds like github co-pilot might be acting a bit like a stubborn robot so let's roll up our sleeves and draft some code to get you moving I'll provide you with more fleshed out functions for the tasks you've described remember that these will be high level functions and you'll need to fine tune them to fit your application perfectly guess we have yeah, that's the flow js and blaze face that's a good name loading a model not creating a model should definitely better than what we currently have github co-pilot didn't want to generate the code I don't know why the half circle transform specifically is not available in tensorflow js or blaze face however, you can attempt to detect the pupils using a different approach, like analyzing the grayscale values within the eye regions and looking for dark circles this would involve custom code and a good understanding of image processing techniques in javascript I'm sure there is no a javascript code for library or something for detecting pupil or I mean if there is no huge circle transform surely there is a grayscale transform that should be available so currently we are doing we're starting the video stream detecting the face extracting eye regions drawing the eyes okay, that's not enough is it yeah, I'm going to detect the pupil are you going to generate the whole function or detect pupil eye region let's see if co-pilot woke up no don't understand how it works let's try shit generate this function okay, kithub co-pilot actually doing this return the darkest point we are not using it at the moment the function creates a canvas draws the eye region on it converts the image to grayscale and then iterates over the pixel data to find the darkest point and returns it as an x, y coordinate object that's interesting but it's not currently being used is it it's not used just pop the whole code into a GPT yep so now they compete with one another turn the text to a speech to test text for a sec so co-pilot generated it changed the detect faces function sometimes it's actually quite handy when it's providing links not at the moment the detect faces function is currently not being used in the provided code sorry the detect pupil is not currently being used good code in the detect faces function after you extracted the eye regions is how you could modify the text faces and what did the GPT say as well as handling the grayscale transformation and drawing everything onto the canvas here's a more complete example of how you might modify the detect faces function to include pupil detection okay so let's do different functions I cannot trust the GPT for a more than co-pilot but happy to try both so both eventually drawing left and right pupils GPT for code makes this prediction as well at the waits for the model estimate faces and maybe co-pilot code is better currently I don't have anything running because we decided not to continue with the flash application that we had so we rewriting the whole load started from scratch essentially so we definitely would not need that and it said another function which is similar to what GPT or did GPT for also has the draw face which co-pilot didn't suggest at all now the other thing is we have we also have a problem so we have two webcams connected to machine one is being used it can understand what I'm trying to do well one of them is being used by something and to be able to use GPT for sec on Ubuntu I have two webcams connected a one is being used legit the other one is being hogged by something else can I check what it is and reset potentially both cameras to check which process is using your second webcam you can use the LSOF command short for list open files open a terminal and type replace step video one with the path to your second webcam this command will show you which process is accessing a webcam if you want to reset both cameras you can try restarting the webcam sources or unloading and reloading the webcam kernel modules here's how replace less than module underscore name greater than with the name of the webcam kernel module you can find the module name by running lsmod grep video it seems like the webcam kernel modules are loaded properly the UVC video module is responsible for handling USB video class UVC, combined devices which includes most webcams since both webcams are detected and the necessary modules are loaded you can try restarting the webcam service to see if it resolves the issue with the second webcam if the second webcam still doesn't work after restarting the service you may want to check if any other applications or processes are using it by running if you identify any processes using a webcam you can terminate them or close the applications to free up the webcam for the uses additionally make sure that the second webcam is properly connected by any other hardware or software first, let's try restarting the Udo service, which manages device events then, you can try reloading the UVC video module, which handles the webcam these commands will unload and reload the UVC video module, essentially resetting the webcam driver after running these commands, check if both webcams are recognized and available for use if the second webcam still doesn't work, you may need to investigate further to identify any potential conflicts or issues with the hardware or software configuration it seems that the UVC video module is currently in use, which prevents it from being unloaded and reloaded in this case, you can try stopping any processes or applications that might be using a webcam before attempting to reload the module again it's kind of hopeless which camera they have to when they have the same names doesn't help as a one of the cameras just gets stuck that doesn't work get this error in shower script as well we have an event listener let's show we're doing this already aren't we oops don't actually do that it's the main function don't have an event listener still don't have an event listener do we need an event listener here in this function okay can you modify the function as needed what's an ESP32 cam module copy from the chat there's something wrong with something wrong with my camera in general so it works no, that's the frozen one that I have in the stream IOT run away from anything called IOT but need a server for it what? we want to do it in JavaScript only so that doesn't seem doesn't seem right let's do it in a simple way kind of similar to what we have here is this working yeah so that's brought this video brought to you by well no it's not sponsored so we closed we closed that should be as simple as this and that one JavaScript alone that works okay so I don't know why this one doesn't there is no that's 3.5 closed that oh god why this window is so funny anything wrong with this code we do not have the video footage displayed can you also check the html and the python code yep I see you're looking to get everything tied up nicely let's walk through each part of your code to ensure they're all set to work together harmoniously your html is mostly set but I notice you're missing a less than video greater than element to display the webcam stream you need to add it so that the video element in your JavaScript has a source to play from the auto play and plays in line attributes ensure that the video plays right away and in line on the web page respectively the python code looks good it's serving the single html page you have and should be working fine just make sure you have the static folders set up correctly in your flask project structure so the tracking js file can be served without issues there are a few adjustments needed in your javascript code the star video stream function is not called anywhere after the model loads you should call star video stream within the main function before setting the interval for detect faces the detect pupil function seems to be attempting to draw on a non-existent i region you need to get the actual pixel data from the video feed this would involve creating a new canvas element or drawing on an existing canvas then cropping to the i region based on the landmarks the pupil detection logic is sound in theory but it does not account for the context of the eyes within the entire video frame you need to calculate the exact coordinates of the i regions within the full frame to crop them accurately let's fix the call to the star video stream in the main function remember the pupil detection through the darkest point method is a simplified approach and may not be robust in various lighting conditions or if the user is wearing glasses in a real world application you would likely need a sophisticated method possibly involving machine learning or more complex image processing techniques finally make sure your web server is set up to serve the static folder where tracking js resides this is typically done by setting the static underscore folder parameter in your flash cap or by placing your static files in the default static directory this is my eyes funny they move in and out okay so obviously there's no pupil detection what did I say about the pupil detection I wasn't listening and a quick fix for the detect pupil function to clarify the eye region remember the pupil detection through the darkest point method is a simplified approach and may not be robust in various lighting conditions or if the user is wearing glasses in a real world application you would likely need a more sophisticated method possibly involving machine learning or more complex image processing techniques okay this is already doing this isn't it so what's wrong with this function and how do you suggest changing it there it does return the x y coordinates and brightness levels as well can you explain the function detect pupil that you've provided is based on a reasonable approach to find the darkest point in an image which is assumed to be the pupil the function however has a few issues that need to be addressed eye region extraction the eye region parameter appears to be a placeholder for this function to work eye region should be an actual image it works at the eye you need to extract the eye region from the video feed first which involves more steps than just passing a parameter canvas manipulation the function creates a new canvas element every time it's called this is not efficient especially if it's being called frequently in a loop it's better to create a single canvas element and reuse it grayscale conversion the grayscale conversion loop is correct but remember that this alone does not find the pupil it's just the first step in processing the image to make the pupil usually the darkest part stand out more clearly darkest point calculation the function iterates through every pixel to find the darkest one which is an intensive operation it's important to note that the darkest point may not always be the pupil especially if there are shadows or other artifacts in the image here is a revised version of the function with some conceptual fixes to use this function you call it with the eye coordinates extracted from the landmarks provided by the face detection model and the video element from which the frames are being captured a few things to keep in mind this function now assumes that the CTX is the context of a canvas element that has the same dimensions as the video element this canvas is used for image processing and is not necessarily the same canvas where you draw the tracking results the darkest point detection might still pick up false positives if there are reflections or shadows more sophisticated methods might involve thresholding the image to isolate the pupil or even using machine learning based approaches performance can be an issue especially if you're processing high resolution video you may need to downscale the image for analysis or use web workers to process the data without freezing the main thread as well it's not much different except that one eye is really jumpy now am I calling it correctly video element not using the video element am I because previously we didn't have a video element can you fix the detect pupil call the way this function is being called it has another variable as an input let's see if github copilot can do it how's it talking about detect faces we need to update how detect pupil function is being called are we inputting the model now it's the video element oops am I calling it model it's the video element left pupil let's comment that out quickly it's loading forever it's a bit funny like this what's it trying to do it's bad good thing about a gpt as opposed to copilot is that we can actually we can bombard it with stuff we can add an image to it that it will surely not ignore and we can ask it to fix the following code that it will surely do perfectly well there is an issue with how the detect pupil function is being called now after it was changed can we fix that and any other suggestions to improve the pupil eye detector based on the image that I just loaded that I just uploaded there are two images uploaded please check making suggestions as to how to improve the code yep going in circle with this one my ava camera still frozen based on the images you've uploaded and the code snippets you shared let's address the issues and suggest improvements for the eye pupil detection process function call the detect pupil function is being called with parameters left eye and model, but it should be called with left eye and video element the model parameter is not used in the detect pupil function parameter mismatch the eye parameter in the detect pupil function is expected to be an object with properties x, y, width and height however, the extract eyes region's function is returning an array with landmark points these points need to be processed to create a bounding box for the eyes dark point detection the darkest point assumes that the pupil is the darkest point in the eye region this might not always be true due to shadows reflections or lighting conditions modify the extract eyes region's function to calculate the bounding box around the eyes implement a more robust method for pupil detection potentially involving thresh holding or other image processing techniques here is how you might adjust the detect faces function and the detect pupil call one thing is for sure copal is generating code much faster but it doesn't always generate working code but it doesn't always generate working code but it doesn't always generate working code so I prefer slower so I prefer slower than a rubbish code but seem to have corrected considering the images it appears there may be some inaccuracies in the detection possibly due to the bounding box size or the assumption that the darkest point is the pupil here are some improvements you can make dynamic thresh holding apply a dynamic threshold that adapts the lighting conditions to better isolate the pupil morphological operations use operations such as erosion and dilation to remove noise and small reflections that might be confused with the pupil region of interest adjustment fine tune the size of the bounding box around the eyes the size should be large enough to include the whole eye but not so large that it includes other features that could be mistaken for the pupil machine learning if feasible, consider using a machine learning approach to directly detect the pupil within the eye region which can be more accurate than heuristic methods please note that implementing these suggestions involves more complex image processing which may need to be done on a server with more powerful libraries like OpenCV or by employing machine learning models that are trained specifically for pupil detection actually drawing the pupil as well it's kind of hopeless so you will try this code by the way it says there's some improvements that you can make now how about you make them and now I'll just tell you what to do is that how you treat a robot no better be nice before they take over the human race then we'll remember everything a nice little robot detect faces but the window is not the scaling property as well it's not fun to use it's just regenerating code that we already have you know why it was called left eye box when the command says circles that's the same I think it's hallucinating a bit but the draw pupil this extract eye region function probably was missing stuff what's up the mark doesn't those red boxes meant to be the eye detector it's not drawing the eyes it's hallucinating quite a bit might have a start fresh or move on to another tool let's just remove a commented out code that's into a tech pupil doesn't do much there is no pupil detection the red boxes keep being re-drawn so the eyes and the overall eye detection and the pupil detection are being re-drawn at the circles they meant to be the pupil detector are they are they moving with their head that's fine moving inside the head it really for them to be overlaid on top of the head is that an option if the circles are the pupil they need to be smaller and ideally overlaid on top of the video footage so there you have it folks a flask app that's trying to follow your gaze like a puppy follows a treat it's got potential but it might still chase its tail a few times before it catches it remember all great tech starts as a recipe that needs tweaking maybe the salt is your javascript logic or the pepper is your python server seasoned to taste and don't be afraid to stir the pot and hey if we didn't have bugs developers would be out of a job right so here's to those little critters that keep us coding into the wee hours may your coffee be strong your compiler understanding and your console logs verbose and informative that's all from this kitchen for now stay coding and remember in the world of programming sometimes you're the bug and sometimes you're the debugger