 Hey, good evening everyone Welcome back to the evening session So for the evening talk, you know, we're gonna start with Abhirami Ravishankar who's gonna talk about Self-driving cars using open CV. So I know there are a lot of curious business specifically for this talk because It's about open CV. And that's you know, really up in the market at this point. So over to you All right. Thank you so much. So good evening everyone My name is Abhirami and I'm from Bangalore and we all know the traffic situation here and the number of accidents that happen on a Really basis during the peak hours. Well, at least before the pandemic began Life would be a lot simpler with self-driving cars and I was always keen to work on it I got introduced to open CV in my first year of engineering and it has a wide range of applications for both human and non-human detection I began this project as a step closer to innovating what already exists today and started it as a weekend project So today I'm going to be talking about set driving cars using open CV So the objective here is to basically understand the use of open CV for self-driving cars and to understand the needs of the obstacle present in front of it and Time it takes to basically move away from the obstacle and the nature of path that needs to be taken after encountering the obstacle So first what is a self-driving car a self-driving car is also known as an automatic vehicle or a robotic car Which is basically a vehicle that is capable of sensing its environment and moving safely with little to no human input Self-driving cars combine a variety of sensors to perceive their surroundings like radar, LiDAR, Sonar, GPS and other inertial measurement units Advanced control systems interpret sensory information to identify appropriate navigation paths as well as obstacles and relevant signage So the concept of self-driving cars has actually been around for a few decades now Scientists in the beginning were actually brainstorming through a few ideas and faced problems One wouldn't be able to figure out without implementing it in the real world I would like to walk you through a small introduction into the history of self-driving cars Problems faced using certain techniques and what we know as a self-driving car today So a few things that have been here since the 1980s are basically hybrid navigation Homogenization and decoupling, vehicle communication systems, pre-programmability and digital races Now a lot of these terms might seem unfamiliar and are no longer used But I would like to explain them in this particular context to understand the evolution of self-driving cars today So firstly hybrid navigation Now there are different systems that help control the car Systems that need improvement include car navigation systems, control systems, location systems, knowing the environment, vehicle control Knowing the speed of the vehicle, the direction and other control methods Next, homogenization and decoupling Homogenization indicates the fact that all digital information assumes the same form The concept of homogenization also applies to autonomous vehicles In order for autonomous vehicles to proceed their surroundings, they have to have different techniques Each of their own accompanying digital information that is from like a radar, GPS, motion sensors or using computer vision Homogenization requires that digital information in these forms are transmitted and stored in the same form This means that their differences are decoupled and digital information can be transmitted, stored And computed in a way that vehicles and their operating systems can understand and act upon it Homogenization also helps to take advantage of the exponential increase in computing power of both hardware and software Which is basically the Moore's law and also supports autonomous vehicles to understand and act upon digital information In a more cost-effective way and therefore lowering the marginal costs of the vehicle Next, vehicle communication systems Individual vehicles may benefit from information obtained from other vehicles in the vicinity Especially information relating to traffic congestion and safety hazards Vacular communication systems use vehicles on the roadside and other roadside units Communicating nodes from a peer-to-peer network, providing each other with information This is rather ineffective and ideal for a world only with connected vehicles and no people involved Another useful factor is reprogrammability Another characteristic of autonomous vehicles is that the core product will have greater emphasis on software and its possibilities Instead of the chassis and the engine which are basically mechanical parts This is because autonomous vehicles have software systems that drive the vehicle Meaning that updates through reprogramming or editing the software can enhance the benefits of the owner That is, update a better distinguishing between a blind person and a not blind person So that the vehicle can take extra caution while approaching the blind person By recognizing a stick along with the person, which also works in the case of other differently-abled individuals A characteristic of this programmable part of the autonomous vehicle is that updates don't need to come only from the supplier Because through machine learning, smart autonomous vehicles can learn certain updates and install them accordingly Like new navigation maps, and in-tech intersection computer systems These reprogrammable characteristics of digital technology and the possibility of smart machine learning Gives the manufacturer of automobiles an opportunity to differentiate themselves on software This also implies that autonomous vehicles would never be finished because the product can continuously be improved Next, digital traces. Now autonomous vehicles are equipped with different sorts of sensors and radars And as said, this allows them to connect and interpolate with computers from other autonomous vehicles and roadside units This implies that autonomous vehicles have digital traces when they are connected And the data that comes from these traces can be allowed to develop new products or updates To enhance autonomous vehicles driving safety and its ability to actually perform well So what are the challenges faced? Well, all of the scientists somehow forgot that in the real world, especially in busy urban areas There would be hundreds of people walking across the street Even for an ideal first world country, where people usually follow traffic rules perfectly It is impossible to eliminate a scenario where there are no people or pets on the street Now coming to a very important factor where open CV plays a crucial role That is the human factor Self-driving cars are already exploring the field of difficulties in determining the intentions of pedestrians, bicyclists and people owning pets And the model and models the behavior that must be programmed into the driving algorithm Human road users also have the challenge of determining the intentions of autonomous vehicles Where there is no driver to which you can make eye contact or exchange hand signals Drive.ai is testing a solution to this problem That basically involves LED boards mounted outside the vehicle announcing Like going slow to not cross or waiting for you to cross This is rather counterproductive and makes the whole process much slower than if a human were to actually drive Now the handoff from automated driving to manual driving Which can become necessary because of unforeseeable road conditions Or if the vehicle has limited capabilities Now a sudden handoff could leave the human driver dangerously unprepared for the moment And in the long term humans would have less practice at driving And might have a lower skin level and thus be more dangerous in the manual mode Semi-automated cars have shown to suffer from this very problem For example Tesla autopilot ignoring the and people usually ignore the road And use their phones and perform other activities inside you know their car Against the advice of the company as the car is not capable of being completely autonomous In the near future pedestrians and bicyclists may travel in the street in a much riskier fashion When because they would believe that self-driving cars are capable of avoiding them In order for people to buy self-driving cars and allow them on the road The technology must be trusted as safe And as an engineering student I also came across Vimo and Uber's incident And it actually got me thinking how hard can it be? Well it turns out a lot Anyway this is how I built the primary prototype So the plan of action here was to basically find the depth of the obstacle present in front of it Calculating the total time the object spends in front of the camera Detecting the nature of obstacle present in front of it Whether it's a moving object or if it's a stationary object And the movement of the car based on this data received by the computer And works real-time and mostly using data fed into it over a period of time So the solution here is to basically use OpenCV Now what is OpenCV and why do we use it? OpenCV is basically an open-source library which is aimed at real-time computer vision The library was developed by Intel and its cross-platform It would support basically Python C++ Java and other programming languages Computer vision is a cutting-edge field of computer science That aims to enable computers to understand what is being seen in front of the image OpenCV is one of the most widely used libraries for computer vision tasks Like face recognition, motion recognition, object detection etc Several people have been working on this for a few decades now But only for specific applications And more to do with their software than software and hardware integration So I decided to attempt a combination of these applications Like lane detection, traffic light and signal detection Pedestrian detection into one prototype And also keeping in mind only the software point of view And trying to perfect it and later plan on scaling it into hardware If it's okay I'll take a little water break So one of the first things I did was basically traffic light detection So three things must be kept in mind Identifying the regions of interest Training a classifier, tracking and optimization So images from urban areas usually have a lot of things going on There are cars, pedestrians, traffic lights And a lot of information that needs to be processed at a quick rate The detection limits the amount of time Basically the amount of information that needs to be analyzed Into much smaller regions of interest Now the regions of interest presents a list of potential traffic light candidates And if it is implemented correctly The detection should improve the system speed and performance I would also like to talk about a few methods that I tried And also explain why I wouldn't recommend it So a lot of you might have heard about sliding windows Now I will explain why I did not use it So sliding windows is basically a box or also known as slides Across the image at every interval And the classifier checks if the contents inside the window Is the traffic light or if it is something else Sliding windows are simple, intuitive and easy to implement The purpose of the region of interest Is to limit the work that the classifier has to do If you were to use this approach We would need relatively small windows to catch the traffic light To handle the range of traffic light sizes We would have to run the sliding window approach Multiple times using different window sizes Or unconstruct an image pyramid So this would basically be an extremely complicated task Now color thresholding is also extremely popular Traffic lights emit one of the three colors Basically red yellow and green Or green The basic idea behind color thresholding Is basically to limit to where these three colors are present Anywhere where the image is not red, yellow or green Is set to basically black To effectively isolate the colors we are interested in A few considerations need to be covered Like color space, threshold cutoff And variation in illumination Typically images are present in the RGB color space However, RGB mixes the color and intensity information Through its channels This makes the RGB format sensitive to changes in lighting To detect the traffic lights We can't have variation in lighting Depending on whether what stops the threshold From picking out the right colors So to compare this Many opt to convert color spaces That separate color from image intensity Now some of the more viable options Are like spotlight detection So a spotlight is basically a bright area In an image surrounded by a darker area The idea behind this detection method Is that traffic lights will appear brighter Than their immediate surroundings The image is transformed into grayscale And a wild top hat filter is applied Now the wild top hat filter Highlights the areas that are brighter than their surroundings The spotlight detection method Is robust to variations in illumination Because of the structuring element Which is basically the kernel is applied locally Meaning uneven background illumination Would not be a problem We can find blob in the image And filter them by shape or size Unlike before we do not rely on color information Now another very advanced method Is blob analysis and morphological filters Blob basically stands for binary large object And refers to a group of connected pixels in an image So the image shown here Basically a dark connected region Known as blob And another example is a red thresholding Image basically shown here And so every red object is basically An image and can be viewed as a blob With important properties like shape and size So the color and blob based methods Customarily use morphological filters To refine the information and scene So morphology is a broad set of image processing And basically processes the images Based on its shape So the erosion and dilation Are two fundamental morphological operations So in dilation the object is in the foreground Is enlarged while the erosion erodes away The boundary of the foreground object So dilation is basically great For joining broken parts of an object And erosion comes in handy at later stages To remove noise Blob analysis can also help filter Non-traffic lights by comparing it To the blob's property of what we expect A traffic lights properties should be So the traffic light should be circular With the exception of like turn signals Because they are also circular in shape And would not take up a large part of the image So we select blobs that are somewhat circular But not too large And the final product is a set of traffic light candidates That need the shape and size criteria So next traffic sign detection Road sign recognition is a significant And essential part in intelligent vehicle navigation systems So the text and logo embedded in a road sign Usually contains a lot of information Such as guided direction And current traffic conditions of that particular road However it is a hard task to find An extract road signs exactly from a single natural image Due to complex backgrounds Variable light condition Appearance degeneration Perspective effects caused by the camera And the recognition of road signs is a challenging problem That has engaged the attention of the computer vision community For more than 30 years Since the road sign is normally highly contrasted In their backgrounds And rectangles are often designed for road signs Previously the approaches were divided into two categories That is color-based and shape-based methods The system recognizes and interprets various signs Using visual only information And therefore signs must be obscured By signs which may be basically obscured Is like blocked by other vehicles or trees Will not be recognized Here the algorithm is based on its core competence In vehicles and pedestrian detection The algorithm shares the attention, classification And tracking framework of these modules And uses robust classifiers Developed in these applications Trained on different examples So the algorithm used to detect the HSV Is for an image extraction Gaussian filter Is used The traffic sign is detected using the HSV And the HSV representation of colors Basically the hue determines the color that you want Saturation determines how intense the color And value determines the light Lightness of the image High variance in sign appearance has made the detection And recognition of road signs A computer vision problem over which many studies Have recently been performed Road signs use particular colors and geometric shapes To attract the driver's attention However, the difficulty recognizing road signs Are largely because the colors might fade Or peel off because of long duration of exposure To the sun and basically they can get damaged Air pollution and air condition can decrease The visibility of road signs Outdoor lighting conditions may vary from day to night And affect the colors being perceived by the camera Obstacles like trees, poles, buildings and pedestrians Or vehicles that are present in front Can basically block the signs from being viewed By the camera Also video images of road signs Basically suffer from blurring In the view that the camera order Is mounted on top of a moving vehicle So a solution to this is basically the images Could be pre-processed in stages With an image processing technique Like a threshold technique, caution filter Canny edge detection, contour or fit ellipse And these stages are performed to recognize The traffic sign patterns And the main reason to select this method Is to reduce the computational cost In order to facilitate clear time implementation So the first strategy is to reduce the number of MLP Also known as multi-layer perception inputs By pre-processing traffic sign image And the second strategy is to search For the best network architecture Which reduces the complexity By selecting a suitable error criteria for training The system was trained Using a training dataset Validated with a validating dataset To find the best network architecture So the cross-variation technique Is implemented with a training dataset Validating dataset As well as a test dataset So recent experiments show consistent results With accurate classifications Of traffic sign backgrounds With complex background images Now moving on to pedestrian detection Pedestrian detection is a very important area of research Because it can enhance the functionality Of pedestrian protection system In a self-driving car We can extract features Like one head, two arms and two legs From the image of a human body And pass it into a machine learning model To train it And after training The model can be used to detect And track humans in images and video streams But lucky for us OpenCV has a built-in method To detect pedestrians And it has a pre-trained HOG HOG is basically a histogram of oriented gradients And a linear SVM model To detect pedestrians in images and video streams So what is an HOG or a histogram of oriented gradients? So the algorithm basically checks Directly surrounding pixels Of every single pixel And the goal is to check how darker The current pixel is Compared to the surrounding pixels So the algorithm draws And arrows each of the adjoining pixels Which are basically darker And it repeats the process For each and every pixel in the image At last every pixel is now replaced by an arrow And these arrows are called gradients And these gradients show the flow of light From a lighter region to a darker region And by using these gradients Further analysis can be performed Now lane detection There are multiple ways we can perform lane detection We can use a learning-based approach Like training a deep learning model Or on an annotated video data set Or by using a pre-trained model Capturing and decoding the video file is the first step We can capture the video using video capture object And after the capturing has been initialized Every video frame is decoded That is converted into a sequence of images The second step is grayscale conversion of the image So the video frames that are in RGB format That is red, green and blue RGB is converted to a grayscale Because processing a single channel image Is faster than processing a three-color image Then the third step is to reduce noise Noise can create false edges and therefore Before going further, it is extremely important To perform image smoothening A logic filter is used to perform this process Next, canny edge detector It computes the gradients in all directions Of our blurred image And traces the edges with large changes in intensity Then region of interest This step takes into account only the regions Covered by the road lane A mask is created Which is of the same dimension as the road image Further, a bitwise and operator is performed Between each pixel of the canny image and this mask It ultimately masks the canny image And shows the region of interest Traced by the polygonal contour of the mask Then the hue line transform The hue line transform is a transform Used to detect state lines And the parallelistic hue line transform Used here gives an output As the extremes of the lines are detected So the library is used for this task At basically a cross-open series Mathplotlib and NumPy So to conclude, I would like to say It is extremely important to keep in mind The human behavioural aspect of self-driving cars Before venturing into it And I hope you learnt quite a bit from my talk If you have any questions or would like to know more Please feel free to contact me And also I did a good amount of research Into this topic for a few months Before dipping my feet into it And I made it a point to learn from articles About failed projects And why it didn't work And why it has taken about 40 years People, since people started working on it To achieve a small amount of autonomy That these, you know, cars have Thank you Hey, great, great talk, Abirami So I know there are a lot of questions actually And we've asked them to, you know, Reach out to you on Zulep So thanks a lot, again, and up next