 Data science, but what we mean by accessibility is how easy it is for visually impaired person to do the task whatever we are talking about That's what accessibility means in general Before we get started, we would like to also say that the aim of this talk is to get you excited about the challenges that are there and maybe let's work together and propose and do some innovative solution that could work for all of us Before we get started with talks, let's do with this introduction My name is Sunil Chaudhary and I am currently working at Morgan Stanley and I have been doing the programming for last five years Hi everyone, this is Rishabh Jain here I am currently a student pursuing my integrated master's degree in applied mathematics from IIT Roorkee and I will be joining Microsoft as a data scientist from next year So both of us are iSTEM co-founders iSTEM basically is an organization which empowers people with disabilities and its aim is to make them technology innovators, not only technology consumers We do it by various activities, we conduct activities related to research and development in accessibility space We also do hackathons for accessibility problems and we work on conducting technical programs for visually impaired people as well So that's about iSTEM hackathon So let's get started with the main talks How do you think visually impaired person access the computer? How many do you know? What technology do they use? Any idea? Does anyone know? Yeah, so they use a piece of technology which is called skin reader This is a kind of an assistive technology that they use Now you will be going to hear what how does it work So one of the sample video that you can see how the skin reader works You are currently on web content To enter the web area, press control, option, shift, down arrow Link, like page, button Link, pushing love struck cat face Play video, button You are currently in the group to exit link, pushing love like page, button Link, video, lunar new year, duration, seven seconds Public button, like, like toggle button You are love toggle button You are currently on a toggle button inside of a toolbar To select or deselect this checkbox press control, option, space To navigate items on this toolbar press control, option, and then up arrow Ha, toggle button Wow, toggle button So what you just heard is a skin reader which basically was reading out a particular web page It was telling you what are different controls for there What you can do with those controls, what buttons can you press So by using these technology, a visually impaired person is easily able to access the computer But there are these widely used skin reader, NVDA and JAWS NVDA is mostly written in Python only And have been developed by mainly visually impaired people or developers themselves And coming to the topic of the talk It is about how a skin reader does read a code, a piece of code Again for that we would like you to experience how does it feel like while listening to the code So for that we have a video as well So there are a few things to be noticed You saw that there was a syntax error in the code But did you feel that skin reader was announcing something special That hey you did something wrong? No unfortunately not Similarly you saw that there was a change in indentation level Or change in the nesting level But again skin reader was not announcing it All of it was happening in a linear fashion So that was the basic problem that underlies readings with skin reader Any kind of piece of code So any programmer when he does coding So the main thing he looks is what are the different activities that he does Reading, writing, debugging, navigating the code And all of these are the activities If they are accessible to visually impaired person They can easily code and do their activities Throughout the talk we will be talking about What are the different challenges in all of these fields In these activities So here is a slide which basically a kind of a wish list That an accessible IDE can have So that a visually impaired person can code easily So these are some of the features That we found out with our own experience As well as we done a research papers survey In this space, the people the research they have published So these are some of the features that they have come with The thing to be noted here that all of these features are not implemented in every IDE Some of them are presenting IDE But again it's not a uniform experience throughout the IDE So talking about these features The first one is how you access different windows and other things So if you have a keyboard shortcut for that It's really nice because most of the visually impaired person Use only keyboard to access their laptop They hardly use mouse Well it may sound surprising but they don't use mouse So if they are keyboard shortcuts for them This is very easy Also there are indentation changes that are happening in the code That conveys cementing meaning to you for the code For example in python it's very crucial So if there are some audio beeps for that That would be very helpful That could be another feature that IDE can include Then suppose you are reading a very large code base Now as a site developer you can fold a particular piece of a code And get a high level overview of the code How does it work, what are the different steps involved But because skin reader is reading all these things linearly So what happens is that they need to read it line by line That's very time consuming One of the solution could be that you present this code In a form of a tree structure or hierarchical structure Where you can expand or collapse nodes That are not interesting to you and go on reading the code And one more thing about is this bookmarking and tagging feature So for example you are right now working on some code And you thought I need to see this function definition maybe So what you can do you can jump on that Or maybe you do it by keyword search whatever the method But the problem is suppose you want to just come back to the place But again because you need to find out all that place It would be good if there is some bookmarking feature That person can place before going to any other place And then come to the place in the code That's why he was working on Similarly suppose you search some keyword in your code Now when you are searching some keyword You can land anywhere that keyword appears So one thing that any person would like to know When he has appeared into the place How does the context look like Where am I right now So if there is some feature that would tell you That you are at this particular function This particular class in this particular for loop So that would be also helpful for visually impaired For knowing the context There are other things as well that could be done For example there is this syntax error happening in the code In the video that we saw So could be there could be some audio feedback That is beeping in the background Well hey you did something wrong just fix it But yeah right now it's not there So you just build it and then you get to know Oh my god I did an error And that is something frustrating sometimes Because you get to realize it very late Okay so this is the feature that are Now we will be talking about different Commonly used IDEs in the python space And how does their accessibility look like So first one is IDLE It's very basic tool that comes with python itself But how many do you think it is accessible Any number Anything Well it seems that it is not It's yeah you are right It's not that much accessible So though it has very simple interface For example the command line And the interactive prompt they call Also the editor where You write the code the editor interface Even that is not accessible Skin data just goes silent As simple as that So you just you don't hear any video Any audio I mean From the skin data about the text That could be there Coming to the Jupyter That we just heard it back in the top So Jupyter is quite well As compared to IDLE So it has an HTML like interface As Jack was telling But what happens is that All these elements are well labeled So you get to know what are the buttons What are the different areas That are there for visualization And edit box everything is there But the main accessibility That appears in Jupyter Is it's editor interface Or where the code the cell Where you write the code So when you go through it By using skin data What happens is that you are not Able to read it as multi-liner So and you Write down what blind people do then They just write their code in notepad And then just copy paste it in the cell Because cell itself is not accessible So they can't write it there If they want to correct it And read out what they wrote So that's how inefficient it is right now Coming to the another Famous ID which is studio code It has again very nice features So you can jump between different windows And these different Commands are there to run the code But again for example The video that you saw was in video studio code And it was not treating you That there was a syntax error Giving you any feedback So it's not that the people Are not doing anything in this field So there are some approaches Or some initiatives that are being taken So I'll be going through three of them here So one thing to note here is That these approaches Could be taken at three levels One is the skin data level itself Another is you make some changes in the IDE Maybe you make some changes in the programming language itself So it becomes more accessible Let's see how Quorum Quorum is a programming language Which has an evidence oriented Programming design So what they do is that they Look for their new features Through research or user survey studies Where they try to find out How could I implement or design my language So that it's accessible to all the users So they did a user study with blind people And they started designing this language And they designed it in such a way that It is more user friendly with Visually impaired person Again what the point to be noted here is That these quorum language Has for example For for loop it has a syntax Let us say repeat time times or something So you can Instead of reading left and right And all those things You can just Intuitively Similarly is an approach that is Taken by Scala Whereby custom scripts are written At screen reader level And these custom scripts Read the code in more intuitive way As you can see on the slide So there is this Scala again Has very scary syntax To be true But yeah these scripts That are written They read it very in very natural Or intuitive way For example for this generic thing They would say context bounded by this For parameters they would say Parametrized by a Int and something like that Instead of reading left bracket Left bar and this That the screen reader was doing In the video that you saw There is a thing this thing called Auto So it is an IDE extension Developed by Microsoft For community and professional edition It has many cool features With respect to accessibility For example if there is some error It will keep beeping But hey you did something wrong Just fix it So I will not go away until you don't fix it So that's a very cool feature Another thing is about debugging So debugging is very inaccessible Area right now But this is the first of the Very few approaches that people have taken So this is one of them So there they introduce talk points Talk points is basically an Audio debugging point Which basically reads out the Value of the variables Whatever you want to see at your break point And also it tells you Some audio beeps that Hey your code passed in through this Particular execution context So by that you can see So moving to the next part Hi everyone Now we already look into the IDs how IDs are accessible Now let's look into The experience of visually impaired programmer Real experience when they do programming So ashtray conducted technical training Program every year So in that program We run two python courses last year One of them when the Student took programming first time Another one when they have Certain experience like in C++ and Java And they are switching in python to use Tools like web development and machine learning So going through the different Search paper and like Research paper and our survey And even our community feedback We came up with these kind of findings So number first of that is The python is very intuitive and It's quite understanding language And people love to do different It's fun to do different stuff in python Because people got It's very engaging them like People used to got the skills Such as function, class and all that Very easily So it's quite increasing for the new learner Another one you can easily Access python by the command prompt And command prompt is inherently Very excessive So this is again very increasing thing And people used to build very cool stuff In python when they are using this As their first language One of the major challenge that Our student and community face Is indentation which is very Visually appealing like it's Good for a visual thing But it's a very painful task To debug or write a code In an indentation fashion Because we also notice that When the person switching from Different languages like c++ They have the clear indication By the left, right or something like that So they are moving into a particular block So this is the major challenge That our viprogrammers face And this is quite discouraging So one of the solution that On the screen reader level Implemented what they have done Is they have provided the audio queues Like the higher you are in the indentation level The higher the pitch of the audio queue Another approach they took At each line what they do They tell you whether your code Is at 4 space or 8 space level So this is the approach Taken by NVDA It's nice to Like people are working in this kind of Approaches To look On to the buzz word of this era Which is data science So I want to ask Do you guys think is data science Is accessible for person with viprogrammers Show of hands please Okay Quite few and yeah This space is very inaccessible So let's walk through Some of the key points In this So here as you know This is very inaccessible This is the All the data science folks know That this is a diagram of the inception That by Google So yeah first of the things Understanding the different architecture And the different algorithm To get hold of that Is quite inaccessible Like even it took me 3 or 4 days To understand what is this architecture So now let's move On to a data science workflow And what are the challenges in that So like What we do in a data science workflow First we do EDA like exploratory data analysis So and then we Train some model and then we have We test the model on different metrics So So yeah this is like one of the example Like this is the data of Like different countries Their GTPs and their life expectancy And population So like I want to know Like which like Cited person can plot easily And can draw a very useful inference Like in the first graph you can see The Africa has a very lower Life expectancy rate And also you can plot like the pair plots Or the scatter plots to find the correlation Between two different features Like whether in the Africa Life expectancy is related to GTP or not you can easily do by visual Thing but as These visual graph is not accessible to a V.I. person what can we do Like we can route a different library Or something like that which can describe This kind of information to a V.I. person To pull out some statistical quantity Or something like that One other major challenge the V.I. person Face is suppose You are Suppose you are building A binary classifier And what the people usually do They apply some algorithms like PCA or some dimensional direction And then they plot The data into the 2d grid and get The field and they get the hold of the data Whether it is linearly separable Or something like that and then They build their model according to that So also we can pull out some statistical quantities In that space so that we can Describe this kind of information So yeah another major challenge Let's move to next slide So another major challenge In this space is to visualize time series data Because there is a lot of visual Information when you are playing with the time series data Suppose we have a seasonality We can compare the 2 plots very easily So let's look some of the possible Solution we can have in this space So one of them is We can have an audio cue So I will explain how it can work Suppose we are Making a plot of our stock price And we have price on the y axis And time standard on the x axis So what we can do With the price of the stock We can have a higher audio cue So how it helps Suppose I listen the same kind of pitch After a time interval I can feel Yeah there might be a seasonality trend And also it can help In getting the fluctuation Like there is a lot of fluctuation Or there is some anomaly So the pitch is very high or low At that point so the audio cue Help in that fashion But audio cue has a very big problem Because you can't play 2 audio At the same time and you can't compare Using this audio cues So for that And this is an open challenge problem Like we can work on it and build something Cool which can solve this problem Another thing we can do is pull out Some statistical quantities like what we do When we are designing some auto ML model We pull out some We apply some algorithms like change of Point algorithm or some We run a window and extract some Different features that we can show To a vi person in some tabular or something In that form Now let's move to the next slide That is our feature engineering So yeah So feature engineering what a sighted person Used to do suppose you are given with data He can easily scroll down a particular column And see and compare Even 2 columns how they are correlated Or how the things are changing But vi person lose this ability because He had to navigate a table like cell by cell So we can also build something In that space which likes to bring We can do it we can do all this stuff Like I can I can have 2 columns print out a pandas or By a numpy but yeah If can be done on the table level That could be good if we compare 2 columns and draw out the inference And also like Suppose there are some anomaly or Suppose the data suppose That is example the price is changing At a particular point When I move to price column What my library or At the skin data level do It's describe me hey the all of the values Most of the values in this column are 0 but at a index 255 I found a value 1 or something Which look like an anomaly to me Like something like that we can build So yeah after having a feel of a data Like how my data look like So we can easily Take machine learning models Like whether my data linear Like model according to that Then one of the major challenges to compare our models Suppose I am I have to build a model with high precision Or say high recall or something like that And I have trained my tools Different models out like SVM Or decision tree something like that What the sided person do he can easily plot The ROC curve and other these kind of plots And have an inference hey If I want to have a high recall I will do the assemble of this method Or I can choose this method Which is totally visually based And we can also print out like We can print out the area after A particular value of like probability Like 0.2, 0.4 In a tabular format So the VI person can easily Access those things So after you Build some beautiful models the things To deploy these things on the clouds So there are three main services Right now Amazon Web Services Microsoft Azure and Google Cloud So these are accessible but not Quite good like you Some of the features are not accessible And like and we should work Upon this and this come under the The web accessibility So next are the conclusions So what we find Yeah okay yeah So what in the conclusion So what we find that the there are Three levels of accessibility one at The state leader level and one at The like programming language And one at the ID like if Three are well integrated Seamlessly integrated every language Can become accessible And another conclusion is The graphs the graph data is The visualization are mostly inaccessible So we can work something out to make Those accessible so now It's to call to action Like ISTEM ISTEM do annual hackathon where We build accessible cool products So you can come and build All products along with us Another call to action is the jupiter thing Because that multi line thing If we fix that thing the Vani Vi community will Like it will help the vi community To grow faster and people can Do work seamlessly and And their productivity will increase Okay so yeah We are open for the questions and yeah Thanks very much So we are only going to be taking one question Hi I did thank you for the talk I did not See you mentioned any terminal based Ideas and you also mentioned That command line is inherently accessible So Is there any work where you use Things like Vim or I Python directly in the terminal Which makes does it make it a little easier So currently right now The I Python itself is inaccessible Like it's go silent When you open with the stream leader Even the idle command Like it's go silent It doesn't tell you what you are typing What you are typing there But the windows command prompt That we talked is inherently accessible So you can use These features as well But the main point out that we wanted To convey here is that Ideas do are important It's not that you can do everything on terminal And what functionality that you get Is let's be true They are not present on These terminal based interfaces So we want to take advantage of that Because you see the problem is not that Blind people can't do these things The point is about productivity and efficiency Because they need to do it in very different ways They need to figure out other things And they need to do it again again So what we want is Can we build some solution together And then increase their productivity and efficiency