 Okay, good evening. Welcome to second first talk this semester. We have Kameshwari who is going to talk to us about a somewhat different topic. It is not the usual where you find it first, but she is going to talk about developmental projects. Okay, thanks all for coming. So I will start the talk with a definition and a disclaimer and then I will continue. So before I begin, let me begin by defining what I mean by developmental projects. So any project that you have worked on will have some research, will have some development, but these are projects that are heavier on the developmental side by which I mean you use the existing state of the art and try to use it to solve some very specific local problem. So when you start out with these development projects, to begin with you think you know exactly what you should do, but halfway through you realize you don't know things have become complex and then you thankfully what exists some research behind how to solve the problem thankfully already exists. So it's a method of adopting it and adapting it to the specific problem and then moving on. So that is how the development projects typically go on. Now coming to the disclaimer portion of it, typically a life cycle of a project involves you get an idea, you execute the idea, you try it out and then based on the feedback you refine it and then you push for wider adoption. So the disclaimer is that while I have experienced with the first four, five is something that I am trying to do. So whatever are the lessons learned, you should keep, I didn't really finish the life cycle. I am trying to complete it, but there are hurdles and not just that I have been focusing on this only in the last three years more seriously. So whatever I am presenting is based on my limited experience. So it's not like I have a lot of yarn to hand out, but whatever lessons I have learned, I will share it with you. So as I mentioned, these are projects that take a pain point and try to solve it. So to ground the discussion, I am going to start looking at one specific problem that we try to solve and we will see the life cycle of solving this particular problem. So the problem we looked at was that the engineering education in India is rather poor, whether we can use technology to somehow help solve this problem. So let's start with what the background is. So we have around 3,345 engineering colleges that are producing 1.5 million engineering students, but unfortunately majority of these are unemployed or underemployed and we always hear that the industry often has to retrain the workforce, they recruit, then they train them before they can contribute. Not just this, we also in the academy are complaining a lot about that we don't get good enough PhD students where they don't have proper training. Again you have to train them here. So all this involves a lot of expenditure both in terms of money as well as time. So this is a big problem. So when we started trying to solve this problem rather wanted to solve this problem, I had some very vague ideas. I'll tell you what these ideas are and how they got concretized later. So the vague idea like I was looking at this Khan academy which was a major source of inspiration. Khan academy was doing things for school education. I thought whether something similar can be done for engineering education. So that is where the motivation was. So the idea was to create some video material. Now why video material? Because the video material is more accessible to a student because there is the way we teach. If you read a textbook, I mean things may not be very clear but the same thing if you explain via video by emphasizing your voice or drawing a figure or pointing to something, it makes it easier for a student to understand. So that's why the video material. So all this resulted in what I call multimedia book. I'll tell what I mean by this book. So the idea was let's create this good multimedia books and then we will ask the teachers in the engineering colleges that are out there to use this multimedia books. Typically, I don't know how much you have interfaced with the teachers outside. Their loads are very high. They often teach, we teach one course a semester but they often teach three or four courses a semester. So the idea is if you are providing such multimedia books, hopefully it will reduce the teaching load. That way they can spend more time interacting with the students in a flipped class setting. How many of you don't know what a flipped class is? People know. All of you know? Everyone knows? Okay. So let me explain what a flipped class is. This is again taken from Wikipedia actually from some place there. So traditional classes, what happens is a teacher comes to the class, imparts some instruction and then the students go home and then they try to solve some homework problems whatever. That is the typical model. But in flipped class, the instruction is taken outside the classroom. So you are imparting the information whatever it is that you want to convey the knowledge via videos that are to be watched before students come to the class. And during the class you actually solve the problem. So the idea is when you try to apply what you have learned, that is when you realize you really didn't understand much. And the instructor is handy to help you solve that particular, I mean clarify whatever questions you have. So that is the flipped class mode of operation. So ideas teachers will use this multimedia books and then conduct such flipped classes. So the end result of whatever of these thoughts was this platform called Bodhi Tree. I'll get into a bit more details about what this Bodhi Tree is. But the talk is driven that this platform provides three interfaces. One of content developers. Content developers are those who are creating the textbook or rather I should say multimedia book. Instructors are those who are using the multimedia books to teach students. So these are the three interfaces that are going to arise. So this multimedia book is I mean some aspects of it is there. But again let's look at the origin of how knowledge has gotten disseminated. So printing press was a major revolution. So we have been using the textbooks that have resulted out of it until the 2000s. Of late people use PDFs or they refer to the HTML, the web content. We believe that the future will be more on what I call multimedia books. Now these multimedia books are a sophisticated form of books where you are just like a textbook. You are organizing whatever you want to convey in the form of chapters and sections. But each section in turn is made up of not just videos it could be made up of animations, virtual labs, reference docs, everything you want to convey. You are putting it in a more sophisticated form. And one other thing I want to emphasize is that videos are not like just information dump type of a videos. These are interactive videos. So you want to mimic in a classroom setting, typically in a traditional classroom setting, a teacher asks questions. The students there is some discussion and then the teacher proceeds. So you do want some aspect of that interactivity to be captured as part of these videos. So these videos are embedded with questions which when a student answers he can get some feedback also. Like okay fine you answered this but that's not the right answer because of these reasons. So you could actually provide that kind of feedback to the student as well. So these are those interactive videos. Not just the videos, these books also come with discussion forums. They can come with a progress tracking like how well are you doing just how well you yourself are doing, how well are you doing relative to others who are also following this multimedia book. So it's like kind of a very powerful medium, I mean compared to a textbook this is like a lot more powerful way of disseminating information. So the idea is to create such multimedia books and then you go ahead to these colleges, ask the teachers there, see you have been using traditional books. Why don't you try these multimedia books? They are a lot more sophisticated. And importantly this flipped class mode in the teaching pedagogy has undergone a lot of research. Many claim it's a lot better way of teaching. Of course there is research going on. It may not work out for all courses but definitely for some courses a lot of people feel that this is a good way of teaching. So what this instructor when he's doing this flipped class mode of offering is he's going to pull this content from multiple multimedia books. And this I think is again a very important aspect because in India again each engineering college will have its own syllabus. You cannot say okay fine this is what I'm teaching you take. So it's more like everyone has to follow the syllabus. If there were two, three such multimedia books a specific instructor can assemble what I call a course specific multimedia book by pulling content from multiple multimedia books and then offer this to his students. So that is also as I said it's particularly important and then you could track the students progress who have watched the videos, how well are they doing in the class, what is their quiz performance. So this Bodhi tree platform provides support for hosting such multimedia books supporting this flipped class mode of operation and it has the usual support like model like functionality like sharing marks, email support Piazza style functionality like facilitating discussion students can chat with each other while they're like for example you're watching you didn't understand your friend is online you can actually chat and get some things clarified. So all this is provided as part of the Bodhi tree interface. One other recent thing that we have added is this lab report grading because lot of our courses also have a lab component. The way typically it is done as you give some lab people type either upload a PDF or a text file and then the TS have to manually grade it. But often the students come back with a crib saying for this why were my marks detected or how much did I get for the specific question within a list of questions. So the idea here is again to ease the manual grading they will provide a form based lab exam I'll show you a demo of it where you actually type it and as the TS or also it's like a web form you can just enter the marks you can enter comments everything gets collated puts into a spreadsheet and it gets shared with the students based on what marks they have got. So that is the report grading feature so it's not that we are really the first or something to do in the space people have been working on this problem for many many years government has invested a lot of money so NPTEL CD are some of the things you may be familiar with. So these are the studio classroom recordings where the focus is mostly on students so you can get these videos from YouTube or ITV server even there was CDs that were distributed earlier. Of course Coursera edX everyone knows these are big time books players who try to target worldwide students. By the way both the tree is a contemporary of Coursera edX we kind of started about roughly about the same time but their focus was on MOOCs whereas our focus from the beginning was on this multimedia interactive books so the NPTEL CD were not quite can be considered as this multimedia interactive books and more importantly our focus was on this local instructors so that they can do this flip class mode of teaching so that is the related work. So in our life cycle I've covered what the idea is the idea is to use video material, flip class for teachers and I'll now talk about how we went about executing it so this both the tree platform is based on the Django python react.js these are basically web development framework tools. We also have an android version of it so this has arisen mainly because we wanted to provide offline access to the material even if you don't have you have internet access you download some material then you don't have internet access you do whatever you want and then later you sync with the server and the state gets synced so you could do all your quizzes offline you could do all your watching of the video offline but whatever marks you have got gets synced with the main server so that's where the android app had come into play. So we built the system based on all these tools so let me talk about scalability first so we had conducted this teacher training workshop using this platform it had about 9000 participants and when we started running it at some point when there were roughly about 500 simultaneous users of the particular system things just started to fall apart things were all connections were getting timed out so it was basically a scalability problem so we then dug in to see what the issues are so our first system so we did some experiments if it were a static file which is basically some simple file that you are passing to the system the server that we have used was able to support 1 lakh requests per second the moment this entire complexity of this Django everything came into play our analysis this was done sometime back showed that we were not able to support more than 50 requests per second I mean it has come down from 1 lakh to 50 requests per second so that's the so that shows the way the code was written as well so we didn't really pay much attention to scalability but then we had to begin a lot of current state of the art to see how we can employ it to solve this scalability problem so memcache radars are some caching solutions we also looked a lot or in fact in the process of looking as well as how to optimize code such that things happen it scales better we looked at database query optimization affinity scheduling is spinning a process to a particular CPU core so all these aspects had to be looked into so it was a major learning experience where when you are working on this the students got to so you cover a lot of stuff in theory but this is a project where when you are working on it you got to experiment with tools that are across a wide range of areas and a lot of I learnt a lot the students also had learnt a lot apart from this we also started log collection to ensure that things were working fine mainly for scalability issues so if you want to debug it's important to have logs and we also used some machine learning based user models derived from the log so that we could conduct some of the scalability tests via jmeter so it involved this entire range of technology that we had to use to ensure that the system got built as well as that it gets k so let me show a very quick demo of how the platform looks so these are in fact we are using this currently to run all those courses so I will just show you a student view so if you can see this is a chapter physical layer is a chapter within this chapter we have these as the sections so if you look at a specific section in turn is made up of these two videos some reference talk as well as a quiz this video I won't play it but it also has lot of quizzes as you can see it's an interactive video these quizzes will keep popping up and it also has out of video quizzes also just these are like the homework problems that you give so this is the interface and it has again as I said discussion forum it has to track maybe this I shouldn't show so it has some assignments you could upload the lab report all these are there as part of the system so let me go back when you said multimedia textbook slides plus the video everything whatever you are seeing here so this is a multimedia book so when you click on it whatever appears here this is the organization of chapters and sections but internally when you click it gives you for that particular concept can be explained through multiple videos slides this doesn't have a lab but it could also have a virtual lab it could have animations whatever it is you are putting everything in one place so moving on so after our execution we moved on to the field trials so far Bodhi tree has been used by 13,000 teachers as well as students so that's something we are quite proud of so within IIT we have conducted nine flip classes predominantly in CSE in fact we didn't take it out till we were quite comfortable only this semester that to a bit late we had offered to outside department people only two had shown interest next time on I think we should be we should do it before the semester began so we now have multimedia books and computer networks architecture wireless networks as well as software lab this was also used to conduct a teacher training workshop this was part of MHRD initiative called teach 10 KT so this covered about 9,000 engineering college teachers where there was one month of online activity followed by five days of practice today some Q3PLE workshops were also conducted which covered about 3,000 students so we have seen interest from others like some faculty in math want to use it to do some Maharashtra wide math teacher training program even IDC they want to do pan India wide design course based offering a few startups also have approached us for using it for life skill development so the I'll tell you how I organize it typically we have three hours of contact hours per week for the students so what I do is two of the hours I give for offline access so they are supposed to watch for a given week a specific material so that schedule is kind of fixed so just to show so I'm basically telling something like this to this you watch all those videos so the schedule is given to them and because we deal with large class sizes which are hundred students which I don't particularly like so what I do is I divide up the class into 35 students each and every week I interact for one hour with each of the batch so from the students perspective they have two hours of watching the videos and one hour face to face but for me this is three hours because I'm dealing with all the students so these face to face sessions typically start out with a quiz it's a small what I call tutorial quiz the reason that the quiz is very important is because if you don't have it people don't watch videos that counts towards a grade so that way they watch the videos and this is a very simple quiz you watch the video you will get full marks in it it's nothing complicated than that because I didn't I mean I shouldn't ask questions where they are not being given opportunity for me to explain in case they didn't understand so these are very simple so after the quiz I provide a very quick summary of what has happened and then multiple things sometimes I actually say okay you learn this here is a problem how will you apply this in this it becomes a discussion session sometimes I just take two or three of the tougher problems which are not like it's good if some instructor is there I make them I pose it they're all solving it I go around looking at how they are solving it the small class so I tend to cover most of them and while they work you know they make common mistakes because as I said I choose somewhat of a tricky problems in class I just say okay see you didn't understand that is why you're doing it but this is how you should do it so this is the typical mode either some discussion on using this concept how do you extend it further or it is a combination of many of these things so that is how the face-to-face tutorials are here you could employ if you are familiar with lot of this teaching pedagogy like think, share or whatever it is all those can be applied here just to make it more lively and engaging to the students so the local instructor has to take on that particular role so the multimedia book is taking care of the two hours so the local instructor has to engage the students in the tutorial slot for which it's the same way as I am doing that is no that 9000 whatever I am doing was for more of a trial this is not the solving of the problem I'll get to how to solve the original problem shortly that was just done to use the system earlier it used to be one intense week of five hours five days face-to-face and lot used to get crammed into them it just becomes too much for them so what we did was you said take one month do it your leisure whatever you want to learn then come and let's drive it face to so when it was so that 9000 also is a hierarchical model so we had 250 center coordinators who were managing the centers locally while I am directing but the 250 center coordinators I did face-to-face with them but at a later date after one month the center coordinators helped the local people while I was overseeing the entire operation no it is a different safe is the technology we use I'll get to that shortly so just to kind of show the feedback that we have got from using this system so this 2 to 4 is recent spring 2015 this is the survey based on that about 63 students out of a class of about 95 filled in the survey in the teacher training workshop those 225 are the center coordinators I didn't collect the survey for those remaining who they were guiding I because I interacted lot more closely with these center coordinators so out of 250 225 had filled the survey so these are some of the questions we have asked so we were asking when you were using Bodhi tree in this where the instructor is not physically present did you feel the need for an instructor so as you can see that very much is a very small portion of the teacher workshop they felt our students didn't feel that much need for instructor but sometimes yes a good member felt that it would be nice if the instructor was present sometimes so the question one is about did you feel the need for instructor presence and red means very much blue means sometimes green means not at all the left is institute offering the right is teachers workshop so for institute offerings our own students teachers workshop is the MHRD is the teach 10 KT program that is 60 so this institute offering 63 students filled the survey for that 225 teachers filled the survey yeah yeah this is from Google survey that's what it projected I just took it from yeah one class this is for a single class so this portion this portion is our CS 224 which is our own students one class this is the teacher training workshop that is also one workshop that I have conducted this is the coordinators who did they all came here to me they are across spread across 250 centers across all of India but I interacted with them face to face as well as through flip so to summarize most I mean they did feel the need for an instructor so because you are removing the physical instructor it is important to know whether they felt the need for a physical instructor so this is the result so we also ask this interactivity portion of the video how many of them found it useful again as you can see many the green says very much many of them found it very useful and the quiz outside is even more they all thought it's very good that they got to practice a lot this also shows how much time did you spend on a 20 minute video good of course there is some portion who spend less than 20 minutes these are the guys who watch the video at 1.5x or something but good number spent 20 to 30 but there are some decent number who have spent 30 to 40 as well so people were watching it at their own place so in fact that is one of the advantages of flip class you can watch things at the pace that is defined by you so we also conducted so this is a comparison with the traditional so we said okay you have experienced traditional classes before now you are experiencing flip classes how do you think your subject understanding whether it increased decreased relative to traditional classes like I mean did you feel that flip classes was reducing your understanding compared to traditional classes so these are some of the questions we have asked so if you see 60% felt that flip classes led to better understanding and 12% and the remaining are actually they have no they found both same only I mean if at all you want to say okay let's not go for flip only 12% actually felt that it decreased their understanding of the subject so a majority were in favor of flipped classes whether this increased the load in our own institute 61% felt it did not increase the load much they were happy with the way things were but for the teacher training workshop in fact only 25% felt it did not increase the load many felt that it increased the load significantly of watching but the reason for that is they all had a daytime job and then they had to watch this as well so it's lot of work for them exam preparedness doesn't apply to the teacher training workshop but within our own institute 70% found that flip classes works better for them to prepare for the exams which means they are basically telling I prefer in a way that traditional class teaching helps me understand things better than flipped that is what exactly what they are trying to tell these people also experience of course it's not for the same course they have experienced traditional classes as well so it's relative to the traditional classes they have experienced versus the flipped class that they are experiencing currently yeah it's not the same instructor do so yeah it's all subjective in fact I'll put lot of disclaimers in the end yeah I mean not well the 60% is basically you are counting in fact I'm putting the lower bar in fact our own students haven't gone all the way till 80% the 60% is more from the teacher's training workshop perspective our own students in fact found it much better so I'll get to that I'll show you one other I don't know if you meant by that but I'll show you one other figure so again finally we asked the question that you know traditional you know flipped how would you rate it 3 means flipped is much better 2 means okay it's better 1 is slightly better about same negative is it's like much worse so this is what we have done so as you can see 83 to 93% prefer flipped mode of teaching to traditional mode institute offering is about 83 teacher training is 93 also try to do is like I have taught this course in a traditional fashion before I've also taught in a flip class before I just wanted to compare how the average class marks were changing across these lot of disclaimers here it's a very sample small sample set I don't want you to I'm not just using this to justify flip class is better it could be a function of the batch it could be a function of the specific instructor I may have put in more effort or in that year as opposed to earlier years there are many variables all this is a thing it looks positive try it out for yourself and then make your own conclusions I'm no way justifying that flipped is it looks good so let me tell what I mean by this so I had done 2 offerings in traditional spring 2009 2012 and then I did flip autumn 2013 and spring 2015 I like to focus on those two red ones because this is my second offering based on I gained some experience and then I'm offering so it's good to compare a traditional second versus flipped second because I've gained some experience in doing it so if you see the A is there isn't that much difference but the B is there is a good 10% increase in the sorry AB's and the class average has also increased not just that the max marks also have increased and I like to believe that I have set the similar exam difficulty there versus here but again as I said this is just a sample point very subjective so I just wanted to share it but you really cannot draw any major conclusions from it you just have to try it yourself get your own feel for it okay so going back so we have finished the field trials so based on our personal experience surveys, suggestions, boxes whatever it is we went again, refined it, I won't spend too much time into it so let me talk about what the so originally I started with this problem let's improve engineering education now what all I've talked about I just built this tool but to really improve the quality of engineering education outside you need to ask the question what motivates an average student quality material there is plenty, NPTEL has lot of excellent nice textbooks but people really it's not really helping much why just because people care about good like for example you have high quality instruction but it is not part of the syllabus for your exams you're not going to pay attention to that high quality instruction even if you put high quality instruction as part of the syllabus but if your exams are just a repeat of the previous exams from some common question bank that is also not going to help you so you really need a method to ensure that the examination process is also in order in order to really do well outside so that is so our three requirements are quality instruction should be part of the syllabus examination process should also test understanding and not general rote memorization and these both should also scale to lakhs of students because engineering students are really quite a lot out there so again I'm now going to start another tool and then I'll pull together for the wider adoption so let me talk about this tool so originally we called the safe which stands for smart authenticated fast exams the origins of safe were very simple so as you see every flip class at the beginning we were having to conduct a quiz and hundred students in a week take it so the TS had to manually grade it was just turning out to be too much we wanted something which is very fast such that the students also do not get an opportunity to cheat so this arose where we thought we'll work on a bring your own device models students get their own smartphones in the class they will authenticate themselves download the exams attempt the exam submit everything gets auto graded because the kind of questions we ask are can be fit to objective type of questions because these are just checking whether they have watched the videos or not so in terms of features it is based on smart devices plus wifi in most cases it is bring your own device it comes with auto grading if it's an objective exam the secure authentication which is very important it's not very easy to achieve the scalable and secure in fact we had to you know you really need to think for just to illustrate the kind of challenges so student can self-cheat by which it means he can access resources from the internet friends via phone chat or bluetooth he can get resources by that particular fashion you could also have to address this problem of impersonation where a student's friend is giving an exam on behalf of the students where either both are in the class they are giving exams to each other or a friend in class is giving exam for both or someone outside is who has access to all these resources is giving exam on behalf of the student and then the means to cheating is you could compromise the app you could compromise the operating system you could exploit bugs in the app of operating system you could pass on authentication quiz information to a friend who is outside but who is using a valid app and operating system so it wasn't a simple thing like okay fine let's do the because the security aspect is tough we had to tackle a lot of these things I won't get given the time I'm not going to get into a lot of detail so the way we have managed this is through extensive logging on the android system where we more or less monitor whatever the student is doing if he's trying to get out we'll force him back into the app we've also tried the use of visual manual passwords to ensure that impersonation doesn't get done we try to create bluetooth maps to figure out the connectivity between the things we per exam app means you're giving an app specific to the depends upon the security you want each quiz will have a specific app which you have to download custom fit for that I mean if you had a previous app it won't work it has to be an exam specific app if you break the operating system there is nothing you can do like if you root your phone and a fiddle around with it so the way we thought that could be handled is in an exam you do random sitting and you just ask people to just swap the phones and take the exam so that way you don't know where you're going to land up with even if you did something to your phone very likely you are going to get someone else's phone to give the exam so it's not going to help you much so all and there are also a lot of this wi-fi bottlenecks the moment you get wi-fi there's a lot of traffic wi-fi not to scale so we had to handle a lot of this wi-fi bottleneck issues in fact you're in the process of handling all that it may not work in all settings but in some settings we feel like once you open that you're stuck anyway to the app it's not that you're yeah it may not work in all settings I agree there may be privacy concerns and stuff but to compensate that for those type of settings you could see one of the things about this application is that it's very cold low cost low maintenance easy setup and highly whatever scalable is you could have like a stock of phones right just like you're investing in a test center with desktops and blah blah you are now investing in smart phones where you have let's say 5000 smart phones that are there for your exam that may not really work but the costs are going down like it's not so much like 4000-5000 you'll get a very decent as opposed to investing 30-40 thousand for a desktop so it's still going to be lot more cheaper even if you go with your own smart phone no no no no these are for high end exams like not really because we do it's you really need to break the OS and do a lot of stuff I mean we are for example JEE I can understand people but we are not really thinking there yet but our students we feel so so we just we don't really do the swapping but we do this extensive logging so that and the app forces them to come back they cannot get out of that they have to decline it because if they accept it it will show and I can pull them up so they are given all these instructions if phone call comes you have to decline if you accept it your exam is like so all these instructions have to be given that is one point in fact one of the things but it has a Wi-Fi connectivity some phones it permits but not all phones the Wi-Fi is a we need Wi-Fi access but you are looking into that in fact Logging is the case where for example whether the app doesn't focus what kind of BSS idea are you connected with who are the neighboring Bluetooth whatever things that you are listening to are you trying to access the phone whatever it is whatever activity you are doing on the phone you are trying to log it no it comes as a dashboard so for example that also I can kind of show you a demo off so this is the dashboard and this is what we are logging so for example of course this is all saying did not submit quiz within the extended time he is connected to some other Wi-Fi ID I mean all these things kind of show up this is the dashboard for that so you could like while the exam is going I am looking at the dashboard if anything I will quickly go there and check what is happening yes in the interest of okay fine so this we feel is a highly scalable system because it is very low cost all you are requiring is a Wi-Fi in a room and thereby it is not like a very complicated power plugs and LAN cabling and desktop setup so we believe that this will scale much better so you could use it for classroom quizzes company placement is like our own PhD admission is as well as so this idea of course certification came only after some time into the thing we did not really think of it being used for course certification earlier but since we were dealing with Bodhi tree and its related issues we thought this could also be used for course certification I won't get into this in the interest of time so let me talk about how so this is Django python server again in fact we had an older version in PHP but we decided to move to this for scalability purposes android coding in this was one of the toughest because as I said you really need to get hold of lot of this logging information which is not android is not been designed for that kind of a thing it tries to hide many things so it was not very easy to achieve but we managed but again as the servers keep changing then the API is changed so again we have to go back and see whether something else is being exposed now so it's been a bit difficult so Wi-Fi scalability was also something so for this we in fact developed a crowd source another android app where students assemble the server actually tells send traffic based on this pattern and we actually do the Wi-Fi scalability of testing for the access point for example using this access point I want to ask the question how many users will this access point support given that the user profile is based on the quiz app in which case what we do is in a classroom like my lab where there are some 60 70 students they're just doing the labs we are just using their phones they are kind of volunteering their phones during the thing we'll just ask them to start this app the servers pushes some user traffic profile to them they'll just generate the traffic profile automatically we collect all the scalability related information and we can otherwise investing in like 100 phones whatever is not an easy thing for us to do so this is something that came about out of this so we have used it so far in real quizzes for five courses four in CSC in fact it's been used by Professor Kannan Maudhigalya in his course almost every week these are real quizzes that count towards their final grade we have also seen some interest from some companies for their placement as well as other but they haven't really used it yet so we are trying to push for it so now let me go back to the original problem I started all this saying let me solve the problem of engineering education now let me try to pull together both the tools and explain how we intend to do this is our current plan our plan keeps changing based on whatever hurdles we experience so it's a forever changing plan the current plan is this so in our ecosystem there are basically three players industry which feels the pain of a poor workforce and then there are this academia which consists of both content developers as well as students and teachers so we are thinking that all these players can interface via a non-for-profit company or for-profit company whatever works out easier in the longer run so the idea is the following so we'll currently focus on five to six core subjects in CSCE because these are the top engineering most of the engineers belong to these disciplines and we will involve the industry from the beginning to ensure that the syllabus that we create kind of meets their needs so that way whoever you are producing will satisfy the needs of the industry of course as academicians we also want to emphasize on concepts so that is also something that we will do once we create the syllabus in cooperation with industry then the content developers are going to create the multimedia books based on the syllabus and then the company is going to do this outreach where it will try to push for adoption of these multimedia books in local engineering colleges as well as target the students directly and then because as I said examination is also very important process so this company will also conduct this high quality exams a bit like gate and also a bit like a university model but this are core specific exams it will do pan India these exams are said by content developers are competent in teachers it can in fact be vetted by the industry also to check whether it's all being fine and this conduction of the exam could be via safe to scale it but it could also happen with local engineering colleges with their support whatever that logistics have to be figured out so we through safe means objective exam because it's easy to grade but we as again instructors we like subjective exams also because not all things can be tested by objective but objective exam could be a filter for that I mean if you didn't even clear the objective exams and forget about subjective only those who have good performance in objective can be given a subjective test that has to be manually graded for which we'll again try to rely on the student workforce or local local engineering teachers or whatever it is to grade so the certification is also provided by the company it could be per course or bunch of courses there's actually a grade assigned to it again here we can work with local colleges to include this in their college transcripts or you can do it independently but more importantly to close the loop so that everyone buys into the system this will also run a placement cell in fact a Pan India placement cell where it's trying to put together the industry needs will be conveyed and it'll put the students in touch with the industry so this is the Envision Plan I mean we are trying to see if it will work out by talking to different people so that's where things are okay quickly so let me so finally I've told a lot of stuff so that is more or less the life cycle of a development project I told idea how we implemented it how we did field trials and then we got some feedback we refined it and we are currently in that wider adoption phase so that's how things are in the process what is it that we have learned so the major hurdles that we see in our system are that these projects typically span many years because there's a lot of stuff you think they'll finish but many things get added they become complex in fact as you go along but we work with a moving student population so some work only for once and some work for a year rarely do we get anyone who works more than a year and project staff we haven't been able to hire good project staff even project staff they also live within one two years it's not that they are sticking around for that long the problem with this is that most coders are inexperienced I mean they are also still learning in fact we offer this as a learning experience for them like do this you learn about a lot of these things so the end result of it is that you get inefficient code buggy code poor documentation they don't follow any coding standards and when you are trying to use it a lot of issues this tends to create the number of work hours also if it's an R&D these are good students work these many hours I mean of course there are students who may not even work these many hours but R&D means they'll work 6 hours a week BTT means 612 MTT is 20 to 30 rarely 40 so this is how it is and you may think we are dealing with students we are in a position of power they can be quite unforgiving I mean if they feel that the grade is getting affected because you are experimenting with all these new things they are scathing in their anonymous reviews if not directly on your face so one has to be either you will have this unforgiving end users where they expect polished products buggy products or you will have this clueless users which we have seen in some of the teacher training is they are very not tech savvy so then it becomes a customer support kind of a thing that you have to provide so you better be very user friendly such that you can avoid all those kind of technical support you have to provide for them and not just all these users will come with different devices, different operating systems, different browsers, different internet speeds these are also lot of things we had to handle as part of the project so what are the lessons that we have learnt faculty can definitely take up projects without knowing or I am like an example of I don't know anything about Django I don't know anything about react.jl I know very little about android or python but still I have been able to manage these projects as well as many other projects I think it mainly is possible because you need to ask the right questions so whatever logic they want to do you need to vet it through some flow charts and whatever code they are doing you need to ask questions like okay how many by the way this is all composed experience it didn't start out like this these are all lessons learned now I insist but I didn't do it before how many database queries are they making how many requests are getting generated what is your execution time all these and you have to think of all corner cases and test it I mean the burden is on you so you have to be ready to do all this. Another thing which has been invaluable in fact anyone who wants to do this kind of work I will insist that they should invest in a software management tool. The kind of tool that we use is from Facebook it's open source sorry it should be Facebook it's called fabricator it's basically a wrapper around a version control like git or bit bucket the nice thing about this tool is that it imposes a coding standard all this documentation or indentation or naming conventions everything it will ensure that way in this lobby code you cannot push it also permits peer code review you can assign it like this can be pushed only someone else approves the code that is also something very important it also supports this unit testing it's kind of painful to write them but in the long run it's important because if something failed when you run the unit test you know exactly what module it's not working so it's for module people code they have to specify how to test that module also those are the unit testing so it will help you track bugs it has a chat room it has Viki all this tools it comes with this is something which is worth investing in so again we didn't really think of scalable when we started it was like a project we thought okay fine we'll do something about it but then our final goal in many of this projects is that we do want it to scale it gives scalability enough weight from the beginning it will come and bite you I mean we did both the tree and now we have there are many times where we thought let's crap all this code start from scratch just because it's turning out to be so messy because we didn't pay enough attention but then it's not easy to do it because the 50,000 lines of code with lot of I mean it's known as there who is so you have to work with the code so from the beginning you have to pay attention to scalability and logging we have tried so what we are now insisting is like there are a lot of tools which you can use during coding to quantify your performance and push only when it meets a certain bar again as I said for say for example we in fact ditched our previous implementation we have started from scratch based on our experience and we are recoding it there are a bunch of students I have assigned certain pieces of code to abandon it because there wasn't enough reason to use it or temporarily suspended thinking okay if the need arises I'll visit it another thing which again in a research institute I think we have to pay attention to but these type of things publications are hard these are using existing ideas to solve some problems so the community may not think these are particularly novel so it's difficult to sell novelty but there are other venues where you can publish but that is something which you are not familiar with so it takes time to get to know what the community expectations are and then write and publish but finally I think personally I feel it's basically a trade off publication needs lot of rigor it needs lot of effort you have to spend adoption also needs in fact ten times more work you have only so much time you decide whether you want to push for this or push for that so that is something to keep in mind when you are trying to publish this type of work it's doable it's lot of work so as I said for wider adoption we are in the process we haven't yet learned many lessons there but one thing which I have felt in retrospect is had I involved all the players in the ecosystem from the beginning maybe things would have moved on much faster but this is a lesson which I think many people in developmental projects are aware of it but since we were the producers we were the consumers because I teach I also am building it so it was working out fine but if you want to take it out you do need to involve other players from the beginning so this is another lesson quickly I'll just take five more minutes and wrap up and we'll it's already kind of exceeding the time so whatever I've covered so far are again what we call educational tools that went out started to solve that specific problem we have a bunch of other tools simple apps that we are also in fact they are at a state of quite maturity in fact we are at the field trial stage for all of these apps let me motivate this in this particular setting suppose I have this presentation and you all want to have this present like typically in a conference setting someone gives a talk and you want access to that particular person's file right now it's like okay someone will upload you download but this is a method where if you have this app and I just say okay share this file and it kind of goes to everyone so there are usages for that kind of file sharing in classroom settings where I want to share there isn't prior preparation on the fly I just want to share something conference talk settings so the sharing is based on multicast multicast is a very old problem in theory everyone understands it very well but it's not been widely adopted so a lot of standards APs don't support the required functionality needed for it so technically solving this has been very difficult because of limitations of the smartphone AP in exposing some of the apis as well as you run into lot of scalability issues so we are kind of trying to build on such app where everyone within a room can share files in a convenient fashion so this is something we have seen today afternoon in such a room I have the mic you don't have mic you have some questions to ask someone runs around with the mic passing the mic around again this is an app that tries to help that particular problem by so anyone so there is a speaker with the mic there is audience if the audience want to ask a question it's a local Skype call between you and me and it kind of plays out through my smartphone through my mic so that way everyone gets to so there is a floor control that is needed but it's a Skype call that is going via playing out via my mic so this is also useful in this kind of talk settings as well as large classes when there are students asking questions you just do some floor control the instructor can do some full control again doubts clear app is something again that arose in the afternoon so I think Sudarshan was trying to say how many want this question answered so similarly in this setting also let's say the lot of questions often what happens is we randomly pick someone and that may not be a boring question it may not be of relevance to many others so the idea here is while a talk is going on whatever you have a question you type it in the app it goes to everyone then they will like it and whatever is the most popular question is what an instructor can kind of take it up and discuss it with the audience so this also has usage in large class settings as well as conference room settings so these are some of the apps that we are working on I in fact mentioned some social work I knew there won't be enough time so I didn't touch up on that so I will do it some other time so just to summarize developmental projects use current technology to solve some local pain points gives lot of satisfaction in fact I have been quite happy with this work feels great to see your work being used in practice and these can be very technically challenging as well I mean these were not like easy I have learnt a lot in the process I have learnt many things across a variety of topics that's been a very intellectually stimulating exercise as well but they do come with lot of hurdles technical hurdles naturally are easier to cross but there are other hurdles which are like business plans and ventures and whatever are not in your control and naturally I should conclude this with a big world of thanks to the amazing set of I mean I have been very fortunate you know when you take up a project it's a lot dependent on the kind of students who are working on it and I have had consistently good students who have done amazing they wound it up they viewed it as their own baby and worked and I think that is what has led to the success of these projects so very big thanks to many years has been three years every year I have some bunch of students who have done outstandingly well so a big thanks to them so to conclude again thanks if your faculty please consider adopting some of these tools as part of your teaching and if your students please cooperate with the faculty when they use these tools I mean we finally want to experiment with new ideas kind of try to push it to the outside world some bugs you have to put up with but overall we will try to make it a pleasant experience and again if your faculty again a sincere request would be that develop such multimedia books so that we can take it out to the outside world and if your students help us give feedback to such books so that they are in a good shape so that we can take it out rest assured will make the task easy and enjoyable so that I will stop any questions or okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay okay Yeah, I don't know, I haven't really, maybe it could reduce my load significantly. Sure. In Moodula, you know, of the shelf things that I can download and get going on day one. Bodhi Tree is a multimedia book. Requires, I mean, the most important thing is the book. So I have to develop my own book if I'm a faculty member, right? Are there some, you know, I think it might be useful to have some important ways and tools and mechanism best practices about how to go about developing this. Yeah. Because, I mean, if you remove Bodhi Tree, if you remove the multimedia book out of the Bodhi Tree, then it doesn't seem to have that many features. No, no, no. Am I right? No. So, for example, the IER professor who is using Bodhi Tree, he has absolutely no video content. He's just using it in place of Moodle. And it is, I wouldn't know what all, whatever Moodle supports currently is also supported. But there are other things like discussion forum and this, what is it? This report grading if you want or this monitoring in case you are assigning quiz outside homework kind of quizzes. It'll monitor how the students are performing doing auto grading and everything. So I like, I mean, I view this as an advanced version. If even forget about multimedia book, you don't want to do multimedia book, you don't do it. But still, I think it is a more sophisticated platform than Moodle. And which will help you. You don't have to really do. But you are losing a good portion of the stuff. But still, there are enough features within it which will work well as a replacement to Moodle. There are many people who are using, the two people, not many. Two people who are using it outside our department are using it as a replacement for Moodle. They just wanted to try it out. I guess they're the. I would recommend this. For what reason you have a question? So one thing which I don't know. I've heard Moodle supports it, but I personally have never used it is for me to share marks with the students. I just upload a CSV. Each person gets to see his own whatever marks. I don't know if Moodle, I have not used any such aspect of Moodle. So that is something this thing supports. The other thing is the discussion forum also like typically I don't want my content to be on Moodle and then use Piazza. This is like in one single integrated system where you have your discussion forum. In fact, it is per concept discussion forum. Also, like if in a given concept you are covering putting three PDFs and you want people to discuss that, then it will be per that as well as overall. So there are all these things that are there which may help an individual instructor. The two things that you've said it is currently supported by Moodle. Yeah, I know that is what but that's why I used it so I don't really know. The discussion forums which are topic-wise and other things that also is being supported by Moodle with very different features. My question would have been, okay, now this is a very hypothetical situation as if you enter into market and go into compete against Moodle which is highly... If I'm entering the market, I'm entering it for flip-class support. I'm not entering for anything else. Moodle offers enough plugins and there are developers, dedicated developers who are actually working on it and developing more, what do you say, better solutions. So how would you try to compete against them would have been one question. To Moodle, see there is so multimedia book as I said is an organization of chapter sections and stuff. Moodle may permit something. It's also a clean interface thing wherein in a way in fact other than Moodle your question should be how would you compete with Coursera or MOOCs because Coursera or edX because it's 90% or 95% is very similar. Moodle is like for a different thing. This is very similar. Doesn't support for example an interactive video where while you are watching the video, please pop up. The HIP plugin, the questions pop up and all these features are currently supported by Moodle. This is like a three-year-old. At that time, I eat Moodle. So when this thing started, when in fact one of the reasons why we have started building it here was there wasn't anything out there to do this. So edX was open source one year after this particular project was in fact I already used it one course offering. That is when edX was open. So there was nothing out there for me to do. All said and done, there are many systems out there. I don't think when you are doing this business, it's not going to be about a lot of things out there. Finally what it boils down to is how you are able to implement and push it. Features, you're not going to sell based on this. It's more about what business model you have, how are you outreaching, what kind of marketing you're doing, what are you pushing. Those all matter lot more than the specific technology platform. You replace Bodhi Tree with edX. It still will not solve that specific problem that I'm talking about. So this arose at that time because there wasn't anything out there. We were all contemporaries in doing all this. This has evolved with this specific nitty gritty details may vary. But right now I think what is important is how exactly are going to use it to solve that specific problem. This tool or that tool is less important than actually doing it out in the outside world. Computationally intensive modules in your system. So is it just supporting large number of requests or are there other computations? So for example, most computationally intensive is this. So you are tracking the, there is a feature called scorecard which or maybe the progress. Like when you are displaying what marks I've got across an entire textbook, you have to make a lot of database queries to get individual, this chapter, this concept, all marks have you got and then aggregate all of it and then present it to you. So that is one thing. But again, as I said, it's been, whatever, designed by people who don't. These are the inexperienced coders. Maybe there is enough database, whatever optimization you could throw in. In fact, that's what we are trying to do now is to reduce the number of database queries where, like when we did a in-depth study, we saw that for some of it is taking 325 database queries for one request. I mean it's, I don't know. I mean that's how people have quoted and then through optimization, we got it down to 20. So we are kind of paying attention to that kind of level of detail as well. Sorry. I see as a bottleneck is you are saying that the instructors have to customize their course. You know, by pulling together the videos and materials. I don't think our teachers have the capability in the colleges. You have to, you know, give them the whole thing in a platter. Even then they will not be easy. So how do you expect teachers to look at their curriculum and then saying that this course will need these modules, pull it all together. That's why it's quite a bit of, you know, you should have some designing capabilities. No, no, no. So, no, I think you are saying, see, for example, if you take a regular textbook, you have two textbooks in your hand, you go through the table of content. You will say, I will use this section, this chapter, these two sections. And under this textbook, this chapter, these two sections. You're just reading the whatever title. All you're doing here is just the same specification. When you say, I want to create a specific textbook or multimedia book for my offering, it is just giving you a list of the table of content for both. And you're just selecting what you want. And then it will merge everything together and display it to you. So it's not whether, I mean, if that itself is not able to do, then it's a different, you don't have to do it also. Then you say, stick with this book that I'm giving you and be whatever happy that also you can do. But it's not like that they have to create. If they want to create, all you are telling is a way to specify what chapter, what sections within the chapter you want to be included in your offering. It's just a selection. That's it. And the system will take care of everything else. No, no, we right now, as I said, we are trying to recruit content developers to create more such multimedia books. So that's what we are trying right now. Last part. So there is a lot of interaction happening. So like, did you look at interface design or interaction design at some point in other than technical design? There is a bunch of other design things which might be important. The UI design you were talking about. So we have paid some attention to it. But as I said, it is attention from me as a whatever, but I do care about whether when someone clicks something, how many clicks does the person has to click to reach some required information. That level of user in the detail. I have given some attention to it, but it's not like some IDC person. If they do it, that kind of systematic study has not happened. But I do whenever I the friend to UI because I teach. I know what it means for a teacher to the level of depth he has to go or from a student perspective. So again, because we are using it within our own thing, the students who are doing my course in flipped class are the coders. So they know from their perspective what are the pain points and they fix it. You check out the site. I'm personally quite happy with the job that they have done because I also, you know, you could do well, but many things have worked quite smoothly without any hassle and all interfaces are very well done. I mean, some of, as I said, there's a lot of people who have worked on it are these hacking coders like whose bread and butter is coding. I mean, they are immersed in it. So those are the kind of people who have worked on this project. All of them have got placed in like Facebook, Twitter, all like top whatever. They are, you know, they are quite capable. In fact, that gave me hope, you know, because it's not an easy thing to manage, but the kind of students who have worked on it have been amazing. So I haven't had too many issues. And as I said, these people have experienced the system themselves. So they know, I mean, that this is a pain point for me. I'm not able to visualize it properly or it's not displaying properly. They come up with their own ideas and implement it. And I just, they've just run it by, but I agree. Like it's not undergoing a proper IDC. In fact, since they are part of the system, they are offered to do the design, the design, whatever, to check that the design matches the IDC background. So hopefully with their input, it'll improve further. Okay, thanks. Engaging topic and our conversation has started. It can certainly continue, but for the moment outside of. So please join me now in the round of applause for.