 Hello, so this is some work. I did last year what I was doing a contract at the University of Montreal at the group, the Russia Access Information. This is a joint group with Jing, that is over there. And so this is a research challenge. Research challenge, yes. Research challenge is very, very exciting things. Imagine your boss comes to you and says, well, you have a month and a half to program this thing, but you don't really know how to do it. And not only you have to do it, but also do it in a way nobody did it before, so it's interesting and you can publish. So it's a very, very frantic. It really pushed you to write the words called you ever wrote. And one of the things that I have learned with the open source world is, okay, don't be ashamed of your worst code and even like in general, people who write this type of research challenge, they never share the code because it's horrible. So you have it all in GitHub now, you can see it and say, oh my God, this guy Pablo and Beijing is better. But Pablo is like terrible, how can he write this type of stuff? But okay, so the task itself is you take one query that come from real search engine query logs from Microsoft Japan, actually, Microsoft Research Japan, and 200 web pages rank from the real search engine over that query and you have to get all the relevant information in a thousand characters. So you are summarizing 200 pages in a thousand characters. And to simplify the task a little bit, the queries belong only to eight set types. So for example, celebrities, location, how to, things like that, but they don't tell you which type is which for each query. Okay, so here is an little example. The query is Whitney Houston death and the examples they provide as they never give us an output, they give us the evaluation queries. So the way it's evaluated says, well you have to return something that has this good nuggets of information like February 11, 2012 for Suite 434 and you will be evaluated like that. The results haven't come up yet, so we still don't know how bad we did, but we know we didn't do that well. So here are some of the actual queries that came during the evaluation, so you have a little better idea of what the task is about. So for example, Ron Paul Tea Party or Markingay Influence. So you really, it's your guess, my guess what these queries are really about. Yes, I mean, if you talk about Nancy Pelosi, okay, you want a little bio, but if you talk about Ron Paul and the Tea Party, well, do they love each other? Do they have tea every day? I don't know. So our approach is before moving here, I was in IBM Research and helped build the IBM job body system. So the intention here was try to apply the same deep QA architecture to the one-click task. And one of the main things of the deep QA system was not to use type information explicitly. So one of the big changes of normal question answering systems is, okay, if you say who was the president in 1978? So you will start saying, oh, you're asking for a president or you're asking for a person. One of the big things about the job body systems that doesn't do that. That's why it could fail miserably but could also use more evidence. Okay, so the approach on Hunter-Gatherer is a simplification of the deep QA architecture where you hunt for text nuggets and then gather evidence for those nuggets, score them using the evidence and try to assemble a final output using the highly score nuggets. So here is what we mean with hunting text nuggets. You get a passage as returned by a search engine on top of these 200 pages. And then we chunk them in a smart way to find which things could be relevant pieces of information for this query. Then we actually do recursive searches where we take the original query plus the nugget, like Whitney Houston depth suite 434 and gather more extra passages to use as evidence. Then we score these things with a variety of scoring methods and we find the sentences that contain these highly score nuggets. And we wrote all these in a month and a half. And because we wrote all these in a month and a half, the output looks horrible. Yes, so this is for one of our outputs and this was the output that I thought was good enough to show. And imagine for a month and a half I spend looking output like this daily until my brain bleed internally. So one of the big problems is spam, yes. I know for many people it's not a problem, it's a way of living, but for us that trying to filter these things, it was very, very difficult. So we are looking for Hillary Clinton first lady and we start talking about North America, Turkey, first ladies, visit airport, jeep, blah, blah, blah. So we didn't go in time to include a component that filters things that make no sense and their libraries that do that. But around here it's targeting nice like Clinton was elected to the United States, becoming the first lady elected to public office, et cetera. So we will definitely score points from this one. The other ones looks even worse. Okay, but the important part is why Python? Yes, so I believe that Hunter-Gatherer really goes back to the duct tape language origins of Python. We have two people working very closely together under a tight lead line, integrated in a large number of existing tools and libraries. We use Indrid, there's a research search engine written in C++, NLTK that was presented by one of our speakers, a CCL parser that is a C++ supervised parser, GLPK that is what we're speaking today, mallet that is a Java machine learning library and boilerplate to extract tests, the text from pages plus others. Very, very exploratory coding. And the most important part that I want to bring about the goodness of Python is we have absolutely no documentation. All code, the code was the only documentation we had. And from that perspective, Python was amazingly successful. Besides that, Jing is from China and from Argentina, but we could understand the code much better than we understand speaking each other. So the case study, the thing I wanted to share with you today that I think is very, very nice you could use it for your own work, this library called GLPK. Yes, the state of the art summarization, a state of the art summarization technique uses integer linear programming and express the selection of sentences for a summary as an optimization problem. So if we have N nuggets and M sentences with some sentences containing certain nuggets and we have a score for each nugget, we want to select sentences up to certain length so to maximize the scores of the contained nuggets. And there is this very nice library called GNU Linear Programming Toolkit, GLPK, GLK. And it has a small domain specific language for linear programming. So the problem we have is expressed in GLPK that way. You say, I will have this metrics that tells me which nugget appears in which sentence. I have the length of each sentence and I have the weight, how good is each nugget? And what I'm trying to find is these, which sentence will make the output? Yes, and I'm trying to maximize the sum of the scores of the sentence and make it to the output so that the length of all the sentences is less than the total length and everything, I mean, all the variables are saying. So on the good side of the Python bindings is you don't need to execute the program outside Python but the Python bindings don't do much more than executing the program outside Python. You do have to give a file that contains the whole program inside. So you will need to do a lot of printouts that produce that file. But once you execute them, so you give a file name that contains the file, then you say update solve. And from there, you can read this constraints object. We'll give you back, where are you? Contrains.s contains the array s inside the matrix and you can manipulate it from Python. And some concluding runs. Okay, so the first thing is, if you go to the code and look at line 100 of parser.py has this function called mixed brackets. This is one of the most horrible functions I ever wrote in my life. When the first speaker, talking about LTK, was saying, well, gate lets you do a notation. So around 2000, the NLP community got together and decided that changing the text while you are doing NLP is a bad idea. You want to do standoff annotations. You want to let the text fixed and say, oh, between this offset and this offset, I'm going to put this annotation. And LTK takes us back, I don't know, 15 years and just give you back list of objects, list of strings for many tasks. So when you want to run two tools over the same text, you will have to mix this list of strings. And this list of strings that have a change in subtle ways, like capitalization or a dot attached to it is basically, you have to do a sequence and alignment for biology to solve the mix in the two things. And so that's there in a very ugly format. And the other thing is I really come from more the Java world and Python is not bringing the good out of me. I wrote the whole code with no types. I was a return list of tuples. And then poor Jean has to read it and make sense out of it. I'm really proud of the guy. But it's a little bit of, I still need to get more into really how to write Python in a way that is not noxious for the world. But overall, I'll definitely use it again if I end up in another of these research competitions. This is my six or seven research competitions so far. I always say when I participate one, I'll never do it again. But well, and take a look at the code. This is definitely a non-trivial piece of research code. If you manage to run it, I will be very surprised. Thank you very much.