 And for those of you who are still here, really, really thank you, you know, really thank you for staying for my talk. I'm going to talk about knowledge extraction in retail or the Internet of Things applied to the retail store. It's a talk about shopping. Some sociologists claim that we people only care about two things. One is shopping. The other, I won't say because we are being recorded. But since shopping is what's really relevant to the talk, let me emphasize. Shopping is procuring objects and objects define us. The objects we buy nourish our body, keep us from nudity, define, project the image of who we want. Objects we buy change, turn our houses into homes. Objects we buy as presents establish bonds with our families and relatives. So shopping from a personal point of view is really important. But from an economic point of view is extremely important. Shopping in macroeconomic terms is called consumer spending, which together with public spending and industrial investment and the housing market are the main drivers of the economy. If consumer spending goes down, everybody panics. So it's important to take a good look at shopping and our research is basically on two hypotheses. First, shopping is a very important industry which is very, very far from efficient. There's a great deal of productivity we can still squeeze out of the shopping process. And the second, there is a great deal of knowledge completely untapped in the shopping industry. At least in digital brick and mortar shopping, in online shopping, which is a model for us, there's a great industry dedicated to applying machine learning and artificial intelligence to analyze every click, what's called click stream analysis of what happens in online shopping. But all of that up to now, it's completely untapped, underutilized in traditional shopping. But first, why does all this have to do with the Internet of Things? Well, very quickly, the Internet was originally designed to put computers in communication with each other. Very quickly it became a phenomenal success at connecting people with each other and people with computers. And that's the Internet we know and love and use right now. Well, the Internet of Things is about a third player in this game, which are objects, things. Most of the IoT Internet of Things we know are smart cities, smart farms, which are basically sensorizing objects and capturing this data into information systems. Well, what we are interested here is in a different kind of relationship is the interaction between objects and people, which is a less studied relationship that emerges from the Internet of Things. And when we talk about Internet of Things, a lot of people think about smart refrigerators and cars and this. Well, we have chosen the most difficult problem, which is the problem of interaction with people and simple objects. Clothes, books, things we buy in the retail store. And we do have a few technologies, the most important being RFID, that allows us to detect a lot of these interactions and capture this data for further analysis. Well, the two hypotheses were that shopping is far from an efficient process. It's actually like a hurdle race. Well, we start by being called potential thieves as we enter the store. You know, there are some gates there that say, we're watching you, we're watching you. And then we see lines. Many times we don't even enter the store because of the lines. It's very complicated to find things in a store. It's very hard sometimes. And then when we find, finally the shelf or the product should be, very often, more often than we would like, the size of the color we wanted is not there. Which is extremely frustrating. If we need to try something on, it's also extremely frustrating. We go there, it's the wrong size. We have to undress, redress, hunt for the size we wanted, go back. It's really, you know, a miracle we end up shopping at all. And finally, lines. After all that, lines. And after all the design of the product, the marketing of the product, the having the store, having done everything, it turns out 30% of the people abandon the shop with the product already selected and ready to pay after five minutes' wait. No wonder online shopping is such a success. Because all these problems don't exist. They don't call you a thief. There are no stockouts. Everything is available. You have plenty of information. There are no lines. Of course it has some drawbacks. Of course it has some drawbacks. Because you cannot feel the object, the product. You have to wait a few days. Less and less anymore. But you have to wait to have your, there is no immediate satisfaction of your shopping needs. But still, you know, the fact that shopping online is so much more convenient is making online shopping go in double digits while traditional shopping, what's called brick and mortar shopping, is declining in fact. But first hypothesis I remind you was it's a very inefficient process shopping in a store, but also there's a lot of opportunities. And the technology behind online shopping is amazing. We don't know as users, but they are analyzing every single click we're taking there, you know. Machine learning algorithms have progressed tremendously. They had to be adapted because they were not satisfied with the statistical learning from shopper behavior. They want to take actions immediately. They are analyzing your clickstream and making decisions on exactly what your next screen is going to look like to maximize the probability of you actually buying something. What's called conversion ratios. Conversion ratios are the ratios of the different steps from landing in the landing page of the store until checking out in the shopping basket. Well, for instance, recommendations. We've heard a lot of talks here about recommendations. Well, in shopping recommendations, it's extremely, extremely important. 70% of Amazon's homepage is devoted to recommendations because 35% of their sales are originated in the recommendations. That means that the jump in sales is plus 50%. Amazon sells 50% more thanks to recommendations. Well, you go to a retail store and none of that is there. No recommendations, no convenience, why? Well, not because there are no technologies that could do that. This is pretty much a green field, a blue ocean of opportunity for us to, if we can, generate something equivalent to the clickstream in the retail store to take all that wealth of knowledge and analysis into the physical retail store and save a lot of businesses and save a lot of jobs in the process. Well, one way to analyze that is not to antagonize online and offline retail models, but to start thinking about a single model. Well, brick and mortar model retail is about putting shoppers and products in contact through a physical space. Online commerce is basically doing the same thing, but in a virtual space. For several years, they've lived completely separate lives. But more and more, through something that's known by the buzzword of Omni Channel, more and more these two things are convergent. And the vision of our research group is there will be a platform that will get inputs of all kinds and produce outputs of all kinds to create a mixed media or a mixed platform shopping experience. We'll be able to shop online, get the stuff in the store or the opposite, shop in the store, do part of our shopping there and get an online order from the store that shipped to our home, start shopping in our phones and end in the store, all possibilities. But in order for these two worlds to merge, the customers we should find in the real store something similar to what we find in the online commerce platform. And this is basically a lot of interactivity. Online shopping is really interactive while a traditional shopper is far from interactive. We should have a personalized shopping experience which online it's getting more and more personalized and in traditional shops it's completely anonymous and also contextualized. Everything that happens should be about what we're interested in. So these are the different technologies that create this interactivity personalization and contextualization of shopping. RFID is a technology, cheap technology that by adding a five cent antenna and chip to the label of the products or to the products themselves allows the product to make its presence known in any space. That's simple. A shirt, a book, a pair of shoes, anything can make its presence known in a space. And that announcement of the present triggers events that until now did not happen. So we don't need a very complicated technology. All we need is a certain item to make its presence known. RFID is a technology that identifies every item, not every reference, not every class of products, but every specific object. So we can track the life cycle of that product from manufacturing to stocking in the store, buying, maybe returns, the whole thing. So we're talking about life cycle. We're somehow giving the concept of life to objects. We're making objects active. The real goal, what we really wish we could do and we're only starting to accomplish the technology is we would like to click in the store. We go to the screen online, we click and we find information. We click, we find opinions. We click and we find recommendations. Well, we touch the shelf and no matter how much, nothing happens, right? Nothing happens. Well, we have not completely solved that problem yet. We're only starting, you know, some only half satisfactory solutions to it. But although we don't have a solution, we have a name for this. We have coined the Pormanto term Cric. Cric is a combination of brick and mortar clicks. So Cric is the abstract term of being able to point to an object and create events, interactive, contextualize, and personalize events about this object. Some of our early attempts have been using augmented reality, augmented reality and RFID. If we put RFID antennas and RFID labels on the objects, the shelves know what objects are more or less where with a precision of 25 centimeters. So if we point an augmented reality software to the shelf, we can map the screen coordinates to the shelf coordinates, ask the shelf what contents are in that particular position, and then find information about this product and create whatever interactive experience is called for in that particular instance. Well, that pretty much works. It has its limitations. But as always, you know, limitations are not so important if you really need it. Who really needs that? Well, people with physical mobility limitations. If you are in a wheelchair, you cannot browse a shelf. You can go to a store, you know? The chairs can do pretty much everything. They can drive, they can take the bus, they can access any building, but as shocking as it may be, they cannot shop independently because anything that's outside arms reach, they cannot browse, they cannot find information. So they need somebody to go with them with the consequent lack of privacy. And this is a product that after certain surveys of people with mobility limitations, you know, they have said, yeah, I would like to go by myself, not with somebody next to me. You know, that's watching over my shoulder what I am doing at every moment. So either with a handheld or even a lot of people with a bound to a wheelchair have also serious arm mobility problems. So with Google glasses, and we've been able to fix the gaze of these people in a certain position on the shelf and give them information. That doesn't completely solve the problem because at the end, if they are interested in something, they need help to get it. But at least the browsing part of shopping they can do by themselves. And the obvious question is why don't online, all they do, they do all the time, but if you talk to them, they say, yeah, but I want to go to the store like everybody else and they have the right and it's understandable. Well, now, after once we've learned and we keep working on this better and more efficient and more usable implementations of creaking, I'm going to show you a lot of different strategies we're following to capture extraction. We are in the phase, sorry, to capture information, to extract knowledge from the store. We are in the phase of solving the problem of capturing the data. Eventually when we have all the data available then we'll worry about what to do with it. But that'll be a simple problem in the sense that online commerce is already working on that problem. So our idea is to provide from physical shopping in the physical store the same or similar levels of information that we have from click stream analysis and then machine learning algorithms, recommendation algorithms, all these things can be run on this data. But extracting data from click stream from a mouse click is a completely different problem than extracting data from people shopping in the store. And there is a lot of hardware and a lot of software that needs to be developed in order to obtain that information. Well, there have to be a lot of solutions to coding. It's very important that the right number is given to every tag. But the first problem is knowing where things are. That may seem like something that's not necessary. Don't stores know what they have and where it is? Well, they don't. Or if they do, they know at best with 80% accuracy, which is far from what they need. If Decadlon wants to compete with Amazon, they are going to be sourcing online orders from the stores. So 80% is not sufficient. They need 99.9% accuracy in the information of what they have and they need to know where to get it. This is why, for instance, Decadlon, Inditex, Macy's, Marks & Spencer, Tesco, all the big retailers already RFID tags in all their objects. We have several solutions, but one solution that is actually relevant to the minds to context because it's a solution that will require cooperation with other groups in the department is the idea of a robot that will go around store capturing information about inventory and location. The question is what products are where? This is like the basic information we need for anything else to happen. And the fact is that this is far, far from solved right now. You can go around with a handheld either with a little antenna and electronics detecting the tags, but you're in a huge department store that's a task that goes beyond the capabilities of humans. They get lost, so you can send a robot. Right now we have certain limitations with the robot. We need to provide the robot with a map. Once we have a map, we need to tell the robot more or less what path to follow. Of course we would much prefer a robot to be left in an unknown space and the robot first to explore the space to create a map and second to plan its own path along the map. So we're talking about exploration, planning, all things that if you've been today in these talks you know, ring a bell. So making the state of the art in robotics for inventory there are only two or three companies in the world that can do that. One is here, we're collaborating with them. But the state of the art is that the robot cannot plan. So we are trying to push that state of the art beyond its current limits. Okay, then we have the smart shelves. We've already talked about that. The smart shelves of course provide inventory. We are also applying machine learning techniques to detect when a customer interacts with an object. The augmented reality creaking technique works but it requires a technology in between. The ideal would be we take a book from the shelf and immediately something happens. Well it turns out it's a difficult problem because RFID to read that the shelf continues to be identifying that book and there are changes, subtle changes in the power and phase parameters of the RFID detection that can be inferred as being an interaction with that. And we have applied the classics, support vector machines and regression and different techniques and have obtained good enough results. It's good enough to publish. Good enough to publish. So I think we are pushing in the state of the art of interaction detection using RFID. Shelfs also displays. I could get into the importance of displays. Displays are more for brands and they are business changing but let me continue. Then there are other interesting cases like for instance the fitting room. A place where the presence of a certain object can trigger an interactive contextualized experience is a fitting room. And there are systems already developed which when a customer enters a fitting room the products that she or he is wearing appear on the screen. The first reaction is what a coincidence. The marketing content is exactly what I have. Maybe the second time or the third time they will say maybe it's not a coincidence. It's not a coincidence. This is another way of quick-cricking. The mere presence of a garment inside a fitting room cricks an event to happen. And here we have an opportunity of showing opinions, showing recommendations, showing but the key here is not to expect a customer to do anything. A lot of people say why don't you have a barcode reader and you read the barcode. Then you require a very small intervention by the consumer which means that 99% of the time it won't happen. We are lazy. People are lazy. So it has to be completely automatic. An RFID allows that. Well we want to determine things like an object leaving the store or entering the store. Well it turns out these problems seem are easy and look easy but turn out to be extremely difficult. You require to determine which is a problem that we are currently working on to determine whether a product enters or leaves the store. You require a combination of face to ray antennas and certain algorithms. We're trying machine learning, we're trying more simple algorithms but it's very hard with a high reliability distinguish between an object that is leaving the store entering the store or just sitting next to the door. Because even a 1% false positive means that the mannequins next to the door are going to be creating false events all the time. So we're going to need parts per million in that detection. We want that to be useful at all. Getting to 90% is easy, fairly easy. Getting to very accurate results is very, very challenging. Well also the payment is very important because at the end at the end online measures through clickstream all the conversion ratios and conversion ratios are finally in the denominator we have how many of these initial shopping processes ended up in a sale. So if we had the payment point we detect, let me just show you a video it's not very from a technology point of view this is not very important or very interesting but from data analysis point of view is crucial to detect when the item leaves. So right now we are able to sit with the big retailers, with Inditext and with Deconon and say ok we have the technology that allows you to know that particular product that went that checked out at a certain time did it go through the fitting room or not. In exactly what shelf was it sitting so we can measure the productivity of every single fixture or part of the store we can assign Euro value in sales per hour for every shelf and you can understand this is immensely valuable to shoppers we can also we can calculate conversion ratios of fitting rooms, they have no idea now they have the fitting rooms because they have to help them but they don't know how many sales if fitting rooms are increasing the conversion ratio maybe they need to put twice as many as they need to close half of them they have no idea now because they are not measuring anything so see how a simple technology can extract so much information some of it might be easier to analyze, some of it is much more difficult to analyze and requires much more complicated algorithms also after payment checking what happens in the door is important too if you are more interested in our research in detecting objects coming in and out there is an entire industry dedicated to analyzing people coming in and out which turns out to be also a surprisingly difficult problem because the light of day the shadows, the lighting counting how many people what's called footfall counting how many people enter the store should be an easy problem well it's not but people that know about it tell me it's complicated there's no more slides I guess well there are more but it's late so let me conclude let me conclude the idea of our research basically it's an ambitious it's an ambitious goal we have it's an ambitious goal we have and not a lot of research groups that are doing this problem it's basically developing the systems made of hardware and software algorithms that will extract the data sets from consumer activity in the retail store that will allow an analysis of the same level of affecting business and business impact is available to online stores and that starts in knowing what we have and where we have it in the stores and just to illustrate how these apparently simple systems problems are actually hard to solve is just to know what products and where they are in a store we need to develop a robot that explores, that plans that applies sophisticated algorithms to from the antenna readings honing into the position of the objects or another simple object and another simple data set is interactions that happened in a shelf we need to apply machine learning algorithms to the different features of the RFID parameters such as power read counts phase in order to decide whether it's just random fluctuations or it's actually an interaction and happening because even we have the products sitting there all day even a small percentage, very small percentage of false positive is going to be creating false events all day all day the problem is complicated because of the level of precision F factor of 0.8 that's not nearly that gets you a paper published, in fact it did for us but it's not nearly good enough for actual use in an actual retail well, thank you for your attention thank you for being here can you hear me? I was wondering about the role of humans that work in the shops in this technology because that's something there's no online shops where there's no human being present in which way can maybe integrate well first, the change in the role of humans in the shopping it's part of the reason why we're doing that it used to be people working in stores were knowledgeable, had been working there for a long time and were actually helpful well, that is part of the past they are just most of the time people working or are just human mannequins they are just hired to wear clothes and to be part of the decoration I'm exaggerating a little bit but there's a part of truth in that exactly, but these people they need some tools so all these tools they can be targeted to the customer but it turns out that in many cases they are targeted as tools to help these people are called associates in the retail lingo to help associates do their job better without a need for training or long term experience for instance if just give you an example certain customer enters a fitting room and that person is going with a certain garment then that event triggers an order to one of the associates to say this person would probably like to try out the associate goes find it takes it to the fitting room and it turns out that person ends up buying that the customer is happy the associate is happy because their variable pay increase their commission increase the retailer is happy the brand is happy it's a win or a round situation it's a little bit like creating cyber associates associates that are not very smart or very experienced or very good at what they're doing but they have the technology that tells them exactly what to do when to do it and where to do it so all they need to do is execute that and they create more sales and they do their job better and that's actually happening not so precise RFID technology is so good that it's almost magic that's a reaction that people have when they see it working wow and there was a time long time ago when RFID we were worried about not detecting enough now the problem is exactly the opposite our problem with RFID technology it reads too much too much it's too good so we need to write a lot of software code to tell from the interesting reads to the stray reads as we call them for instance techniques for an antenna overhead a fitting room it's also going to be reading from the other so we need to compare the reads from the different antennas apply algorithms and filter and make decisions decision making based on this product cannot be in two fitting rooms at the same so which is the most likely we use probabilistic algorithms to decide in which fitting room it's actually so RFID technology being too good and about price is incredibly cheap you have a attack that costs 5 cents and it has a radio frequency transceiver in it that can run four sessions at the same time can be talking to four different readers at the same time to me it's mind blowing you can do that for 5 cents thank you