 Thank you for the introduction. Can you all hear me cool? So I'm really glad you're all here today, and I will tell you a little bit about touchable visualizations now So it really ties in neatly with the whole responsive topic as well And first of all you might be wondering. So who's this guy here and a little bit about myself I'm my name is Dominik as power and I'm an interaction designer and as you can see I come from the beautiful city of Munich, Germany Which is known for the Oktoberfest and not much else I guess But as you can see on this picture especially is Munich is very very fruitful for intellectual endeavors so I spent a couple of years in academia before doing my own thing as a freelancer and I worked in information visualization and Of course the German accent helped with that career choice and as Irene already mentioned I mostly worked in personal visualization. So what do we do with all the data that's collected about us? How can we make sense of that and bring it into a format that every one of us can understand and Here's one example a project that I did with Frederick Seifert and last history here takes a music listening history and your photos and your calendar entries and It merges all of them into a single timeline and this way you can explore the music that you've listened to and put it into the perspective of your life at that point in time and It lets you reminisce based on that and it even builds neat little slideshows out of the photos that you took at a certain point And the music that you listen to so it's really fun and You can go check it out. It's still available online for free even though it's Mac only sorry Another project that I want to briefly touch on is virtual projection and here We thought about the problem of small screens on portable devices because they're really really small if we've as we've seen in the previous talk And this is one issue that's not bound to go away. Unfortunately, even though phones seemingly no no limit when it comes to growing display size But the idea behind virtual projection was to provide an easy way to transfer and from information from a handheld device to a stationary display and What's easier than having a projector built into your phone? Here in this project however everything is just simulated so the phone and the display act as if the phone had a built-in projector so the neat thing about that is that the whole thing works with regular iPhones and TVs and all the magic only happens in software and As the mobile device and the stationary display are still coupled and everything simulated you have full control over the results So you can for example use your handheld device as an interactive image filter Just by moving it in front of the of the large display and you can also have these little Magic lens effects here where you see a satellite image on the handheld and you see the regular map on the large screen Okay, so that was some of my research work but now I'm really excited about all these new fangled touch devices that have appeared in our backs and pockets and The chances that they give us especially from a data visualization perspective Okay, so quick survey just to get you all awake again after lunch who here owns a touch control device of any kind Okay, wow Okay, but Second question who here has already developed some type of data visualization specifically geared towards touch-enabled devices. Oh Few of you cool. Okay for the rest. I'm here to fix that So I hope that after this talk you have a clearer understanding of what you have to be aware of when you're designing for touch Okay, so let's start at the start and this is probably gonna get me in trouble But I'll do it anyway. So what is data whether it's small or big or whatever Well, first of all data means facts provided or learned about something or someone So it's information on an object a person process, etc and so while we only recently started raving about the power of data the concept has been around us always because Data is information about our surroundings It lets us find food and shelter and spot predators before they spot us and so on But how do you gather this data these information about an object? Well easy through visualization, of course and Conveniently most objects provide us with an instant visualization of their properties So just looking at this avocado here already gives you various infos about it like its size whether it has been damaged and so on But in nature it's mandatory to gather as much information about our surroundings as possible to increase the chances of survival Of course, so we're not only relying on a single sense namely site But we're using all of them everything that we have available and also most objects don't really tell us everything Simply through their looks so in this example here if I asked you to tell me whether this avocado was ripe What would you do? Well, I guess you will probably take it into your hand and gently squeeze it and you could feel if it was ripe so This is what I mean with touchable visualizations So visualizations that you can manipulate using only your fingers that you can examine turn around through touch But also of course do your regular data with stuff like filtering highlighting and so on And as we've just seen it seems that supporting touch in your visualization will soon be a necessity as everybody uses touch control devices So as visualization designers, what do you have to be aware of in this case? So first touchable visualizations are about interaction So there's no need to panic if you only produce static images because they still work fine on tablets Just display them and then we have these pinch to zoom gestures and so on to interact with them So that's perfectly fine. Just treat them as images and you're all set Second even for interactive visualizations touch support so far is mostly optional Because often it's enough to just enlarge these buttons a little bit make them easier to be touched and you're all set and While this explicit touch support isn't needed in I think it's definitely wasted potential if you're not thinking about that Because your visualizations could become much more engaging and interesting when you're being aware of that So Here's a recent study by Drucker and colleagues from Microsoft research Which was published at CHI this year and what they did was they compared two bar chart Visualizations for tablets one mainly controlled through gestures and the other using the regular good old button control panel And the results of the study showed that not only people prefer the gesture controlled version Which was not too surprising But their task performance was much better using gestures than buttons So that's one thing to think about and if I haven't convinced you yet You still need convincing hands Rosling here certainly looks like he would love his visualizations to be touch enabled So think of him next time you design something Okay, so anyway, what are some examples for touchable visualizations? What's what's out there? What can inspire you? So one of the first apps to integrate smart visualizations into an app was the companion app to Nicholas Felton's datum and Datum is a live logging service that lets you track anything You can create your own items and categories and use it every time you do or eat or watch or think something and The mobile app is of course a great addition to the service as you have your computer with you always pretty much But the app even lets you access and analyze your data. So The way this looks like You can see that here so we have a set of categories and if you if you're interested in coffee, for example If you want to see your coffee consumption in this app You arrive at this screen here and you have this little mini map at the bottom there that you can drag around to navigate within the data and Then you see these stacked graphs appearing there at the top That show give you an idea of the amount of coffee that you've consumed and then you can tap up there to read the actual values And I think it's in milliliters or something. So don't be alarmed And that that's a really neat app. It's great to use and it's a lot of fun Another example is high coup deck and high coup deck as an app for the iPad that lets you create presentations and while it of course has bullet points in there like every great presentation has to a Great presentation also needs charts and you can also add them in high coup deck and the way this looks like is You can select a certain chart and of course you take the donut chart because it's the most powerful chart available and Then you you arrive at this interface here that lets you give a name to your chart and so on But what's really nice is that you have these little These little handles there that let you adjust the data in real time so you can touch on them and adjust that and You can also add more sectors by tapping the plus in the middle and One thing that might be a bug or a feature. I'm not sure you can't create a donut chart That has more than a hundred percent which might be really annoying for people from Fox news for example, so Yeah, anyway, don't use it if you're up for something like that Okay, so these were two examples from commercial products And they of course come with their own restrictions to be practical to have a market and so on and we can find much more interesting examples of touchable visualizations and research Especially as researchers are really fond of these these big-ass tablets that they call interactive table tops and these table tops Which should arrive in all in all of our living rooms any day now? Naturally controlled to touch so we need ways to interact with them But of course simply controlling things on a tabletop is trivial again Just make those buttons big enough the power lies in those much more interesting things that we can do through touch and Here's one example from the University of Calgary that was published at ITS and this visualization here is a reservoir visualization, so it's geological data you have these layers and You can use your fingers to peel them apart and look within that 3d visualization at the underlying layers and you can hold parts with the one hand and Move stuff with the other and you can even You can even peel these layers and flip them like the pages of a book All the while while being in 3d, so this is a this is a really great use of touch I think and it's it's very powerful idea to be able to manipulate your data in that way Okay, so I hope that after these examples you're excited to learn a bit more about interaction design for touchable and Also mobile devices you can't really separate the two cleanly yet There aren't really that many examples of touchable visualizations yet So I will mostly talk about general app design, but the recommendations and guidelines Still apply of course if you if you have touch and visualization in your app Okay So you want to develop this touchable visualization? Let's call it an app and you're wondering what things you have to think about and what do you have to be aware of and The most important aspect and one that you're probably familiar with from your own experience is the fat finger problem So evolution obviously hasn't shaped us for typing on small pieces of glass unfortunately What so when designing for touch we have to take that into account and make these buttons big enough again? And all developer guides suggest slightly different sizes, but they're all around 45 pixels squared or 90 pixels squared for retina this place and so what this means is that if we have an iPad with a resolution of 1024 by 768 pixels then you can't really put an interface like that on there and still expect people to be able to use it So go easy on the buttons Um Also, not only the fingers are fat, but also the hands So if you have smart animations that convey information Don't rely on them being visible because there might be a pair of hands within the site of the user and The screen so that could be occluded and finally handheld devices are much more flexible in their use than desktops and Portrait and landscape orientations here are only the most obvious usage modes So whether an app works with a thumb or requires two hands also makes a huge difference And you've probably all had these situations where you wanted to look up something on Google Maps And you tried zooming with only one hand, which is just not very pretty so The fat finger problem the occlusion problem and the question of usage mode all belong to ergonomics the Limitations that the physical world puts on all people interacting with your app you should be aware of them when designing touch areas and transitions and Of course applying interaction techniques A second set of considerations are contextual ones. So interaction design was really great back in the days when everybody was sitting in front of their PCs Because not much was happening around them But nowadays interaction can happen pretty much everywhere We all carry these little portable computers around and drag them out in pretty much every situation and So you as interaction designers have to be aware of the various situations in which a handheld device might be used And of course, it's not only location It's more and more also the social context who are people currently with while interacting with your app And are they trying to pass the time or are they looking for relevant information and are in a hurry and so on so Context social location marks concern number two for the touchable visualization designer And finally a last thing Again a question for all of you. I won't let you sleep. Sorry. So how do you enlarge this photo? What do you do? right, yeah Exactly you you pinch it you use this pinch to zoom gesture Now isn't that weird I mean when was the last time you pinched an actual photo and what I'm more interested in is what did it do in response? It probably didn't zoom so This is this is a third set of considerations, which is convention So you have to be aware of them people expect to be able to zoom stuff by pinching for example, which is somewhat arbitrary, but it just it still exists and there's also stuff like horizontal swiping for the leading items and Double tapping for zooming in so if you design your app Never ever design counter to these conventions because you'll only frustrate people Okay, so these are kind of the three rock bottom parts that you have to think of when designing for for touch Keeping these in mind is paramount and it leads you to designing a good app But the question is how do you design a great app and that's what this lady here comes in That's a person using your app and to make your app really great You have to put yourself in her shoes. Why is she using your app? What is she trying to accomplish in other words? What's her task? So is she doing some really crazy scientific? data analysis So trying to figure out a trend or development over time or something very complicated and lean forward or Is it more about lounging on the couch and playing around with your personal data trying to relive your last vacation or looking up a song from your listening history, so The the current task that the person is trying to achieve using your app is definitely very important and Then there's another thing another aspect that is especially pressing with touch-controlled visualizations, which is discoverability So this here is a screenshot of figure which is an amazing synthesizer app for the iPhone And it's really fun to create little musical pieces with it But if you look at the interface what parts of it can you press? you probably can't tell me right now without actually using the app and you also can tell what happens when you press them What will you break when you press the wrong button and you might get away with things like that in a more artistic app that? We're pretty much every interaction leads to something interesting, but Discoverability is a much bigger problem for other types of apps Then when you think about the person you're developing for you Also have to know that this app that you're building is completely alien to her when you think back to the issues with fat fingers We haven't really evolved to navigate virtual spaces either Unfortunately, so orientation is also tricky in software. We're freed from any restrictions that the physical world gives us So we as designers can go totally crazy to whatever we like but really what this mainly does is confuse the person in front of the screen and Quickly lost in our virtual world and become frustrated. So orientation is also a very interesting and important aspect Okay, and finally after all these issues and problems There's one last thing that you should never ever forget and it's probably the most important thing Which is let people have fun. Let them play instead of work add things that are unnecessary, but enjoyable to create a really great app and Another more serious aspect of that is be forgiving when it comes to mistakes So don't delete an email draft because of one misplaced swipe for example Okay, so that was probably a lot to take in but you're not the first ones to build applications So there are thousands of inspiring examples out there that you can scrutinize for nice little design ideas And we've actually come a long way since Apple started this whole app store madness in 2008 and really in these five years So much has happened with app design and mobile interaction design that some of these apps that were fantastic back in 2008 Have reached something that I like to call the the geo cities point. So this this effect is something that looked amazing Just a couple of years back as somehow seems a bit weird right now And somewhat outdated So here's some of the best apps from 2008 the year the app store started and only five years ago And they still work great today, of course, but they well they look a little bland. So especially colors are somewhat Reluctant and we have lists. So we have lists everywhere and It's not only the information Architecture and the graphic design. It's also the interaction design itself So everything rests on well-defined screens and you navigate between them and you have all these buttons and Here's some apps from 2013 so five years later They still look somewhat the same especially Facebook and Instagram But the content is very different. So while we still have lists text is no longer the main element photos are front and center and also tabs have become much more subtle and have morphed into more interesting controls and Also, the action no longer happens on clearly separated screen But we have multiple layers of information and spatial organization of that Now I want to show you a couple of examples How app designers have addressed these four aspects that we have before of discoverability task orientation and fun and Probably the most important concept regarding touchable interface phases is direct manipulation and direct manipulation means that on-screen elements are manipulated directly simply by touching them instead of touching a button or using some other external control and I don't know if you've seen these initial versions of the maps application on Android phones because they didn't use Direct manipulation for zooming for example. So we had these buttons there at the bottom and They were really hard to hit and you sometimes hit them Not on purpose and it was it was somewhat frustrating But since then of course Android has also gone to using pinching for zooming which is much easier to use Another aspect that is often used is physicality. So behavior from the real physical world So many apps by now have this drag down to refresh metaphor So that's not only fun to pull it off But it's also really useful and it works really well because you have this huge Touch area that you can use to refresh instead of just a tiny button somewhere When it comes to orientation, I really like ever notes iPhone app ever No, it lets you take down notes and organize them and to keep them all neatly organized They are put into virtual folders and all of these folders are put into Put into a file cabinet here and by tapping the title bar you can access this main navigation and move to other places and You have this any these animated transitions that clearly show you where you are currently so you're never lost in this virtual space but you can always see where you are and which part of the app and you can always go back and Similarly while we had clear transitions between screens before now everything is tied together as in Facebook's app For example where the sidebar menu visibly lies behind its main activity list So it's no longer on separate screens, but they're behind each other Okay, and finally a lot of aspects of apps are geared towards fun. The graphics are appealing certain aspects are simply useless in the best sense of the word and My favorite recent example for that are Facebook Facebook's chat hats. I don't know if you've seen them but in the Facebook app if somebody sends your message they appear as little circles there and You can tap on these circles of course to actually read the message, but you can also just toss them around on the app It's it doesn't fit any purpose It doesn't do anything useful, but you can use the heads of your friends and toss them around in the app. I mean that's Weren't you all waiting for that? Okay, so that is a really nice touch great little thing completely useless and just a lot of fun Okay So now we know about the most important things to keep in mind when designing for touch and we've seen some inspiring examples But the question is where do you how do you get started? What do you start from? so do you just have a couple of ideas and try to tie them together somehow and Especially how do you make something as abstract as data visualization tangible so that people can use that that? this discoverability and orientation for example aren't such big problems and And a good way to do that is to put your app on some solid standing some solid ground Which is mostly a metaphor so One thing that's easy to forget especially since we have whole conferences organized about it Is this that computer interfaces don't actually exist? So they're really here for a moment and you can click on them and tap around in them but if the power has gone for say 24 hours then all of your crazy devices turn back into the the black little mirrors that they are and Because these interfaces are so so strange to us. We need metaphors to make sense of them So the internal workings of the machine are completely alien to us anyway So we take these ideas from the physical world to make them understandable for us and one great definition of metaphors in lake often Johnson's work metaphors we live by and I quote the the essence of metaphors understanding and experiencing one kind of thing in Terms of another which I think neatly sums it up and we can never hope to understand the way that a computer works Or think in the same way and that's why we need them we need metaphors and One could even argue that that's all that data visualization for example is understanding and experiencing data in terms of visual representations But of course when I mentioned metaphor and tablets in the same sentence you might immediately think of something like this year The way we organize our contacts and need little address books and the way we write down notes on little virtual legal pads and the way we play games on little gambling tables with green felt and all and Then you might think of the insult of the day that was invented when fanboys needed a big word to throw around and Everyone soon centered on skew morphism to describe Apple's tendency to overly cling to some physical object for a digital interface Well, originally the word skew or morph only describes ornaments to make an object look like it was made with a different technique For example this great this really fantastic station wagon here that features some fake wood panels as a reminder of the glory days when cars actually had parts made of wood and I think the actual reason that people have become so frustrated with Apple's design is that there's a mismatch at the core of these apps and The source of this frustration is that metaphor comes in two flavors metaphor is for one the visual metaphor so what physical object your app looks like and on the other hand the interaction metaphor or behavioral metaphor so what your app acts and feels like and There's no better way to frustrate people than to show them this but not let them flip the page Especially when that's perfectly doable in another app on the same device By the same software company that looks the same like a book and here you can't flip the pages and you can't in the contacts app So a well-applied metaphor neatly solves all of your issues of task So imagine you're reading a book on a piece of glass. That's that's a clear task Discoverability well, you know how to turn pages of a book and you just do it the same Orientation you're well aware of where you are in the book how much more how many more pages are coming and so on and Finally fun because hey, it's a it's a book on a tablet. Isn't that amazing? so Okay, so these were kind of the the basics of Touch interaction design and I want to make those points clearer with a concrete example I want to talk about a little Project that I did with function Lee and Sheila Carpondale, which is called touch wave and with touch wave We tried to turn an existing visualization technique namely stacked graphs into a touchable visualization And you will see how all of the design aspects that I've explained before come into this design So just first just a brief recap so the basic idea behind stacked graphs The stream graph is probably the most popular incarnation is to take some Area charts that all share the same horizontal axis for example time and stack them on top of some arbitrary Baseline and on top of each other so imagine layer one and layer two here are stocks and the charts show their values over time And by stacking them on top of each other you can also read the sum of them at the same time Which is pretty neat But there's also some heated debate around stacked graphs and whether anybody in their right minds should use them because they have quite some Problems so the main issue is probably how bad we as people are at reading curves and Some of you will probably laugh when I ask you to tell me which of these curves here grows faster a or b Because well, they're actually the same curve, but it's really hard to see and Of course that is a main problem especially for these curvy things called stacked graphs and Similarly, it's very hard to compare parts of stacked graphs visually or make sense of them when there's say more than 30 layers But that's all on that only applies to the static form So interaction neatly solves that as we see in the in the fantastic baby name Voyager here for example So stacked graphs make a perfect target for our touchification of graphs and We of course started with a metaphor and we had this idea of the graph as dough or clay So as some some moldable material that you can shape with your hands and Then we spent a lot of time sketching and playing with that idea in our heads and we had to boil it down to Something that didn't only adhere to the metaphor that we had in our heads But also something that was usable and at the same time overcame the restrictions of the visualization technique that we have picked So we ended up with a couple of general concepts. So the first was Physicality so our graph should become a physical object and act like one which was kind of the central aspect and As you probably know from your experience with physical objects, they don't just appear or disappear They have a certain Continuity to them. So all changes and transitions should reflect that elements just can't disappear and Another thing when it when the touch comes in the graph or the virtual object should react like a physical object to touch So this helps with discoverability, of course But while designing something like that, you also have to think of the interaction conventions of the platform so the pinch-to-zoom for example and Finally as this playroom is scarce and fingers are fat Why not tie all functionality to the graph itself? And we wanted to completely skip buttons and map everything to manipulations of the virtual object or directly touching somewhere else Okay, so here's the result and coming straight from this metaphor of the visualization as a virtual object It was clear that people should be able to directly transform that object They should be able to grab layers drag them out reorganize them and do all of that simply through touch And that's also something new that resulted from the touchable version the option to Reorganize layers in a stack graph, which is very rare in other systems I'm not I'm not aware of any that do that, but it's it's very easy and natural and fun here But then also just mindlessly sticking with a metaphor isn't everything It's also about usability and task and conventions and so on So for zooming and focusing on parts of the graph We relied on the existing convention and used the pinch-to-zoom gesture Of course a bit modified as there as you see there are no parts of the graph Disappearing when you zoom but it it's actually reshaping and you can still see everything so you still keep the context But that's that's still not how dough for example would react to such a treatment if you stuck two fingers in it and Also another convention you can get rid of all these manipulations by swiping horizontally which is again arbitrary but convention and And finally sometimes you need to convey precise information for me for example to be able to compare the values of Layers at a certain point and if we had a device capable of haptic feedback and be able to simulate friction Then we might map that to that, but of course it's much cleaner if To just show the explicit values. So if you touch anywhere on the graph or outside of it these Little vertical rulers here appear and they let you read if the resolution was higher They would let you read the actual values of the graph and you can also use multiple fingers to do that and What's also great? They appear every time you touch anywhere so you can use that to compare parts of the graph just by dragging out the layers and Putting them on top of each other Okay, so much for the interaction in touch wave another challenge that we faced and that I probably should have mentioned earlier when it comes to touchable visualizations is how to implement them as Unfortunately, I mean some of you are probably aware of that our tablets and smartphones aren't really as capable and powerful as our other computers yet, but The first thought that we had when starting with touch wave was of course we were using the web We have all these powerful Java script libraries and it's going to be just beautiful and it's going to run on iOS and Android and PCs and Max and and Google Glass and fridges and it's going to be awesome, but one thing that So we looked at D3 obviously and one thing that we really liked in this D3 example here were these seamless transitions So how this graph is morphing into another one and we thought it was just perfect for our purposes But of course it was meant to be so this video here was taken with my laptop and this is the same thing running on the iPad and If you look closely you can see an animation there, but it's very subtle so It's it's just not the same and So we ended up just writing a native app based on Apple's core graphics and core animation frameworks and those are our hardware accelerated and they perform really nicely and If you look closely you can see these these little transitions of the graph and so on Which is just what we wanted to have, but of course Really sorry for the bad news when it comes to the web So I have a kitten here But I have high hopes that that will change in the near future so we have things like WebGL for example and hardware acceleration in the browser and once that moves to tablets Then I guess we're all set and our beautiful D3 visualizations should work again Okay, so At the end of my talk, where are we going with these touchable visualizations to look a little bit into the future? So will the future of data be little physical sculptures 3d prints of our data that we can actually touch or Will we play around with our food and create data visualizations out of it? And what will it taste like and how many Facebook friends do you need to get a tasty cocktail? Or will we interact with our data and virtual realities and look like completely door like complete dorks while doing it? I guess We will find out. Thank you Any questions? Okay, that's that's a great question. So how difficult was it to develop natively instead of using the web frameworks? um We had a lot of overhead in there. So we had to write a lot for data manipulation, of course and also creates the Pretty much everything from scratch. I am I actually ported the the great stream graph library by Martin Wattenberg and colleagues and ported that to to objective C so I could use it on the iPad So the the drawing was easier, but of course we didn't have much help So I'm not aware of any data of his frameworks for the iPad there might be some but it's not as As great as for the web unfortunately Yeah Okay, I think that's that's actually the main reason that that stuff isn't Working on tablets yet that this hardware acceleration is missing because the graphics chips are actually pretty pretty powerful Even on tablets. I mean just look at the games that you have on the iPad for example so if you could use something like that and Compile your JavaScript to see and run it more or less natively on the machine even if you have the browser layer in between I think that yeah that could help a lot so I guess So touch wave is probably going to be hopefully one of the last native examples and the rest can use the web Sorry, what was the last part? Okay, so the question was how close to the visual and the interaction metaphor have to be to to actually work and That's that's a very good question. I mean you probably don't want to replicate the whole Behavior of a physical object just to adhere to some metaphor But you always have to make sure that the most obvious things that people would like to do with a certain object Should work like flipping a page for example, especially if you I mean the contact contact app is a great example because it's it's so Obviously broken in this regard You probably shouldn't support everything and you don't have to but the as I said probably to do the most important interactive things and Then you have this this benefit of letting people easily discover the functionality So we we haven't in this project here Um, so are you talking about general gesture controlled interfaces? So mapping gestures to any interaction? Yeah, okay So there are some frameworks out there. I think even for for JavaScript, for example, so you can also use that in your browser-based applications Of course gestures always face this problem of discoverability Because how do you discover that you're supposed to draw a circle or a pentagram or whatever? Without anybody actually telling you so I'm I'm a bit torn about gestures to be honest So I'm not really sure if if it's such a great idea to overly rely on them But of course you can you can have help screens You can have little animated tutorials when starting the app for the first time So I think you can definitely use gestures, but probably should go easy on them Actually, we already have webgl on the iPad But only for the ads Unfortunately, so it's actually supported there, but only if if your Visualization or your app is running in this little ad banner in the browser So there is no technical restriction that keeps Apple from allowing webgl in the browser But they just haven't done it yet. I have no idea why But I think it will happen pretty soon. I mean it's already supported on Android tablets So it's just a matter of time the year maybe Okay