 is an artist and a composer. And she works at the interface of social, political and technological infrastructure. She's gonna talk to us about the potential of sound and how it engages with the political questions. So please give a big round of applause to Jasmine Guffand. Thank you. It's, can you hear me? Sorry. Oh yes, now I can hear me. It's an absolute pleasure to be here today and I'm very impressed looking at around at how many people are sitting and waiting to listen to a presentation on a Saturday evening. This is indeed a special Congress. So hi, my name is Jasmine Guffand. I am an artist who works primarily with sound. I also make electronic music and today I will be talking about one project. It's called Listening Back. It's an add-on for the Firefox and Chrome browsers that translates cookies into sound while you browse online. Anyone can get it, download it and install it for Firefox or via the Chrome Store for the Chrome browser. And so the way that this works is that every time a cookie is inserted onto my computer or deleted from my computer or updated, a sound is triggered. Those sounds that you could hear just now were Google cookies and it's interesting to note that the Google analytic cookie is actually the most common cookie across the entire worldwide web and our personal computers. So I'm just gonna load another page. So that kind of stuttering sound that you just heard and that you'll be hearing again during this presentation is actually the limit while my system reaching its limit and that's because browser add-ons were never intended to process large amounts of sound. So when I was in the development phase of this project, a lot of some websites were crashing the browser due to the sheer amount of cookie activity generating sound at one time. So I had to set a cap on how many cookies could trigger sound at once and it's currently set to 43, but it's interesting to note that the Internet Engineering Task Force, which is the de facto standardization body for the Internet, has written a proposed standard for cookies and actually this is a really interesting document. It's more than just a technical document. It traces a five and a half year process in writing a standardization for a rapidly expanding tracking technology and in it they suggest that browsers should be able to host at least 50 cookies per domain and at least 3,000 cookies in total. So that's a lot of cookies and cookies are actually just one of many different online tracking technologies that are implemented through the technical protocol of the World Wide Web. Our personal data is thereby captured, collected, aggregated, compiled and sold. Such data can include our IP address, type of computer or mobile phone, operating system, the add-ons we have installed, our searches, our likes, the websites we visit, what we buy, what we watch and how long our cursor lingers on a page. Some of the lesser known online surveillance technologies include web bugs, audio beacons, web RTC, IP discovery, third party HTTP requests and device fingerprinting. The device fingerprinting method can be broken down into a further subset of tracking techniques, including Canvas, browser, font, audio context and battery API fingerprinting. But when I started this project, I had only ever heard of cookies because our access to the World Wide Web is visually mediated through screen devices. The web browser, as an interface for the World Wide Web, marks a point where technology becomes apparent to the user. Online monitoring technologies such as cookies are largely obscured from the user, making these systems difficult to approach, analyze and understand. By sonifying this largely invisible tracking technology, so that is by transforming the data generated by cookies into sound, I'm interested in sound's ability to interrupt the visual surface and highlight a disconnect between the graphical interface of the World Wide Web and the sociopolitical implications of background algorithmic processes of data capture. So here, sound is functioning as a mediator of the invisible. By enabling the opportunity to listen back to otherwise imperceivable monitoring infrastructures, I ask, what is the potential of sound as a means for revealing asymmetrical relationships of power inherent to surveillance societies? And how does it feel to be routinely monitored? When I use the word feel, I'm interested in how our understanding of data is affected by experiencing it through sound. How does a potentially emotional and bodily experience of data influence our understanding? So sound is a method of knowledge and thinking, or to put it in another way, knowledge and understanding through sensing experience. I'd like to credit the programmer that I collaborated with for this project, Max Breedon and Brian McLeod, who further helped me with some modifications. And Tom McIntyre, who along with Brian helped me get it onto AMO or Mozilla for Firefox. So that actually just happened in the last couple of weeks, but it's been on the Chrome store for a little while. Max suggested that we use the tombra.js library for designing the sounds. So that's the library I used for programming the sounds. And I'd assign specific signature sounds for some of the most commonly used or the largest web platforms, such as Google, Facebook. I think you get the picture. Amazon, YouTube, and some of the third-party cookies that I found to be most commonly used across a lot of different websites, such as the one you can hear right now. Probably most of you know the difference between first and third-party cookies, because they're definitely some way nerdier techier talks at this Congress, but just in case. A first-party cookie is any cookie that has the domain name of the website you're on. So, for example, with TheGuardian.com, any cookie that has the domain name TheGuardian.com is a first-party cookie, and any cookie with any other domain name is a third-party cookie. And that's what you can hear right now. It's a third-party cookie that's constantly updating on my computer via The Guardian website. It's called krxd.net. It's used for targeting and advertising across 5,926 different websites. Persistent and Session Cookies are another subset of cookies where the Session Cookie is deleted from your computer when you close the browser or log out of a session or close the tab. Whereas Persistent Cookies are there for as long as they've been programmed to be there for, unless, of course, you clear your cache. And so this can be anything from seconds to minutes to hours to days to months to weeks. And here are the top 100 websites found using this cookie out of the 5,926. So if you're interested in doing cookie research, I can actually recommend Cookiepedia as quite a thorough database. Now, as I mentioned before, a sound is triggered whenever a cookie is inserted, deleted or updated. But not every time a cookie is read by the browser or website. So the data I had access to for this project was determined by the browser's API. That is what information Google or Mozilla provide to third-party developers. So in this case, it's every time a cookie is inserted, deleted or updated, but not when a cookie is read by the browser or web server. And what was insightful to me was the extent to which certain technical processes are hidden, even from tech-savvy programmers, because these processes of online data extraction are, in fact, well-kept business secrets. Max, who I collaborated with, found a way to hack into the Chrome API and access the data for how often a cookie is read by the browser. But since there is already an extraordinary amount of cookie activity, just from each time a cookie is inserted, deleted or updated, I decided not to use the data for when a cookie is being read, because I also knew I wanted to be able to make the listing back add-on available for anyone to download via the Chrome Store or Mozilla. As it was important for me that anyone could have access to it if they desired. Now, this project has consequently directed my attention to the history of Internet cookies themselves. And from within this recent historical framework surfaces a direct link between the invention of cookies in 1994. The origins of automated online data collection, the commercialisation of the worldwide web and the emergence of contemporary modes of surveillance capitalism. So surveillance capitalism is a term that's been recently popularised by Shazana Zuboff with a book this year, The Age of... And in brief, or extremely briefly, it refers to the online data broker industry as headed by corporations such as Google. In this big industry, big data is a surveillance asset and profits are derived from a unilateral surveillance of users of online platforms in which the aim is to modify human behaviour, predict and control markets, which I think most of us are familiar with by now. And this image, which I love, I should credit, comes from Internet of Shit's Twitter feed. So if you happen to be on Twitter, I can recommend following Internet of Shit because it's not only hilarious but very poignant. And so another important condition to this culture of surveillance capitalism is the agency of technology to monitor in real time, which provides an unprecedented extraordinary infrastructure for radically distributed opportunities for observation, interpretation, communication, influence, prediction, and ultimately control. Now, I believe that the technical and commercial origins of the online data broker industry lie with the invention of the cookie in 1994. And at this time, the World Wide Web was a stateless infrastructure. So stateless is a computer engineering term used to describe the situation in which, every time a website is visited, there is absolutely no recall of previous visits, preferences, login information or user ID. Without a state mechanism to store records of browsing activity, online shopping, as we know it today, would not have been possible. And this is actually a very important point because this is where it all begins. So of course, advertisers were tracking customers before the web, but this is the origins of online tracking and automated tracking. So online shopping before cookies would have been analogous to using a vending machine as you could only buy one product at a time and there would have been no automated feature to remember your personal information. So stateless web meant that every visit to a website was like the first and any commercial transaction would have to be handled from start to finish in just one single session. Presented with this online condition of memory loss, Netscape Communications Programmer, Lu Mon Tui, developed a state mechanism while working on an e-commerce application for North American telecom giant MCI. The $7 million contract was for a new shopping cart application for online stores. They were developed together with John Giannandria over a six-month period. So remember the standardization process took five and a half years, whereas the development of the cookie itself was only six months. Mon Tui named the state mechanism a cookie after a UNIX protocol already in use called Magic Cookie. And incidentally, there are various origin stories going around about the name Magic Cookie. Probably the most famous one is that it was invented by coders in San Francisco who were working late into the night, as you do. And so they ordered in food from Chinatown and with that came a fortune cookie and apparently that's the namesake for the Magic Cookie. But most of these stories have been debunked. So I did manage to trace it back to a 1979 UNIX programmers manual and a Magic Cookie functions as an opaque identifier that is sent back and forth between different pieces of software unchanged. So I still don't know why it's called a Magic Cookie and if anyone has any information about that, I would love to know. So similar to the Magic Cookie, the Internet Cookie is defined by its invisibility. So remember the Magic Cookie is an opaque identifier and the Internet Cookie or HTML Cookie is a small reference file that is sent by a web server to a browser and then placed by the browser on the user's computer. And every time the user loads the website, for example, the browser sends the cookie back to the server to notify the website. So a cookie is like a reference number or ID that travels between web servers, web browsers and our personal computers. The Internet Cookie allows web servers to identify users without disclosing what and how much information is being collected. Its content is therefore at the discretion of the web server. Inherent to the technical protocol of a cookie is a lack of transparency that aligns knowledge and power in the hands of their creator. And importantly, the cookie provided a means to reliably implement a virtual shopping cart. So cookies literally gave birth to the virtual shopping cart. And significantly, for the first time, a protocol for online automated data collection. A technically termed Persistent Client State Object and Lumontui Paint-Into-Dit as such. So as you can see, inventor Lumontui currently sign in Facebook. Cookies transformed the worldwide web from a system of discontinuous visits to a culture of persistent connectivity. Version 0.9 beta of Mosaic Netscape was released on October 13, 1994, and supported state management. This was followed by the commercial release of the Netscape Navigator browser in 1995, which introduced cookies silently as a default setting. Now, I don't know about you, but I do remember the Netscape Navigator browser. And someone's put together this video. It's been programmed to be on your computer. So it wasn't until 1996 that the privacy implications of cookies first became known to the public. And I should note here, this was very much due to the standardization process, because they were raising concerns about the privacy implications, particularly of third-party cookies. And this led to some newspaper articles, such as the San Jose Mercury News headline, Web Cookies May Be Spying on You. The extraction of user-generated data in exchange for advertising revenue has subsequently developed into a business model key to the sustainability of internet platforms. By signifying a largely invisible tracking technology, the Listening Back plugin critiques a lack of transparency, inherent to online monitoring technologies, and the broader context of opt-in default cultures in transit to contemporary modes of online connectivity. So what I mean by default culture is, for example, with the browser, you either get it with your operating system or perhaps you download it from the web. And the first time you use it, you are by default opted in to being tracked by third-party cookies. But every browser offers the option to block third-party cookies, and this in no way affects the functionality of the World Wide Web. But I found that not many people know about it, and also it's hidden down in a lot of sub-menus. So, for example, with the Chrome browser, you have to go to Settings and then scroll down to Advanced, and then under Privacy and Security, it's in Site Settings, and then Cookies and Site Data, and here you can block third-party cookies. It states that some sites may not work properly, but I've never, ever, ever had a problem by blocking third-party cookies. So you can hear how a lot of the ongoing sound is third-party cookies. Typically, most third-party cookies are there when you load a page, and this is not always the case, but often, and the third-party cookies are usually the regular ones. So I thought it was important to be able to listen to the web and hear the difference between third- and third-party cookies. So there is an interface for this plugin, and you can change the scale, and then you can turn third-party cookies up and down. So if you wanted to listen to the web without third-party cookies, you can turn it down and vice versa. So it's not blocking the cookies, it's just turning down the volume of the sound. And then here, you can enter the domain name for a cookie and turn it up and down or change the octave so I could put it up a couple of. So I've used it for live performance and how am I doing for time? I can show you another couple of other sites that use a lot of cookies that actually fly search engines. So I might even close. So I made a special sound which actually also has that same third-party cookie I was trying to do for now. And this one's quite fun to play around with. And what if this project asks, what is the potential of a sound, of sound as a tool? I think it has been a very entertaining talk. We have some time for questions. If you have any questions, we have three microphones here in each one of the aisles. So feel free to approach and ask a question. Microphone number one. Thank you. Thank you for nice tunes. I have a question. How you composed and decided on which pitch will be for which website and what cookie will sound how? That's a very good question. I mean, in a way, a lot of it was sort of intuitive because I do work a lot with electronic sound and music. And so I just made sounds that I liked. But another deciding factor was, say for this generic cookie sound. Because the cookie activity is so constant, I wanted to make a sound that you didn't find too annoying and therefore turn it off immediately because I think it's important to sort of engage with it for a while. So it's trying to find a balance between, yeah, just to find a sound that you can listen to over and over. And then it was, I just made sense to make individual sounds for the main platforms because I knew everybody would be going to those platforms. And I guess for the Amazon one, I did kind of try and make it a bit more scary than usual. Especially, because I've performed this sometimes, you know, on a PA system in a club with a subwoofer and then there's a lot of bass in that sound that's quite overwhelming. And I guess the decision with, I chose like four keys that work together. So, you know, no matter what key you put it in or still be in tune, except for that generic cookie sound which you can't tune. Because I find if you're listening to a lot of different data, it helps if it's in tune. Otherwise, it can be really confusing to try and work out what sound belongs to what data. So, no matter what you do, you're not going to, it's not going to, it's sort of going to be tuned for. Microphone number two. Thanks for a wonderful presentation. As you mentioned, there are lots of other forms of tracking techniques that can go on a web page. For example, JavaScript fingerprinting. Do you have any plans to roll out different and increasingly creative sounds for those kinds of practices? I think that would be a great idea actually to sort of, you'd have to add all the different tracking technologies. I mean, I think you'd have to have an interface where you could turn them off like the individual data sets because otherwise it would just be way too much sound. Which might be nicer for a moment but then if you want to sort of try and listen to how much, say, web beacons are present or device fingerprinting. Yeah, I don't have any actual plans and if anyone wants to collaborate, I'd be happy to collaborate but yeah, I have thought about and I think that would be a nice idea. Number three. Yes, so thanks for the talk, it was great. I have a question that is related to the first one you got. Did you explore trying to use some of the randomness that is in these cookies to make the sounds more unique? The what? So all of these cookies contain a lot of junk basically, that is unique to each person. Junk? Well, data I guess. Did you try to use any of that to generate the more unique sounds? Well, the data that I have access to, I mean, I can show you. So this background page shows you in a way what data I have access to and so override is when it's updating or it's inserted or deleted. I actually focused so far on the different domain names as for creating sounds and also then the length that a cookie is programmed to be on your computer. That also affects the length of the sound, but Max, who I worked with, said that it couldn't be longer than a couple of seconds just because of the CPU. And like I was saying, the browser add-ons not really meant for unprocessing a lot of sound. But there's a lot of other information that you could sonify, such as the date and time or if it's a session cookie or a persistent cookie. As you can see, it's really hard to know what, I don't know how well you can see it. It's really hard to decipher actually what a cookie is doing, which is part of this sort of lack of transparency. So even if you have access to this background page, which I guess we all do have access to, when you look at the verval, I mean, it's impossible to understand what it means. Or I find it impossible to understand what it means. So this junk or noise that you're talking about, I'm not exactly sure what that is. Or if I would have access to it via the API. Oh, so I was thinking like just using, for example, the barbell here. The what? The, like the value of the barbell. Okay, okay. So like ignoring whatever it's supposed to mean, but just using it. Using it to generate a sound. Yeah, basically. Yeah, that's something you could do. And then it would be more individual every sound. Yeah, that's a good idea. Like for number two. Thanks a lot for this beautiful talk. I was wondering, you have a favorite musically speaking website. A favorite musically speaking website. So like a website, which gives very nice sound. Or just generally music. So with your plugin, when you go to that website, it makes some beautiful sound. Oh, okay. I do quite like Expedia. Okay, thanks. Number one. I have maybe two questions or a suggestion and a question. The first is I'd find it very interesting to make something similar, which doesn't display or doesn't audio lies. Third party cookies, but a third party content in general. So scripts and images and everything else that's loaded from a third party website. And the question, maybe just out of personal interest is, which site have you found that has the most obnoxious, loudest and just longest sound created by that add-on? I'll start with the second question first. So when I was developing it, one of the websites that was crashing the browser was Vogue, actually. And I mean, for the first question, I mean, it was important for me that it was just a sonification of the data, so not to have a visualization. Sorry, that was bad to formulate. I meant the same thing, but with third party content in general. So scripts that are loaded from other sites that these could also have a audio representation. I was kind of like the other question, just like other tracking. Yeah, no, I think it would be great to develop it in that way, yeah. I'll take the last question for microphone number two. Hi, I noticed that when you scroll down the website, there was a lot of noise. I don't have much technical background, but I was just wondering why there would be so many cookies loading up when you do subtraction? Well, it's hard to say. I mean, there's a few websites that do that, whatif.com, which is a flight search engine, is one that does. And whether that is because they're tracking your every single move on the page, or if it's just sometimes with some, which it probably is, I would say, because I imagine they would generate a lot of their revenue from data and selling data. But sometimes you get websites like, say, Deutsche Bahn, and that also has this constant cookie. And I don't think, I mean, like I said, again, I can only speculate. I don't think it's necessarily because they're collecting so much data on you because I assume they're making money by selling tickets, but sometimes it's a kind of, I suppose not as someone who's a programmer, but a clumsy way of designing your sort of backend infrastructure, because cookies aren't just used for tracking, like they are a state mechanism. So, yeah, I can't answer that, I actually can't answer that question because I can't tell. Even if I look back at this background page, which is as much information as I can get for what cookies are happening when I scroll, it doesn't make any sense to me. Okay, so that's all the time we have. Thanks again to Jasmine for a great talk. Thank you.