 So copy and paste in online just so we can see the problem and understand why there might be something worth doing here I'll just show you a few shots of what the competition is doing. I don't know how well you can see that but So this is what happens in Google Docs I got a we have a sample demo document I just used a spreadsheet and You copy and paste it in Google Docs from one tab to another tab and that's what you get As of yesterday It's a little bit unfair because all of the formulae are copied and they refer actually to a data sheet here that isn't there So it's a little bit tricky But people do that typically they have their data on their presentation and separate it. So not not a great job in in crime Excel does a little bit better. So you get the inline values others, this is Excel online, I guess the Office version and you can see some conditional formatting in there as well, which is quite quite impressive and previously we used to do this and in fact shipping products are like, you know 6-2 and 4-0-x for a collaborator Do exactly this So you do a CSV import when you paste which was which is not as good as it could be let's say Not wonderful on the other hand the the very common use case of copying inside the sheet We did with an internal shortcut so you'd essentially send a new no copy and a new no paste and it would just move the stuff Internally and that's really good to to do stuff on the server wherever you can for several reasons I mentioned the formulae that Google Docs breaks now But you know if you're copying columns around or you're cutting and pasting formulae in calc It is it behaves very differently. It rewrites all of the formulae that depend on your data So if you move this formula and you put it here, it's not really just moving a formula It's not really just pasting the content It's also updating the whole rest of the spreadsheet and this is really really really important part of spreadsheet editing So we can do that really well And so copying and pasting and cutting and moving inside. I was absolutely fine doing it on the server And you'll see we prepended the word internal to all of these things to try and make it clear But this is not it's not really a great user experience So yeah, so we only really did plain text when we when we came out of the the document So so we want to improve it. So well what what technologies to be given? What what are the API's look like and the answers are uniformly bad here? Unfortunately, so the API's to copy and paste are frankly terrible Yes in lots of browsers you can copy and cut whatever you like put it on the clipboard any any any kind of thing as long as it's text or Text HTML or text RTF in some cases, but there is a whole load of horrendous security Stuff that's kind of piled up and accumulated So there's a very good API for for sticking mime types with it's not very good But that you can stick a mime type and a blob For each of these types on the clipboard, which is great, but in order to make that actually work There are a whole load of hidden constraints across browsers You know you have to have a focused entry With a content in it and select it and then run a command as if the user is pressing copy and Then handle that event in which you can have an object in which you can somehow set your content Although it doesn't always work. So you then need to catch and you know, huh pretty pretty unfortunate And so in consequence you see these dialogues are showing up here So your browser can't access the clipboard use control C or max C or this kind of thing The top one I think is Google Docs The bottom one is us you see these around the place If you use control C if you're a keyboard lover, you probably don't see this problem But if you use context menu pop-ups, you have this problem and particularly for paste because paste ultimately takes information from your system and You know then uses it in JavaScript and it's the Wild West out there on the web You know JavaScript is just a crazy crazy thing And so wouldn't it be nice if all of your web pages weren't watching your clipboard all the time? You know so there is this fear that someone writes the web page that just sits there and Steals all of your clipboard and uploads it somewhere continually. And so, you know as you edit your finance spreadsheets, you know It's all being you know Sent somewhere else and so IE 11 I have a good thing to say about IE Internet Explorer had basically the right idea here. They put up a dialogue saying ooh, ooh Oh, the JavaScript trying to look at your clipboard. Are you really sure you want to do this? And so you could allow your application to actually access your clipboard, which is basically what you want to do You know, you trust this it signed. It's on a sensible site. It should be accessing your clipboard. I'll have it Unfortunately, everyone else piled up all sorts of stupid ways of trying to work out whether you should or shouldn't use the clipboard And then left them lying around and so Google Google have obviously different teams one of them working on the browser with a security team that's paranoid and Then they have the Google Docs people who want their users to have a nice experience And so this is the compromise, you know, you you fire up a recently downloaded Google Chrome You go to the Google Docs and it says yeah, your Chrome's not good enough You need an extra plug-in that allows us to actually copy and paste, you know This is what's there today pretty silly So yes, we you know the web needs to fix itself get its stuff in order really But there is all sorts of other stupid stuff and so in the end at the end of the day You're gonna end up with a hidden or transparent or off-screen Copy-paste area with a font size. That's you know, and there's a whole load of just weird weird gotchas here And there's also security context So you've got it all working and everything is looking beautiful and you discover that actually The click that you did on your mobile device for various compatibility reasons isn't a click It's a touch event and a touch event doesn't have the security context You know even a tap that you need to do copy and paste So on iOS everything works all of the APIs are there the objects are there You shove your data into the clipboard and it says it succeeded and it's all just thrown on the floor Wonderful, isn't it? You can't make this stuff up and actually what you needed to do It was to have a have a click and it's got to have a pop-up security context So you need instead of a you know some normal html You can look on you need an href to a hash and if you click on this then all is well So you know wasn't our major security win all that complexity because there's an easy work around it at least for now until someone Does something even sillier? So IE 11 also has a terrible API so you can't do HTML copy paste Manually at all you have to actually create an HTML document in a content area and select it That's hidden and then when you press control see your focus is really not on the document It's on the magic area and by combination of three events, you know before copy before you know copy and then something else And an idle handler you can just about Mesh the content in and out as HTML And so you may ask yourself why is HTML? Oh, so the other thing is that we really need to put more data into this that you can't see So text plane is really not going to do it for us text RTF is sort of standard ish Probably we can put comments in but text HTML is pretty much the only place we can put a nice comment in a magic So we can talk to ourself and when we see this stuff coming in we can go. Oh, that's us Let's do something cunning And you might think that in images you could do this as well So often we want to you know we can paste images, but actually images you don't send a PNG through the copy paste API And the security excuses that you can create a tiny PNG that explodes to a gigabyte big So if someone copied and pasted this then they would destroy the machine basically and that would be bad So so all of the images are unpacked by the browser and then re-packed as just pixels So getting any metadata through that channel is practically impossible. So you're dumped Lumbered with text HTML and that's it Um, yeah, and the mobile situation is is is even worse than the iOS, you know the iOS quirk here So they're just getting a button So the idea that there is a system button you can press like control C that we can trust some of security perspective is Difficult when there is no keyboard and no control key So there is a nice button on the keyboard and is that has that Android doesn't have it Gboard does have it. It has a built-in clipboard Magicky thing But it throws everything away except text and while it's converting your HTML to text It also handily throws in a whole load of things like the title and the metadata as actual texts as well So it's just impossibly broken and there's nothing you can do So yeah, even even detecting that the copy has succeeded or failed is really not trivial So yes, so that's one set of real badness the security people have gone wild off the reservation The second problem is that all of the clipboard APIs are synchronous They assume that all the data you want to put on the clipboard is already in the browser when you want to put it that And this is simply not the case you can select a whole spreadsheet and it can have 10 to the 10 to the 13 cells in it something like that and you can't put those all onto the clipboard Before you know someone's going to paste them and gonna paste them somewhere else And this is really horrible for us. It's terrible for VDI clients there are all sorts of aspirational standards that supposedly fix this that are poorly implemented and Underused and the people implementing them say, oh, we've got this brilliant new API But for security reasons, we only use text plain And here's our little demo that can copy text plain more easily and you're like great That wasn't really what we needed So yeah, so this is really bad and it's easy for people like Google Docs that put a lot of the document in the browser But for us we have a lot of document on the server side And so it's just not there to put in The other stupid thing the web API is do is that they demand all of the data be there at once So LibreOffice can paste as a BMP a metafile several kinds of metafile, you know And obviously ODF and RTF and text HTML and all but the cost of generating all of these different formats For a large selection is quite significant So it's it's a bit of a pain and and so normally normal Copy-paste APIs have a negotiation phase where they say hey I've got all these formats you could have if you want one And the other guy goes I preferred this one and he asks for one and gets that Rather than having to do all of this generation shove it and then do it back again But I don't know whoever whoever created the web copy clipboard API is obviously hadn't read or understood Just basic operating system You know clipboard API is because it's all there. It's been there in like windows since Years and years and years are for efficiency reasons because often these old machines quite small So how do we improve things? How do we make a user interface that works on top of this for copy and paste? Well, so the answer is we try and keep it simple. So if you select a little bit of text Not not a vast chunk of stuff We essentially push this at the end of the selection. We send send the results to the client So we've got it there so we can immediately put it on the clipboard Of course, if it's a more complex thing even like a shape or something like this that can appear in multiple formats There's a problem there So instead we put a magic on the on the clipboard which basically says at the bottom of the screen You can see here this it says this to paste outside online Please first click the download button So we put that on the clipboard and we put on it a magic That that is hidden away behind the text that says this is us Hmm. So if you take this and you paste to another online window or you paste it inside the same application We go. Oh, that's us. We can short circuit this all on the server or we can we can download it and reupload it transparently for you But the reality is if you click on a complex chart and you want to paste it into you know Microsoft Word online or something we need to be able to download that as an image and all these different types populate the clipboard And then paste it elsewhere. So they have this nasty Starts download button that pops up and this dialogue on the right that tries to explain the sorry situation We're in you know, like, you know, we it's not us that sucks You know the web is really there's a ruin you but luckily because of this this magic magic inside that gives the origin of the content For any copying and pasting between, you know online instances different tabs and so on now. We can do a really good job We actually have a connection to ourself That's relatively robust and it looks like this. We use actually just the meta origin Which says where it came from and Hopefully you can see some of these these horrible things here, but this basically has the wappy like wappy like URL in it that identifies where it came from the source of the thing and Yeah, the file it's in and then some security magic attack a very large hex number And we rotate those numbers every a couple of minutes so, yeah Not Conceptually horribly difficult. So for our data, we then yeah, there's a whole lot of asynchronicity downloading that because web requests are Asynchronous and sadly is all mime types re-uploading it as all mime types injecting that into a kit process And then sending a new no paste which then uses the existing PC code to sort it out Yeah, if we have Images inside things we we try and download base 64 for that so that we can have some kind of self-standing Because of course serving images as web URLs that are eternal. It's not really very cool It's not that's not really where we want to be so there's a chunk of work to base 64 embed those things And make that nice and so we then have a new clipboard endpoint So you you talk to online and there's a clipboard sub sub point and that either talks to the kit and And gets the data or well it does some other horrible things obviously when you close the document You have this this problem that Well, you know, there was a whole whole chunk of data on the clipboard and You may think closing the document is not a common case that you copy close the document open another document and paste But actually this is the normal case on mobile So if you're using a mobile device you tend to have one window and you copy you close the document effectively load another and paste it and And then what well hopefully it should work and so you know we try and then serialize these things and keep them around But this makes life horrible because as you shut down the process There's a whole extra phase in the state machine to try and interrogate its clipboard serialize it get it out and save it and there's some other gotchas like We're really dealing with multiple clipboard So if you go paste in in LibreOffice, it knows there is nothing on your clipboard And it looks at its system clipboard and there's no system inside the kit and it goes Oh nothing to paste we won't even bother putting the paste bit up. It's super clever But of course, it's not seeing your the paste buffer or the clipboard on the P you know on the client on the on the PC So we basically just clobber it and we ensure there's always a paste item comes up there dialogue pasting again is is pretty horrible. So anyway after the After all of this work, we we move to a more positive place Let's say and I think probably better than either of the previous two Google Docs or Microsoft Office So maybe you can see some of the the conditional formatting flags down there stuff, you know data coming across Formatting because it's essentially doing a rich ODF transfer from what from one side to the other quite why it misses the charts I don't know but of course if you if you manually copy the charts You can get those two as real charts and not images And so we can start to then actually have some something that really really works really nicely So then there's a whole load of other problems like say on the Mac For example, if you paste from Safari, you select something Wikipedia shoot cool. Am I running out of time or? No back you want to get back The camera was not working smile done The slides will be published online Really good So on the Mac, I think you know we discovered, you know the customer is testing and they test on Mac and they copy from Wikipedia And the Safari browser and it it generates RTF, but the RTF doesn't include the images Which is kind of annoying so when you paste it into your document They're like, ah, you didn't paste the image. We're like, yeah so We did exactly what we were told by this dumb browser But one of the great joys of copy paste is you have no idea where it comes from. It's one of these end to end power law testing problems and Yeah, so you just have to disable RTF entirely on the Mac platform It's the only fix Which is slightly depressing really possibly we could introspect it and go Well, that's the really lame safari thing that doesn't work yet and we have no way of fixing And blacklist it but anyway So you discover these little joys as you go along And yeah security wise so we create a new clipboard access key It's a hard random number of a very long long length every couple of minutes And we push that proactively to the clients and we accept that key and the last key Or you can just disable copy paste if you're paranoid Um, unfortunately our text plane is is less good than it was Because we now actually convert html to text on the on the client So we send just one html version And then we do an html to text conversion in the browser and would you believe it Browsers have no good way of converting html to text Nothing at all. Yeah. Whoa browsers api suck But you would have thought that this would have been something that browsers were quite good at I mean honestly, you know, they have whole they spent the whole life trying to do exactly this But there's no good way of doing this a short of like manually Writing rejects to parse out html, you know, and so yeah, we shove it We shove it in an html element inside the dom And then we request it back as text and it does just a terrible job like a really awful Awful job redundant line breaks, you know extra guff poor layouts. Da da da. It's just hideous So it's possible. Yeah, because the sad thing is we actually had very good text copy and paste beforehand Because LibreOffice is doing it and LibreOffice has all this text layout stuff and I can oh, yeah You can actually do things and and so it was really good. But now unfortunately it ain't so Need to fix that. Anyway, so the punchline is we have lots of great copy pastes code around We can do file formats. No one has ever heard of, you know We can we can populate your clipboard with things, you know in just the most weird weird formats But the web then just basically screws us over with bad apis You know synchronicity no content negotiation and just trying to to hurt you at every step with pointless security nonsense And yeah, that's a shame. So it's a bit of an epic fighting this and there were several people ash Was helping here and shim on a hink and Marco and mick lash and erin and various other people trying to get this working against time And beyond that. Yeah, thanks for our customers and partners who uh, You know pay to make all of this possible. We hope so. Uh, that's that any questions How are we doing for time? Time time 10 minutes. I talked too fast. Perfect. Ah, oh no So if you look at the market share statistics of browsers firefox has disappeared pretty much like it's very small but yes, of course we test for it so In in terms of our test matrix. I always been testing on IE 11 on windows firefox on windows chrome on windows edge on windows safari on mac chrome on mac chrome on linux firefox on linux on ios The native browser on ios also the embedded web view on ios inside the next cloud app I should have perhaps talked a bit more about that and the same on android But then of course the embedded thing is also multiple versions of different browsers depending on which android you have So yeah, I mean as I say the test matrix is pretty bad In terms of having a factor of 12 or something across five platforms in it And then you have all of this application to application problem So it's like well I paste a shape from libra office writer into libra office cap So that's one kind of problem. Um, but now I paste it into google docs or the you know another web view or another You know like native applications. What happens and the the test matrix explodes As you as you have these different types on different platforms pasted to each other And you know and and some of these web APIs APIs are so bad That they're timing sensitive so they work when you test them in the browser in in the debugger I mean literally, you know like I so many times I thought yes We finally found the security problem. It works because in the debugger when you debugger it works And then it doesn't work on the actual device There is uh, you know and there's there's no it's just a race condition or an interaction of the debugger Or there's a timing problem that's screwing their security checking And it just it just works when you debug it Yeah, really nice. Um, so so you know, but so yeah pasting for example from mac os into the emulated ios device on That works really nicely But then of course it doesn't work on the actual device So, uh, yeah, it's it's a testing nightmare And it's that and then you get to some points that you know LibreOffice copy and paste is not as loved as it could be anyway on the pc So it's relatively easy for example to select something on the pc LibreOffice draw and paste it into impressed and discover you've got blue boxes instead of Transparent one so like this there's underlying problems even in the easiest possible case of pc to pc copy paste And we haven't fixed them all but we're getting better so uh, yeah If you if you have a death wish for a testing to the nth, you're very welcome to get involved and the Help help report problems But like like all things I think we now have a pretty good infrastructure for doing this I think we understand most of the problems Um, and it's just detail now lots of detail That's my good question. Look at that. Look at that on that guy He rocks so yeah fire foxes fire foxes I use chrome, but you know people use firefox Anything else? Oh, uh, cool It's Yes, yes, it's many man months of work and a hard one. Yes Uh against the deadline so again, you know, it looks easy, but it's it's not quite as easy as you might hope Oh Yeah, yeah, yeah Luckily, I wasn't a javascript programmer in a in a previous smooth and innocent life I was not Not doing javascript and now I wish I hadn't you know sort of scars you for life Yes, javascript is an industry disaster area as far as I can see it needs Yes, come the revolution Lots of people will be put against the wall that created javascript But no, who knows it's I it is what it is I I think you can look at LibreOffice and you can see in document formats what what you have in functionality in javascript It's it's layer upon layer of Previous decisions that you have to live with Good good questions anything else we have another five minutes Ishmael moved strangely there wasn't a question. Okay. Well, you've been very good. Thanks so much