 Okay, so we'll start, so our next speaker will introduce us with subtitles in HTML5, please welcome Andrea Steyer. Thank you, it's also subtitles in XML, you already heard that I like XML, XML is great. I'm from the IRT, IRT is the Institute for Rundfunk Technique in Germany. It's an R&D organization funded by the public broadcasters in Germany, Switzerland and Austria. And we've been doing some work on our IMACJS as part of our European project immersive accessibility. We're working with partners so that access services get appropriate representation in immersive media, especially in 360 degrees. And you will see later how we plan to make this happen with IMACJS. So, this is my third talk here at the open media room of FOSSTEM. And my talk is always around how open source can help adoption of standards. And the last two talks, they were more on the direction, yeah, I would like to see it this way. And I think with IMACJS, I think we are already pretty there, where I think open source can help standards and standards can help open source. So, IMAC stands for Internet Media Subtitles and Caption. It's an XML format for subtitles. It's based on the time text language of the W3C. It's a subset, a profile of TTML. So, all IMAC documents are also TTML documents. If you heard of EBU-TDD, that's also a subset of TTML, specified by the EBU, the European Broadcast Union, who also co-organizes this event. So, IMAC is a superset of EBU-TDD. So, if you have an EBU-TDD document, it's always also an IMAC document. So, an IMAC renderer will always render EBU-TDD. So, IMACJS is written in JavaScript. It has a pretty free open source license, BSD2. And because it covers pretty much of what you would do with TTML, it's not only an IMAC renderer. It's a good TTML renderer. And because it has not only a text profile, but also a profile for images that may be useful if you do not have the font available on the device, especially if you want to deliver some Asian languages, or subcontinent Asian languages, then you may use the image profile. It's, of course, hosted on GitHub. And so, also, it's PLMU. And PLMU is also the editor, the responsible editor of the IMAC spec. This work for standardization, but also for IMACJS is sponsored by MovieLabs. And MovieLabs is an R&D organization funded by three, no, I think six major Hollywood studios. So, I picked three use cases for IMACJS I want to look at. One is for reference, so how you can use IMAC as a reference implementation. How it can bridge the gap in HTML file as polyfill. And just one slide would just point out some ideas how you could use IMACJS as a toolbox. So, what is the value of a reference implementation? Some developers, I see also some developers here in the room, don't like to read long, complex standards. They really like to work by example on the learn by example how to implement it. So, therefore, you need a reference implementation that does the right thing with some input. So, for IMAC, you get some XML subtitle documents and it renders the correct subtitle presentation. And some developers prefer to implement some standards. But this could be done also with something that is not open source, it's just free and you have access to it. But the value of open source here for reference implementation is that you can also look under the hood and you can see how it's done or it's a guidance also how to implement. It shows you in algorithmic form how the standard is built up. And I have to say, after looking in the code of IMACJS, that I learned a lot about the standard myself. And last but not least, of course, you can integrate it in your existing systems and software as you like, especially if you have a free license like BSD. So, what are the process steps of IMACJS? Of course, first it needs to be parsed. It will be translated in an object model. So, you have the document as one object model. Then, from that object model, you can, for every point in time of the play time of the video, you can generate a presentation, a static presentation of the subtitles. It is called ISD, I come later to that. And at the end, of course, you want to render it out somewhere. And I look at it if you want to render it out to HTML. So, that you generate an HTML fragment from the ISD. So, you can build. You can use the IMAC code with the grunt script to one file. You would normally use that if you use it in the browser. But you can also use it with Node.js. It's written in modules. And the parsing model is doc.js. The P generation model is IC.js. And HTML.js covers the function of rendering. So, first, parsing. That's quite simple. I assume you will build IMAC from code with grunt. And then you have IMAC object. And the IMAC object as property piece has the main functions that you need. And one is from XML. You give it a XML string. And then it generates this JavaScript representation of the XML document. Pretty simplified example how it looked like. For example, on the console, if you want to see what is in there. It not only rebuilds the structure. It also already do some helping stuff. It resolves the times. So, there are different time expressions possible in our HTML. So, it all translated to our media time in seconds. And traction of seconds. Yeah, and it does some other helpful stuff that eases processing later. So, ISD generation. What is ISD? ISD, oops, not much. ISD stands for Intermediate Synchronic Document. I give an example to show why you need it. So, if you have word-by-word subtitles. You may know it from US captions and also from some live UK subtitles. Then you would also do it possibly like that. So, you want to have I and at first time coming up word-by-word. You start with with the I and then leave it on screen for the complete time, which is three seconds and at the M start is one seconds and at first time are two seconds. And this could happen pretty much everywhere in the document. So, you could have a lot of different content elements having different timings. And at rendering you really do not want to get track or track all this different independent time content elements. So, you want a static representation of what happens at a certain time. So, you want a representation at one, zero seconds where just the I would be in the content element and the P in the paragraph in this case. In one second it will be I am and three seconds it will be I am at first time. So, in the intermediate synchronized document everything was not relevant. It is a specific point of time where you want the result of the subtitle rendering plane. It is pruned out and you got just what you need to render. And then you can do whatever you want. Pretty simple what to do with I am and C you use the function generate ISD. You gave it a JavaScript model generated before the offset. You want every offset that makes sense from the beginning to the end of the playtime of the video. And then you get this ISD representation and it looks like that. So, it is similar to what we have seen before but all times are gone. So, the ISD has no time at all. It is just a static representation of the rendering plane of the subtitles. And what is also done with this step that is done by ICJS is that all values for style attributes are computed. So, there are different ways for example to specify color. You can use hex code or RGBA with decimal or different values for text alignment. All this is normalized and in the ISD document. So, at this point, you could render a doubt as SVG. You could do it whatever you use in your player. But I think a pretty obvious use case would be to render it as HTML. So, you need a wrapper element where you hook in this generated subtitle DOM fragment. Usually, you would use I think a container where you have the video element in. And then use this container element as wrapper. But you could also just create one element. And then you give it the function render of IMSC. Give it ISD generated before. Yeah, this element, this wrapper element. And if this wrapper element does not have explicit height and width, usually it would have, I think, then you need also to provide this as parameters for the function. So, this is how it looks like. So, all the HTML specific style properties are mapped to CSS as inline styles. It's pretty straightforward, but still some tricks you have to do. So, it's really good to look at HTML.js if you want to do it yourself. IMSC as HTML is not natively supported by browsers. Nonetheless, by a lot of people, possibly not all will agree, it's seen as part of the web platform. So, if you want to have to support it, you need a polyfill. So, a polyfill is not only there that all the browsers support some new stuff. It's also there to have some new features or unsupported features of the web platform supported already in browsers. So, and here IMSC.js does exactly that. It gives you the opportunity to use our HTML in the HTML5 ecosystem, and it does it quite well. And how could you do it? It's really super simple. I assume you have a video element somewhere. You've got it already in the view variable. You add a track to the video. And then you have two choices, possibly. You first go to every point in time where some subtitle presentation changes. So, IMSC has also the get media times function that gives you an array of all these different times. So, you iterate through that. And every time something changes, you create a VTT queue with the appropriate times beginning and end. So, the time itself and the next time when something changes. Or you already know that you have simple subtitle block with beginning and end. And then also from the media time it's quite easy to figure out where's the beginning and where's the end. And then you generate VTT queues from that. And of course nothing in the text. You don't want that VTT itself to render it. We just want the queue and track implementation. So, what we want is to use the callback functions that can be used together with on end and on exit events of the queue. So, every time the queue starts, you would generate an ISD and you would render it out. So, that's all. So, that here goes your subtitle renderer and your video player. And of course you should not forget on exit to remove it again from the wrapper element. And then at the end add it to the text track of course. So, I just touched briefly how to use IMST.js as toolbox. Possibly you do not have a TTML document or you have a really static presentation of your subtitles. So, you may not want to always generate the ISDs from the TTML document. Of course it iterates quite a few times over the structure. So, you can use an ISD directly and just manipulate the text for example or the position or whatever. So, that would be quite simple. You actually could use also VTT. You can load it in the text track and then iterate over the text track queues to just change the ISDs and use IMST.js for that. You also can server-side rendering of HTML fragments. As said, you can use it with Node and to generate the DOM fragments, of course you need some DOM environment on the server side. So, and I think JS DOM would be the first choice for that. So, drink, some water. So, it's used already in operation. So, and the most popular and kind of known framework that uses this is Dash.js. SoLen and EBU, they did some subtitle implementation in Dash.js. SoLen presented it two years ago in the OpenMedia deaf room. And now, Dash.js replaced the subtitle rendering of TTML with IMST.js. You can see it live on the IOT demo streams for CMAP or subtitling.it.d slash CMAP. There's some recorded live stream and this is a real live stream from Bavarian Broadcast. And, yeah, the real live stream, it's, yeah, you can watch it also outside of Germany if it has no geo-blocking in. And I come back to the immersive accessibility project. We are thinking about how to present subtitles in 360 degrees. So, a colleague of mine, Peter Topesh, he worked with IMST.js. He did exactly what I said before. He took the ISD and just changed the position. So, because if you have, for example, some position subtitles that should stay regardless where you are in the, yeah, in this 360 degree environment, then you need to change the position based on the user actions. So, for example, if somebody is playing a guitar and you want to indicate, for deaf and hearing subtitles, somebody is, yeah, somebody is playing or there's some guitar music. You play it next to the guitar player and if the guitar player moves left or you look right, then it moves together with the guitar player to the left. So, yeah, this is already working with IMST.js and works pretty well. If you have some interest in that use case, I think my colleague Peter Topesh will be happy to, yeah, get some feedback or just email him and so you can discuss other use cases. Okay, so I'm already at the end, 54 seconds to go. I will use that. Here are some links. No, there are some links for the IMST specification, IMST.js. There are also other open source projects working on, oh, I see I made a problem. Yeah, mistake. Okay, there's the Time Text Toolkit. There's the Live Toolkit. It's, of course, not the IMST.js link. It's a different one. I replaced it in the slides. They're not uploaded yet on FOSSTEM. Sorry for that. I will do it later and there is a subtitle conversion framework also on GitHub. Okay. Thank you, Lars. We still have five minutes for questions. Is there any question? I have a mic. I jumped. I see you use TTML, but it seems like Web VTT is used a lot on the web mainly. So what's the reason for using IMST in a TTML-based format when, especially towards the end users? I actually had some slides in my presentation that had the headline questions I would ask and one was what is about Web VTT? Because this comes up, of course, every time. Web VTT has been chosen by the browsers as the subtitle formats they want to support. For a lot of organizations, our DVB, EBU, SIMT, ATC, it's very important to get a certain quality of subtitles guaranteed, not just text on the screen. So they have chosen TTML to do that. And if you want some guarantee of presentation that works all over devices, I think IMSTJS and TTML are possibly, from my point of view, the better choice because you have very different implementations at the moment from Web VTT, maybe the changes in the future, and also not all features are supported. And what is also very important for our shareholders, who are the broadcasters, they want to not change format. So they will also produce, most possibly, their subtitles in the future in TTML. BBC already does, so therefore they have no loss of transformation if they also use TTML for presentation. Does it answer your question? Thank you. There's another question. Thank you. Do you have support for multilingual streams? Yes, I mean, this is not a feature of TTML itself. So in one document, you can't have, I mean, you can have possibly, but it's not designed to have more than one language in the subtitled document file. But what you would do normally with streams is to have different tracks, or, yeah, you use dash in HLS and you will have different, what is it, representation, I think representation sets for dash. So it will not be solved actually by the subtitled format itself. One last question, maybe. No? Okay, thank you.