 Hi, I'm Jonathan from Kaltura, I'm architect in a Kaltura open source video platform and like just like cats as well. I'm going to talk about lessons that we learned during the years in Kaltura, how to overcome the non-limitation that REST API is textual and how we deal with many media files using our REST API. So video is used today, media generally is very significant content in the internet, it's affecting your costs, it's affecting your traffic, it's affecting your storage, it's affecting everything that you do and of course it's affecting your API. Next we have to define the enemy, to know what's the problem. I could show you graphs, charts of how media used today in the internet and how it will be used in 10 years and of course all the boring charts always show the same, it's always going up and the conclusion is always the same, you can't ignore media anymore in your application because media is used today not only in media companies, media is used today on your cell phone, media is used today in enterprise companies, in universities and so on. So in Kaltura we are doing, we aspire to do what we call everything video, everything video means that we are doing capturing and conferencing, transcoding, encryption, streaming both live and VOD, edge teaching, client side and server side, recording, playback, distribution, syndication and so on. Everything is done in Kaltura using API, everything is REST API which is text wall. So although REST is HTTP and HTTP although it's originally created for hypertext, since the very beginning HTTP supports media files, we know put action to push files to the server, we know post action that can, sorry post method that able to upload files to the server, get, you can get progressive download, you can get video or image using get method and today you can even stream video over HTTP using HLS, VOD or live REST although it's based on HTTP it's fully text wall, it's an industry standard and what we usually we use get, delete, post, put, all these actions using text wall path, with text wall processing, if you need complex data structure you will need JSON or XML whatever you choose, even in the response everything will be text wall, JSON or XML. Of course there are simple solutions in the industry, some providers use a put to push files to the server, the problem with push is that it supports only one single file and you don't have the ability to add to the request complex data structure, first you could use a very complex data structure in the path or in the query string but it's not designed for that and it could be very uncomfortable for the developer to push complex data structure in the query string. Another solution is the form data, again the problem here although it supports multiple files, the problem here is that you don't have a native support for complex data structure that like we do in JSON or XML. So I decided to investigate a bit further, so I took two specimens of these media providers home and after 10 years that I'm raising them I can tell you that they took over my sofa, my kitchen, probably even my brain. 10 years ago we started with a very simple solution the most simple to get media from the internet is get, you can get the file using progressive download but usually in the industry standards the actions to perform on the server are using HTTP methods like put, patch, head, get whatever I missed. Anyway it's very limited me list, we have five or six HTTP methods and you don't have the ability to add a complex request to the server. What we did in Kaltura is that all our requests are sent using post or get doesn't matter how it's arrived to the server and the action itself, what to do about the object, about the service that we are calling, we just added it to the URL which is a bit different than the classic API, classic request API and it's enabling us many different actions and of course we support the regular crud, create, read, update, delete and we also added another standard, tomorrow standard that's called the serve, the serve instead of returning the object, the XML or the JSON that represents the object it will return the media that's related with the object. As for uploads, we are using a post of course and you can post more than one file and unlike get that enable you only path or query string, the post enable you to send another option that we have in order to upload content to Kaltura, you just inform to Kaltura what the location of the file and Kaltura will pull the file for you. Of course because we are talking about REST API we have to manage the status of the upload of the download in this case, the status if it's failed succeeds, it's progress and so on. So the serve actions enable us many different content types, every object in Kaltura, for example media, entry in Kaltura contains few video renditions, few thumbnails, closed caption files, maybe license files, different related files and a file upload enable us to upload all these files. Now we encountered few issues while using file upload, the first problem with video is that it's huge, we have customers like Walt Disney, Walt Disney of course they have large videos and you don't want to wait four hours for the upload to finish and then it fails five minutes before it's done and then the entire, you have to do the entire process since the beginning, so what we did is to support chunked upload, you can upload the files in chunked and on the server side we combine all the chunks together to one single file. Another problem that we had is we have different data centers, when you call the API you don't know what data center you are calling in fact, one chunk could arrive to different data center and then you couldn't combine them together, so we just redirect, according to the first server that the first chunk that is selected to, all the chunks from this moment on will be redirected to the same data center, in each data center we have multiple machines, we just sold it with a shared disk, chunks may arrive in the wrong order and we manage it in order to know what chunk to put where and another problem that we had is that the file upload could take long time, you don't want to upload the file and only then to do something about it, you don't want to wait few hours until the entire video is uploaded and only then associated it with with thumbnail or with video and so on, so what we did is to support, to enable you to run the upload and forget about it, enable you to associate the object, the file before it's created, so we have some kind of pointer to the file that is not created yet, using that pointer you will relate it to the object that it relates to and later when the upload will be finished automatically we will use that pointer to associate it with the object, these are two examples for a common API call in this case it's upload token get which returns a XML or JSON representing the object, you can see that the JSON, you can't see that it's JSON because it's okay anyway trust me it's JSON, you don't see the all the screen, we just send the JSON using content type application JSON, unlike when we are sending files we will send a multi-part form data with the JSON as one of the fields separated with boundary from the file itself that's the binary data, try to save place on this on the slide so you don't have a lot of binary data here, now if any of you ever tried to implement the API of WebEx or Akamai or other company that deals with media it could be very difficult, not easy to implement the HTTP by yourself and for that in Kultura we generate automatically SDKs in multiple coding languages, in each coding language we try to follow the standard that's the best practice in this coding language, some coding languages using a string path for the to represent files like PHP, some with file objects streams buffer Node.js, C sharp Java each one is different, in some cases we even implemented sometimes two different signatures of each method in order to support a more than one standard in one language this profile download we assume that the customer don't want to download the entire content the entire media to its application memory, unlike the object that you want it in the application memory, the data the media itself you don't want it in the memory, you don't want the binary data, you probably want to do something with it, if it's image that you want to present in your HTML you can using our API instead of returning the media itself we just return URL that from using that URL you can download or get the media itself so every place that we define in our API that action serve action return a media file or any file instead of returning the file the SDK will return URL to that file, later you can use that URL as image source or as to send it to the player and so on. This is a I selected the PHP which is synchronic language the code is shorter here is example I'm using a file path which is the standard in PHP creating upload token which is the pointer to the file that will be uploaded later and the file size is not mandatory but if we are uploading in chunks it's very important because otherwise the server won't know when is the last if the last chunk arrived or not it's comparing to the file size that we defined in the beginning and I'm creating the upload token using client upload token service add action ban the scene everything translated into PHP to sorry to HTTP and then using the created the upload token and just uploading using upload action with the upload token ID in the file path very simple very easy to use the other hand serve action you can send complex data structure to the server our cultural some problems support scaling and trimming and background and many different parameters I just chose two widths the height which are quite standard in this case and I just call the serve action with the complex data structure and what I get in return is remote URL the URL is the URL that I can use to get the content later and in this case I just downloaded the files and saved it to the disk but I could do anything else with that URL now we have large customers with the large amount of the media and very soon we understood that just upload and serve actions are not enough so at the beginning we added a CSV which enabled you to upload a CSV file which is I will talk a bit more about it later XML file which is more complex not so easy for human for human to write it the complex structure because a media entry could be related with a lot of metadata many different files like thumbnails media closed captions and so on it's not very easy for human to write it we support drop folder drop folders could be on the customer servers in this case we will see what's using ftps ftps s3 doesn't matter how we can watch the folders and see what's going on on his server or it could be on our servers and then the customer or customers our enterprise companies many of them use Webex so we just added another ability to scan using apis the Webex server and we are downloading the the files from Webex this is when I implemented the Webex API it wasn't easy okay CSV as I said is designed for human writing it's supporting very simple properties main media file and main one thumbnail file and few more properties like name didn't write all the properties but basically it's named the script and tags very basic properties however the XML could contain many different related objects all of them could be a in a single XML item we enable the drop folder to drop the files manually by humans which is very simple for user just to drop with the file and media entry will be created on our system but it wasn't enough customers wanted to add a tags or category or user ID many different information so we started to enable them Webex to extract data from the file name but it's again not enough so we decided to support the XML which is complex data structure with all the related objects with a lot of metadata data a time related information on the video and with one different from the regular XML that I mentioned before this XML could relate to files that dropped in the drop folder now in control we saw that we have many ways to upload content to the server so we decided to associate this content with the files in many different ways we have asset resource which is in fact linked to another media that already exists on the server drop folder file mentioned a server file which is integration between our systems sometimes one system create a file on the server uploaded file uploaded file the token is two different ways that we do that webcam remote storage which is storage remote storage that predefined for that specific customer string it could be relevant for SRT for example if you want to upload SRT you don't have to upload the file you can just use string resource and url resource which is of course just specifying what the url in what url the file exists and in order to associate the content with the object we just define the resource in this case it's url so I had to set the url property and we call some assets set content and associated some assets with the resource and all these resources supported in all our apis on all the objects that associated with content you can use each one of these resources and also in the xml so the xml can use each one of these resources also in order to use each media doesn't matter how you uploaded it to cultura we also support publishing the content to external system using push we are pushing the content to the media itself to remote system and I will show a few examples soon and syndication which is publishing MRSS of our content and the external systems downloading the content to their systems these are the standards that we support as you can see there is no industry standard but the most important one here is cultural not because we are cultural but because culture expose all the information that we have on the media and you can apply xsl t on that media and using that xsl t you can apply any different standard that exists in the industry and sometimes for small companies that's what we do questions any question most of that code was client side examples what do you have to have on the server side to support that all our systems internally using APIs between them okay but I mean is there a specifics I have to run the culture a server presumably one control server use API to talk to another API said to another server so okay so you're saying it's kind of symmetric yeah okay another question what do you support as backend storage we use our own data center and we use our own share disk it's called icilon but it's just a it thing it's it doesn't really matter you could use any disk we used NFS for many years it was very good as well for a specific amount of content today we use icilon there are many providers in the in the market another question thank you thank you Jonathan