 Hello everybody. I would like to welcome all of you to our second seminar on the Kriyadias environment and some specific topics connected to it. My name is Stanisław Krzyżanowski. I'm a product manager here at CloudFerru and today I would like to talk about the very basics of prototyping with some of the services that are offered within the Kriyadias platform. We will be talking about mainly the Kriyadias Finder and the Kriyadias Jupyter Hub. The aim of today's activity is to show you how to browse, discover, then download data and I hope that we'll manage to perform a very simple visualization of a one selected product. But let's start with a short introduction for anyone who's not familiar with the concept of a Dias with our platform. It's worth pointing out that our today's meeting will be divided into very short presentation. I can count five slides. After that we'll perform a live workshop review during which I will show you how to discover browse, download and finally do a just a bit of processing of the data. So let's start with the very short overview of Kriyadias, what it is, how it works, what's the purpose of its existence. Take a look at the second slide. It is a very high level depiction of the Dias infrastructure. Dias is a chart for data and information access service. European Commission with the European Space Agency a couple years back have decided that they need platforms for data acquisition, dissemination and processing that would lower the bar, lower the entry point for any stakeholders that might be interested in taking advantage or involving their Earth observation data into their processing chains. So our Kriyadias is one of the platforms that was contracted by Issa. We have built it, its first day of its operation was about two years ago, so we are two years up and running, more than two years actually. Main components of the platform are Earth observation data repository that is responsible for collecting, indexing, storing and disseminating the Earth observation data. The second part is the so-called cloud services part. This is a computing cloud that allows our users to access and then process the desired data and create something new, something exciting from it. The Kriyadias environment is supplemented by a set of services about which we will talk more later, but the aim of those services is to allow our users to visualize access, browse the data and manage their own cloud resources. We have talked about the data very briefly, so what types of data you as our users can access. This shirt summary presents what data products, what collections, from which missions are accessible via the Kriyadias platform. As you can see, our main type of data are the Copernicus data. Those are coming from the Sentinel family satellites. Currently we have four types, operationals, those are Sentinel 1, Sentinel 2s, 3s, and Sentinel 5p. P stands for precursor, I think. Additionally, we host data from those who are focused on the coverage of Europe. Landsat are the NASA's American Earth observation satellites. We have envisads as well. This is a different European Earth observation mission focused on marine surveillance. At this slide, I do not highlight that we have also data from the so-called Copernicus services. Those are post-processed Earth observation data with some enhancement and or combined with some in situ data. Copernicus services mainly offer some derivatives of the Earth observation data themselves. Today, we'll be working on the Sentinel 2 data on the Level 1 called L1C, actually. But before we go on to discovering those products, I would like to describe to you what services we'll be using. We'll start with the Kriyadias portal, which serves as an information and navigation hub for any user that would like to start and work with Kriyadias. From this portal, you will see it in a minute, our users can go to any other service. Then we'll use the Earth observation finder in order to brose and discover some products that might be interesting to us. After we determine what products we need, we'll switch to the Kriyadias Chipter notebook, which is, if anyone does know what a Jupiter notebook is. Jupiter notebook is an interactive coding environment accessible from the ordinary internet browser. Our Kriyadias Jupiter instances are hosted within the cloud ferrous infrastructure, so they have access to the whole Earth observation data repository, something that I hope will manage to successfully show in the end of this presentation. One thing to mention, all the services and functionalities that we will be using from now on are accessible to anyone with no fees or charges. However, our users have to register themselves in order to have a Jupiter notebook instance set up, but I do not intend to go through the registration process with you today. I think it was covered during the last webinar. It is quite self-explanatory, I would say, but if you have any questions or you would face any obstacles, please contact us. This is it for the presentation. We'll come back to it later. I have one more slide for you when we will be talking about the Sentinel 2 data, but for now I would like to point your attention to our Kriyadias portal, which I hope I will be able to show you from my screen. Please give me a minute. Due to the fact that we are in a not a very big crowd here, I would suggest that if you have any questions, please ask them of any time you like. I will try to answer. I think I can make some stops. Okay, okay, okay, I hope that you can see the Kriyadias portal website. This is the very starting place of our endeavor. First thing we need to do is log in. I have my login credentials ready, but it's quite easy. As I mentioned before, we have to wait and now we'll get to the Kriyadias Earth observation finder. Currently, you can see the graphical user interface of the E8 of other. It is quite a powerful device service that allows you to find any data you may require, you may need in your processes. This first part will be performing now. We'll do it manually. Afterwards, I will show you how to do exactly the same thing, but in a more automated way that would allow you to implement it in your processes or in your algorithm or code. I was thinking a bit which part of the world we should choose and I've got inspired by the fact that in November, the European Space Agency will be launching its next Sentinel satellite. It will be Sentinel-6 launched from a Wanderberg airfare base in California because it will be launched using the Falcon 9 rocket. First thing we'll do, we type a place that we are interested in, then we have to select a date of observation. That's something I'm doing currently. Subsequently, we want to choose what kind of collection we are interested in. For the sake of this webinar, I would like to work on the Sentinel-2 data. Precisely, it will be Sentinel-2 level 1. As you will see in a couple of steps, this will be the Sentinel-2 level 1 data, which is a bit more post-processed than the level 1b, but this is the data that is available freely and openly on the Creator's platform, so no problem here. As we have selected our place we're interested in dates that we're looking for. As you can see, you're able to select between observed and published. Normally, commonly, the time difference between observation and publishing of data is quite small, so we tend to use observed, but it may come useful at times to have this option. We're looking at one more time for Sentinel-2 data on the first processing level. We do not have any preferences as to the orbit number, so what we do now, now we just press search and fingers crossed. We will find something. It seems that we do not find anything, so as it may happen sometimes, we do not find anything because I've made a spelling mistake, so let's try our luck one more time. Yes, we did find five products that suit our predefined requirements. Those are all, as you can see, Sentinel-2 all 1c products. From now on, if we were to be satisfied with manual selection, we can select one product. We will have a product identifier highlighted here, and using this product identifier, we would be able to access it from the EOData repository within the CreoDias. This is very important. This is a product identifier within the CreoDias, but we are not that easily satisfied, so I will show you how the API works, the programmatic interface. In order to access the programmatic interface, or to access, use the programmatic interface, you have to formulate a query that would ask, let's say, ask the CreoDias finder about the specific data. From the CreoDias portal, you can access a knowledge base in which there is an extensive Finder API documentation that would tell you how to formulate those queries. If you don't want to formulate them, say, you can use the CreoDias finder, select your requirements, and after you press search, the CreoDias finder formulates a query for you. Let's take a look how a response to such a query looks like. First of all, we will put this query in a internet browser to show you how the response looks like. What should you do? You simply paste the query in the browser. Okay, we have it. I hope you can see it clearly. This query should give us the very same results that the finder gave us. So we should have at least five different products listed here. As you can see, we have products ranging from zero to four, so that is five products. Previously, I have selected the last product. The numbering of the products is based on observation time, so this one was observed the earliest, I think. Yes, because I have, as you can see here, the last product, the very early one from the 5th of July, has a approximately 0% cloud cover, and the product we have selected here within the API response has a 0.0171 cloud cover, which is roughly zero. This is the product that we shall use for the next part of our presentation, but we want to take it a step further. We don't want to use the API manually copied within the browser. So let us come back to the Creadias portal. We are back, and now we shall move to the Creadias Jupyter notebook, hosted by the Jupyter Hub within the environment. It can be selected from the tools menu as last position, so called Jupyter Hub. You have to sign in, as I mentioned before, so we are signing in with the Creadias, and let's see what happens here. As we are waiting for the login process to finish, I think it's on my side, it has worked, or yours as well. Any delays you witness here are main due to heavy load on a single laptop when you are presenting and working. So I have prepared for you a basic script for data visualization. This script consists not only of some Python code, but we will use just a tiny, tiny bit of Unix commands in order to get all information about data we require first. So this script allows you to test the S3 capabilities on the Creadias. The S3 is a access protocol that can be used in order to download data or parts of the data. As you remember just before, we have copied, I'm waiting for yours to update. We have copied the query from the Creadias finder interface. We will do one more time. We have put it into the browser to see if it works correctly. We have seen that it, yes, it does. And right now we will utilize the notebook and if we can use the same query from the notebook level to get the information we need. In this particular query, I have decided I don't need all additional data. So this part of code will only produce the product identifiers I have talked before and we will choose one later to download parts of it and process it, visualize it actually. For anyone that is not familiar with Jupyter notebooks, what you can see on your screens right now is a typical Jupyter notebook. This one has our Creadias' layout with our background but the functionalities are only extended not lessened from a normal, let's say generic Jupyter notebook. Each of the cells contains a part of code. It can be executed separately into something we'll be doing for the sake of this presentation. In order to, first to cells that we'll be using are designed, are thought to produce us a working product identifier and then root address to access the data. So let's see if we can get the same result at the finder protocol user interface from the browser answer of the finder API query and from the Jupyter notebook. Quite typically, I think we have obtained what we need. It's actually untypical. It's not in a very well behaved manner but it works. So let's continue. Let's find the product. We may desire as you can see each, can you actually see it? Okay, I hope you can see it now. I don't know what happened with the webinar application but let's hope it's okay now. Yes, you can see it. I hope so. So after running the API query 3D, could you, we have a question from Matjelitewski. Could you describe the advantages of using Jupyter for small businesses and startups? Oh yes, I gladly would but I think that's a question that we will answer after we finish the coding, the coding presentation if you don't mind. So I'm just going to query the query of this finder about the products we may require. So next thing that we will do is we shall copy the product identifier of a product we need. It has been copied and at this moment I would like to come back to the presentation part because I need to show you the structure of Sentinel to L1C products to give you the full view about what we will be doing right now. Please switch your attention to the slide number five. This is a, from the courtesy of the United States Geographical Survey. This is a graph that shows you how the product, the Sentinel to products are structured and what we have managed to achieve is to get the product dot save identifier. The dot save format is a specific format to Sentinel to products. What we now need in order to get a visualization is to define the so-called granule. When we talk about the Sentinel to L1C products, granule is the same, basically the same thing as tile. If you don't know what a tile is, tile is a one portion of a grid that the earth has been divided into each tile has a specific let's say code attached to it so it can be identified. So what we need to is to get a tile let's call it address in order to access the image data to be visualized. The image data is stored in so-called JP2 format which is JPEC 2000 compressed format. It's a typical operation for this data. So I will try to get a pointer. I hope you can see my pointer. The line we are following is from product to product dot save. That's where we are now. We have obtained the product ID so we have the product dot save address and now we are looking for the granule address because we want to get to the tile folder then to get to the image data products. The image data products are divided between bands from 1 to 12. Each band has a separate image and what we will do we will download those bands and then display them within one picture to visualize part of land we are interested in. In this case this will be a place from which the Sentinel 6 will be launched in just three months I think. So as we have clarified this let us come back to the Jupiter notebook. Note that once again I'm waiting for you to see it. Yes I think you have seen it. That's the status you should remember from the last time we've been here. Please recall as well the structure of the data we want to access. As I have shown you a desired, I will highlight it now. I hope you can see it. We have a desired product ID for a one specific Sentinel 6 product but as you can remember we need a let's call it granule or a tile ID in order to access the compressed JP2 data. In order to do so we will take advantage of the fact that each of the Jupiter notebook instances is connected within the cloud with the Earth Observation Data Repository and I can use a very simple as you can see UNIX command to get the information about about the granule as I know the structure of the documents. What I have to do is I have to actually only copy the desired product ID. So I select the product ID copied and then I shall paste it right here and it should produce something to this. That's a those numbers and letters. This is a tile or a granule ID. So let's go to the access through S3 and downloading part. So for access and download we will be using a library card called Boto Free. It's using the S3 protocol in order to access the data. The data is open so we don't have to put any access or secret keys. The host of our data is data.clothferrer.com and the key path of the data we want to access is the same as actually the product ID. So we will just copy the product ID and then we shall paste it instead of this one which I was checking out before the webinar to see if it's working. Then we have to copy the granule ID and put it in here because this is actually a directory that our code will be looking for for the data. We end this key path with slash image data as you can see the product structure. This is the folder where we can find the band specific JP to products. Afterwards we can decide which bands we want to display using the script. This time we will settle on bands 2, 3 and 4 which should produce a true color imagery. Then we have to do one more thing. We have to tell our script how those separate products are named but it's quite easy. We just have to choose a tile ID. This thing is a tile ID and the acquisition date because as you may know or may not know the product identifiers of Sentinel 2 products actually of each of any, maybe not any majority of our observation products, the products identifiers are composed of parts that have a very specific meaning. In this part I think we can quickly, quickly discuss it at least some parts of it. This is the platform that was taking the measurement, the Sentinel 2B. Here we have the MSI, the instrument, then L1C as is the processing level of the product. Next part is the date of the observation. Here we have for example the number of the relative orbit of the platform when the observation was made. The last part is I think the time reference from the ground station. I'm not quite sure. I think so. So we have told our script what to look for, where to look for it, which ones we are interested in and right now we'll just run it and we'll have to wait just a I think approximately 30 seconds in order for it to process and download the specified products. During this time I can show you that when we go back to the home folder of your Jupyter notebook, it should in a second produce more folders that will contain the data we are downloading. Let's see. It's still processing. We don't have anything here, so we have to wait. I will take advantage of this moment. Let's go back here. I will take advantage of this moment and answer the question that has been sent to us by Mr. Matij Litewski. The question was could you describe the advantages of using Jupyter for small business and startups? So there are many advantages to using Jupyter, not only to small businesses and startups, but let's focus on this. First of all, this is a free of charge online programming environment with access to a huge Earth observation data repository. Currently the EOData repository of the Creeodias is more than 17.5 petabytes of pure EOData and such environment does not come by often, so it's something that should be very attractive to any startup and or small business because when you're a startup you don't want to invest in infrastructure, you may not be able, especially at first, you may not be able to purchase lots of cloud resources, so starting your prototyping, starting your work, exploring your possibilities with such an online environment offered to you would be very beneficial. Actually for anyone, even for bigger entities, Jupyter NodeWork that operates within the same cloud environment as the virtual machine and the Earth observation data repository may be very beneficial because they are very cost-effective and quite easy to set up and use user interfaces, so especially initially during prototyping or during prototyping phase, many, many people use Jupyter NodeBooks as user interfaces because they do not demand additional front-end work, they do not demand lots of effort and it's quite easy to present the results of the work using those. Okay, while I was answering to this question, it seems that the script has managed to download the desired products, the desired pens, two, three and four. Let's come back to the home folder of my Jupyter notebook and check if they are here. I hope that you can see that we have three additional files called bo2, bo3, bo4.jp2, that's what we want to see, that's what we are interested in, so we can go to the very last part of our small and basic presentation. This is the visualization of the Sentinel-2 L1C bands that we have downloaded just before. I will start up the script because it may take some time as well, I hope not that much. For this part of the script we use rasteriolibrary and the mutplip library. The script is quite simple actually and the result of the script should be a creation of a Sentinel TrueColor 2.tif file and visualization of it right here, so we have managed to go all the way from data discovery using Decreodias to API query to data download and then visualization using three services available to all Acreodias users. It was the very basic, I hope that in the future webinars, during the future webinars, we will be able to show you more, to give you more tools to utilize within your processes. I hope that this presentation will maybe inspire you or lead you to do more to explore more our environment, our cloud. If you have any questions that came up just recently or during the presentation, please ask them now. I will try to do my best to give you satisfactory answer. As there are no other questions, there is one. Hello, Florian, thank you for your question. Florian asks, is there a portfolio of notebooks with more examples available online? The answer to your question is that we are working on such a portfolio. We want to create a, let's call it, library of notebooks for our users to try on, to try out. But it is not yet ready. I hope it will be within, let's say, two, three months tops. But you can find many notebooks, notebook examples in the internet. But that's something that we will be doing as soon as possible to get you some more some more examples, some more inspiration to your work.