 Hello, this is Hans van de Kwas, senior lecturer at IHE Delft Institute for Water Education. In this video, I'm going to explain GIS file formats. We're going to discuss the common file formats for Rust and Vector, and some internet formats, as well as spatial databases, and I'll also focus on good practice with GIS files and how to organize your data. Since the beginning of GIS, there have been developed many different GIS formats, and these formats are mainly created by government mapping agencies or GIS software developers. There are many different formats for Vector and for Rust data. Fortunately, there is a tool, which is called GDAL in the Geospatial Data Abstraction Library, also called GDAL, or GDAL. GDAL is a very important tool to convert between different GIS formats. It can convert file formats from Rust or Vector, with Vector we call it OGR, or Ogre, and it can also do all kinds of GIS analysis and projection transformations. Basically, GDAL is behind many GIS software tools that we can use. It's not only included in QGIS, but also in ArcGIS and many other tools. It is even available as libraries to use in programming languages such as Python. Let's have a look at common vector formats. A very well-known format, but not the best format, is the SRE Shapefile. The name Shapefile suggests that it is only one file. This is a bit confusing. It consists at least of three files with the extension SHP, SHX and Dbf. The SHP file contains the feature geometry, such as the point, line or polygon geometry. SHX is an index to allow seeking forwards and backwards quickly. Then there is the .dbf file, which contains the attribute table. This is the Debase4 format, and you can also open it in a spreadsheet program. However, there are also more extensions that you can see connected to the Shapefile, such as the .prj file, which contains the projection format. And if you share your file, it is of course very useful to also share the .prj file. There are even other extensions that are useful. So the rule is that if you want to share or to copy the Shapefile, you copy all the files related to the file, not only the .shp, because it then will not work in a GIS. Another format that is often seen is the comma-separated values file. It is also confusing because it does not always contain commas as a delimiter. It can also have semicolon or space. Therefore, it is advisable to always open the CSV files in a text editor to check the file. That it doesn't always use the comma as a separator as to do with language settings on your computer. In many languages, the comma is a decimal separator. So in order to import or export these files into GIS or spreadsheet programs, it is important for you to know what the separator is. In QGIS, we can import these files using Add Layer from the main menu and Add the Delimited Text Layer. Let's have a look at some common raster formats. Very common format is the TIFF, a tagged image file format. That is not only used in GIS, but generally for raster images. In GIS, there is some extension to that which is called the GeoTIFF. And the GeoTIFF allows to include the map projection and all kinds of coordinate information. Sometimes you don't find that information inside the TIFF file, but it comes with an external file, the so-called world file, which has the extension dot tfw. When you want to share a GeoTIFF, then it is important to include also this world file. Often you will also see the Archinfo ASCII grid. Also this file can be simply opened in a text editor. When you open it in a text editor, you can read that it has a header. The header gives the number of columns, in this case 4, the number of rows, 6, the x lower left corner, the y lower left corner, the cell size, and the no data value. And then each line contains the pixel values. So in this case, no data, no data, 5 and 2. And we can see on the picture how it's visualized. Many models output in this format or read this format. A better approach to GIS layers is to store them into spatial databases. You have to distinguish between personal databases, which are files on your hard disk, and databases that multiple users can use on the internet. It is of course good practice when you work in a team, in a department, or in an authority for the government to share the data within the department or outside across different institutes. Spatial databases are often not limited to GIS layers such as Raster and Vector data, but can also contain a lot of other data, such as the styling and all kinds of meta data. Spatial databases are different from normal databases only in the way how it treats spatial data. It uses the bounding boxes to have an optimal way of searching in space. Examples of spatial databases are post-GIS, which is well adopted as open source. And there's also Oracle spatial, SRE Geo database, Spatial Light, and the Geo package, the last three are often used as personal databases. With QGS 3, the Geo package is a default format. It is superior to the shapefile because we can easily use it to share our data without being concerned about that it has to contain different files. We can create new Geo packages and we can rewrite to existing Geo packages. And it has an extension.gpkg. We can, besides the layers, also store styling in a Geo package. There's a tool in the processing toolbox that we can use to wrap all the layers that we have into one Geo package. It's called the package layers tool. Obviously, on the web we don't share files, but we connect to databases, but we also need to render the data. And for that reason, there are so-called open geospatial consortium, OGC web services. For data, there's the WMS, which is basically rendering a picture from the data. And there's WFS, the web feature service, which contains the real vector data, the points, lines, and polygons that we can also edit. There's also the web coverage service that contains the real raster data that we can use for editing the data. Besides these services, there are many more and related to data. There's importance of sharing the metadata using the catalog services for the web, or to do online processing using the web processing service. Very often we also see the tiled web map. Other names for that are the raster tile map or the XYZ tiles. They are used to efficiently visualize online maps, because the layers are divided in many different tiles, while with WMS it is one single image. And we can use it in QGIS. In the browser, we can add the XYZ tiles. If we have a URL, it will add it, like in this example, the Google Satellite. And we can also create them ourselves from data using a plugin by Lutra Consulting, which is called the Tiles XYZ plugin. And then we can generate these tiles with different zoom levels to make it efficient, for example, to export it to mobile phone applications. QGIS also has a very useful plugin, the Quick Map Services plugin, which allows us to connect to different web services that support the XYZ tiles. The different files on databases can be converted from one to another. So we can convert one raster file format to another one using a translate function. We find it in the raster menu under conversion and then translate. Of course it uses GDAL to do that. We can do from vector to another vector format using the Save As option or export in the newer versions of QGIS. We can rasterize vector to raster. And the other way around, we can also polygonize raster to vector. Please note that it is important to do that only with discrete raster sources because if we have a continuous data set where many different gradients exist and different values, many pixels will be converted in a single pixel polygon which costs a lot of time for calculation and is probably not very useful. You can also add a GIS layer, a raster or vector layer to a geodatabase and vice versa. Once you're working with GIS, it is important to regularly save your project in a project file. This has the .qgz extension in QGIS. This file includes links to the layers that you are using in your project, the styling of the layers, the zoom level and the extent that you are in and the on-the-fly reprojection. And basically it saves the state of your project. Because it doesn't contain the files but only the links to the files, you should be careful. And if you move the files or delete the files or have been using temporary files, you will see this error when you open the project again. Handle bad layers. And there you have to indicate with the browse button where the file did go. When you are working with files, it is very important to organize your GIS data in a proper way. There is some good practice. Never save data on the desktop or in my documents. These are Windows determined locations and they are often in a path deep in your computer related to your user profile which might be even a roaming profile. It's better to save it on a dedicated partition. When you buy a computer it's advisable that you ask for a separate partition. So you have a C drive where your program files are stored and a D drive or a Z drive where your data is stored. This helps when your computer crashes only the C drive where your programs are stored will be reinstalled and your other partition will be kept as it is and you don't mix up these things. It's important not to use spaces or strange characters in folder and file names. That is both for RGS and QGI is important because many of the tools will simply crash when these characters are used. Also keep the file names intuitive. Of course you will test many different algorithms but it is better to include a little bit of the name of the algorithm in the trials because otherwise the next time you will work with these files you will not remember anymore what test 2 meant. Should learn where your browser saves downloaded files. By default this goes to some download folder somewhere on your computer but most probably you want to save your files somewhere in your project folder where you want to work. Therefore you should know that you can control this in all these browsers using the settings or the options to customize where to put it and I always use a setting that it has to ask me where I want to save the file. And the data is provided in zip files. These are compressed files and it's very important that you learn how to use the zip files. I can recommend to use the 7zip tool which is open source. When you use zip files it's good to check what is in the file because sometimes it's compressed at the folder level and then when you decompress the folder this whole folder will be put on the hard drive. So in that case you will not make first a sub folder and then extract it otherwise you have it twice. But sometimes it's zipped at the file level and then you need to create a folder and unzip the files in that folder. Some examples of file and folder names so the incorrect ones are on the left. We see a space in my documents. We see that it's on the C drive and we see that there's a minus sign used. The correct way is to save it on the D partition for example using an underscore. In the second example we see that module space to be and then a dot in the folder name. That's also not advisable and some of the tools will crash. So it's better to replace the strange signs or the spaces with underscores. So it's important to start thinking from the beginning how you want to organize your data because if you save your project files and you start moving your files you will have to handle those bad layers. So one thing you can do is organize it in this way where you put the different exercises you do in a different folder and we use underscores instead of spaces.