### Introduction
In this article, we look at different ways to store pictures of the Earth's surface using Egypt's Digital Surface Model (DSM) as an example. We compare TIFF, GeoTIFF, and Cloud Optimized GeoTIFF (COG) to see how they are different and how they are used in maps and online services. This will help us understand how these methods have changed and made it easier to work with maps and satellite images.
![[Pasted image 20240221151153.png]]
### TIFF vs GeoTiff vs Cloud Optimized GeoTIFF (COG)
| Feature | TIFF (Tagged Image File Format) | GeoTIFF | Cloud Optimized GeoTIFF (COG) |
|----------------|-------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| History | Developed in the mid-1980s by Aldus Corporation (later acquired by Adobe Systems). It was designed to store high-quality raster graphics. | An extension of TIFF introduced in the early 1990s. Developed by NASA and other organizations to include geospatial metadata within the TIFF file structure. | Introduced around 2015-2017. An evolution of GeoTIFF designed specifically for efficient web access and use in cloud computing environments. It optimizes GeoTIFF for spatial data access over the internet. |
| File Extension | .tiff or .tif | .tiff or .tif (same as TIFF, but contains additional geospatial metadata) | .tiff or .tif (same as GeoTIFF, but structured for web-friendly access) |
| Author(s) | Aldus Corporation, later development by Adobe Systems | An informal group of developers and organizations, including NASA | An initiative led by developers in the geospatial community, with significant contributions from companies and organizations involved in cloud computing and GIS technology. |
| Key Features | - Versatile image storage with support for multiple images and data in a single file<br>- Supports various color spaces and compression schemes | - Inherits all TIFF features<br>- Embeds geospatial metadata directly into the file, allowing images to be geo-referenced to the Earth's surface<br>- Supports various coordinate systems and map projections | - Inherits all GeoTIFF features<br>- Optimized for fast, random access of large files over a network<br>- Supports efficient partial reading of image data, useful for web services and cloud applications<br>- Compatible with standard web protocols |
| Use Cases | - General-purpose image storage<br>- Desktop publishing<br>- Graphic design | - Geospatial analysis<br>- Mapping and GIS applications<br>- Remote sensing | - Cloud-based GIS services<br>- Web mapping applications<br>- Remote sensing data distribution and analysis in cloud environments |
### TIFF
#### TIFF Metadata with gdalinfo
To examine the metadata of a TIFF file:
1. open your terminal and try to detect any downloaded (.tif/tiff) file from the internet to get the the metadata of the file with :
```bash
gdalinfo Harby_Main.tiff
```
This command reveals the absence of geospatial metadata (e.g., CRS, spatial resolution), indicating it's a standard image:
```bash
Driver: GTiff/GeoTIFF
Files: Harby_Main.tiff
Size is 4500, 4501
Metadata:
TIFFTAG_DATETIME=2024:02:03 20:48:13
TIFFTAG_RESOLUTIONUNIT=2 (pixels/inch)
TIFFTAG_SOFTWARE=Adobe Illustrator 28.1 (Windows)
TIFFTAG_XRESOLUTION=300
TIFFTAG_YRESOLUTION=300
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=PIXEL
PREDICTOR=2
Corner Coordinates:
Upper Left ( 0.0, 0.0)
Lower Left ( 0.0, 4501.0)
Upper Right ( 4500.0, 0.0)
Lower Right ( 4500.0, 4501.0)
Center ( 2250.0, 2250.5)
Band 1 Block=4500x9 Type=Byte, ColorInterp=Red
Band 2 Block=4500x9 Type=Byte, ColorInterp=Green
Band 3 Block=4500x9 Type=Byte, ColorInterp=Blue
```
### GeoTIFF
#### Download and Merge DSM of Egypt and Convert It to COG Using GDAL and Python
To obtain a DSM for Egypt with a spatial resolution of 30 meters (ideally seeking 1 meter for detailed analysis of Ideal Locations to Implement Solar Panels because we have [[How to Download the Egypt Buildings Dataset]]):
- https://www.eorc.jaxa.jp/ALOS/en/aw3d30/data/index.htm
- https://portal.opentopography.org/datasetMetadata?otCollectionID=OT.112016.4326.2
A script has been developed to automate the download from OpenTopography's API.
Github Repo: https://github.com/Youssef-Harby/dsm2cog/
#### Prerequisites
- Register and obtain your secret key from [OpenTopography](https://opentopography.org/).
- Git
- Python 3.8+
#### 1. **Clone the Repository and Set Up the Virtual Environment**
Clone the repository and create a virtual environment. Install the required packages from `requirements.txt`.
```bash
git clone https://github.com/Youssef-Harby/dsm2cog.git
cd dsm2cog
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```
#### 2. **Run the Script**
Execute the main script to start the process.
```bash
python main.py
```
*dont't forget to fill your secret key in main.py here: [https://github.com/Youssef-Harby/dsm2cog/blob/main/main.py#L9-L10](https://github.com/Youssef-Harby/dsm2cog/blob/main/main.py#L9-L10)*
#### 3. **Navigate to the DSM Data Directory**
Change into the directory containing the DSM data for Egypt.
```bash
cd dsm_data_egypt
```
*check them in QGIS for example*
![[Pasted image 20240219000137.png]]
#### 4. **Create a List of TIFF Files**
Generate a text file listing all TIFF files in the directory.
```bash
ls -1 *.tif > tiff_list.txt
```
#### 5. **Merge TIFF Files Using gdal_merge.py**
Use gdal_merge.py to combine all the raster data into a single TIFF file.
```bash
gdal_merge.py -o mosaic.tif --optfile tiff_list.txt
```
*file size is : 3.14 GB*
*check them in QGIS for example*
![[Pasted image 20240219000345.png]]
#### 6. We know can make a `gdalinfo` on the `mosaic.tif` to know more about **GeoTIFF** still not **COG**
```bash
gdalinfo mosaic.tif
```
Result:
```bash
Driver: GTiff/GeoTIFF
Files: mosaic.tif
mosaic.tif.aux.xml
Size is 39600, 39600
Coordinate System is:
GEOGCRS["WGS 84",
ENSEMBLE["World Geodetic System 1984 ensemble",
MEMBER["World Geodetic System 1984 (Transit)"],
MEMBER["World Geodetic System 1984 (G730)"],
MEMBER["World Geodetic System 1984 (G873)"],
MEMBER["World Geodetic System 1984 (G1150)"],
MEMBER["World Geodetic System 1984 (G1674)"],
MEMBER["World Geodetic System 1984 (G1762)"],
MEMBER["World Geodetic System 1984 (G2139)"],
ELLIPSOID["WGS 84",6378137,298.257223563,
LENGTHUNIT["metre",1]],
ENSEMBLEACCURACY[2.0]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
CS[ellipsoidal,2],
AXIS["geodetic latitude (Lat)",north,
ORDER[1],
ANGLEUNIT["degree",0.0174532925199433]],
AXIS["geodetic longitude (Lon)",east,
ORDER[2],
ANGLEUNIT["degree",0.0174532925199433]],
USAGE[
SCOPE["Horizontal component of 3D system."],
AREA["World."],
BBOX[-90,-180,90,180]],
ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (25.000000000000000,33.000000000000000)
Pixel Size = (0.000277777777778,-0.000277777777778)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( 25.0000000, 33.0000000) ( 25d 0' 0.00"E, 33d 0' 0.00"N)
Lower Left ( 25.0000000, 22.0000000) ( 25d 0' 0.00"E, 22d 0' 0.00"N)
Upper Right ( 36.0000000, 33.0000000) ( 36d 0' 0.00"E, 33d 0' 0.00"N)
Lower Right ( 36.0000000, 22.0000000) ( 36d 0' 0.00"E, 22d 0' 0.00"N)
Center ( 30.5000000, 27.5000000) ( 30d30' 0.00"E, 27d30' 0.00"N)
Band 1 Block=39600x1 Type=Int16, ColorInterp=Gray
Min=-425.000 Max=2511.000
Minimum=-425.000, Maximum=2511.000, Mean=255.691, StdDev=263.865
Metadata:
STATISTICS_APPROXIMATE=YES
STATISTICS_MAXIMUM=2511
STATISTICS_MEAN=255.69077310604
STATISTICS_MINIMUM=-425
STATISTICS_STDDEV=263.86475013193
STATISTICS_VALID_PERCENT=100
```
### Cloud Optimized GeoTIFF (COG)
#### 7. **Convert to Cloud Optimized GeoTIFF (COG)**
Convert the merged TIFF file to Cloud Optimized GeoTIFF format with LZW compression.
```bash
gdal_translate mosaic.tif output_cog.tif -of COG -co COMPRESS=LZW
```
*file size is : 1.32 GB*
Now we can make `gdalinfo` on the **Cloud Optimized GeoTIFF (COG)**
```bash
gdalinfo output_cog.tif
```
Result should be something like :
```bash
Driver: GTiff/GeoTIFF
Files: output_cog.tif
Size is 39600, 39600
Coordinate System is:
GEOGCRS["WGS 84",
ENSEMBLE["World Geodetic System 1984 ensemble",
MEMBER["World Geodetic System 1984 (Transit)"],
MEMBER["World Geodetic System 1984 (G730)"],
MEMBER["World Geodetic System 1984 (G873)"],
MEMBER["World Geodetic System 1984 (G1150)"],
MEMBER["World Geodetic System 1984 (G1674)"],
MEMBER["World Geodetic System 1984 (G1762)"],
MEMBER["World Geodetic System 1984 (G2139)"],
ELLIPSOID["WGS 84",6378137,298.257223563,
LENGTHUNIT["metre",1]],
ENSEMBLEACCURACY[2.0]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
CS[ellipsoidal,2],
AXIS["geodetic latitude (Lat)",north,
ORDER[1],
ANGLEUNIT["degree",0.0174532925199433]],
AXIS["geodetic longitude (Lon)",east,
ORDER[2],
ANGLEUNIT["degree",0.0174532925199433]],
USAGE[
SCOPE["Horizontal component of 3D system."],
AREA["World."],
BBOX[-90,-180,90,180]],
ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (25.000000000000000,33.000000000000000)
Pixel Size = (0.000277777777778,-0.000277777777778)
Metadata:
AREA_OR_POINT=Area
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=BAND
LAYOUT=COG
Corner Coordinates:
Upper Left ( 25.0000000, 33.0000000) ( 25d 0' 0.00"E, 33d 0' 0.00"N)
Lower Left ( 25.0000000, 22.0000000) ( 25d 0' 0.00"E, 22d 0' 0.00"N)
Upper Right ( 36.0000000, 33.0000000) ( 36d 0' 0.00"E, 33d 0' 0.00"N)
Lower Right ( 36.0000000, 22.0000000) ( 36d 0' 0.00"E, 22d 0' 0.00"N)
Center ( 30.5000000, 27.5000000) ( 30d30' 0.00"E, 27d30' 0.00"N)
Band 1 Block=512x512 Type=Int16, ColorInterp=Gray
Overviews: 19800x19800, 9900x9900, 4950x4950, 2475x2475, 1237x1237, 618x618, 309x309
```
You can see the **Overviews** part `19800x19800, 9900x9900, 4950x4950, 2475x2475, 1237x1237, 618x618, 309x309` and `COMPRESSION=LZW`, `INTERLEAVE=BAND` ,`LAYOUT=COG`
Read more about COG Validation here : https://github.com/rouault/cog_validator
![[Pasted image 20240218223856.png]]
These simplified versions are especially helpful when someone wants to quickly see the entire image without needing the full detail. Because of this, the process is much faster instead of loading the highly detailed original image, the system can load one of these simpler overviews, allowing for a speedy preview of the image.
### Upload to Storj
I am using [Storj](https://www.storj.io/), which offers 25 GB of free storage. You can sign up and upload the COG file as follows:
![[Pasted image 20240218225949.png]]
### WebMap preview
#### Interactive Demo - *Zoom into any part of Egypt to display the DSM*
https://www.cogeo.org/map/#/url/https%3A%2F%2Flink.storjshare.io%2Fraw%2Fjxw4q7vcaeaxlf5hwheejfxxynxq%2Ftruemaps-public%2Fcog%2Foutput_cog.tif/center/30.5,27.638/zoom/6
<iframe src="https://www.cogeo.org/map/#/url/https%3A%2F%2Flink.storjshare.io%2Fraw%2Fjxw4q7vcaeaxlf5hwheejfxxynxq%2Ftruemaps-public%2Fcog%2Foutput_cog.tif/center/30.5,27.638/zoom/6" width="100%" height="420" style="border:1px solid black;">
</iframe>
### Open the COG file in QGIS
#### 1. Go to Layer > Data Source Manager (CMD/CTRL + L)
![[Pasted image 20240218230315.png]]
#### 2. Choose `Raster` >> `Protocol HTTP(S), Cloud, etc.` and paste your COG file URL (ex. `https://link.storjshare.io/raw/juj37qat4melrpmooioq65fzgo7q/truemaps-public/aw3d30/output_cog.tif`) then click Add
![[Pasted image 20240218230542.png]]
Result :
![[Pasted image 20240218231716.png]]
### Open the COG file in ArcGIS Pro
#### 1. Go to Insert > Connections > Cloud Store > New Cloud Storage Connection
![[Screenshot 2024-02-19 at 2.33.45 AM.png]]
#### 2. Fill in the information for your object storage (I am using Storj, but it can be AWS, GCP, Azure, MinIO, etc.)
![[Screenshot 2024-02-19 at 2.59.08 AM - 2.png]]
#### 3. A new folder called "Cloud Stores" will appear if the connection is successful, and the connection name you entered will be displayed under "Cloud Stores.":
![[Pasted image 20240219031703.png]]
#### 3. Drag and drop the COG file or Right Click >> Add to Current Map and start your analysis !
![[Pasted image 20240219030359.png]]
![[Pasted image 20240219030559.png]]
### Youtube Video (Arabic)
### REF:
- https://developers.planet.com/docs/planetschool/an-introduction-to-cloud-optimized-geotiffs-cogs-part-1-overview/
- https://www.cogeo.org/
- https://en.wikipedia.org/wiki/TIFF
- https://en.wikipedia.org/wiki/GeoTIFF
- https://opengislab.com/blog/2021/4/25/accessing-cloud-optimized-geotiffs-in-arcgis-pro