{ "cells": [ { "cell_type": "markdown", "id": "confidential-bargain", "metadata": {}, "source": [ "This user guide will show you how to work with metadata in evalscripts. We will focus on using objects [`scenes`](/evalscript/v3/#scenes), [`inputMetadata`](/evalscript/v3/#inputmetadata), and [`outputMetadata`](/evalscript/v3/#outputmetadata). Use cases, covered with the examples below, include accessing metadata and using it in processing, passing the metadata to an output file `userdata.json`, and adding your own metadata to the file.\n", "\n", "Note that metadata normally provided in raster format is available as bands in Sentinel Hub. Such metadata can be accessed and processed in evalscript in the same manner as any other input band. This is not covered in this guide, but you can find basic examples and such metadata listed in the `Data` section for each data collection e.g. [sunAzimuthAngles](https://docs.sentinel-hub.com/api/latest/data/sentinel-2-l1c/#available-bands-and-data).\n", "\n", "Each example below begins with a description that highlights the important points of the example. All examples output also processed satellite images (average values of NDVI or band B02) but we do not display them here, since the focus is on metadata. To run the examples, you only need to have Python installed on your machine and an active Sentinel Hub account. You will always need to run the code in the chapter \"Authentication\" while the rest of the examples can be run independently.\n", "\n", "The jupyter notebook with all examples can be downloaded [here](user-guides/metadata/notebook/metadata_sh_docs.ipynb)." ] }, { "cell_type": "markdown", "id": "square-preserve", "metadata": {}, "source": [ "## Authentication" ] }, { "cell_type": "markdown", "id": "prescribed-couple", "metadata": {}, "source": [ "First, we need to fetch an access token, which we will use to authenticate all Sentinel Hub requests. To do so, replace `` and `` in the code snippet below with your client id and client secret, respectively and run the code. To learn how to get your client id and client secret, read this [documentation](https://docs.sentinel-hub.com/api/latest/api/overview/authentication/#registering-oauth-client)." ] }, { "cell_type": "code", "execution_count": null, "id": "intimate-determination", "metadata": {}, "outputs": [], "source": [ "from oauthlib.oauth2 import BackendApplicationClient\n", "from requests_oauthlib import OAuth2Session\n", "import os, io, tarfile, json, requests\n", "\n", "client_id = ''\n", "client_secret = ''\n", "client = BackendApplicationClient(client_id=client_id)\n", "oauth = OAuth2Session(client=client)\n", "oauth.fetch_token(token_url='https://services.sentinel-hub.com/auth/realms/main/protocol/openid-connect/token',\n", " client_secret=client_secret, include_client_id=True)" ] }, { "cell_type": "markdown", "id": "quarterly-incentive", "metadata": {}, "source": [ "The access token is stored in the `oauth` object, which will be used to send all subsequent requests." ] }, { "cell_type": "markdown", "id": "vertical-tulsa", "metadata": {}, "source": [ "## Check which metadata is available" ] }, { "cell_type": "markdown", "id": "settled-anatomy", "metadata": {}, "source": [ "The metadata is stored in two objects, which we call `inputMetadata` and `scenes`. Their properties are documented [here](/evalscript/v3/#inputmetadata) and [here](/evalscript/v3/#scenes), respectively. However, the properties of the `scenes` object can be different depending on the selected:\n", "- mosaicking (e.g. ORBIT or TILE),\n", "- data collection (Sentinel-2 L2A, Sentinel-1, Sentinel-5p, ...),\n", "- function in the evalscript (`evaluatePixel`, `preProcessScenes`, `updateOutputMetadata`).\n", "\n", "A convenient way to check which metadata is for your request available in `scenes` is to dump (i.e. write) all properties of the object to userdata.json file. This can be achieved with the Processing API as shown in this basic [example](https://docs.sentinel-hub.com/api/latest/data/sentinel-2-l1c/examples/#true-color-and-metadata-multi-part-response-geotiff-and-json). The two examples below show few more tricks that can be used to explore `scenes` object." ] }, { "cell_type": "markdown", "id": "relative-dependence", "metadata": {}, "source": [ "### Properties of scenes object and mosaicking ORBIT" ] }, { "cell_type": "markdown", "id": "amateur-lender", "metadata": {}, "source": [ "This example shows:\n", "- How to access metadata when mosaicking is ORBIT using `scenes.orbits`.\n", "- How to pass metadata from `scenes` to userdata.json file using `outputMetadata.userData` in `updateOutputMetadata` function." ] }, { "cell_type": "code", "execution_count": null, "id": "introductory-primary", "metadata": {}, "outputs": [], "source": [ "url = 'https://services.sentinel-hub.com'\n", "\n", "evalscript = \"\"\"\n", "//VERSION=3\n", "function setup() {\n", " return {\n", " input: [\"B02\", \"dataMask\"],\n", " mosaicking: Mosaicking.ORBIT,\n", " output: {\n", " id: \"default\",\n", " bands: 1\n", " }\n", " }\n", "}\n", "\n", "function evaluatePixel(samples, scenes, inputMetadata, customData, outputMetadata) {\n", " //Average value of band B02 based on the requested scenes\n", " var sumOfValidSamplesB02 = 0\n", " var numberOfValidSamples = 0\n", " for (i = 0; i < samples.length; i++) {\n", " var sample = samples[i]\n", " if (sample.dataMask == 1){\n", " sumOfValidSamplesB02 += sample.B02\n", " numberOfValidSamples += 1\n", " }\n", " }\n", " return [sumOfValidSamplesB02 / numberOfValidSamples]\n", "}\n", "\n", "function updateOutputMetadata(scenes, inputMetadata, outputMetadata) {\n", " outputMetadata.userData = {\n", " \"inputMetadata\": inputMetadata\n", " }\n", " outputMetadata.userData[\"orbits\"] = scenes.orbits\n", "}\n", "\"\"\"\n", "\n", "request = {\n", " \"input\": {\n", " \"bounds\": {\n", " \"bbox\": [13.8, 45.8, 13.9, 45.9]\n", " },\n", " \"data\": [{\n", " \"type\": \"sentinel-2-l1c\",\n", " \"dataFilter\": {\n", " \"timeRange\": {\n", " \"from\": \"2020-12-01T00:00:00Z\",\n", " \"to\": \"2020-12-06T23:59:59Z\"\n", " }\n", " }\n", " }]\n", " },\n", " \"output\": {\n", " \"responses\": [{\n", " \"identifier\": \"default\",\n", " \"format\": {\n", " \"type\": \"image/tiff\"\n", " }\n", " },\n", " {\n", " \"identifier\": \"userdata\",\n", " \"format\": {\n", " \"type\": \"application/json\"\n", " }\n", " }\n", " ]\n", " },\n", " \"evalscript\": evalscript\n", "}\n", "\n", "headers = {\n", " 'Content-Type': 'application/json',\n", " 'Accept': 'application/x-tar'\n", "}\n", "\n", "response = oauth.post(f\"{url}/api/v1/process\", headers=headers, json = request)\n", "tar = tarfile.open(fileobj=io.BytesIO(response.content))\n", "userdata = json.load(tar.extractfile(tar.getmember('userdata.json')))\n", "userdata" ] }, { "cell_type": "markdown", "id": "entire-cambodia", "metadata": {}, "source": [ "### Properties of scenes object and mosaicking TILE" ] }, { "cell_type": "markdown", "id": "requested-collectible", "metadata": {}, "source": [ "This example shows how to:\n", "- Access scenes metadata when mosaicking is TILE using `scenes.tiles` and write it to userdata.json file.\n", "- How to calculate a maximum value of band B02 and write it to userdata.json file. Note that we use a global variable `maxValueB02` so that we can assign a value to it in `evaluatePixel` function but write its value to metadata in `updateOutputMetadata` function. The advantage of this approach is that `maxValueB02` is written to metadata only once and not for each output pixel." ] }, { "cell_type": "code", "execution_count": null, "id": "waiting-culture", "metadata": {}, "outputs": [], "source": [ "url = 'https://services.sentinel-hub.com'\n", "\n", "evalscript = \"\"\"\n", "//VERSION=3\n", "function setup() {\n", " return {\n", " input: [\"B02\", \"dataMask\"],\n", " mosaicking: Mosaicking.TILE,\n", " output: {\n", " id: \"default\",\n", " bands: 1\n", " }\n", " }\n", "}\n", "\n", "var maxValueB02 = 0\n", "\n", "function evaluatePixel(samples, scenes, inputMetadata, customData, outputMetadata) {\n", " //Average value of band B02 based on the requested tiles\n", " var sumOfValidSamplesB02 = 0\n", " var numberOfValidSamples = 0\n", " for (i = 0; i < samples.length; i++) {\n", " var sample = samples[i]\n", " if (sample.dataMask == 1){\n", " sumOfValidSamplesB02 += sample.B02\n", " numberOfValidSamples += 1\n", " if (sample.B02 > maxValueB02){\n", " maxValueB02 = sample.B02\n", " }\n", " }\n", " }\n", " return [sumOfValidSamplesB02 / numberOfValidSamples]\n", "}\n", "\n", "function updateOutputMetadata(scenes, inputMetadata, outputMetadata) {\n", " outputMetadata.userData = { \"tiles\": scenes.tiles }\n", " outputMetadata.userData.maxValueB02 = maxValueB02\n", "}\n", "\"\"\"\n", "\n", "request = {\n", " \"input\": {\n", " \"bounds\": {\n", " \"bbox\": [13.8, 45.8, 13.9, 45.9]\n", " },\n", " \"data\": [{\n", " \"type\": \"sentinel-2-l1c\",\n", " \"dataFilter\": {\n", " \"timeRange\": {\n", " \"from\": \"2020-12-01T00:00:00Z\",\n", " \"to\": \"2020-12-06T23:59:59Z\"\n", " }\n", " }\n", " }]\n", " },\n", " \"output\": {\n", " \"responses\": [{\n", " \"identifier\": \"default\",\n", " \"format\": {\n", " \"type\": \"image/tiff\"\n", " }\n", " },\n", " {\n", " \"identifier\": \"userdata\",\n", " \"format\": {\n", " \"type\": \"application/json\"\n", " }\n", " }\n", " ]\n", " },\n", " \"evalscript\": evalscript\n", "}\n", "\n", "headers = {\n", " 'Content-Type': 'application/json',\n", " 'Accept': 'application/x-tar'\n", "}\n", "\n", "response = oauth.post(f\"{url}/api/v1/process\", headers=headers, json = request)\n", "tar = tarfile.open(fileobj=io.BytesIO(response.content))\n", "userdata = json.load(tar.extractfile(tar.getmember('userdata.json')))\n", "userdata" ] }, { "cell_type": "markdown", "id": "suspended-richards", "metadata": {}, "source": [ "## Output metadata into userdata.json file" ] }, { "cell_type": "markdown", "id": "innovative-federation", "metadata": {}, "source": [ "In this example, we write several pieces of information to the userdata.json file:\n", "- A version of the software with which the data was processed. We take this information from `inputMetadata`.\n", "- Dates when the data used for processing was acquired. We take this information from `scene.tiles`.\n", "- Values set by user and used for processing, such as thresholds (e.g. `ndviThreshold`) and array of values (e.g. `notAllowedDates`).\n", "- Dates of all tiles available before we filtered out those acquired on dates given in `notAllowedDates` array. These dates are listed in `tilesPPSDates` property of userData. Note how we used a global variable `tilesPPS`: we assigned it a value in `preProcessScenes` and output it in `updateOutputMetadata` function.\n", "- Dates of all tiles available after the filtering. These dates are listed in `tilesDates` property of userData.\n", "- Description of the processing implemented in the evalscript and links to external resources." ] }, { "cell_type": "code", "execution_count": null, "id": "humanitarian-republican", "metadata": {}, "outputs": [], "source": [ "url = 'https://services.sentinel-hub.com'\n", "\n", "evalscript = \"\"\"\n", "//VERSION=3\n", "function setup() {\n", " return {\n", " input: [\"B08\", \"B04\", \"dataMask\"],\n", " mosaicking: Mosaicking.TILE,\n", " output: {\n", " id: \"default\",\n", " bands: 1\n", " }\n", " }\n", "}\n", "\n", "// User's inputs\n", "var notAllowedDates = [\"2020-12-06\", \"2020-12-09\"]\n", "var ndviThreshold = 0.2\n", "\n", "var tilesPPS = []\n", "function preProcessScenes(collections) {\n", " tilesPPS = collections.scenes.tiles\n", " collections.scenes.tiles = collections.scenes.tiles.filter(function(tile) {\n", " var tileDate = tile.date.split(\"T\")[0];\n", " return !notAllowedDates.includes(tileDate);\n", " })\n", " return collections\n", "}\n", "\n", "function evaluatePixel(samples, scenes, inputMetadata, customData, outputMetadata) {\n", "\n", " var valid_ndvi_sum = 0\n", " var numberOfValidSamples = 0\n", " for (i = 0; i < samples.length; i++) {\n", " var sample = samples[i]\n", " if (sample.dataMask == 1){\n", " var ndvi = (sample.B08 - sample.B04)/(sample.B08 + sample.B04)\n", " if (ndvi <= ndviThreshold){\n", " valid_ndvi_sum += ndvi\n", " numberOfValidSamples += 1\n", " }\n", " }\n", " }\n", "\n", " return [valid_ndvi_sum / numberOfValidSamples]\n", "}\n", "\n", "function updateOutputMetadata(scenes, inputMetadata, outputMetadata) {\n", " outputMetadata.userData = {\n", " \"inputMetadata.serviceVersion\": inputMetadata.serviceVersion\n", " }\n", "\n", " outputMetadata.userData.description = \"The evalscript calculates average ndvi \" +\n", " \"in a requested time period. Data collected on notAllowedDates is excluded. \" +\n", " \"ndvi values greater than ndviThreshold are excluded. \" +\n", " \"More about ndvi: https://www.indexdatabase.de/db/i-single.php?id=58.\"\n", "\n", " // Extract dates for all available tiles (before filtering)\n", " var tilePPSDates = []\n", " for (i = 0; i < tilesPPS.length; i++){\n", " tilePPSDates.push(tilesPPS[i].date)\n", " }\n", " outputMetadata.userData.tilesPPSDates = tilePPSDates\n", "\n", " // Extract dates for tiles after filtering out tiles with \"notAllowedDates\"\n", " var tileDates = []\n", " for (i = 0; i < scenes.tiles.length; i++){\n", " tileDates.push(scenes.tiles[i].date)\n", " }\n", " outputMetadata.userData.tilesDates = tileDates\n", "\n", " outputMetadata.userData.notAllowedDates = notAllowedDates\n", " outputMetadata.userData.ndviThreshold = ndviThreshold\n", "}\n", "\"\"\"\n", "\n", "request = {\n", " \"input\": {\n", " \"bounds\": {\n", " \"bbox\": [13.8, 45.8, 13.9, 45.9]\n", " },\n", " \"data\": [{\n", " \"type\": \"sentinel-2-l1c\",\n", " \"dataFilter\": {\n", " \"timeRange\": {\n", " \"from\": \"2020-12-01T00:00:00Z\",\n", " \"to\": \"2020-12-15T23:59:59Z\"\n", " }\n", " }\n", " }]\n", " },\n", " \"output\": {\n", " \"responses\": [{\n", " \"identifier\": \"default\",\n", " \"format\": {\n", " \"type\": \"image/tiff\"\n", " }\n", " },\n", " {\n", " \"identifier\": \"userdata\",\n", " \"format\": {\n", " \"type\": \"application/json\"\n", " }\n", " }\n", " ]\n", " },\n", " \"evalscript\": evalscript\n", "}\n", "\n", "headers = {\n", " 'Content-Type': 'application/json',\n", " 'Accept': 'application/x-tar'\n", "}\n", "\n", "response = oauth.post(f\"{url}/api/v1/process\", headers=headers, json = request)\n", "\n", "tar = tarfile.open(fileobj=io.BytesIO(response.content))\n", "userdata = json.load(tar.extractfile(tar.getmember('userdata.json')))\n", "userdata" ] }, { "cell_type": "markdown", "id": "theoretical-lender", "metadata": {}, "source": [ "## Filter tiles based on metadata" ] }, { "cell_type": "markdown", "id": "compatible-behalf", "metadata": {}, "source": [ "### Satellite (S2A vs S2B)" ] }, { "cell_type": "markdown", "id": "rocky-gather", "metadata": {}, "source": [ "Here we parse original tile ids to get the information with which satellite, S2A or S2B, this tile was collected. Then we filter out the tiles acquired from the satellite S2A and only process the data acquired from the satellite S2B." ] }, { "cell_type": "code", "execution_count": null, "id": "normal-tyler", "metadata": {}, "outputs": [], "source": [ "url = 'https://services.sentinel-hub.com'\n", "\n", "evalscript = \"\"\"\n", "//VERSION=3\n", "function setup() {\n", " return {\n", " input: [\"B02\", \"dataMask\"],\n", " mosaicking: Mosaicking.TILE,\n", " output: {\n", " id: \"default\",\n", " bands: 1\n", " }\n", " }\n", "}\n", "\n", "function getSatelliteFromProductId(productId) {\n", " textParts = productId.split(\"_\")\n", " satellite = textParts[0];\n", " return satellite\n", "}\n", "\n", "// Filter by satellite\n", "function preProcessScenes(collections) {\n", " collections.scenes.tiles = collections.scenes.tiles.filter(function (tile) {\n", " return getSatelliteFromProductId(tile.productId) == \"S2B\"});\n", " return collections;\n", "}\n", "\n", "function evaluatePixel(samples, scenes, inputMetadata, customData, outputMetadata) {\n", " outputMetadata.userData = {\n", " \"tiles\": scenes.tiles\n", " }\n", "\n", " //Average value of band B02 based on the requested tiles\n", " var sumOfValidSamplesB02 = 0\n", " var numberOfValidSamples = 0\n", " for (i = 0; i < samples.length; i++) {\n", " var sample = samples[i]\n", " if (sample.dataMask == 1){\n", " sumOfValidSamplesB02 += sample.B02\n", " numberOfValidSamples += 1\n", " }\n", " }\n", " return [sumOfValidSamplesB02 / numberOfValidSamples]\n", "}\n", "\"\"\"\n", "\n", "request = {\n", " \"input\": {\n", " \"bounds\": {\n", " \"bbox\": [13.8, 45.8, 13.9, 45.9]\n", " },\n", " \"data\": [{\n", " \"type\": \"sentinel-2-l1c\",\n", " \"dataFilter\": {\n", " \"timeRange\": {\n", " \"from\": \"2020-12-01T00:00:00Z\",\n", " \"to\": \"2020-12-06T23:59:59Z\"\n", " }\n", " }\n", " }]\n", " },\n", " \"output\": {\n", " \"responses\": [{\n", " \"identifier\": \"default\",\n", " \"format\": {\n", " \"type\": \"image/tiff\"\n", " }\n", " },\n", " {\n", " \"identifier\": \"userdata\",\n", " \"format\": {\n", " \"type\": \"application/json\"\n", " }\n", " }\n", " ]\n", " },\n", " \"evalscript\": evalscript\n", "}\n", "\n", "headers = {\n", " 'Content-Type': 'application/json',\n", " 'Accept': 'application/x-tar'\n", "}\n", "\n", "response = oauth.request(\"POST\", f\"{url}/api/v1/process\", headers=headers, json = request)\n", "tar = tarfile.open(fileobj=io.BytesIO(response.content))\n", "userdata = json.load(tar.extractfile(tar.getmember('userdata.json')))\n", "userdata" ] }, { "cell_type": "markdown", "id": "embedded-batch", "metadata": {}, "source": [ "### Relative orbit id" ] }, { "cell_type": "markdown", "id": "peripheral-holly", "metadata": {}, "source": [ "This example shows how to filter tiles based on relative orbit id. The steps are:\n", "- We parse absolute orbit id from product id, which is available in `scenes` object and looks like `'S2B_MSIL1C_20201206T100409_N0209_R122_T33TUL_20201206T111219'`. This is done with the function `getRelativeOrbitIdFromProductId`.\n", "- Once we have a relative orbit id for each tile, we use `preProcessScenes` function to select tiles from the relative orbit 122. " ] }, { "cell_type": "code", "execution_count": null, "id": "posted-tennis", "metadata": {}, "outputs": [], "source": [ "url = 'https://services.sentinel-hub.com'\n", "\n", "evalscript = \"\"\"\n", "//VERSION=3\n", "function setup() {\n", " return {\n", " input: [\"B02\", \"dataMask\"],\n", " mosaicking: Mosaicking.TILE,\n", " output: {\n", " id: \"default\",\n", " bands: 1\n", " }\n", " }\n", "}\n", "\n", "function getRelativeOrbitIdFromProductId(productId) {\n", " textParts = productId.split(\"_\")\n", " relativeOrbitId = parseInt(textParts[4].substring(1));\n", " return relativeOrbitId\n", "}\n", "\n", "// Filter by relative orbit id\n", "function preProcessScenes(collections) {\n", " var allowedRelativeOrbits = [122]\n", " collections.scenes.tiles = collections.scenes.tiles.filter(function(tile) {\n", " var relativeOrbitId = getRelativeOrbitIdFromProductId(tile.productId);\n", " return allowedRelativeOrbits.includes(relativeOrbitId)\n", " })\n", " return collections;\n", "}\n", "\n", "function evaluatePixel(samples, scenes, inputMetadata, customData, outputMetadata) {\n", " outputMetadata.userData = {\n", " \"scenes\": scenes.tiles\n", " }\n", "\n", " //Average value of band B02 based on the requested tiles\n", " var sumOfValidSamplesB02 = 0\n", " var numberOfValidSamples = 0\n", " for (i = 0; i < samples.length; i++) {\n", " var sample = samples[i]\n", " if (sample.dataMask == 1){\n", " sumOfValidSamplesB02 += sample.B02\n", " numberOfValidSamples += 1\n", " }\n", " }\n", " return [sumOfValidSamplesB02 / numberOfValidSamples]\n", "}\n", "\"\"\"\n", "\n", "request = {\n", " \"input\": {\n", " \"bounds\": {\n", " \"bbox\": [13.8, 45.8, 13.9, 45.9]\n", " },\n", " \"data\": [{\n", " \"type\": \"sentinel-2-l1c\",\n", " \"dataFilter\": {\n", " \"timeRange\": {\n", " \"from\": \"2020-12-01T00:00:00Z\",\n", " \"to\": \"2020-12-06T23:59:59Z\"\n", " }\n", " }\n", " }]\n", " },\n", " \"output\": {\n", " \"responses\": [{\n", " \"identifier\": \"default\",\n", " \"format\": {\n", " \"type\": \"image/tiff\"\n", " }\n", " },\n", " {\n", " \"identifier\": \"userdata\",\n", " \"format\": {\n", " \"type\": \"application/json\"\n", " }\n", " }\n", " ]\n", " },\n", " \"evalscript\": evalscript\n", "}\n", "\n", "headers = {\n", " 'Content-Type': 'application/json',\n", " 'Accept': 'application/x-tar'\n", "}\n", "\n", "response = oauth.request(\"POST\", f\"{url}/api/v1/process\", headers=headers, json = request)\n", "\n", "tar = tarfile.open(fileobj=io.BytesIO(response.content))\n", "userdata = json.load(tar.extractfile(tar.getmember('userdata.json')))\n", "userdata" ] } ], "metadata": { "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 5 }