Precipitation Variability and Extreme Events

Content creators: Will Gregory, Laura Paccini, Raphael Rocha

Content reviewers: Paul Heubel, Jenna Pearson

Content editors: Paul Heubel

Production editors: Paul Heubel, Konstantine Tsafatinos

Our 2024 Sponsors: CMIP, NFDI4Earth

Project Background

In this project, you will explore rain gauge and satellite data from the CHIRPS data set to extract rain estimates and land surface reflectance, respectively. These data will enable identification of extreme events in your region of interest. Besides investigating the relationships between these variables, you are encouraged to study the impact of extreme events on changes in vegetation.

Project Template

Data Exploration Notebook

Project Setup

# google colab installs

#!pip install --quiet s3fs boto3 botocore

# Imports

import numpy as np
import matplotlib.pyplot as plt
import xarray as xr
import pooch
import os
import tempfile
import pandas as pd
import s3fs
import boto3
import botocore
import datetime

Helper functions

Show code cell source Hide code cell source

# @title Helper functions

def pooch_load(filelocation=None,filename=None,processor=None):
    shared_location='/home/jovyan/shared/Data/Projects/Precipitation' # this is different for each day
    user_temp_cache=tempfile.gettempdir()

    if os.path.exists(os.path.join(shared_location,filename)):
        file = os.path.join(shared_location,filename)
    else:
        file = pooch.retrieve(filelocation,known_hash=None,fname=os.path.join(user_temp_cache,filename),processor=processor)

    return file

Figure settings

Show code cell source Hide code cell source

# @title Figure settings

import ipywidgets as widgets       # interactive display

%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/neuromatch/climate-course-content/main/cma.mplstyle")

CHIRPS Version 2.0 Global Daily 0.25°

The Climate Hazards Group InfraRed Precipitation with Station data (CHIRPS) is a high-resolution precipitation dataset developed by the Climate Hazards Group at the University of California, Santa Barbara. It combines satellite-derived precipitation estimates with ground-based station data to provide gridded precipitation data at a quasi-global scale between 50°S-50°N.

Read more about CHIRPS here:

• The climate hazards infrared precipitation with stations—a new environmental record for monitoring extremes

• Climate Hazard Group CHG Wiki

• Data storage location

Indices for Extreme Events

The Expert Team on Climate Change Detection and Indices (ETCCDI) has defined various indices that focus on different aspects such as duration or intensity of extreme events. The following functions provide examples of how to compute indices for each category. You can modify these functions to suit your specific needs or create your own custom functions. Here are some tips you can use:

• Most of the indices require daily data, so in order to select a specific season you can just use xarray to subset the data. Example:

daily_precip_DJF = data_chirps.sel(time=data_chirps['time.season']=='DJF')

• A common threshold for a wet event is precipitation greater than or equal to \(1 \text{ mm}/\text{day}\), while a dry (or non-precipitating) event is defined as precipitation less than \(1 \text{ mm}/\text{day}\).

• Some of the indices are based on percentiles. You can define a base period climatology to calculate percentile thresholds, such as the 5th, 10th, 90th, and 95th percentiles, to determine extreme events (cf. in particular W2D3).

def calculate_cdd_index(data):
    """
    This function takes a daily precipitation dataset as input and calculates
    the Consecutive Dry Days (CDD) index, which represents the longest sequence
    of consecutive days with precipitation less than 1mm. The input data should
    be a DataArray with time coordinates, and the function returns a DataArray
    with the CDD values for each unique year in the input data.
    Parameters:
    ----------
      - data (xarray.DataArray): The input daily precipitation data should be
      a dataset (eg. for chirps_data the DataArray would be chirps_data.precip)
    Returns:
    -------
      - cdd (xarray.DataArray): The calculated CDD index

    """
    # create a boolean array for dry days (PR < 1mm)
    dry_days = data < 1
    # initialize CDD array
    cdd = np.zeros(len(data.groupby("time.year")))
    # get unique years as a list
    unique_years = list(data.groupby("time.year").groups.keys())
    # iterate for each day
    for i, year in enumerate(unique_years):
        consecutive_trues = []
        current_count = 0
        for day in dry_days.sel(time=data["time.year"] == year).values:
            if day:
                current_count += 1
            else:
                if current_count > 0:
                    consecutive_trues.append(current_count)
                    current_count = 0
        if current_count > 0:
            consecutive_trues.append(current_count)
        # print(consecutive_trues)
        # CDD is the largest number of consecutive days
        cdd[i] = np.max(consecutive_trues)
    # transform to dataset
    cdd_da = xr.DataArray(cdd, coords={"year": unique_years}, dims="year")
    return cdd_da

# code to retrieve and load the data

years = range(1981,2024) # the years you want. we want 1981 till 2023
file_paths = ['https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.' + str(year) + '.days_p25.nc' for year in years] # the format of the files
filenames = ['chirps-v2.0.'+str(year)+'.days_p25.nc' for year in years] # the format of the files

downloaded_files=[ pooch_load(fpath,fname) for (fpath,fname) in zip(file_paths,filenames)] # download all of the files

#### open data via xarray
chirps_data = xr.open_mfdataset(
    downloaded_files, combine="by_coords"
)  # open the files as one dataset

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1981.days_p25.nc' to file '/tmp/chirps-v2.0.1981.days_p25.nc'.

SHA256 hash of downloaded file: abf5aa47db1a47b1bb0358591d02adb7cdc765af1de83834aa0a36157828f0cb
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1982.days_p25.nc' to file '/tmp/chirps-v2.0.1982.days_p25.nc'.

SHA256 hash of downloaded file: 5264a039135cbe00fff7c2b8296123ddc652db67a3086ac1d84bc7cfd91d1b3c
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1983.days_p25.nc' to file '/tmp/chirps-v2.0.1983.days_p25.nc'.

SHA256 hash of downloaded file: 8083ed549a4027fddad74e4a3be75c8587ad0830de3be0200c56b295d81cbe26
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1984.days_p25.nc' to file '/tmp/chirps-v2.0.1984.days_p25.nc'.

SHA256 hash of downloaded file: e1c3edab18b4bd69a659a42e95ca9a97051b7055738fdf4b63796ff199bbf2a4
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1985.days_p25.nc' to file '/tmp/chirps-v2.0.1985.days_p25.nc'.

SHA256 hash of downloaded file: 795f6ef239d5a88b0afc682875ad5567f61d694980346c656db5445af428b3ab
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1986.days_p25.nc' to file '/tmp/chirps-v2.0.1986.days_p25.nc'.

SHA256 hash of downloaded file: 3ad6497e71ee565d2ecdb895993f2ca9b4db6ed0efb702eb000b934b7ea4438f
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1987.days_p25.nc' to file '/tmp/chirps-v2.0.1987.days_p25.nc'.

SHA256 hash of downloaded file: a47dd1314d10589213b7d807fb8f1486972f2a0fcf081d899b414304c09aad8d
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1988.days_p25.nc' to file '/tmp/chirps-v2.0.1988.days_p25.nc'.

SHA256 hash of downloaded file: 07b7a89ac7e7f1e16cad1db1100217a60a124964b77693e6323b0c1316f813e3
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1989.days_p25.nc' to file '/tmp/chirps-v2.0.1989.days_p25.nc'.

SHA256 hash of downloaded file: a81c7a0916bf81e4047cda8d6320cc5c94720f758f232c50b80502a840702ab8
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1990.days_p25.nc' to file '/tmp/chirps-v2.0.1990.days_p25.nc'.

SHA256 hash of downloaded file: 10a95daa84df0e1934c462c3687cff0accff907557f92c6c1e7ee0d2077ed0d4
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1991.days_p25.nc' to file '/tmp/chirps-v2.0.1991.days_p25.nc'.

SHA256 hash of downloaded file: f549808c69601cf288b4f0f42b8ac5337c348f06d1fb55aa86b67e9bbd3837f9
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1992.days_p25.nc' to file '/tmp/chirps-v2.0.1992.days_p25.nc'.

SHA256 hash of downloaded file: ae26310b18369c1015e6b70db6dfb7ad1c138f846bd7c322e0b6794a26423f56
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1993.days_p25.nc' to file '/tmp/chirps-v2.0.1993.days_p25.nc'.

SHA256 hash of downloaded file: 292ff168ef7f4420d36606f7ed84aa5380dd979a90351762926ef4c569e90f56
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1994.days_p25.nc' to file '/tmp/chirps-v2.0.1994.days_p25.nc'.

SHA256 hash of downloaded file: f856963801db2a62fa5c35537810f1649f19d32c84f1c4f2bb3fd3dcc82ba7ec
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1995.days_p25.nc' to file '/tmp/chirps-v2.0.1995.days_p25.nc'.

SHA256 hash of downloaded file: 54c7a408d49112465479a06a3d1e5b37cb10b526c2cf20215f2b75579c8c2a0e
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1996.days_p25.nc' to file '/tmp/chirps-v2.0.1996.days_p25.nc'.

SHA256 hash of downloaded file: b0a4604e5a213671e6d591ad959c2156685ee026b54fbc8ad87860e35f3db4df
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1997.days_p25.nc' to file '/tmp/chirps-v2.0.1997.days_p25.nc'.

SHA256 hash of downloaded file: a04019a26e20988e35d8d14e88f1083aabb8104434c556c21e3d29e1c30ae731
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1998.days_p25.nc' to file '/tmp/chirps-v2.0.1998.days_p25.nc'.

SHA256 hash of downloaded file: 15f3e33707c14b3ef26b525cb116766b847f2295c60aa41c6ae73df10e748e98
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.1999.days_p25.nc' to file '/tmp/chirps-v2.0.1999.days_p25.nc'.

SHA256 hash of downloaded file: dbb0a6a0ee1ff5e3ba5439101bedc24dad0951810a03a6f5cdc96e4b08c06716
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2000.days_p25.nc' to file '/tmp/chirps-v2.0.2000.days_p25.nc'.

SHA256 hash of downloaded file: 57014c9ff72c8997b5b141793c06ffd1a8d8f5c6d1308789ac8e12c3860e5790
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2001.days_p25.nc' to file '/tmp/chirps-v2.0.2001.days_p25.nc'.

SHA256 hash of downloaded file: 8550c96c7f8c29cbc6a1be3be56d7f769125d15b9ac7bc6c4f0fc4a76034f0fa
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2002.days_p25.nc' to file '/tmp/chirps-v2.0.2002.days_p25.nc'.

SHA256 hash of downloaded file: 8027aaf7e005d74073d2c744d9c8a0343ffade272c1d1f8c420972bcbade2231
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2003.days_p25.nc' to file '/tmp/chirps-v2.0.2003.days_p25.nc'.

SHA256 hash of downloaded file: 5b5a538e187ca651bc1fcc25eb445d31a05067b5a109bd8bb4651db11346cf6f
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2004.days_p25.nc' to file '/tmp/chirps-v2.0.2004.days_p25.nc'.

SHA256 hash of downloaded file: 5e671526f27567e9218d275f7a47b6c0d7e0acf718edfee64c5843f5ee497e93
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2005.days_p25.nc' to file '/tmp/chirps-v2.0.2005.days_p25.nc'.

SHA256 hash of downloaded file: 2185afd5c3d110da52a9a81eb97f25b37f30db5c0478e536a84c957351e44b53
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2006.days_p25.nc' to file '/tmp/chirps-v2.0.2006.days_p25.nc'.

SHA256 hash of downloaded file: 3f78d5e733ea38bef25e47af9b3a907a96e27b8334cc04e4898837239e6539e6
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2007.days_p25.nc' to file '/tmp/chirps-v2.0.2007.days_p25.nc'.

SHA256 hash of downloaded file: c4594aeaafd7b9e7065577563799bca461582723e6b32df332206b43cb853138
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2008.days_p25.nc' to file '/tmp/chirps-v2.0.2008.days_p25.nc'.

SHA256 hash of downloaded file: 6fe5dc8facd7750b6c08ecd0cadaaf19570acccd6a15f20efcbdd960230b5551
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2009.days_p25.nc' to file '/tmp/chirps-v2.0.2009.days_p25.nc'.

SHA256 hash of downloaded file: e6ccdc38aa146793f2c92858412122ef14a7a34332940bec03bd520cc77dc235
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2010.days_p25.nc' to file '/tmp/chirps-v2.0.2010.days_p25.nc'.

SHA256 hash of downloaded file: f6b6bd962626c038959a743b5f6e6c22b0bd6647b89b4a08b0f2305ac77f7277
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2011.days_p25.nc' to file '/tmp/chirps-v2.0.2011.days_p25.nc'.

SHA256 hash of downloaded file: 45f437d8f1c7237b7043ebf4fec800461252af02e7cbcc87a844ddf0d1a911d2
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2012.days_p25.nc' to file '/tmp/chirps-v2.0.2012.days_p25.nc'.

SHA256 hash of downloaded file: 27792b3d028ca5e745a8fe45c4f133a4183469512d2e55ead950d54a5a5a890b
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2013.days_p25.nc' to file '/tmp/chirps-v2.0.2013.days_p25.nc'.

SHA256 hash of downloaded file: d531a704ed1a40be2d509b1d984913194015f84721edc555f2de8c000bcab638
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2014.days_p25.nc' to file '/tmp/chirps-v2.0.2014.days_p25.nc'.

SHA256 hash of downloaded file: c3225de85da99422bc5fde9dbd1bdabff206fc2902d811944169348c8352d521
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2015.days_p25.nc' to file '/tmp/chirps-v2.0.2015.days_p25.nc'.

SHA256 hash of downloaded file: 0c770adf07884a25610ea00d66d5e221529159746d3d76d9affea69c77666233
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2016.days_p25.nc' to file '/tmp/chirps-v2.0.2016.days_p25.nc'.

SHA256 hash of downloaded file: 8ff4f6284c1d37b50e8b1e47dbd9f6e108a9ee3b17a6752bb18cc0b7384526d3
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2017.days_p25.nc' to file '/tmp/chirps-v2.0.2017.days_p25.nc'.

SHA256 hash of downloaded file: 9692ca1fee495b8091084b1e5d29ea618483e67886a281f82b4a03555861d6b8
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2018.days_p25.nc' to file '/tmp/chirps-v2.0.2018.days_p25.nc'.

SHA256 hash of downloaded file: 05b0850b8b8c16a2d5d33ddc6ba89d503084dc7374845214289be1b411a30d3f
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2019.days_p25.nc' to file '/tmp/chirps-v2.0.2019.days_p25.nc'.

SHA256 hash of downloaded file: 311723f2f8d2af3307da198ebafece0c081200f336444a4261516539348fed38
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2020.days_p25.nc' to file '/tmp/chirps-v2.0.2020.days_p25.nc'.

SHA256 hash of downloaded file: 2ab3b142141c5cfc708bc666dc89900005b7452b66324515fca1baa8dff00402
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2021.days_p25.nc' to file '/tmp/chirps-v2.0.2021.days_p25.nc'.

SHA256 hash of downloaded file: 1b2db33508896ed1932b9db7e6dad6136e90687d3fe630f08066dbd279e0dabb
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2022.days_p25.nc' to file '/tmp/chirps-v2.0.2022.days_p25.nc'.

SHA256 hash of downloaded file: 5f9b88f0623083759ba351a263bf23afd7fb26b13e23d051d5bb22a55aa0bcb9
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'https://data.chc.ucsb.edu/products/CHIRPS-2.0/global_daily/netcdf/p25/chirps-v2.0.2023.days_p25.nc' to file '/tmp/chirps-v2.0.2023.days_p25.nc'.

SHA256 hash of downloaded file: 8430cc8afd1a69205ecf757a2d8d95f3e6fbc44544326b27b4d534a82ee9c14c
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

We can now visualize the content of the dataset.

# code to print the shape, array names, etc of the dataset
chirps_data

<xarray.Dataset> Size: 36GB
Dimensions:    (latitude: 400, longitude: 1440, time: 15705)
Coordinates:
  * latitude   (latitude) float32 2kB -49.88 -49.62 -49.38 ... 49.38 49.62 49.88
  * longitude  (longitude) float32 6kB -179.9 -179.6 -179.4 ... 179.6 179.9
  * time       (time) datetime64[ns] 126kB 1981-01-01 1981-01-02 ... 2023-12-31
Data variables:
    precip     (time, latitude, longitude) float32 36GB dask.array<chunksize=(61, 67, 240), meta=np.ndarray>
Attributes: (12/15)
    Conventions:       CF-1.6
    title:             CHIRPS Version 2.0
    history:           created by Climate Hazards Group
    version:           Version 2.0
    date_created:      2015-10-07
    creator_name:      Pete Peterson
    ...                ...
    reference:         Funk, C.C., Peterson, P.J., Landsfeld, M.F., Pedreros,...
    comments:           time variable denotes the first day of the given day.
    acknowledgements:  The Climate Hazards Group InfraRed Precipitation with ...
    ftp_url:           ftp://chg-ftpout.geog.ucsb.edu/pub/org/chg/products/CH...
    website:           http://chg.geog.ucsb.edu/data/chirps/index.html
    faq:               http://chg-wiki.geog.ucsb.edu/wiki/CHIRPS_FAQ

xarray.Dataset

• Dimensions:
  • latitude: 400
  • longitude: 1440
  • time: 15705
• Coordinates: (3)
  • latitude
    (latitude)
    float32
    -49.88 -49.62 ... 49.62 49.88

    units :

    degrees_north

    standard_name :

    latitude

    long_name :

    latitude

    axis :

    Y

    array([-49.875, -49.625, -49.375, ...,  49.375,  49.625,  49.875],
          dtype=float32)

  • longitude
    (longitude)
    float32
    -179.9 -179.6 ... 179.6 179.9

    units :

    degrees_east

    standard_name :

    longitude

    long_name :

    longitude

    axis :

    X

    array([-179.875, -179.625, -179.375, ...,  179.375,  179.625,  179.875],
          dtype=float32)

  • time
    (time)
    datetime64[ns]
    1981-01-01 ... 2023-12-31

    standard_name :

    time

    axis :

    T

    array(['1981-01-01T00:00:00.000000000', '1981-01-02T00:00:00.000000000',
           '1981-01-03T00:00:00.000000000', ..., '2023-12-29T00:00:00.000000000',
           '2023-12-30T00:00:00.000000000', '2023-12-31T00:00:00.000000000'],
          dtype='datetime64[ns]')

• Data variables: (1)
  • precip
    (time, latitude, longitude)
    float32
    dask.array<chunksize=(61, 67, 240), meta=np.ndarray>

    units :

    mm/day

    standard_name :

    convective precipitation rate

    long_name :

    Climate Hazards group InfraRed Precipitation with Stations

    time_step :

    day

    geostatial_lat_min :

    -50.0

    geostatial_lat_max :

    50.0

    geostatial_lon_min :

    -180.0

    geostatial_lon_max :

    180.0

    Array Chunk Bytes 33.70 GiB 3.74 MiB Shape (15705, 400, 1440) (61, 67, 240) Dask graph 9288 chunks in 87 graph layers Data type float32 numpy.ndarray

• Indexes: (3)
  • latitude
    PandasIndex

    PandasIndex(Index([-49.875, -49.625, -49.375, -49.125, -48.875, -48.625, -48.375, -48.125,
           -47.875, -47.625,
           ...
            47.625,  47.875,  48.125,  48.375,  48.625,  48.875,  49.125,  49.375,
            49.625,  49.875],
          dtype='float32', name='latitude', length=400))

  • longitude
    PandasIndex

    PandasIndex(Index([-179.875, -179.625, -179.375, -179.125, -178.875, -178.625, -178.375,
           -178.125, -177.875, -177.625,
           ...
            177.625,  177.875,  178.125,  178.375,  178.625,  178.875,  179.125,
            179.375,  179.625,  179.875],
          dtype='float32', name='longitude', length=1440))

  • time
    PandasIndex

    PandasIndex(DatetimeIndex(['1981-01-01', '1981-01-02', '1981-01-03', '1981-01-04',
                   '1981-01-05', '1981-01-06', '1981-01-07', '1981-01-08',
                   '1981-01-09', '1981-01-10',
                   ...
                   '2023-12-22', '2023-12-23', '2023-12-24', '2023-12-25',
                   '2023-12-26', '2023-12-27', '2023-12-28', '2023-12-29',
                   '2023-12-30', '2023-12-31'],
                  dtype='datetime64[ns]', name='time', length=15705, freq=None))

• Attributes: (15)

  Conventions :

  CF-1.6

  title :

  CHIRPS Version 2.0

  history :

  created by Climate Hazards Group

  version :

  Version 2.0

  date_created :

  2015-10-07

  creator_name :

  Pete Peterson

  creator_email :

  pete@geog.ucsb.edu

  institution :

  Climate Hazards Group. University of California at Santa Barbara

  documentation :

  http://pubs.usgs.gov/ds/832/

  reference :

  Funk, C.C., Peterson, P.J., Landsfeld, M.F., Pedreros, D.H., Verdin, J.P., Rowland, J.D., Romero, B.E., Husak, G.J., Michaelsen, J.C., and Verdin, A.P., 2014, A quasi-global precipitation time series for drought monitoring: U.S. Geological Survey Data Series 832, 4 p., http://dx.doi.org/110.3133/ds832.

  comments :

  time variable denotes the first day of the given day.

  acknowledgements :

  The Climate Hazards Group InfraRed Precipitation with Stations development process was carried out through U.S. Geological Survey (USGS) cooperative agreement #G09AC000001 "Monitoring and Forecasting Climate, Water and Land Use for Food Production in the Developing World" with funding from: U.S. Agency for International Development Office of Food for Peace, award #AID-FFP-P-10-00002 for "Famine Early Warning Systems Network Support," the National Aeronautics and Space Administration Applied Sciences Program, Decisions award #NN10AN26I for "A Land Data Assimilation System for Famine Early Warning," SERVIR award #NNH12AU22I for "A Long Time-Series Indicator of Agricultural Drought for the Greater Horn of Africa," The National Oceanic and Atmospheric Administration award NA11OAR4310151 for "A Global Standardized Precipitation Index supporting the US Drought Portal and the Famine Early Warning System Network," and the USGS Land Change Science Program.

  ftp_url :

  ftp://chg-ftpout.geog.ucsb.edu/pub/org/chg/products/CHIRPS-latest/

  website :

  http://chg.geog.ucsb.edu/data/chirps/index.html

  faq :

  http://chg-wiki.geog.ucsb.edu/wiki/CHIRPS_FAQ

NOAA Fundamental Climate Data Records (FCDR) AVHRR Land Bundle - Normalized Difference Vegetation Index

As we learned in the W1D3 tutorials, all the National Atmospheric and Oceanic Administration Climate Data Record (NOAA-CDR) datasets are available both at NOAA National Centers for Environmental Information (NCEI) and commercial cloud platforms. See the link here. We are accessing the data directly via the Amazon Web Service (AWS) cloud storage space.

For this project, we recommend using the Normalized Difference Vegetation Index (NDVI). It is one of the most commonly used remotely sensed indices. It measures the “greenness” of vegetation and is useful in understanding vegetation density and assessing changes in plant health. For example, NDVI can be used to study the impacts of droughts, heatwaves, and insect infestations on plants covering Earth’s surface. A good overview of this index from this particular sensor can be accessed here.

Recall what we learned in W1D3 Tutorial 3, the data files on AWS are named systematically:

Sensor name: AVHRR
Product category: Land
Product version: v005
Product code: AVH13C1
Satellite platform: NOAA-07
Date of the data: 19810624
Processing time: c20170610041337 (This will change for each file based on when the file was processed)
File format: .nc (netCDF-4 format)

In other words, if we are looking for the data of a specific day, we can easily locate where the file might be.

For example, if we want to find the AVHRR data for the day of 2002-03-12 (or March 12, 2002), you can use:

s3://noaa-cdr-ndvi-pds/data/2002/AVHRR-Land_v005_AVH13C1_*_20020312_c*.nc

The reason that we put * in the above directory is that we are not sure about what satellite platform this data is from and when the data was processed. The * is called a wildcard, and is used because we want all the files that contain our specific criteria, but do not want to have to specify all the other pieces of the filename we are not sure about yet. It should return all the data satisfying that initial criteria and you can refine further once you see what is available. Essentially, this first step helps to narrow down the data search. You can then use this to create datasets from the timeframe of your choosing.

Note the download might take up to hours depending on your time frame of interest and your device. Choose your time frame wisely before taking action to avoid unnecessary data retrieval amounts.

# we can access the data similar to Tutorial 3 of W1D3
# to access the NDVI data from AWS S3 bucket, we first need to connect to s3 bucket

fs = s3fs.S3FileSystem(anon=True)
client = boto3.client('s3', config=botocore.client.Config(signature_version=botocore.UNSIGNED)) # initialize aws s3 bucket client
# choose a date of interest, we pick its year or month while defining the file_location
date_sel = datetime.datetime(2001,1,1,0)

file_location = fs.glob('s3://noaa-cdr-ndvi-pds/data/'
                        + date_sel.strftime('%Y')
                        + '/AVHRR-Land_v005_AVH13C1_*'
                        + date_sel.strftime("%Y%m") # or "%Y" for the entire year
                        + '*_c*.nc') # the files we want for a whole year/month
files_for_pooch=[pooch_load('http://s3.amazonaws.com/'+file,file) for file in file_location]

ds = xr.open_mfdataset(files_for_pooch, combine="by_coords", decode_times=False)  # open the file
ds

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010101_c20170608112729.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010101_c20170608112729.nc'.

SHA256 hash of downloaded file: bea1b2cfeead6742dd8e6006533d447a1cfd6f7311fbc8a662f5f0b58ac65500
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010102_c20170608114253.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010102_c20170608114253.nc'.

SHA256 hash of downloaded file: 3b8037793f3699a79d3d2c6d5413966a356d4b1e27d853555231961616fe40b6
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010103_c20170608115841.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010103_c20170608115841.nc'.

SHA256 hash of downloaded file: a92fd5e6188556cd35511394312fb4f5ecf4d7e7153fc4002f29d23316463340
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010104_c20170608121434.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010104_c20170608121434.nc'.

SHA256 hash of downloaded file: 21c0f5e7b361eeed45bad648c12b3d88c670da77e81ed36d0c83e7daa59e0ece
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010105_c20170608123036.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010105_c20170608123036.nc'.

SHA256 hash of downloaded file: fc7d244edf3dd0eec1609ed9aeca5d94c479f8115a38c9cf879efa889046a66d
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010106_c20170608124647.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010106_c20170608124647.nc'.

SHA256 hash of downloaded file: 260c939a2b10f35eafe6ebd67c9b9d260062e7c455a1b7268f6bafaec4e9edf0
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010107_c20170608130258.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010107_c20170608130258.nc'.

SHA256 hash of downloaded file: 85f0e58ce20cf1fcacdb8856b2bc805a073e7d349de2816c4c71d18c35f88de7
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010108_c20170608130658.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010108_c20170608130658.nc'.

SHA256 hash of downloaded file: 3dc1a63efd5439b5c7c30421dfc3f19918d03719ee824c5bba9f28842a133748
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010109_c20170608131430.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010109_c20170608131430.nc'.

SHA256 hash of downloaded file: 5c1ddfbe3f562b8fcd84a5f2c294aa5ae804f6a9c686ea21e7a019c2434b2ce1
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010110_c20170608133019.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010110_c20170608133019.nc'.

SHA256 hash of downloaded file: 4c270d4cd8c43a3e8830477f681fdc7f492acdbeb517fbb9fa002246955d6468
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010111_c20170608134807.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010111_c20170608134807.nc'.

SHA256 hash of downloaded file: dbb9c7c2a17fa376727a6d6efed7dba915222439678c9fe28fef8f93c283f9a0
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010112_c20170608140615.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010112_c20170608140615.nc'.

SHA256 hash of downloaded file: 2443ca0d7294da9555a462c2c6af5620342688522ee9a6197606ca91551848b6
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010113_c20170608142322.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010113_c20170608142322.nc'.

SHA256 hash of downloaded file: a91b0ccd4216729443aaba8875de29ddcf0968789ce86b2abf06984abcb1ab9b
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010114_c20170608144105.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010114_c20170608144105.nc'.

SHA256 hash of downloaded file: 80aacb1b012bc917fd0ebbcccf793062f8a490031a21cede4f8e6ab6bd59f45e
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010115_c20170608145913.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010115_c20170608145913.nc'.

SHA256 hash of downloaded file: e3b97d2052ece810cf14f3c7ad428d30e9e9acf98aa2082684f8f3986a6f05aa
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010116_c20170608151715.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010116_c20170608151715.nc'.

SHA256 hash of downloaded file: 317ece4c8925f74e93d6111c15f8a5d78231da91f47c98b3432ad9a9d6dd5e0c
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010117_c20170608153425.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010117_c20170608153425.nc'.

SHA256 hash of downloaded file: d1a98e00eb1cf5524eab70e5d0cefb793f789c653c010c5af5fd7b641a0453ec
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010118_c20170608155135.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010118_c20170608155135.nc'.

SHA256 hash of downloaded file: 48710d5c1cc845f35ba23c528ac9e592fd4d9b7c9961d1ddb31ac1ce7880f829
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010119_c20170608160850.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010119_c20170608160850.nc'.

SHA256 hash of downloaded file: 1a8dbe37b9f48e28e1ad24c7002cdc4e132413668e63f856ff30a34ab2e49bf5
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010120_c20170608162614.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010120_c20170608162614.nc'.

SHA256 hash of downloaded file: 6419dc7f01e8725ebc3db108b770d64c6221fc61acb824959bbe5a246faec097
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010121_c20170608164215.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010121_c20170608164215.nc'.

SHA256 hash of downloaded file: 365b748dce06c1731a03162855d25ee4467fcb94fa7c623f82f8c407fa4ea88e
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010122_c20170608165946.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010122_c20170608165946.nc'.

SHA256 hash of downloaded file: 2d1904144d4459967c49f4b197a4e35ca1fea352f1eea41732daa679af81378f
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010123_c20170608171732.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010123_c20170608171732.nc'.

SHA256 hash of downloaded file: 83d678770250b0fc8169c72b648da9ad6d11843b8c73250b4857c5ee18b5287c
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010124_c20170608173542.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010124_c20170608173542.nc'.

SHA256 hash of downloaded file: 3cc82111ca3ce5f58948c31af1778d46401ed4ec61403b8a374d1d690ce7f9ec
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010125_c20170608175403.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010125_c20170608175403.nc'.

SHA256 hash of downloaded file: 04618e326461457b9ec9495ad527a82845bd7e363f53705f4e47fdac6383d83d
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010126_c20170608181206.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010126_c20170608181206.nc'.

SHA256 hash of downloaded file: 1022477d48371e7df468f8699bebd9716ea19ee104e397919ce61cd2b178bec2
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010127_c20170608182926.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010127_c20170608182926.nc'.

SHA256 hash of downloaded file: 81f12834cf32df6171e65db33afd90fbfd65027330aa0b21cbd0ea6088c8280a
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010128_c20170608184655.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010128_c20170608184655.nc'.

SHA256 hash of downloaded file: cf1b2f7bb2ede9ff76917f77681d8c0cf8c8e9b8cbaa43b315613e848ea2d6ab
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010129_c20170608190436.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010129_c20170608190436.nc'.

SHA256 hash of downloaded file: 20b5d6c84a736057d6fa8def6857a7e507fd95ae5c265da893692788b59c7497
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010130_c20170608192209.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010130_c20170608192209.nc'.

SHA256 hash of downloaded file: c2122ad87a08bf38ce1b38f6d03bdf783209431dcd45a1d37be07c4385f88eb5
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

Downloading data from 'http://s3.amazonaws.com/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010131_c20170608193958.nc' to file '/tmp/noaa-cdr-ndvi-pds/data/2001/AVHRR-Land_v005_AVH13C1_NOAA-16_20010131_c20170608193958.nc'.

SHA256 hash of downloaded file: d5c1b61b554941b79caf7731fe49a6dfcddeed08777dd516c3cf6505c6834ad1
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

<xarray.Dataset> Size: 11GB
Dimensions:    (latitude: 3600, longitude: 7200, time: 31, ncrs: 1, nv: 2)
Coordinates:
  * latitude   (latitude) float32 14kB 89.97 89.93 89.88 ... -89.93 -89.97
  * longitude  (longitude) float32 29kB -180.0 -179.9 -179.9 ... 179.9 180.0
  * time       (time) float32 124B 7.305e+03 7.306e+03 ... 7.334e+03 7.335e+03
Dimensions without coordinates: ncrs, nv
Data variables:
    crs        (time, ncrs) int16 62B dask.array<chunksize=(1, 1), meta=np.ndarray>
    lat_bnds   (time, latitude, nv) float32 893kB dask.array<chunksize=(1, 3600, 2), meta=np.ndarray>
    lon_bnds   (time, longitude, nv) float32 2MB dask.array<chunksize=(1, 7200, 2), meta=np.ndarray>
    NDVI       (time, latitude, longitude) float32 3GB dask.array<chunksize=(1, 1200, 2400), meta=np.ndarray>
    TIMEOFDAY  (time, latitude, longitude) float64 6GB dask.array<chunksize=(1, 1200, 2400), meta=np.ndarray>
    QA         (time, latitude, longitude) int16 2GB dask.array<chunksize=(1, 3600, 7200), meta=np.ndarray>
Attributes: (12/48)
    title:                                  Normalized Difference Vegetation ...
    institution:                            NASA/GSFC/SED/ESD/HBSL/TIS/MODIS-...
    Conventions:                            CF-1.6, ACDD-1.3
    standard_name_vocabulary:               CF Standard Name Table (v25, 05 J...
    naming_authority:                       gov.noaa.ncei
    license:                                See the Use Agreement for this CD...
    ...                                     ...
    PercentValidDaytimeData:                33.79
    PercentValidDaytimeLand:                33.79
    PercentValidClearDaytimeLand:           4.54
    PercentValidDaytimeLandInCloudShadow:   0.85
    PercentValidClearDaytimeWater:          0.00
    PercentValidDaytimeWaterInCloudShadow:  0.00

xarray.Dataset

• Dimensions:
  • latitude: 3600
  • longitude: 7200
  • time: 31
  • ncrs: 1
  • nv: 2
• Coordinates: (3)
  • latitude
    (latitude)
    float32
    89.97 89.93 89.88 ... -89.93 -89.97

    units :

    degrees_north

    long_name :

    latitude

    axis :

    Y

    standard_name :

    latitude

    bounds :

    lat_bnds

    array([ 89.975  ,  89.925  ,  89.875  , ..., -89.875  , -89.925  , -89.97499],
          dtype=float32)

  • longitude
    (longitude)
    float32
    -180.0 -179.9 ... 179.9 180.0

    units :

    degrees_east

    long_name :

    longitude

    axis :

    X

    standard_name :

    longitude

    bounds :

    lon_bnds

    array([-179.975  , -179.925  , -179.875  , ...,  179.875  ,  179.92499,
            179.975  ], dtype=float32)

  • time
    (time)
    float32
    7.305e+03 7.306e+03 ... 7.335e+03

    units :

    days since 1981-01-01 00:00:00

    long_name :

    time

    axis :

    T

    standard_name :

    time

    calendar :

    gregorian

    array([7305., 7306., 7307., 7308., 7309., 7310., 7311., 7312., 7313., 7314.,
           7315., 7316., 7317., 7318., 7319., 7320., 7321., 7322., 7323., 7324.,
           7325., 7326., 7327., 7328., 7329., 7330., 7331., 7332., 7333., 7334.,
           7335.], dtype=float32)

• Data variables: (6)
  • crs
    (time, ncrs)
    int16
    dask.array<chunksize=(1, 1), meta=np.ndarray>

    grid_mapping_name :

    latitude_longitude

    epsg_code :

    EPSG:4326

    longitude_of_prime_meridian :

    0.0

    semi_major_axis :

    6378137.0

    inverse_flattening :

    298.257223563

    Array Chunk Bytes 62 B 2 B Shape (31, 1) (1, 1) Dask graph 31 chunks in 94 graph layers Data type int16 numpy.ndarray

  • lat_bnds
    (time, latitude, nv)
    float32
    dask.array<chunksize=(1, 3600, 2), meta=np.ndarray>

    Array Chunk Bytes 871.88 kiB 28.12 kiB Shape (31, 3600, 2) (1, 3600, 2) Dask graph 31 chunks in 94 graph layers Data type float32 numpy.ndarray

  • lon_bnds
    (time, longitude, nv)
    float32
    dask.array<chunksize=(1, 7200, 2), meta=np.ndarray>

    Array Chunk Bytes 1.70 MiB 56.25 kiB Shape (31, 7200, 2) (1, 7200, 2) Dask graph 31 chunks in 94 graph layers Data type float32 numpy.ndarray

  • NDVI
    (time, latitude, longitude)
    float32
    dask.array<chunksize=(1, 1200, 2400), meta=np.ndarray>

    long_name :

    NOAA Climate Data Record of Normalized Difference Vegetation Index

    units :

    1

    valid_range :

    [-1000 10000]

    grid_mapping :

    crs

    standard_name :

    normalized_difference_vegetation_index

    Array Chunk Bytes 2.99 GiB 10.99 MiB Shape (31, 3600, 7200) (1, 1200, 2400) Dask graph 279 chunks in 63 graph layers Data type float32 numpy.ndarray

  • TIMEOFDAY
    (time, latitude, longitude)
    float64
    dask.array<chunksize=(1, 1200, 2400), meta=np.ndarray>

    long_name :

    Time since Start of Data Day

    units :

    hours since 2001-01-01 00:00:00

    valid_range :

    [ 0 2399]

    grid_mapping :

    crs

    Array Chunk Bytes 5.99 GiB 21.97 MiB Shape (31, 3600, 7200) (1, 1200, 2400) Dask graph 279 chunks in 63 graph layers Data type float64 numpy.ndarray

  • QA
    (time, latitude, longitude)
    int16
    dask.array<chunksize=(1, 3600, 7200), meta=np.ndarray>

    flag_masks :

    [ 2 4 8 16 32 64 128 256 512 1024 2048 4096 8192 16384 -32768]

    flag_meanings :

    cloudy cloud_shadow water sunglint dense_dark_vegetation night ch1_to_5_valid ch1_invalid ch2_invalid ch3_invalid ch4_invalid ch5_invalid rho3_invalid BRDF_corr_problem polar_flag

    long_name :

    Quality Assurance

    grid_mapping :

    crs

    Array Chunk Bytes 1.50 GiB 49.44 MiB Shape (31, 3600, 7200) (1, 3600, 7200) Dask graph 31 chunks in 63 graph layers Data type int16 numpy.ndarray

• Indexes: (3)
  • latitude
    PandasIndex

    PandasIndex(Index([  89.9749984741211,  89.92500305175781,             89.875,
            89.82499694824219,   89.7750015258789,   89.7249984741211,
            89.67500305175781,             89.625,  89.57499694824219,
             89.5250015258789,
           ...
           -89.52499389648438, -89.57499694824219,            -89.625,
           -89.67500305175781, -89.72499084472656, -89.77499389648438,
           -89.82499694824219,            -89.875, -89.92500305175781,
           -89.97499084472656],
          dtype='float32', name='latitude', length=3600))

  • longitude
    PandasIndex

    PandasIndex(Index([-179.97500610351562,  -179.9250030517578,            -179.875,
            -179.8249969482422, -179.77499389648438, -179.72500610351562,
            -179.6750030517578,            -179.625,  -179.5749969482422,
           -179.52499389648438,
           ...
            179.52499389648438,  179.57501220703125,             179.625,
            179.67498779296875,  179.72500610351562,  179.77499389648438,
            179.82501220703125,             179.875,  179.92498779296875,
            179.97500610351562],
          dtype='float32', name='longitude', length=7200))

  • time
    PandasIndex

    PandasIndex(Index([7305.0, 7306.0, 7307.0, 7308.0, 7309.0, 7310.0, 7311.0, 7312.0, 7313.0,
           7314.0, 7315.0, 7316.0, 7317.0, 7318.0, 7319.0, 7320.0, 7321.0, 7322.0,
           7323.0, 7324.0, 7325.0, 7326.0, 7327.0, 7328.0, 7329.0, 7330.0, 7331.0,
           7332.0, 7333.0, 7334.0, 7335.0],
          dtype='float32', name='time'))

• Attributes: (48)

  title :

  Normalized Difference Vegetation Index parameters derived from NOAA-16 GAC data for day 2001/001

  institution :

  NASA/GSFC/SED/ESD/HBSL/TIS/MODIS-LAND > MODIS Land Science Team, Terrestrial Information Systems, Hydrospheric and Biospheric Science Laboratory, Earth Sciences Division, Science and Exploration Directorate, Goddard Space Flight Center, NASA

  Conventions :

  CF-1.6, ACDD-1.3

  standard_name_vocabulary :

  CF Standard Name Table (v25, 05 July 2013)

  naming_authority :

  gov.noaa.ncei

  license :

  See the Use Agreement for this CDR available from the NOAA CDR webpage

  cdm_data_type :

  Grid

  time_coverage_start :

  2001-01-01T00:00:00Z

  time_coverage_end :

  2001-01-01T23:59:59Z

  product_version :

  v5r0

  platform :

  NOAA-16 > National Oceanic & Atmospheric Administration-16

  sensor :

  AVHRR > Advanced Very High Resolution Radiometer

  keywords_vocabulary :

  NASA Global Change Master Directory (GCMD) Science Keywords

  platform_vocabulary :

  Global Change Master Directory (GCMD) Platform Keywords

  instrument_vocabulary :

  Global Change Master Directory (GCMD) Instrument Keywords

  keywords :

  EARTH SCIENCE > BIOSPHERE > VEGETATION > VEGETATION INDEX

  source :

  NSS.GHRR.NL.D01001.S0000.E0048.B0143233.GC, NSS.GHRR.NL.D01001.S0043.E0237.B0143334.WI, NSS.GHRR.NL.D01001.S0232.E0426.B0143435.WI, NSS.GHRR.NL.D01001.S0421.E0615.B0143536.WI, NSS.GHRR.NL.D01001.S0611.E0756.B0143637.WI, NSS.GHRR.NL.D01001.S0751.E0945.B0143738.GC, NSS.GHRR.NL.D01001.S0941.E1114.B0143839.GC, NSS.GHRR.NL.D01001.S1110.E1255.B0143940.GC, NSS.GHRR.NL.D01001.S1250.E1444.B0144041.GC, NSS.GHRR.NL.D01001.S1439.E1616.B0144142.GC, NSS.GHRR.NL.D01001.S1611.E1741.B0144243.WI, NSS.GHRR.NL.D01001.S1736.E1922.B0144244.WI, NSS.GHRR.NL.D01001.S1918.E2112.B0144345.GC, NSS.GHRR.NL.D01001.S2107.E2301.B0144546.GC, NSS.GHRR.NL.D01001.S2256.E0037.B0144647.GC, NSS.GHRR.NL.D01001.S0000.E0048.B0143233.GC.txt, NSS.GHRR.NL.D01001.S0043.E0237.B0143334.WI.txt, NSS.GHRR.NL.D01001.S0232.E0426.B0143435.WI.txt, NSS.GHRR.NL.D01001.S0421.E0615.B0143536.WI.txt, NSS.GHRR.NL.D01001.S0611.E0756.B0143637.WI.txt, NSS.GHRR.NL.D01001.S0751.E0945.B0143738.GC.txt, NSS.GHRR.NL.D01001.S0941.E1114.B0143839.GC.txt, NSS.GHRR.NL.D01001.S1110.E1255.B0143940.GC.txt, NSS.GHRR.NL.D01001.S1250.E1444.B0144041.GC.txt, NSS.GHRR.NL.D01001.S1439.E1616.B0144142.GC.txt, NSS.GHRR.NL.D01001.S1611.E1741.B0144243.WI.txt, NSS.GHRR.NL.D01001.S1736.E1922.B0144244.WI.txt, NSS.GHRR.NL.D01001.S1918.E2112.B0144345.GC.txt, NSS.GHRR.NL.D01001.S2107.E2301.B0144546.GC.txt, NSS.GHRR.NL.D01001.S2256.E0037.B0144647.GC.txt, NCEP_REANALYSIS_SURFACE_2001001.hdf, NCEP_REANALYSIS_SURFACE_2001001.hdf, TOMS_2001001.hdf, GLOBAL_LAND_SEA.4KM, NOAA-16.OLE.txt

  date_created :

  2017-06-08T11:27:29Z

  spatial_resolution :

  0.050000 degrees per pixel

  geospatial_lat_min :

  -90.0

  geospatial_lat_max :

  90.0

  geospatial_lon_min :

  -180.0

  geospatial_lon_max :

  180.0

  metadata_link :

  https://doi.org/10.7289/V5ZG6QH9

  program :

  NOAA Climate Data Record Program for satellites

  cdr_variable :

  NDVI

  Process :

  LTDR_GAPS

  PostProcessingVersion :

  2.9

  PFIIVersion :

  3.5.45

  Satellite :

  NOAA-16

  Instrument :

  AVHRR

  InputDataType :

  GAC

  summary :

  AVHRR GAC data from NOAA-16 for 2001, days 1 to 1, processed by the Long-Term Land Data Record (LTDR) project (v3.5.45) into normalized difference vegetation index (NDVI) and quality-control flags.

  ProductionDate :

  2017-06-08T11:27:29.000000Z

  ESDT :

  AVH13C1

  LocalGranuleId :

  AVHRR-Land_v005_AVH13C1_NOAA-16_20010101_c20170608112729.nc

  id :

  AVHRR-Land_v005_AVH13C1_NOAA-16_20010101_c20170608112729.nc

  RangeBeginningDate :

  2001-001

  RangeBeginningTime :

  00:00:00.0000

  RangeEndingDate :

  2001-001

  RangeEndingTime :

  23:59:59.9999

  PercentMissingData :

  16.33

  PercentValidDaytimeData :

  33.79

  PercentValidDaytimeLand :

  33.79

  PercentValidClearDaytimeLand :

  4.54

  PercentValidDaytimeLandInCloudShadow :

  0.85

  PercentValidClearDaytimeWater :

  0.00

  PercentValidDaytimeWaterInCloudShadow :

  0.00

If the above downloading procedure is not working as expected, please retrieve the following sample to execute the remaining cells of the notebook. Ask for the help of your project TA to download your data of interest.

# uncomment to get a subsample of AVHRR-Land_v005_AVH13C1_NOAA-16 NDVI data if data retrieval above is not working properly

#fname_ndvi_200101 = 'AVHRR-Land_v005_AVH13C1_NOAA-16_200101.nc'
#link_id_ndvi = "a9v7h"
#url_ndvi_200101 = f"https://osf.io/download/{link_id_ndvi}/"
#
#ds = xr.open_dataset(pooch_load(url_ndvi_200101, fname_ndvi_200101))
#ds

Note that the downloaded satellite data is quality-controlled and flagged accordingly. Please refer to W1D3 Tutorial 3 and others to decipher it. Choose the feasible helper functions, e.g. get_quality_info_AVHRR(QA) depending on the sensor of interest. VIIRS is a state-of-the-art sensor that produces measurements that have been available since 2014. We provide an example of usage in the following.

# AVHRR: function to extract high-quality NDVI data
def get_quality_info_AVHRR(QA):
    """
    QA: the QA value read in from the NDVI data

    High-quality NDVI should meet the following criteria:
    Bit 7: 1 (All AVHRR channels have valid values)
    Bit 2: 0 (The pixel is not covered by cloud shadow)
    Bit 1: 0 (The pixel is not covered by cloud)
    Bit 0:

    Output:
    True: high quality
    False: low quality
    """
    # unpack quality assurance flag for cloud (byte: 1)
    cld_flag = (QA % (2**2)) // 2
    # unpack quality assurance flag for cloud shadow (byte: 2)
    cld_shadow = (QA % (2**3)) // 2**2
    # unpack quality assurance flag for AVHRR values (byte: 7)
    value_valid = (QA % (2**8)) // 2**7

    mask = (cld_flag == 0) & (cld_shadow == 0) & (value_valid == 1)

    return mask

# get the quality assurance value from NDVI data
QA = ds.QA

# create the high quality information mask
mask = get_quality_info_AVHRR(QA)

# check the quality flag mask information
#mask

# mask ndvi data
ndvi_masked = ds.NDVI.where(mask)
ndvi_masked

<xarray.DataArray 'NDVI' (time: 31, latitude: 3600, longitude: 7200)> Size: 3GB
dask.array<where, shape=(31, 3600, 7200), dtype=float32, chunksize=(1, 1200, 2400), chunktype=numpy.ndarray>
Coordinates:
  * latitude   (latitude) float32 14kB 89.97 89.93 89.88 ... -89.93 -89.97
  * longitude  (longitude) float32 29kB -180.0 -179.9 -179.9 ... 179.9 180.0
  * time       (time) float32 124B 7.305e+03 7.306e+03 ... 7.334e+03 7.335e+03
Attributes:
    long_name:      NOAA Climate Data Record of Normalized Difference Vegetat...
    units:          1
    valid_range:    [-1000 10000]
    grid_mapping:   crs
    standard_name:  normalized_difference_vegetation_index

xarray.DataArray

'NDVI'

• time: 31
• latitude: 3600
• longitude: 7200

• dask.array<chunksize=(1, 1200, 2400), meta=np.ndarray>

  Array Chunk Bytes 2.99 GiB 10.99 MiB Shape (31, 3600, 7200) (1, 1200, 2400) Dask graph 279 chunks in 139 graph layers Data type float32 numpy.ndarray

• Coordinates: (3)
  • latitude
    (latitude)
    float32
    89.97 89.93 89.88 ... -89.93 -89.97

    units :

    degrees_north

    long_name :

    latitude

    axis :

    Y

    standard_name :

    latitude

    bounds :

    lat_bnds

    array([ 89.975  ,  89.925  ,  89.875  , ..., -89.875  , -89.925  , -89.97499],
          dtype=float32)

  • longitude
    (longitude)
    float32
    -180.0 -179.9 ... 179.9 180.0

    units :

    degrees_east

    long_name :

    longitude

    axis :

    X

    standard_name :

    longitude

    bounds :

    lon_bnds

    array([-179.975  , -179.925  , -179.875  , ...,  179.875  ,  179.92499,
            179.975  ], dtype=float32)

  • time
    (time)
    float32
    7.305e+03 7.306e+03 ... 7.335e+03

    units :

    days since 1981-01-01 00:00:00

    long_name :

    time

    axis :

    T

    standard_name :

    time

    calendar :

    gregorian

    array([7305., 7306., 7307., 7308., 7309., 7310., 7311., 7312., 7313., 7314.,
           7315., 7316., 7317., 7318., 7319., 7320., 7321., 7322., 7323., 7324.,
           7325., 7326., 7327., 7328., 7329., 7330., 7331., 7332., 7333., 7334.,
           7335.], dtype=float32)

• Indexes: (3)
  • latitude
    PandasIndex

    PandasIndex(Index([  89.9749984741211,  89.92500305175781,             89.875,
            89.82499694824219,   89.7750015258789,   89.7249984741211,
            89.67500305175781,             89.625,  89.57499694824219,
             89.5250015258789,
           ...
           -89.52499389648438, -89.57499694824219,            -89.625,
           -89.67500305175781, -89.72499084472656, -89.77499389648438,
           -89.82499694824219,            -89.875, -89.92500305175781,
           -89.97499084472656],
          dtype='float32', name='latitude', length=3600))

  • longitude
    PandasIndex

    PandasIndex(Index([-179.97500610351562,  -179.9250030517578,            -179.875,
            -179.8249969482422, -179.77499389648438, -179.72500610351562,
            -179.6750030517578,            -179.625,  -179.5749969482422,
           -179.52499389648438,
           ...
            179.52499389648438,  179.57501220703125,             179.625,
            179.67498779296875,  179.72500610351562,  179.77499389648438,
            179.82501220703125,             179.875,  179.92498779296875,
            179.97500610351562],
          dtype='float32', name='longitude', length=7200))

  • time
    PandasIndex

    PandasIndex(Index([7305.0, 7306.0, 7307.0, 7308.0, 7309.0, 7310.0, 7311.0, 7312.0, 7313.0,
           7314.0, 7315.0, 7316.0, 7317.0, 7318.0, 7319.0, 7320.0, 7321.0, 7322.0,
           7323.0, 7324.0, 7325.0, 7326.0, 7327.0, 7328.0, 7329.0, 7330.0, 7331.0,
           7332.0, 7333.0, 7334.0, 7335.0],
          dtype='float32', name='time'))

• Attributes: (5)

  long_name :

  NOAA Climate Data Record of Normalized Difference Vegetation Index

  units :

  1

  valid_range :

  [-1000 10000]

  grid_mapping :

  crs

  standard_name :

  normalized_difference_vegetation_index

# draw data to an example plot
ndvi_masked.isel(time=30).coarsen(latitude=10).mean().coarsen(longitude=20).mean().plot()

<matplotlib.collections.QuadMesh at 0x7f9aec422700>

FAO Data: Cereal Production

Cereal production is a crucial component of global agriculture and food security. The Food and Agriculture Organization collects and provides data on cereal production, which includes crops such as wheat, rice, maize, barley, oats, rye, sorghum, millet, and mixed grains. The data covers various indicators such as production quantity, area harvested, yield, and production value.

The FAO also collects data on “area harvested”, which refers to the area of land that is being used to grow cereal crops. This information can be valuable for assessing the productivity and efficiency of cereal production systems in different regions, as well as identifying potential areas for improvement. Overall, the FAO’s data on cereal production and land under cereals production is an important resource for policymakers, researchers, and other stakeholders who are interested in understanding global trends in agriculture and food security.

filename_cereal = 'data_cereal_land.csv'

url_cereal = 'https://raw.githubusercontent.com/Sshamekh/Heatwave/f85f43997e3d6ae61e5d729bf77cfcc188fbf2fd/data_cereal_land.csv'

path_cereal = 'shared/data/'
if os.path.exists(path_cereal + filename_cereal):
    ds_cereal_land = pd.read_csv(path_cereal + filename_cereal)
else:
    ds_cereal_land = pd.read_csv(pooch_load(url_cereal,filename_cereal))

ds_cereal_land.head()

Downloading data from 'https://raw.githubusercontent.com/Sshamekh/Heatwave/f85f43997e3d6ae61e5d729bf77cfcc188fbf2fd/data_cereal_land.csv' to file '/tmp/data_cereal_land.csv'.

SHA256 hash of downloaded file: 0d71645aeeb9e1cca8abe179c525c496f3b2b02867119069679762c0f9f1da47
Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future.

	Country Name	Country Code	Series Name	Series Code	1972 [YR1972]	1973 [YR1973]	1974 [YR1974]	1975 [YR1975]	1976 [YR1976]	1977 [YR1977]	...	2012 [YR2012]	2013 [YR2013]	2014 [YR2014]	2015 [YR2015]	2016 [YR2016]	2017 [YR2017]	2018 [YR2018]	2019 [YR2019]	2020 [YR2020]	2021 [YR2021]
0	Afghanistan	AFG	Cereal production (metric tons)	AG.PRD.CREL.MT	3950000	4270000	4351000	4481000	4624000	4147000	...	6379000	6520329	6748023.28	5808288	5532695.42	4892953.97	4133051.85	5583461	6025977	4663880.79
1	Afghanistan	AFG	Land under cereal production (hectares)	AG.LND.CREL.HA	3923100	3337000	3342000	3404000	3394000	3388000	...	3143000	3182922	3344733	2724070	2793694	2419213	1911652	2641911	3043589	2164537
2	Albania	ALB	Cereal production (metric tons)	AG.PRD.CREL.MT	585830	625498	646200	666500	857000	910400	...	697400	702870	700370	695000	698430	701734	678196	666065	684023	691126.7
3	Albania	ALB	Land under cereal production (hectares)	AG.LND.CREL.HA	331220	339400	334040	328500	350500	357000	...	142800	142000	143149	142600	148084	145799	140110	132203	131310	134337
4	Algeria	DZA	Cereal production (metric tons)	AG.PRD.CREL.MT	2362625	1595994	1480275	2680452	2313186	1142509	...	5137455	4912551	3435535	3761229.6	3445227.37	3478175.14	6066252.82	5633596.78	4393336.75	2784017.29

5 rows × 54 columns

# example
country = 'Brazil'
unit = 'MT' # metric tons

ds_cereal_land_brazil = ds_cereal_land[
(ds_cereal_land["Country Name"] == country) & (ds_cereal_land["Series Code"] == f'AG.PRD.CREL.{unit}')
].drop(
    ['Country Name', 'Country Code', 'Series Name', 'Series Code'], axis=1
).reset_index( drop=True ).transpose()

ds_cereal_land_brazil.head()

	0
1972 [YR1972]	22703928
1973 [YR1973]	23721606
1974 [YR1974]	26240014
1975 [YR1975]	26238419
1976 [YR1976]	31143200

We can now visualize the content of the dataset.

# example
plt.plot(np.arange(1972,2021,1),ds_cereal_land_brazil[1:].astype(float).values)

# aesthetics
plt.title(f'{country}')
plt.xlabel(f'Time (years)')
plt.ylabel(f'Cereal production ({unit})')

Text(0, 0.5, 'Cereal production (MT)')

Now you are all set to address the questions you are interested in!

Q1

Plot the annual mean of precipitation for three example years (e.g., 2001, 2005, 2016) using the CHIRPS data sets.

Hint: Check out the tutorials from W1D1 (e.g. T4), W1D2 (e.g. T2) and W1D3 (e.g. T5) to recall averaging with xarray.

Q2

Plot the global mean of land precipitation as a function of time for the last 40 years.

Hint: Check out the tutorials that contain time series calculations e.g. from W1D2 (e.g. T1), W1D3 (e.g. T5) and W1D5 (e.g. T8).

Compute seasonal means for June-August (JJA) and December-February (DJF).

Hint: Check out the code snippet in the Indices for Extreme Events section of this notebook. Furthermore, recall Tutorial 5 of W1D1.

Q3

Repeat the analysis in Q2 for a selected region of interest. How does this compare with the respective mean precipitation in the northern and southern hemispheres? How does this compare to the global mean?

Hint: Check out tutorials that showed how to select certain regions of interest, e.g. T8 and T9 of W1D1, T1 of W1D2 and others.

Q4

Examine the time series in your region of interest and identify extreme values.

Hint: Select a method from a predefined list of indices (provided above in Section Indices for Extreme Events) based on relative or absolute thresholds.

Do extreme events occur more often in wet or dry seasons?

Hint: We briefly discussed wet and dry season differences in Tutorial 4 of W1D3 already, however for another data set of monthly frequency namely the Global Precipitation Climatology Project (GPCP) Monthly Precipitation Climate Data Record (CDR) (cf. Tutorial 4 of W1D3). You also examined extreme events on W2D3.

Further Reading

• Anyamba, A. and Tucker, C.J., 2012. Historical perspective of AVHRR NDVI and vegetation drought monitoring. Remote sensing of drought: innovative monitoring approaches, 23, pp.20. https://digitalcommons.unl.edu/nasapub/217/

• Schultz, P. A., and M. S. Halpert, 1993. Global correlation of temperature, NDVI and precipitation. Advances in Space Research 13(5). pp.277-280. doi.org/10.1016/0273-1177(93)90559-T (not open access)

• Seneviratne, S.I. et al., 2021: Weather and Climate Extreme Events in a Changing Climate. In Climate Change 2021: The Physical Science Basis. Contribution of Working Group I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change [Masson-Delmotte, V. et al. (eds.)]. Cambridge University Press, Cambridge, United Kingdom and New York, NY, USA, pp.1513–1766, https://www.ipcc.ch/report/ar6/wg1/chapter/chapter-11/

• IPCC, 2021: Annex VI: Climatic Impact-driver and Extreme Indices [Gutiérrez J.M. et al.(eds.)]. In Climate Change 2021: The Physical Science Basis. Contribution of Working Group I to the Sixth Assessment Report of the Intergovernmental Panel on Climate Change [Masson-Delmotte, V. et al. (eds.)]. Cambridge University Press, Cambridge, United Kingdom and New York, NY, USA, pp. 2205–2214, https://www.ipcc.ch/report/ar6/wg1/downloads/report/IPCC_AR6_WGI_AnnexVI.pdf

• Zhang, X., Alexander, L., Hegerl, G.C., Jones, P., Tank, A.K., Peterson, T.C., Trewin, B. and Zwiers, F.W., 2011. Indices for monitoring changes in extremes based on daily temperature and precipitation data. Wiley Interdisciplinary Reviews: Climate Change, 2(6), pp.851-870. doi.org/10.1002/wcc.147 (not open access)