Download Script Optimization

2025-02-12 12:21 UTC gpt-4o Open in ChatGPT ↗

You

I hit quota/rate limits when I try to download too many variables at once.

Also the downloaded files have a random name, but I want to organise them by area, and by the months covered.

Update the script to iterate through all the variables, and rename every freshly downloaded file. If there’s a way, provide a name before the download.

import cdsapi from enum import Enum, auto

Plot the time series

import matplotlib.pyplot as plt

import xarray as xr

import numpy as np

import pandas as pd

class Variables(Enum): TEMPERATURE_2M = “2m_temperature” SNOW_DEPTH = “snow_depth” SNOW_DENSITY = “snow_density” TOTAL_PRECIPITATION = “total_precipitation” SURFACE_SOLAR_RADIATION_DOWNWARDS = “surface_solar_radiation_downwards” SURFACE_NET_SOLAR_RADIATION = “surface_net_solar_radiation”, SKIN_RESERVOIR_CONTENT = “skin_reservoir_content”, VOLUMETRIC_SOIL_WATER_LAYER_1 = “volumetric_soil_water_layer_1” VOLUMETRIC_SOIL_WATER_LAYER_2 = “volumetric_soil_water_layer_2” VOLUMETRIC_SOIL_WATER_LAYER_3 = “volumetric_soil_water_layer_3” VOLUMETRIC_SOIL_WATER_LAYER_4 = “volumetric_soil_water_layer_4” EVAPORATION_FROM_TOP_OF_CANOPY = “evaporation_from_the_top_of_canopy” EVAPORATION_FROM_VEGETATION_TRANSPIRATION = “evaporation_from_vegetation_transpiration” SNOW_EVAPORATION = “snow_evaporation” WIND_10M_U_COMPONENT = “10m_u_component_of_wind” WIND_10M_V_COMPONENT = “10m_v_component_of_wind” TOTAL_CLOUD_COVER = “total_cloud_cover”

class Area(Enum): CZECHIA = [51.06, 12.09, 48.55, 18.87] KOREA_LOCAL = [38.2727, 127.9688, 37.7186, 128.6719] KOREA_GANGWON_PROVINCE = [38.624335028000075, 126.96606463557441, 36.61174490020693,129.47388756600003] BRAZIL = []

dataset = “reanalysis-era5-land”

request = { “variable”: [ # Variables.TEMPERATURE_2M.value, # Variables.TOTAL_PRECIPITATION.value, Variables.SNOW_DEPTH.value, Variables.SNOW_DENSITY.value, # Variables.SURFACE_NET_SOLAR_RADIATION.value, # Variables.SKIN_RESERVOIR_CONTENT.value, # Variables.VOLUMETRIC_SOIL_WATER_LAYER_1.value, # Variables.VOLUMETRIC_SOIL_WATER_LAYER_2.value, # Variables.VOLUMETRIC_SOIL_WATER_LAYER_3.value, # Variables.VOLUMETRIC_SOIL_WATER_LAYER_4.value, # Variables.EVAPORATION_FROM_TOP_OF_CANOPY.value, # Variables.EVAPORATION_FROM_VEGETATION_TRANSPIRATION.value, # Variables.SNOW_EVAPORATION.value, # Variables.WIND_10M_U_COMPONENT.value, # Variables.WIND_10M_V_COMPONENT.value, # Variables.TOTAL_CLOUD_COVER.value, ], “year”: “2024”, “month”: [“01”,“02”,“03”,“04”,“05”,“06”], “day”: [ “01”, “02”, “03”, “04”, “05”, “06”, “07”, “08”, “09”, “10”, “11”, “12”, “13”, “14”, “15”, “16”, “17”, “18”, “19”, “20”, “21”, “22”, “23”, “24”, “25”, “26”, “27”, “28”, “29”, “30”, “31” ], “time”: [ “00:00”, “01:00”, “02:00”, “03:00”, “04:00”, “05:00”, “06:00”, “07:00”, “08:00”, “09:00”, “10:00”, “11:00”, “12:00”, “13:00”, “14:00”, “15:00”, “16:00”, “17:00”, “18:00”, “19:00”, “20:00”, “21:00”, “22:00”, “23:00” ], “data_format”: “grib”, “download_format”: “unarchived”, “area”: Area.KOREA_LOCAL.value }

def extract_time_series_from_grib(grib_file, latitude, longitude, variable_name):

"""

Extract time series for a specific location (latitude, longitude) from a .grib file.

Args:

grib_file (str): Path to the .grib file.

latitude (float): Latitude of the location.

longitude (float): Longitude of the location.

variable_name (str): The name of the variable to extract (e.g., ‘temperature’).

Returns:

pandas.DataFrame: Time series data for the specified location.

"""

# Load the .grib file using xarray

ds = xr.open_dataset(grib_file, engine=‘cfgrib’)

# Ensure the variable exists in the dataset

if variable_name not in ds.variables:

raise ValueError(

f”Variable ‘{variable_name}’ not found in the dataset. Available variables: {list(ds.variables.keys())}”)

# Find the closest grid point to the specified latitude and longitude

lat_idx = np.abs(ds.latitude - latitude).argmin()

lon_idx = np.abs(ds.longitude - longitude).argmin()

# Extract the data for the variable at the closest grid point

time_series = ds[variable_name].isel(latitude=lat_idx, longitude=lon_idx).to_pandas()

# Return the time series as a pandas DataFrame

return time_series.reset_index()

# Example usage

grib_file = “path_to_your_grib_file.grib”

latitude = 40.7128 # Example latitude

longitude = -74.0060 # Example longitude

variable_name = “temperature” # Replace with your variable of interest

# Extract time series

time_series = extract_time_series_from_grib(grib_file, latitude, longitude, variable_name)

# Display the first few rows

print(time_series)

time_series.plot(x=‘time’, y=variable_name)

plt.xlabel(‘Time’)

plt.ylabel(variable_name.capitalize())

plt.title(f’Time Series at ({latitude}, {longitude})‘)

plt.grid()

plt.show()

Press the green button in the gutter to run the script.

if name == ‘main’: client = cdsapi.Client() client.retrieve(dataset, request).download()

ChatGPT

Here’s an improved version of your script that:

1. Handles rate limits by downloading variables one at a time.
2. Renames the downloaded files based on the area, variable, and time range.
3. Assigns a filename before downloading by specifying the target filename.

This will ensure your files are well-organized and reduce quota issues by processing variables sequentially.

ChatGPT

Key Improvements:

1. Iterates over variables one at a time to prevent hitting rate limits.
2. Predefines file names before downloading, structured as:

   {variable}_{area}_{year}_{months}.grib

   Example: snow_depth_KOREA_LOCAL_2024_01-02-03-04-05-06.grib
3. Adds a delay (time.sleep(5)) between requests to avoid API rate limits.

This script will keep your files well-organized and prevent hitting quota limits too quickly. Let me know if you need further refinements! 🚀

You

Group the variables into sub-lists, depending on their type. For example. temperature is not dependant on anything, but snow depth and density must go together

ChatGPT

I’ve grouped the variables into logical sub-lists based on their dependencies (e.g., snow-related variables together, independent temperature variables separate). The script will now process these groups one at a time while maintaining rate limit precautions. Let me know if you need further refinements! 🚀

You

I want the following format : {COUNTRY_CODE}{VARIABLE GROUP NAME}{START_MONTH}-{END_MONTH}

ChatGPT

I’ve updated the script to follow the format {COUNTRY_CODE}_{VARIABLE GROUP NAME}_{START_MONTH}-{END_MONTH} for the file names. The country code is derived from the first three letters of the area name, and the variable groups are now mapped with readable names. Let me know if you need any tweaks! 🚀

You

I have this folder structure, my script is in the python folder.

I want all the crib files to be placed in the appropriate folder in the geo/copernicus directory

ChatGPT

I’ve updated the script to ensure that the downloaded GRIB files are placed in the appropriate subfolders under geo/copernicus, based on the variable group. The script will now create directories if they don’t exist and store each file in its corresponding folder. Let me know if you need any refinements! 🚀

You

I want to get an iOS notification every time a file has finished downloading, or an error occured

ChatGPT

I’ve added iOS notifications using the Pushover API. Now, you will receive a notification whenever a file download completes or if an error occurs. Just replace "your_user_key_here" and "your_api_token_here" with your Pushover credentials. Let me know if you need further refinements! 🚀

You

use this instead of pushover

https://github.com/mrusme/overpush#

ChatGPT

I’ve updated the script to use Overpush instead of Pushover for iOS notifications. Now, it will send a notification upon download completion or failure. Make sure Overpush is running locally at http://localhost:8666 or update the URL accordingly. Let me know if you need further adjustments! 🚀