arctichoke.path

Submodules

• arctichoke.path.file_lists
• arctichoke.path.manipulate_paths
• arctichoke.path.model_paths
• arctichoke.path.variable_paths

Functions

list_available_models([data_dir, project, ...])	List the names of the models available alphabetically.
get_model_path(source_id[, data_dir, project, activity_id])	Find the file path for the specified model.
get_variable_path(source_id, variable_id[, data_dir, ...])	Find the file path for the specified model and variable.
list_variable_files(source_id, variable_id[, ...])	Get a list of data files for the specified variable for the specified model.
select_files_by_time(data_filepaths, start, end[, test])	Filter the list to be only the files in the given date range.
verify_path(path)	Verify that the filepath exists.
remove_non_empty_directory(base_dir)	Remove a non-empty directory and all its contents.
make_file_path(path)	Create a file path.
get_model_path(source_id[, data_dir, project, activity_id])	Find the file path for the specified model.
list_available_models([data_dir, project, ...])	List the names of the models available alphabetically.
list_available_variables(source_id[, experiment_id, ...])	List the names of the variables available for the specified model.
get_variable_path(source_id, variable_id[, data_dir, ...])	Find the file path for the specified model and variable.
list_variable_modifications(variable_path[, ...])	List the modifications made to variable files in the specified path.

Package Contents

arctichoke.path.list_available_models(data_dir: str = '/arctichoke_data/bergybits/data', project: str = 'CMIP6', activity_id: str = 'HighResMIP', institution_id: str, [str] = None)

List the names of the models available alphabetically.

Search the data directory that has been populated by esgpull and return a list of all the available models, or source_id’s. This assumes the esgpull convention of subdirectories: data/project/activity_id/institution_id/source_id.

Parameters:

• data_dir (str, optional) – The absolute file path to the data directory. The esgpull convention means this should end in /data. Default is /arctichoke_data/bergybits/data.

• project (str, optional) – The name of the project in which to search for available models. Default is CMIP6.

• activity_id (str, optional) – The name of the activity ID in which to search for available models. Default is HighResMIP.

• institution_id (str, list of str, or None, optional) – The name(s) of the institution ID(s) in which to search for available models. If None is given, all available institutions are included. Default is None.

Returns:

model_names – A list, sorted alphabetically, of the names of the available models.

Return type:

List of str

Examples

>>> from arctichoke.path.find_data import list_available_models
>>> list_available_models()
['AWI/AWI-CM-1-1-HR', 'AWI/AWI-CM-1-1-LR', 'BCC/BCC-CSM2-HR', 'EC-Earth-Consortium/EC-Earth3P', 'EC-Earth-Consortium/EC-Earth3P-HR', 'MOHC/HadGEM3-GC31-HH', 'MOHC/HadGEM3-GC31-HM', 'MOHC/HadGEM3-GC31-LL', 'MOHC/HadGEM3-GC31-MM', 'NCAR/CESM1-CAM5-SE-HR', 'NCAR/CESM1-CAM5-SE-LR', 'NERC/HadGEM3-GC31-HH', 'NERC/HadGEM3-GC31-HM']
>>> list_available_models(institution_id = 'EC-Earth-Consortium')
['EC-Earth-Consortium/EC-Earth3P', 'EC-Earth-Consortium/EC-Earth3P-HR']
>>> list_available_models(institution_id = 'MOHC')
['MOHC/HadGEM3-GC31-HH', 'MOHC/HadGEM3-GC31-HM', 'MOHC/HadGEM3-GC31-LL', 'MOHC/HadGEM3-GC31-MM']
>>> list_available_models(institution_id = ['MOHC', 'NERC'])
['MOHC/HadGEM3-GC31-HH', 'MOHC/HadGEM3-GC31-HM', 'MOHC/HadGEM3-GC31-LL', 'MOHC/HadGEM3-GC31-MM', 'NERC/HadGEM3-GC31-HH', 'NERC/HadGEM3-GC31-HM']

arctichoke.path.get_model_path(source_id: str, data_dir: str = '/arctichoke_data/bergybits/data', project: str = 'CMIP6', activity_id: str = 'HighResMIP')

Find the file path for the specified model.

Search the data directory that has been populated by esgpull for a model’s source_id and return the file path. This assumes the esgpull convention of subdirectories: data/project/activity_id/institution_id/source_id.

Parameters:

• source_id (str) – The name of the source ID (model) for which to find the file path.

• data_dir (str, optional) – The absolute file path to the data directory. The esgpull convention means this should end in /data. Default is /arctichoke_data/bergybits/data.

• project (str, optional) – The name of the project in which to search for available models. Default is CMIP6.

• activity_id (str, optional) – The name of the activity ID in which to search for available models. Default is HighResMIP.

Returns:

model_filepaths – A list of the file path(s) for the specified model.

Return type:

List of str

Examples

>>> from arctichoke.path.find_data import get_model_path
>>> get_model_path(source_id='AWI-CM-1-1-HR')
['/arctichoke_data/bergybits/data/CMIP6/HighResMIP/AWI/AWI-CM-1-1-HR']
>>> get_model_path('HadGEM3-GC31-HM')
['/arctichoke_data/bergybits/data/CMIP6/HighResMIP/MOHC/HadGEM3-GC31-HM', '/arctichoke_data/bergybits/data/CMIP6/HighResMIP/NERC/HadGEM3-GC31-HM']

arctichoke.path.get_variable_path(source_id: str, variable_id: str, data_dir: str = '/arctichoke_data/bergybits/data', project: str = 'CMIP6', activity_id: str = 'HighResMIP', experiment_id: str = 'hist-1950', variant_label: str = None, **kwargs)

Find the file path for the specified model and variable.

Search the data directory that has been populated by esgpull for a model’s source_id and return the file path. This assumes the esgpull convention of subdirectories: data/project/activity_id/institution_id/source_id/experiment_id/variant_label/table_id/variable_id.

Parameters:

• source_id (str) – The name of the source ID (model) for which to find the variable path.

• variable_id (str) – The name of the variable ID for which to find the variable path.

• data_dir (str, optional) – The absolute file path to the data directory. The esgpull convention means this should end in /data. Default is /arctichoke_data/bergybits/data.

• project (str, optional) – The name of the project in which to search for available models. Default is CMIP6.

• activity_id (str, optional) – The name of the activity ID in which to search for available models. Default is HighResMIP.

• experiment_id (str, optional) – The name of the experiment ID in which to search for available variables. Default is hist-1950.

• variant_label (str, None, optional) – The name of the variant label (ensemble member) for which to get the variable path. If None is given, the file path for the first variant with the variable is returned. Default is None.

• **kwargs – Keyword arguments to pass to get_model_path().

Returns:

variable_path – The file path to the directory for the specified variable from the specified model.

Return type:

str

Examples

>>> from arctichoke.path.find_data import get_variable_path
>>> get_variable_path(source_id = 'HadGEM3-GC31-HM', variable_id = 'areacello')
'/arctichoke_data/bergybits/data/CMIP6/HighResMIP/MOHC/HadGEM3-GC31-HM/hist-1950/r1i1p1f1/Ofx/areacello'
>>> get_variable_path(source_id = 'EC-Earth3P-HR', variable_id = 'siage')
'/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r3i1p2f1/SImon/siage'
/workspace/src/arctichoke/path/find_data.py:396: UserWarning: (get_variable_path) More than one file path found: ['/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r1i1p2f1/SImon/siage', '/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r2i1p2f1/SImon/siage', '/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r3i1p2f1/SImon/siage']
Returning first result in list.
'/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r1i1p2f1/SImon/siage'
>>> get_variable_path(source_id = 'EC-Earth3P-HR', variable_id = 'siage', variant_label = 'r3i1p2f1')
'/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r3i1p2f1/SImon/siage'

arctichoke.path.list_variable_files(source_id: str, variable_id: str, with_modification: str = None, **kwargs)

Get a list of data files for the specified variable for the specified model.

Search for all the files for the given model and variable and return them in a list. This assumes the esgpull convention of subdirectories: data/project/activity_id/institution_id/source_id/experiment_id/variant_label/table_id/variable_id/gn/version/data_file.nc.

Parameters:

• source_id (str) – The name of the source ID (model) for which to get the variable files.

• variable_id (str) – The name of the variable ID for which to get the variable files.

• with_modification (str, None, optional) – The prefix of a modification to the data files to find. Ex: trim_NWP_. If None, returns only the original files. Default is None.

• **kwargs – Keyword arguments to pass to get_variable_path().

Returns:

data_filepaths – A list, sorted alphabetically, of the filepaths of the variable data files.

Return type:

List of str

Examples

>>> from arctichoke.path.find import list_variable_files
>>> list_variable_files(source_id = 'HadGEM3-GC31-HM', variable_id = 'areacello')
['/arctichoke_data/bergybits/data/CMIP6/HighResMIP/MOHC/HadGEM3-GC31-HM/hist-1950/r1i1p1f1/Ofx/areacello/gn/v20190301/areacello_Ofx_HadGEM3-GC31-HM_hist-1950_r1i1p1f1_gn.nc']
>>> list_variable_files(source_id = 'EC-Earth3P-HR', variable_id = 'siage')
/workspace/src/arctichoke/path/find_data.py:399: UserWarning: (get_variable_path) More than one file path found: ['/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r1i1p2f1/SImon/siage', '/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r2i1p2f1/SImon/siage', '/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r3i1p2f1/SImon/siage']
Returning first result in list.
['/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r1i1p2f1/SImon/siage/gn/v20181212/siage_SImon_EC-Earth3P-HR_hist-1950_r1i1p2f1_gn_195001-195012.nc',
...
'/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r1i1p2f1/SImon/siage/gn/v20181212/siage_SImon_EC-Earth3P-HR_hist-1950_r1i1p2f1_gn_201401-201412.nc']

arctichoke.path.select_files_by_time(data_filepaths: [str], start: str, int, end: str, int, test: bool = False)

Filter the list to be only the files in the given date range.

From the given list of datafiles, find the ones that are in between the given start and end dates and return those in a list. If the start and end are given as years, the returned list will include both the start and end years, if available.

Parameters:

• data_filepaths (List of str) – A list, sorted alphabetically, of the filepaths of the variable data files.

• start (str, int) – The start date of the time range, either a string in the format YYYY-MM-DD or an integer of a year.

• end (str, int) – The end date of the time range, either a string in the format YYYY-MM-DD or an integer of a year.

• test (bool, optional) – If True, the function skips verifying the data_filepaths exist for use in testing. Default is False.

Returns:

data_filepaths – A list, sorted alphabetically, of the filepaths in the date range.

Return type:

List of str

Examples

>>> from arctichoke.path.find import select_files_by_time
>>> select_files_by_time(source_id = 'HadGEM3-GC31-HM', variable_id = 'areacello')
['/arctichoke_data/bergybits/data/CMIP6/HighResMIP/MOHC/HadGEM3-GC31-HM/hist-1950/r1i1p1f1/Ofx/areacello/gn/v20190301/areacello_Ofx_HadGEM3-GC31-HM_hist-1950_r1i1p1f1_gn.nc']

arctichoke.path.verify_path(path)

Verify that the filepath exists.

Check if the path to the data files exists and is valid. If not, raise an error.

Parameters:

path (str) – Relative path to the directory containing data files.

Raises:

FileNotFoundError – If the specified path does not exist.

Returns:

path – The verified path to the data files.

Return type:

str

Examples

>>> verify_path('README.md')
'README.md'

arctichoke.path.remove_non_empty_directory(base_dir)

Remove a non-empty directory and all its contents.

Recursively delete all files and directories in the given path.

Parameters:

base_dir (str) – Relative path to the directory to be removed.

arctichoke.path.make_file_path(path)

Create a file path.

Create the specified directory structure if the given path doesn’t exist.

Parameters:

path (str) – Relative path to make.

Returns:

path – The verified path to the data files.

Return type:

str

Examples

>>> make_file_path('datafiles/some/more/data/a_file.txt')
'datafiles/some/more/data/'

arctichoke.path.get_model_path(source_id: str, data_dir: str = '/arctichoke_data/bergybits/data', project: str = 'CMIP6', activity_id: str = 'HighResMIP')

Find the file path for the specified model.

Search the data directory that has been populated by esgpull for a model’s source_id and return the file path. This assumes the esgpull convention of subdirectories: data/project/activity_id/institution_id/source_id.

Parameters:

• source_id (str) – The name of the source ID (model) for which to find the file path.

• data_dir (str, optional) – The absolute file path to the data directory. The esgpull convention means this should end in /data. Default is /arctichoke_data/bergybits/data.

• project (str, optional) – The name of the project in which to search for available models. Default is CMIP6.

• activity_id (str, optional) – The name of the activity ID in which to search for available models. Default is HighResMIP.

Returns:

model_filepaths – A list of the file path(s) for the specified model.

Return type:

List of str

Examples

>>> from arctichoke.path.find_data import get_model_path
>>> get_model_path(source_id='AWI-CM-1-1-HR')
['/arctichoke_data/bergybits/data/CMIP6/HighResMIP/AWI/AWI-CM-1-1-HR']
>>> get_model_path('HadGEM3-GC31-HM')
['/arctichoke_data/bergybits/data/CMIP6/HighResMIP/MOHC/HadGEM3-GC31-HM', '/arctichoke_data/bergybits/data/CMIP6/HighResMIP/NERC/HadGEM3-GC31-HM']

arctichoke.path.list_available_models(data_dir: str = '/arctichoke_data/bergybits/data', project: str = 'CMIP6', activity_id: str = 'HighResMIP', institution_id: str, [str] = None)

List the names of the models available alphabetically.

Search the data directory that has been populated by esgpull and return a list of all the available models, or source_id’s. This assumes the esgpull convention of subdirectories: data/project/activity_id/institution_id/source_id.

Parameters:

• data_dir (str, optional) – The absolute file path to the data directory. The esgpull convention means this should end in /data. Default is /arctichoke_data/bergybits/data.

• project (str, optional) – The name of the project in which to search for available models. Default is CMIP6.

• activity_id (str, optional) – The name of the activity ID in which to search for available models. Default is HighResMIP.

• institution_id (str, list of str, or None, optional) – The name(s) of the institution ID(s) in which to search for available models. If None is given, all available institutions are included. Default is None.

Returns:

model_names – A list, sorted alphabetically, of the names of the available models.

Return type:

List of str

Examples

>>> from arctichoke.path.find_data import list_available_models
>>> list_available_models()
['AWI/AWI-CM-1-1-HR', 'AWI/AWI-CM-1-1-LR', 'BCC/BCC-CSM2-HR', 'EC-Earth-Consortium/EC-Earth3P', 'EC-Earth-Consortium/EC-Earth3P-HR', 'MOHC/HadGEM3-GC31-HH', 'MOHC/HadGEM3-GC31-HM', 'MOHC/HadGEM3-GC31-LL', 'MOHC/HadGEM3-GC31-MM', 'NCAR/CESM1-CAM5-SE-HR', 'NCAR/CESM1-CAM5-SE-LR', 'NERC/HadGEM3-GC31-HH', 'NERC/HadGEM3-GC31-HM']
>>> list_available_models(institution_id = 'EC-Earth-Consortium')
['EC-Earth-Consortium/EC-Earth3P', 'EC-Earth-Consortium/EC-Earth3P-HR']
>>> list_available_models(institution_id = 'MOHC')
['MOHC/HadGEM3-GC31-HH', 'MOHC/HadGEM3-GC31-HM', 'MOHC/HadGEM3-GC31-LL', 'MOHC/HadGEM3-GC31-MM']
>>> list_available_models(institution_id = ['MOHC', 'NERC'])
['MOHC/HadGEM3-GC31-HH', 'MOHC/HadGEM3-GC31-HM', 'MOHC/HadGEM3-GC31-LL', 'MOHC/HadGEM3-GC31-MM', 'NERC/HadGEM3-GC31-HH', 'NERC/HadGEM3-GC31-HM']

arctichoke.path.list_available_variables(source_id: str, experiment_id: str = None, list_var_mods: bool = False, **kwargs)

List the names of the variables available for the specified model.

Search the data directory that has been populated by esgpull and return a list of all the available varibles, or variable_id’s for the specified model. This assumes the esgpull convention of subdirectories: data/project/activity_id/institution_id/source_id/experiment_id/variant_label/table_id/variable_id.

Parameters:

• source_id (str) – The name of the source ID (model) for which to find the variable paths.

• experiment_id (str or None, optional) – The name of the experiment ID in which to search for available variables. If None is given, all available experiments are included. Default is None.

• list_var_mods (bool, optional) – Whether to include information about modifications made to the datafiles. Default is False.

• **kwargs – Keyword arguments to pass to get_model_path().

Returns:

avail_var_dict – A dictionary of experiment ID’s whose keys are dictionaries of variant labels whose keys are lists of available variables.

Return type:

dict

Examples

>>> from arctichoke.path.find_data import list_available_variables
>>> list_available_variables(source_id = 'HadGEM3-GC31-HM')
{'MOHC/HadGEM3-GC31-HM': {
    'control-1950': {
        'r1i1p1f1': ['areacello']},
    'highres-future': {
        'r1i1p1f1': ['areacello', 'siu', 'siv', 'sithick', 'siconc', 'siage'],
        'r1i3p1f1': ['siconc', 'siage', 'siv', 'sithick', 'siu']},
    'hist-1950': {
        'r1i1p1f1': ['areacello', 'siage', 'siv', 'siu', 'siconc', 'sithick'],
        'r1i3p1f1': ['siconc', 'sithick', 'siu', 'siage', 'siv']}},
'NERC/HadGEM3-GC31-HM': {
    'highres-future': {
        'r1i2p1f1': ['siv', 'siu', 'siconc', 'sithick', 'siage']},
    'hist-1950': {
        'r1i2p1f1': ['siconc', 'siu', 'sithick', 'siv', 'siage']}}}
>>> list_available_variables(source_id = 'HadGEM3-GC31-HM', experiment_id = 'hist-1950')
{'MOHC/HadGEM3-GC31-HM': {
    'hist-1950': {
        'r1i1p1f1': ['areacello', 'siage', 'siv', 'siu', 'siconc', 'sithick'],
        'r1i3p1f1': ['siconc', 'sithick', 'siu', 'siage', 'siv']}},
'NERC/HadGEM3-GC31-HM': {
    'hist-1950': {
        'r1i2p1f1': ['siconc', 'siu', 'sithick', 'siv', 'siage']}}}

arctichoke.path.get_variable_path(source_id: str, variable_id: str, data_dir: str = '/arctichoke_data/bergybits/data', project: str = 'CMIP6', activity_id: str = 'HighResMIP', experiment_id: str = 'hist-1950', variant_label: str = None, **kwargs)

Find the file path for the specified model and variable.

Search the data directory that has been populated by esgpull for a model’s source_id and return the file path. This assumes the esgpull convention of subdirectories: data/project/activity_id/institution_id/source_id/experiment_id/variant_label/table_id/variable_id.

Parameters:

• source_id (str) – The name of the source ID (model) for which to find the variable path.

• variable_id (str) – The name of the variable ID for which to find the variable path.

• data_dir (str, optional) – The absolute file path to the data directory. The esgpull convention means this should end in /data. Default is /arctichoke_data/bergybits/data.

• project (str, optional) – The name of the project in which to search for available models. Default is CMIP6.

• activity_id (str, optional) – The name of the activity ID in which to search for available models. Default is HighResMIP.

• experiment_id (str, optional) – The name of the experiment ID in which to search for available variables. Default is hist-1950.

• variant_label (str, None, optional) – The name of the variant label (ensemble member) for which to get the variable path. If None is given, the file path for the first variant with the variable is returned. Default is None.

• **kwargs – Keyword arguments to pass to get_model_path().

Returns:

variable_path – The file path to the directory for the specified variable from the specified model.

Return type:

str

Examples

>>> from arctichoke.path.find_data import get_variable_path
>>> get_variable_path(source_id = 'HadGEM3-GC31-HM', variable_id = 'areacello')
'/arctichoke_data/bergybits/data/CMIP6/HighResMIP/MOHC/HadGEM3-GC31-HM/hist-1950/r1i1p1f1/Ofx/areacello'
>>> get_variable_path(source_id = 'EC-Earth3P-HR', variable_id = 'siage')
'/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r3i1p2f1/SImon/siage'
/workspace/src/arctichoke/path/find_data.py:396: UserWarning: (get_variable_path) More than one file path found: ['/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r1i1p2f1/SImon/siage', '/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r2i1p2f1/SImon/siage', '/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r3i1p2f1/SImon/siage']
Returning first result in list.
'/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r1i1p2f1/SImon/siage'
>>> get_variable_path(source_id = 'EC-Earth3P-HR', variable_id = 'siage', variant_label = 'r3i1p2f1')
'/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r3i1p2f1/SImon/siage'

arctichoke.path.list_variable_modifications(variable_path: str, list_filenames: bool = False, verbose: bool = False)

List the modifications made to variable files in the specified path.

Search the given directory of data files and return a dictionary indicating whether any modifications have been made and how many files in each modification category exist. This assumes modifications are made to files by adding a prefix to the file name.

Parameters:

• variable_path (str) – The filepath to the variable in question. Expected format: data/project/activity_id/institution_id/source_id/experiment_id/variant_label/table_id/variable_id.

• list_filenames (bool, optional) – Whether to list all the file names for each modification or to give the number of files. Default is False, which gives the number of files.

• verbose (bool, optional) – Whether to output additional information while the function is running. Default is False.

Returns:

var_mod_dict – A dictionary of modification prefixes, the values of which are either the number of files under that modification or a list all the files under that modification.

Return type:

dict

Examples

>>> from arctichoke.path.find_data import list_variable_modifications
>>> list_variable_modifications(variable_path = '/arctichoke_data/bergybits/data/CMIP6/HighResMIP/AWI/AWI-CM-1-1-HR/hist-1950/r1i1p1f2/Ofx/areacello', list_filenames = True)
{'': ['areacello_Ofx_AWI-CM-1-1-HR_hist-1950_r1i1p1f2_gn.nc']}
>>> list_variable_modifications(variable_path = '/arctichoke_data/bergybits/data/CMIP6/HighResMIP/EC-Earth-Consortium/EC-Earth3P-HR/hist-1950/r2i1p2f1/SImon/siconc')
{'': 65, 'trim_NWP_': 65}