utilities.py

Download and management utilities for syncing time and auxiliary files

• Can list a directory on a ftp host

• Can download a file from a ftp or http host

• Can download a file from CDDIS via https when NASA Earthdata credentials are supplied

• Checks MD5 or sha1 hashes between local and remote files

Source code

General Methods

ATM1b_QFIT.utilities.get_data_path(relpath)

Get the absolute path within a package from a relative path

Parameters

relpath: str,

relative path

ATM1b_QFIT.utilities.file_opener(filename)

Platform independent file opener

Parameters

filename: str

path to file

ATM1b_QFIT.utilities.get_hash(local, algorithm='MD5')

Get the hash value from a local file or BytesIO object

Parameters

local: obj or str

BytesIO object or path to file

algorithm: str, default ‘MD5’

hashing algorithm for checksum validation

• 'MD5': Message Digest

• 'sha1': Secure Hash Algorithm

ATM1b_QFIT.utilities.url_split(s)

Recursively split a url path into a list

Parameters

s: str

url string

ATM1b_QFIT.utilities.get_unix_time(time_string, format='%Y-%m-%d %H:%M:%S')

Get the Unix timestamp value for a formatted date string

Parameters

time_string: str

formatted time string to parse

format: str, default ‘%Y-%m-%d %H:%M:%S’

format for input time string

ATM1b_QFIT.utilities.even(value)

Rounds a number to an even number less than or equal to original

Parameters

value: float

number to be rounded

ATM1b_QFIT.utilities.ceil(value)

Rounds a number upward to its nearest integer

Parameters

value: float

number to be rounded upward

ATM1b_QFIT.utilities.copy(source, destination, move=False, **kwargs)

Copy or move a file with all system information

Parameters

source: str

source file

destination: str

copied destination file

move: bool, default False

remove the source file

ATM1b_QFIT.utilities.check_ftp_connection(HOST, username=None, password=None)

Check internet connection with ftp host

Parameters

HOST: str

remote ftp host

username: str or NoneType

ftp username

password: str or NoneType

ftp password

ATM1b_QFIT.utilities.ftp_list(HOST, username=None, password=None, timeout=None, basename=False, pattern=None, sort=False)

List a directory on a ftp host

Parameters

HOST: str or list

remote ftp host path split as list

username: str or NoneType

ftp username

password: str or NoneType

ftp password

timeout: int or NoneType, default None

timeout in seconds for blocking operations

basename: bool, default False

return the file or directory basename instead of the full path

pattern: str or NoneType, default None

regular expression pattern for reducing list

sort: bool, default False

sort output list

Returns

output: list

items in a directory

mtimes: list

last modification times for items in the directory

ATM1b_QFIT.utilities.from_ftp(HOST, username=None, password=None, timeout=None, local=None, hash='', chunk=8192, verbose=False, fid=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, mode=509)

Download a file from a ftp host

Parameters

HOST: str or list

remote ftp host path

username: str or NoneType

ftp username

password: str or NoneType

ftp password

timeout: int or NoneType, default None

timeout in seconds for blocking operations

local: str or NoneType, default None

path to local file

hash: str, default ‘’

MD5 hash of local file

chunk: int, default 8192

chunk size for transfer encoding

verbose: bool, default False

print file transfer information

fid: obj, default sys.stdout

open file object to print if verbose

mode: oct, default 0o775

permissions mode of output local file

Returns

remote_buffer: obj

BytesIO representation of file

ATM1b_QFIT.utilities.http_list(HOST, timeout=None, context=<ssl.SSLContext object>, parser=<lxml.etree.HTMLParser object>, format='%Y-%m-%d %H:%M', pattern='', sort=False)

List a directory on an Apache http Server

Parameters

HOST: str or list

remote http host path

timeout: int or NoneType, default None

timeout in seconds for blocking operations

context: obj, default ssl.SSLContext()

SSL context for url opener object

parser: obj, default lxml.etree.HTMLParser()

HTML parser for lxml

format: str, default ‘%Y-%m-%d %H:%M’

format for input time string

pattern: str, default ‘’

regular expression pattern for reducing list

sort: bool, default False

sort output list

Returns

colnames: list

column names in a directory

collastmod: list

last modification times for items in the directory

ATM1b_QFIT.utilities.from_http(HOST, timeout=None, context=<ssl.SSLContext object>, local=None, hash='', chunk=16384, verbose=False, fid=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, mode=509)

Download a file from a http host

Parameters

HOST: str or list

remote http host path split as list

timeout: int or NoneType, default None

timeout in seconds for blocking operations

context: obj, default ssl.SSLContext()

SSL context for url opener object

timeout: int or NoneType, default None

timeout in seconds for blocking operations

local: str or NoneType, default None

path to local file

hash: str, default ‘’

MD5 hash of local file

chunk: int, default 16384

chunk size for transfer encoding

verbose: bool, default False

print file transfer information

fid: obj, default sys.stdout

open file object to print if verbose

mode: oct, default 0o775

permissions mode of output local file

Returns

remote_buffer: obj

BytesIO representation of file

ATM1b_QFIT.utilities.attempt_login(urs, context=<ssl.SSLContext object>, password_manager=True, get_ca_certs=False, redirect=False, authorization_header=False, **kwargs)

attempt to build a urllib opener for NASA Earthdata

Parameters

urs: str

Earthdata login URS 3 host

context: obj, default ssl.SSLContext()

SSL context for url opener object

password_manager: bool, default True

Create password manager context using default realm

get_ca_certs: bool, default False

Get list of loaded “certification authority” certificates

redirect: bool, default False

Create redirect handler object

authorization_header: bool, default False

Add base64 encoded authorization header to opener

username: str, default from environmental variable

NASA Earthdata username

password: str, default from environmental variable

NASA Earthdata password

retries: int, default 5

number of retry attempts

netrc: str, default ~/.netrc

path to .netrc file for authentication

Returns

opener: obj

OpenerDirector instance

ATM1b_QFIT.utilities.build_opener(username, password, context=<ssl.SSLContext object>, password_manager=True, get_ca_certs=True, redirect=True, authorization_header=False, urs='https://urs.earthdata.nasa.gov')

build urllib opener for NASA Earthdata with supplied credentials

Parameters

username: str or NoneType, default None

NASA Earthdata username

password: str or NoneType, default None

NASA Earthdata password

context: obj, default ssl.SSLContext()

SSL context for url opener object

password_manager: bool, default True

Create password manager context using default realm

get_ca_certs: bool, default True

Get list of loaded “certification authority” certificates

redirect: bool, default True

Create redirect handler object

authorization_header: bool, default False

Add base64 encoded authorization header to opener

urs: str, default ‘https://urs.earthdata.nasa.gov’

Earthdata login URS 3 host

ATM1b_QFIT.utilities.check_credentials()

Check that entered NASA Earthdata credentials are valid

ATM1b_QFIT.utilities.cddis_list(HOST, username=None, password=None, build=True, timeout=None, parser=<lxml.etree.HTMLParser object>, pattern='', sort=False)

List a directory on GSFC CDDIS archive server

Parameters

HOST: str or list

remote https host

username: str or NoneType, default None

NASA Earthdata username

password: str or NoneType, default None

NASA Earthdata password

build: bool, default True

Build opener and check Earthdata credentials

timeout: int or NoneType, default None

timeout in seconds for blocking operations

parser: obj, default lxml.etree.HTMLParser()

HTML parser for lxml

pattern: str, default ‘’

regular expression pattern for reducing list

sort: bool, default False

sort output list

Returns

colnames: list

column names in a directory

collastmod: list

last modification times for items in the directory

ATM1b_QFIT.utilities.from_cddis(HOST, username=None, password=None, build=True, timeout=None, local=None, hash='', chunk=16384, verbose=False, fid=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, mode=509)

Download a file from GSFC CDDIS archive server

Parameters

HOST: str or list

remote https host

username: str or NoneType, default None

NASA Earthdata username

password: str or NoneType, default None

NASA Earthdata password

build: bool, default True

Build opener and check Earthdata credentials

timeout: int or NoneType, default None

timeout in seconds for blocking operations

local: str or NoneType, default None

path to local file

hash: str, default ‘’

MD5 hash of local file

chunk: int, default 16384

chunk size for transfer encoding

verbose: bool, default False

print file transfer information

fid: obj, default sys.stdout

open file object to print if verbose

mode: oct, default 0o775

permissions mode of output local file

Returns

remote_buffer: obj

BytesIO representation of file