morphfits.input =============== .. py:module:: morphfits.input .. autoapi-nested-parse:: Obtain input from the DJA archive. .. rubric:: References 1. [DJA v7 Mosaic Release Details](https://dawn-cph.github.io/dja/imaging/v7/) 2. [DJA v7 Mosaic Release](https://s3.amazonaws.com/grizli-v2/JwstMosaics/v7/index.html) Attributes ---------- .. autoapisummary:: morphfits.input.BASE_URL morphfits.input.FILE_LIST_ENDPOINT morphfits.input.FILE_LIST_PATH morphfits.input.logger Classes ------- .. autoapisummary:: morphfits.input.DownloadProgressBar Functions --------- .. autoapisummary:: morphfits.input.get_file_list morphfits.input.get_src_dest morphfits.input.setup_input_dirs morphfits.input.get_files morphfits.input.unzip_files morphfits.input.main Module Contents --------------- .. py:class:: DownloadProgressBar(iterable=None, desc=None, total=None, leave=True, file=None, ncols=None, mininterval=0.1, maxinterval=10.0, miniters=None, ascii=None, disable=False, unit='it', unit_scale=False, dynamic_ncols=False, smoothing=0.3, bar_format=None, initial=0, position=None, postfix=None, unit_divisor=1000, write_bytes=False, lock_args=None, nrows=None, colour=None, delay=0.0, gui=False, **kwargs) Bases: :py:obj:`tqdm.tqdm` Decorate an iterable object, returning an iterator which acts exactly like the original iterable, but prints a dynamically updating progressbar every time a value is requested. :param iterable: Iterable to decorate with a progressbar. Leave blank to manually manage the updates. :type iterable: iterable, optional :param desc: Prefix for the progressbar. :type desc: str, optional :param total: The number of expected iterations. If unspecified, len(iterable) is used if possible. If float("inf") or as a last resort, only basic progress statistics are displayed (no ETA, no progressbar). If `gui` is True and this parameter needs subsequent updating, specify an initial arbitrary large positive number, e.g. 9e9. :type total: int or float, optional :param leave: If [default: True], keeps all traces of the progressbar upon termination of iteration. If `None`, will leave only if `position` is `0`. :type leave: bool, optional :param file: Specifies where to output the progress messages (default: sys.stderr). Uses `file.write(str)` and `file.flush()` methods. For encoding, see `write_bytes`. :type file: `io.TextIOWrapper` or `io.StringIO`, optional :param ncols: The width of the entire output message. If specified, dynamically resizes the progressbar to stay within this bound. If unspecified, attempts to use environment width. The fallback is a meter width of 10 and no limit for the counter and statistics. If 0, will not print any meter (only stats). :type ncols: int, optional :param mininterval: Minimum progress display update interval [default: 0.1] seconds. :type mininterval: float, optional :param maxinterval: Maximum progress display update interval [default: 10] seconds. Automatically adjusts `miniters` to correspond to `mininterval` after long display update lag. Only works if `dynamic_miniters` or monitor thread is enabled. :type maxinterval: float, optional :param miniters: Minimum progress display update interval, in iterations. If 0 and `dynamic_miniters`, will automatically adjust to equal `mininterval` (more CPU efficient, good for tight loops). If > 0, will skip display of specified number of iterations. Tweak this and `mininterval` to get very efficient loops. If your progress is erratic with both fast and slow iterations (network, skipping items, etc) you should set miniters=1. :type miniters: int or float, optional :param ascii: If unspecified or False, use unicode (smooth blocks) to fill the meter. The fallback is to use ASCII characters " 123456789#". :type ascii: bool or str, optional :param disable: Whether to disable the entire progressbar wrapper [default: False]. If set to None, disable on non-TTY. :type disable: bool, optional :param unit: String that will be used to define the unit of each iteration [default: it]. :type unit: str, optional :param unit_scale: If 1 or True, the number of iterations will be reduced/scaled automatically and a metric prefix following the International System of Units standard will be added (kilo, mega, etc.) [default: False]. If any other non-zero number, will scale `total` and `n`. :type unit_scale: bool or int or float, optional :param dynamic_ncols: If set, constantly alters `ncols` and `nrows` to the environment (allowing for window resizes) [default: False]. :type dynamic_ncols: bool, optional :param smoothing: Exponential moving average smoothing factor for speed estimates (ignored in GUI mode). Ranges from 0 (average speed) to 1 (current/instantaneous speed) [default: 0.3]. :type smoothing: float, optional :param bar_format: Specify a custom bar string formatting. May impact performance. [default: '{l_bar}{bar}{r_bar}'], where l_bar='{desc}: {percentage:3.0f}%|' and r_bar='| {n_fmt}/{total_fmt} [{elapsed}<{remaining}, ' '{rate_fmt}{postfix}]' Possible vars: l_bar, bar, r_bar, n, n_fmt, total, total_fmt, percentage, elapsed, elapsed_s, ncols, nrows, desc, unit, rate, rate_fmt, rate_noinv, rate_noinv_fmt, rate_inv, rate_inv_fmt, postfix, unit_divisor, remaining, remaining_s, eta. Note that a trailing ": " is automatically removed after {desc} if the latter is empty. :type bar_format: str, optional :param initial: The initial counter value. Useful when restarting a progress bar [default: 0]. If using float, consider specifying `{n:.3f}` or similar in `bar_format`, or specifying `unit_scale`. :type initial: int or float, optional :param position: Specify the line offset to print this bar (starting from 0) Automatic if unspecified. Useful to manage multiple bars at once (eg, from threads). :type position: int, optional :param postfix: Specify additional stats to display at the end of the bar. Calls `set_postfix(**postfix)` if possible (dict). :type postfix: dict or *, optional :param unit_divisor: [default: 1000], ignored unless `unit_scale` is True. :type unit_divisor: float, optional :param write_bytes: Whether to write bytes. If (default: False) will write unicode. :type write_bytes: bool, optional :param lock_args: Passed to `refresh` for intermediate output (initialisation, iterating, and updating). :type lock_args: tuple, optional :param nrows: The screen height. If specified, hides nested bars outside this bound. If unspecified, attempts to use environment height. The fallback is 20. :type nrows: int, optional :param colour: Bar colour (e.g. 'green', '#00ff00'). :type colour: str, optional :param delay: Don't display until [default: 0] seconds have elapsed. :type delay: float, optional :param gui: WARNING: internal parameter - do not use. Use tqdm.gui.tqdm(...) instead. If set, will attempt to use matplotlib animations for a graphical output [default: False]. :type gui: bool, optional :returns: **out** :rtype: decorated iterator. .. py:method:: update_to(b=1, bsize=1, tsize=None) .. py:data:: BASE_URL :value: 'https://s3.amazonaws.com/grizli-v2/JwstMosaics/v7' URL to DJA v7 Mosaics archive. .. py:data:: FILE_LIST_ENDPOINT :value: 'index.csv' Endpoint for filelist CSV. .. py:data:: FILE_LIST_PATH Path to downloaded filelist CSV. .. py:data:: logger Logger object for this module. .. py:function:: get_file_list() -> dict[str, dict] Get the DJA file list as a dictionary indexed by filename. :returns: DJA file list as a dictionary indexed by filename. :rtype: dict[str, dict] .. py:function:: get_src_dest(morphfits_config: morphfits.config.MorphFITSConfig, file_list: dict[str, dict]) -> list[tuple[str, str]] Get the required filenames for the given FICLs. :param morphfits_config: Configuration object for this program run. :type morphfits_config: config.MorphFITSConfig :param file_list: DJA file list as a dictionary indexed by filename. :type file_list: dict[str, dict] :returns: List of source URLs and their corresponding destination paths. :rtype: list[tuple[str, str]] .. py:function:: setup_input_dirs(morphfits_config: morphfits.config.MorphFITSConfig) Create input directories where missing. :param morphfits_config: Configuration object for this program run. :type morphfits_config: config.MorphFITSConfig .. py:function:: get_files(src_dest: list[tuple[str, str]], file_list: dict[str, dict]) Download files from DJA. :param src_dest: List of source URLs to download from, and their corresponding destination paths. :type src_dest: list[tuple[str, str]] :param file_list: DJA file list as dictionary indexed by filename. :type file_list: dict[str, dict] .. py:function:: unzip_files(src_dest: list[tuple[str, str]], file_list: dict[str, dict]) Unzip downloaded files. :param src_dest: List of source URLs to download from, and their corresponding destination paths. :type src_dest: list[tuple[str, Path]] :param file_list: DJA file list as dictionary indexed by filename. :type file_list: dict[str, dict] .. py:function:: main(morphfits_config: morphfits.config.MorphFITSConfig) Identify, download, and unzip files from the DJA archive for given FICLs. :param morphfits_config: Configuration object for this program run. :type morphfits_config: config.MorphFITSConfig