reprox package

Submodules

reprox.core module

Shared common methods for reprocessing, not useful in itself

reprox.core.check_user_is_admin(admin_group='xenon1t-admins')[source]

Check that the user is an xenon1t-admin

reprox.core.format_context_kwargs(minimum_run_number, maximum_run_number)[source]
reprox.core.get_context(package='cutax', context='xenonnt_v7', output_folder='/home/docs/checkouts/readthedocs.org/user_builds/reprox/checkouts/latest/test_folder/strax_data', config_kwargs: Union[None, dict] = None, minimum_run_number=17900, maximum_run_number=None)[source]
reprox.core.log_versions()[source]

Log versions (nested import makes the arg parsing quick)

reprox.core.parse_args(description='nton reprocessing on dali', include_find_args=False, include_processing_args=False, include_workflow_args=False)[source]

Parse arguments to return to the user

reprox.find_runs module

reprox.find_runs.determine_data_to_reprocess(st: strax.context.Context, targets: Union[str, tuple, list] = (), special_modes: Union[List[str], Tuple[str]] = ('LED', 'noise', 'pmtap', 'pmtgain', 'exttrig'), keep_detectors: Union[str, tuple, list] = ('tpc',), exclude_from_invalid_cmt: Optional[str] = 'global_v7', _max_workers: int = 50, ignore_runs=()) → pandas.core.frame.DataFrame[source]
Find data that we can process. This data needs to:

  1. (optional) be within the validity of a specified CMT version. Disable with exclude_from_invalid_cmt=False

  2. Don’t be some calibration mode (led/noise etc. data)

  3. Not be available already (why would you want to reprocess that?)

  4. Have the data which we need in order to compute this target.

Parameters

  • st – Context to run with

  • targets – Data types to produce

  • special_modes – list of modes to exclude to determine here (usually you can do this trivially, so no need to use this function)

  • exclude_from_invalid_cmt – A CMT version whereof we will check that the CMT version extends to those ranges where we would like to reprocess.

  • _max_workers – Max workers for finding the stored data

Returns

reprox.find_runs.find_data(targets: Union[str, list, tuple], exclude_from_invalid_cmt_version: Union[bool, str] = 'global_v7', context_kwargs: Union[None, dict] = None, keep_detectors: Union[str, tuple, list] = ['tpc'], ignore_runs=()) → None[source]

Determine which data to process, see determine_data_to_reprocess :param targets: List of targets to process :param exclude_from_invalid_cmt_version: A CMT version (optional) to

exclude runs that lie outside it’s validity from

Parameters

context_kwargs – Any context kwargs

Returns

reprox.process_job module

class reprox.process_job.ProcessingJob(run_id, targets, submit_kwargs)[source]

Bases: object

Class for starting jobs and keeping an eye on their status

get_run_job_state(read_last=10, ignore_patterns=['tensorflow', 'UserWarning', 'module compiled against']) → str[source]

Get the state of the current job

submit(**extra_kwargs)[source]

Submit the job to be run

submit_message = None

reprox.submit_jobs module

reprox.submit_jobs.can_submit_more_jobs(nmax='100')[source]
reprox.submit_jobs.cycle_queue(queues=('xenon1t', 'dali', 'broadwl'))[source]
reprox.submit_jobs.get_rundoc(run_id)[source]
reprox.submit_jobs.n_jobs_running()[source]
reprox.submit_jobs.submit_jobs(submit_kwargs: Union[None, dict] = None, targets: Union[str, List[str], Tuple[str]] = ('event_info', 'event_pattern_fit'), break_if_n_jobs_left_running: Union[None, int] = None, clear_logs: bool = False, sleep_s_when_queue_full: int = 60, submit_only: Union[None, int] = None, known_partitions: Union[tuple, list] = ['dali', 'xenon1t']) → List[reprox.process_job.ProcessingJob][source]

Submit jobs to the queue for the given options

Parameters

  • submit_kwargs – dict of options that are passed on to the job submission

  • targets – List of datatypes to produce

  • break_if_n_jobs_left_running – threshold when to stop reporting the status

  • clear_logs – If true, clear the logs from previous jobs

  • sleep_s_when_queue_full – sleep this many seconds if the

  • submit_only – maximum number of jobs to submit

  • known_partitions – list of partitions this user can submit to

Returns

a list of all the jobs that were submitted

reprox.validate_run module

Validate that the data can be loaded successfully and move the data to the production folder

class reprox.validate_run.RunValidation(path: str, context: Optional[strax.context.Context] = None, mode: Union[int, reprox.validate_run.ValidationLevel] = ValidationLevel.SHALLOW)[source]

Bases: object

Check that a directory (corresponding to a single datatype is

find_error() → str[source]

Run several checks on a path to see if the processing was done correctly

class reprox.validate_run.ValidationLevel(value)[source]

Bases: enum.IntEnum

An enumeration.

DEEP = 1
SHALLOW = 0
reprox.validate_run.change_ownership(path, group)[source]
reprox.validate_run.move_all(source_folder: str = '/home/docs/checkouts/readthedocs.org/user_builds/reprox/checkouts/latest/test_folder/strax_data', destination_folder: str = '/home/docs/checkouts/readthedocs.org/user_builds/reprox/checkouts/latest/test_folder', **move_kwargs)[source]

Move data from all folders in <source_folder> into the destination folder and change the ownership of the folder

Parameters

  • source_folder – The main folder where to look for folders to move

  • destination_folder – The folder where the <path> folder should be moved to

  • move_kwargs –

    Takes the following kwargs: :group: Name of the group that the permissions should be set to :validation_level: the level at which to validate the data:

    • ValidationLevel.SHALLOW: <0> for basic validation

    • ValidationLevel.DEEP: <1> where we actually try loading the

      data with (requires a context)

    context

    for when the validation_level is set to ValidationLevel.DEEP

Returns

None

reprox.validate_run.move_folder(path: str, destination_folder: str = '/home/docs/checkouts/readthedocs.org/user_builds/reprox/checkouts/latest/test_folder', group='xenon1t-admins', validation_level: int = ValidationLevel.SHALLOW, context: Optional[strax.context.Context] = None) → Optional[str][source]

Move data from path into the destination folder and change the ownership of the folder

Parameters

  • path – The folder to move

  • destination_folder – The folder where the <path> folder should be moved to

  • group – Name of the group that the permissions should be set to

  • validation_level –

    the level at which to validate the data: - ValidationLevel.SHALLOW: <0> for basic validation - ValidationLevel.DEEP: <1> where we actually try loading the

    data with (requires a context)

  • context – for when the validation_level is set to ValidationLevel.DEEP

Returns

error string if an error occurred

Raises

FileExistsError – when there is already a folder at the destination path

Module contents