Vh:dZddlmZddlmZmZddlmZddlmZddl Z ddl m Z ddl mZd e jd efd Zd e jd e fd ZeGddZeGddZeGddZeGddZdS)z' Basic configuration for Dora is here. ) Namespace) dataclassfield)fnmatch)PathN) DictConfig) OmegaConfdataargsc|jD]L}t|tsJt||r#t ||}|t |||MdS)z>Update the given dataclass from the argument parser args. N)__dict__ isinstancestrhasattrgetattrsetattr)r r keyvalues O/var/www/html/movieo_spanner_bot/venv/lib/python3.11/site-packages/dora/conf.pyupdate_from_argsrso}**#s##### 4   *D#&&E c5))) **cfgc8tj|d}t|tsJ|D]W\}}t|t sJt ||rt|||>td|j d|dS)z6Update the given dataclass from the hydra config. T)resolvezObject of type z does not have an attribute N) r to_containerrdictitemsrrrAttributeError __class__)r rdctrrs rupdate_from_hydrar!s  d 3 3 3C c4   iikkFF U#s##### 4   F D#u % % % % "E4>"E"E?B"E"EFF F FFrc~eZdZUdZdZeed<dZeed<dZ eed<dZ eed <d Z e j eed <d Zeed <d Ze j eed<eeZe jeed<dZeed<dZeed<dZeed<dZeed<d Ze j eed<d Ze j eed<d Ze j eed<d S) SlurmConfiga+ Configuration when scheduling a job. This differs slightly from Slurm/Submitit params because this will automatically scale some values based on the number of GPUs. Args: gpus (int): number of total GPUs to schedule. Number of nodes and tasks per nodes will be automatically inferred. mem_per_gpu (float): amount of memory in GB to schedule per gpus. time (int): maximum duration for the job in minutes. cpus_per_gpu (int): number of cpus per gpu, this will set the `cpus_per_task` automatically, based on the number of gpus and `one_task_per_node`, unless `cpus_per_task` is explicitely provided. cpus_per_task (int or None): number of cpus per task. partition (str): partition name comment (str): comment for the job. setup (List[str]): list of shell commands to execute before the actual command. Use it for `module load`. max_num_timeout (int): maximum number of requeue. one_task_per_node (bool): if True, schedules a single task per node, otherwise, will schedule one task per gpu (default is False). array_parallelism (int): when using job arrays, how many tasks can run in parallel. qos: (str or None): qos param for slurm. account: (str or None): account param for slurm. ..warning:: this assumes one task per GPU. Set `one_task_per_node` if you do not want that. Tasks without any gpus are not really supported at the moment. gpus( mem_per_gpuitime cpus_per_gpuN cpus_per_tasklearnlab partitioncommentdefault_factorysetupmax_num_timeout constraintFone_task_per_nodearray_parallelismexcludeqosaccount)__name__ __module__ __qualname____doc__r%int__annotations__r'floatr(r*r+tpOptionalr-rr.rlistr1Listr3r5r6boolr8r9r:r;rrr#r#.sD@D#MMMKD#L#&*M2;s#***Is $GR[ $$$%555E273<555OSJ#t### s   $GR[ $$$ CS    $GR[ $$$$$rr#cLeZdZUdZdZeed<dZeed<dZeed<dZ eed<dS) SubmitRulesa: Submit rules describe in which case Shepherd will schedule new jobs. Args: retry (bool): if true, all failed or canceled jobs will be rescheduled. update_pending (bool): if true, all pending jobs whose Slurm parameters have changed will be replaced. update (bool): if true, all pending or running jobs whose Slurm parameters have changed will be replaced. replace (bool): if true, all running jobs will be replaced by new jobs. replace_done (bool): if true, all done jobs will be relaunched. Fretryupdatereplace replace_doneN) r<r=r>r?rKrGrArLrMrNrHrrrJrJas^  E4FDGTL$rrJcheZdZUdZdZeed<dZeed<dZeed<dZ eed<dZ eed <d Z eed <d S) ShepConfigz Configuration for Shepherd. Mostly naming conventions for folders and files. There should be little reasons to change that. zjob.pkljob_fileby_idorphanssubmititsubmitit_folderlatestlatest_submititarraysN) r<r=r>r?rQrrArRrSrUrWrXrHrrrPrPvs~HcE3GS%OS%%%#OS###FCrrPcXeZdZUdZedZeed<eeZ e j e ed<dZ eed<dZe jeed <dZe je ed <d Ze ed <d Ze ed <eeZeed<dZe ed<dZeed<dZe ed<dZe ed<de defdZfdZxZS) DoraConfiga< Main Dora configuration. The main parameters to change are the following. Args: dir (Path or str): path where Dora will save all useful informations, logs. This is also where you should store your checkpoints (see `dora.xp.XP`). exclude (List[str]): list of patterns of argument names to ignore when computing the XP signature and doing deduplication. For instance 'num_workers', etc. git_save (bool): when True, experiments can only be scheduled from a clean repo. A shallow clone of the repo will be made and execution will happen from there. This does not impact `dora run` unless you pass the `--git_save` flag. shared (Path or None): if provided, the path to a central repository of XPs. For the moment, this only supports sharing hyper-params, logs etc. will stay in the per user folder. grid_package (str or None): if provided, package to look for grids. Default to the package with the `train.py` module followed by `.grids`. z ./outputsdirr/r9Fgit_saveNshared grid_packagez history.jsonhistoryxpsshepzrendezvous.txtrendezvous_fileuse_rendezvousgrids_gridscodes_codesarg_namereturnc@|jD]}t||rdSdS)zUReturn True if the given argument name should be excluded from the signature.TF)r9r)selfrhpatterns r is_excludedzDoraConfig.is_excludeds8|  Gx)) tt urc|dvr ddlm}|t||}t||dS)N)r[r]r$)to_absolute_path)r\rorsuper __setattr__)rknamerrors rrqzDoraConfig.__setattr__s` $ $ $ 2 2 2 2 2 2 --e4455 D%(((((r)r<r=r>r?rr[rArrEr9rCrFrr\rGr]rDr^r_r`rPrarbrcrergrmrq __classcell__)rs@rrZrZsb$[!!C!!!!E$777GRWS\777Hd $FBK $$$%)L"+c")))"GS!!!CuZ888D*888+OS+++ ND   FCFCCD)))))))))rrZ)r?argparser dataclassesrrrpathlibrtypingrComegaconf.dictconfigr omegaconfr Anyrr!r#rJrPrZrHrrr{s((((((((++++++*26***** FBF F F F F F /%/%/%/%/%/%/% /%d  (           1)1)1)1)1)1)1) 1)1)1)r