#Sh$ddlZddlZddlmZddlmZmZmZmZm Z ddl m Z ddl m Z mZddlmZdejeefd ejeefd dfd ZGd d e Zdee de eefded e fdZdS)N)Path)AnyListOptionalTypeUnion)plugins)ExecutorJob)DelayedSubmissionkwargsdeprecated_argsreturnct|D]M}||}|s||||<tjd|d|dNdS)Nz Setting 'z' is deprecated. Use 'z ' instead.)listgetpopwarningswarn)rrargnew_args X/var/www/html/movieo_spanner_bot/venv/lib/python3.11/site-packages/submitit/auto/auto.py_convert_deprecated_argsrs}F||RR!%%c**   **S//w P#PPWPPPQQQQ RRc neZdZdZdddZddeeefdeede d dffd Z e d efd Z d e eeefd dfd Zed ejeeffdZed ejefdZde d dfdZdej ed ej eej fdZxZS) AutoExecutoraMAutomatic job executor This class is used to hold the parameters to run a job on the cluster corresponding to the environment. It can also be used to run job locally or in debug mode. In practice, it will create a bash file in the specified directory for each job, and pickle the task function and parameters. At completion, the job will also pickle the output. Logs are also dumped in the same directory. Executor specific parameters must be specified by prefixing them with the name of the executor they refer to. eg: - 'chronos_conda_file' (internal) - 'slurm_max_num_timeout' See each executor documentation for the list of available parameters. Parameters ---------- folder: Path/str folder for storing job submission/output and logs. warn_ignored: bool prints a warning each time a parameter is provided but ignored because it is only useful for the other cluster. cluster: str Forces AutoExecutor to use the given environment. Use "local" to run jobs locally, "debug" to run jobs in process. kwargs: other arguments must be prefixed by the name of the executor they refer to. {exname}_{argname}: see {argname} documentation in {Exname}Executor documentation. Note ---- - be aware that the log/output folder will be full of logs and pickled objects very fast, it may need cleaning. - use update_parameters to specify custom parameters (gpus_per_node etc...). If you input erroneous parameters, an error will print all parameters available for you. slurm_max_num_timeoutchronos_conda_file)max_num_timeout conda_fileNfolderclusterrrc  |p||_tj}|j|vrt d|jt ||jd}dd|d}|D]L}d|vsJ|d|d||dd }||vsJ|d|d |d |Mt||j|fi||_ | fd | D|_ t|j j|j jdS) Nz-AutoExecutor doesn't know any executor named zHExtra arguments must be prefixed by executor named, received unknown argKnown executors: , ._z 'z'. rz', and 'z' executor is also unknown. c\i|](\}}|D]}|v||d|)Sr()_valid_parameters).0ex_nameexrvalids r z)AutoExecutor.__init__..Sse! ! ! ++--! ! % G##c## r)whichr#r get_executors ValueErrorr_ctor_deprecated_argsjoinsplit flexible_init _executorr+items_deprecated_argssuper__init__r" parameters) selfr"r#r executorserr err_ex_listnameprefixr/ __class__s @rr<zAutoExecutor.__init__@s.$**,, )++ *>AAA  Y YD$;;;3 @ @$ @ @; @ @;;;ZZ__Q'F)###XXXXvXX;XX$###&y'>QQ&QQ&&((! ! ! ! (00! ! !  .0IJJJJJrctjtfd}|dkr%t dd|S)z%Returns what is the detected cluster.c8|SN)affinity)r.r?s rz$AutoExecutor.which.._s " 0F0F0H0Hr)keyrz*Did not found an available executor among r')r r2maxrH RuntimeErrorkeys)best_exr?s @rr1zAutoExecutor.which[st)++ i%H%H%H%HIII W  & & ( (A - -_INNL\L\___`` `rfolderscxt|jdd}| ||dStjddS)aArchive a list of folders to be untarred in the job working directory. This is only implemented for internal cluster, for running job on non-installed packages. This is not useful on slurm since the working directory of jobs is identical to your work station working directory. folders: list of paths The list of folders to archive and untar in the job working directory register_dev_foldersNzZIgnoring dev folder registration as it is only supported (and needed) for internal cluster)getattrr8rr)r>rOregisters rrQz!AutoExecutor.register_dev_foldersfsW4>+A4HH   HW      Ml     rc httttttttdS)N)rB timeout_minmem_gbnodes cpus_per_task gpus_per_nodetasks_per_nodestderr_to_stdout)strintfloatboolclss r_typed_parameterszAutoExecutor._typed_parametersxs+  ! $   rcht|SrG)setrbrMr`s rr+zAutoExecutor._valid_parameterss(3((**//11222rc  dDt D]R\}}|tkrttf}|vr0t ||sJd|d|d|dSt j fdD}g}tj }|D]}t|dkr | d|d d 5|\}} ||vr | d |d |d | d^|| } | | vr@| vr<| d| d|d |d | d dzd | z|re|dd |d dgt!d |jdi fd|Dj fdD} |D]$\}} | vr|jkr|d | | | <%jjdi| dS)aUpdates submission parameters to srun/crun. Parameters ---------- AutoExecutors provides shared parameters that are translated for each specific cluster. Those are: timeout_min (int), mem_gb (int), gpus_per_node (int), cpus_per_task (int), nodes (int), tasks_per_node (int) and name (str). Cluster specific parameters can be specified by prefixing them with the cluster name. Notes ----- - Cluster specific parameters win over shared parameters. eg: if both `slurm_time` and `timeout_min` are provided, then: - `slurm_time` is used on the slurm cluster - `timeout_min` is used on other clusters ci|] \}}||| SrG)r,kvs rr0z.sCCC41aQ]!Q]]]rz Parameter "z" expected type z (but value: "z")cDg|]}|v|ddS)r()r6)r,xgenericss r z.s/IIIq7H7HAGGCOO7H7H7Hrr z Parameter 'rz*' need to be prefixed by an executor name.zUnknown executor 'z' in parameter 'r(z'.zUnknown argument 'z' for executor 'z Valid arguments: r&r%z=As a reminder, shared/generic (non-prefixed) parameters are: r' cji|]/\}}|jk|v||d|0Sr*)r#r)r,r.rrmrr>s rr0z.sRB%%#//VZZ2  ..*9//rc*i|]}|v||Srgrg)r,rhrmrs rr0z.s+8g8g8g!YZ^fYfYfF1IYfYfYfrNrg)r9rrbr^r] isinstancerr:r r2lenappendr+r5extendrM NameErrorupdater8_convert_parametersr#_internal_update_parameters) r>rrB expected_typespecificinvalidr?ex_argr.rr/r=rms `` @rryz(AutoExecutor._internal_update_parameterssf$DC6<<>>CCC1133#+>>#3#3   D-%%!$e v~~!&, >>i$ii iiY_`dYeiii> !)>???IIIIVIII)++   F6{{abVAYbbbcccGB""TBTTTTSTTTUUUbM3355E%Cx$7$7^^^b^^RT^^WZ^^^*+ii&&'   0 NNE )..2B2B(C(CEE_T\___    DIIg..// /    '   ^778g8g8g8g8gv8g8g8ghh  8 8GB(""rT\'9'9"(B"7 322@@Z@@@@@rdelayed_submissionsc6|j|SrG)r8_internal_process_submissions)r>r~s rrz*AutoExecutor._internal_process_submissionss~;;.sCeeeFAsPQP\P\]cPdPde!CKKMM*Ceeer)rBr9)rar"rrCs @rr7r7sJ XXZZ# F 3v f feeeefllnneee f ffr)typingrrpathlibrrrrrrcorer core.corer r core.utilsr rr\Mappingrrr7rgrrrsk33333333333333%%%%%%%%******RRWS#X%6RTWY\T\I]RbfRRRR@Q@Q@Q@Q@Q8@Q@Q@QFgtH~guS$Y/?g3gS[ggggggr