Vh&vdZddlZddlmZddlZddlZddlmZddlZ ddl Z ddl m Z m Z ddlmZddlmZmZe jZd Zd ZGd d eZGd deZdgdddddejde jeefde jede je de je jeefdef dZdS)a* DecoratedMain is the main abstraction used inside Dora. This defines the core interface that needs to be fullfilled for Dora to be able to handle an experiment. This is also where the end user will specify how to launch job, default Slurm configuration, storage location, naming conventions etc. N) OrderedDict)Path) DoraConfig SlurmConfig) NamesMixin)XP_contextc||dd\}}tj|}t||S)N.r)rsplit importlib import_modulegetattr) full_name module_namefun_namemodules O/var/www/html/movieo_spanner_bot/venv/lib/python3.11/site-packages/dora/main.py _load_mainr s<%,,S!44K  $[ 1 1F 68 $ $$cV|dkr"tj|j}|dS|jS|S)N__main__)sysmodules__spec__name)rspecs rget_module_namer&s3 {6"+ <49  rcPeZdZdZdZdedefdZdZde j e de fd Z d Zd Zde je defd Zde j e fd ZdefdZde de je fdZde defdZdZde jde j e fdZdede j efdZdefdZdS) DecoratedMaina Main function that will actually run the XP, wrapped with tons of meta-informations. Args: main (callable, taking no argument): main function for the experiment as provided by the user. dora (DoraConfig): configuration for Dora. Fmaindorac||_||_t|j}|d|_d|_n6d|vr$|dd\|_|_nd|_||_|j|_|jdz|jz|_ dS)Nunknowntrainr r) r"r#r __module__package main_moduler r__name__ _full_name)selfr"r#rs r__init__zDecoratedMain.__init__<s  %do66  %DL&D  k!!1<1C1CC1K1K. d..( #. L /C/$-?rcT|}||s|S||}||t j|5|cdddS#1swxYwYdSN) _get_argv _is_active_mainget_xpinit_xpr enter_xp)r,argvxps r__call__zDecoratedMain.__call__Os~~t$$ ::<<  [[   R  r " "  ::<<                  s<BB!$B!r6returncdSNTr,r6s rr1zDecoratedMain._is_activeXstrc*|Sr/r"r,s rr2zDecoratedMain._main[syy{{rc t|jffSr/)rr+r@s r __reduce__zDecoratedMain.__reduce__^sDO---rct)z0Return an XP given a list of arguments. NotImplementedErrorr=s rr3zDecoratedMain.get_xpas"###rc*tjddS)Nr)rr6r@s rr0zDecoratedMain._get_argvfsx|rr7c|jddtj|jt |jd|j|jjjddd|jjddd |jjj d|jj dn#t$rYnwxYwtj|jt |jd |j dn#t$rYnwxYw|S)ag Initialize the XP folder. Once this is done, the XP can be retrieved from its signature. This is done automatically before running, or when using the `--init` flag of the `dora grid` command. This will also initialize the shared XP folder so that the XP hyper-params can be easily shared using its signature. T)exist_okparentswNi)rHrImode) foldermkdirjsondumpr6open _argv_cache_shared_argv_cacheparentchmodPermissionErrorr,r7s rr4zDecoratedMain.init_xplsP t444 "'444555 ,  ! ( / 5 5tTX] 5 ^ ^ ^  ! ( . .dQV . W W W %,399%@@@%,2259999"     IbgtB$93?? @ @ @ %++E2222"     s%AC C,+C,D88 EEsigc`t||jdg}|jr&t jt |jS|j?|jr&t jt |jStd|)zReturns the argv used to obtain a given signature. This can only work if an XP was previously ran with this signature. N)rWr#cfgr6z)Could not find experiment with signature ) r r#rQexistsrNloadrPrR RuntimeError)r,rWr7s rget_argv_from_sigzDecoratedMain.get_argv_from_sigsCdiT ; ; ; > " " R9T".1122 2  " .23H3O3O3Q3Q .9T""78899 9P3PPQQ QrcR|||S)zcReturns the XP from the signature. Can only work if such an XP has previously ran. )r3r])r,rWs rget_xp_from_sigzDecoratedMain.get_xp_from_sigs${{411#66777rcd|jdS)NzDecoratedMain()r?r@s r__repr__zDecoratedMain.__repr__s, ,,,,rargct)a Convert a Python value to argv. arg can be either: - a list, then each entry will be converted and all argv are concatenated. - a str, then it is directly an argv entry. - a dict, then each key, value pair is mapped to an argv entry. rD)r,rcs r value_to_argvzDecoratedMain.value_to_argvs "###rcL|j|jjS)aReturn the metrics for a given XP. By default this will look into the `history.json` file, that can be populated with the Link class. Can be overriden, but metrics should still be returned as a list of dicts, possibly with nested dicts. )linkr[historyrVs rget_xp_historyzDecoratedMain.get_xp_historys  wrctSzEReturn default Slurm config for the launch and grid actions. )rr@s rget_slurm_configzDecoratedMain.get_slurm_configs}}rN)r*r' __qualname____doc___slowMainFunrr-r8tpListstrboolr1r2rBSequencer r3r0r4r]r_rbAnyredictrirrlr<rrr!r!1s E@W@J@@@@&   rws|...$2;s+$$$$$ 273< "6 RS RR[-= R R R R83828888 ---$$BGCL$$$$ +rr!c eZdZdZ ddededejdej e de f fd Z d ej ed efd Zd ejd ejefdZded efdZd e ffd ZxZS) ArgparseMainaImplementation of `DecoratedMain` for XP that uses argparse. Args: main : main function to wrap. dora : Dora config, containing the exclude and dir fields. parser : parser to use, and to derive default values from. slurm : default slurm config for scheduling jobs. use_underscore : if False, scheduling a job as `launcher(batch_size=32)` will translate to the command-line `--batch-size=32`, otherwise, it will stay as `--batch_size=32`. NTr"r#parserslurmuse_underscorecvt||||_||_||_dSr/)superr-rzr|r{)r,r"r#rzr{r| __class__s rr-zArgparseMain.__init__s7 t$$$ , rr6r9c4t|}|j|}g}|jD]:\}}|j||kr|||f;t|j|||}|S)N)r#rYr6delta) listrz parse_args__dict__items get_defaultappendr r#)r,r6argsrkeyvaluer7s rr3zArgparseMain.get_xpsDzz{%%d++---// + +JC{&&s++u44 c5\*** TYDt5 A A A rrccg}t|tr||nt|trp|D]Z\}}|js|dd}|dur|d|?|d|d|[nLt|ttfr|D]}|| |z }ntd||S)N_-Tz--=z5Can only process dict, tuple, lists and str, but got ) isinstancersrrwrr|replacertuplere ValueError)r,rcr6rrparts rrezArgparseMain.value_to_argvs0 c3   \ KK     T " " \!iikk 4 4 U*0++c3//CD==KK S ++++KK 2S 2 25 2 23333  4dE] + + \ 1 1**4000 1ZUXZZ[[ [ rr7cXt}|jJ|jD] \}}|||< |Sr/)rr)r,r7partsrrs rget_name_partszArgparseMain.get_name_partss> x###8  KD%E$KK rc`|j|jStSrk)r{r~rl)r,rs rrlzArgparseMain.get_slurm_configs+ : !: ww'')))rr;)r*r'rmrnrprargparseArgumentParserrqOptionalrrtr-rursr r3rvrrrerrrl __classcell__)rs@rryrys  QUWJ@W K0IM2;s+BGCL& *+**********rryz ./outputsT)direxcluder{sharedr|rzrrr{rr|c :dtffd }|S)aNicer version of `ArgparseMain` that acts like a decorator, and directly exposes the most useful configs to override. Args: parser: parser to use, and to derive default values from. exclude: list of patterns of arguments to exclude from the computation of the XP signature. dir: path to store logs, checkpoints, etc. to. slurm: default slurm config for scheduling jobs. shared: path to the shared XP repository. use_underscore: if False, scheduling a job as `launcher(batch_size=32)` will translate to the command-line `--batch-size=32`, otherwise, it will stay as `--batch_size=32`. **kwargs: extra args are passed to `DoraConfig`. r"ctdtdnttd}t||S)N)rrr)r|r{r<)rrrry) r"r#rrkwargsrzrr{r|s r _decoratorz!argparse_main.._decorator seS !>44tF||MM  D$~UZ[[[[r)rp)rzrrr{rr|rrs``````` r argparse_mainrsa,\\\\\\\\\\\\\ r) rnr collectionsrrrNpathlibrtypingrqrconfrrnamesrr7r r Callablerprrr!ryrUnionrsrurrtrr<rrrs######  )))))))) +%%% @@@@@J@@@F<*<*<*<*<*=<*<*<*@.9.048=A)- (1xT *;s+[1+bhsDy&9:  #' r