Vhc9UdZddlmZddlmZmZddlZddlmZddl Z ddl m Z ddl Z ddlZddlZddlZddlmZmZmZdd lmZmZmZdd lmZdd lmZmZmZdd lm Z m!Z!dd l"m#Z#m$Z$m%Z%ddl&Z'eedZe j(e)gdfe*d<dZ+eGddZ,dZ-de j.defdZ/ede,fdedede)dede j0ede,de j1e fdZ2dZ3de j1e)dede j1e de j1e fd Z4de j.deded!e j1e d"e j(de5f d#Z6dS)$z This defines the `dora grid` action, and provides a `run_grid` API that can be used from a notebook or any other script. When using the API, you can provide the equivalent of the command line flags with the `RunGridArgs` dataclass. )ProcessPoolExecutor) dataclassfieldN)partial)Path) SlurmConfig SubmitRulesupdate_from_args)ExplorerLauncherHerd) DecoratedMain)colorize simple_logfatal)SheepShepherd)import_or_fatalreliable_rmtreetry_loadzGrid:logcdSN)argskwargss O/var/www/html/movieo_spanner_bot/venv/lib/python3.11/site-packages/dora/grid.pyno_printr%sDc~eZdZUdZeeZeje e d<dZ e e d<dZ ee d<dZejee d <d Ze e d <d Ze e d <d Ze e d <d Ze e d<d Ze e d<d Zeje e d<d Ze e d<dZejee d<dZejee d<dZejee d<d Ze e d<dS) RunGridArgsay Arguments to tune the behavior of the `run_grid` function. Args: patterns (list[str]): List of patterns used to filter by name the XPs. monitor (bool): if True, will monitor the advances of the XPs every `interval` minutes, stopping only when all runs completed or failed. interval (float): interval in minutes to wait between updates. trim (int or None): if provided, will trim all XP logs to the epoch of the XP with the provided index. Useful to compare XP started at different times. trim_last (bool): if True, will trim all XP to the least advanced XP. silent (bool): if True, do not print anything (e.g. API usage). dry_run (bool): if True, Dora will simulate the run of the grid, without scheduling or canceling any XP. cancel (bool): if True, will cancel all XPs in the grid. If `patterns` is provided, only XP matching the patterns will be canceled. clear (bool): This will cancel any previous job, clear the XP folder, and reschedule a new experiment. )default_factorypatternsTmonitorintervalNtrimF trim_lastsilentdry_runcancelclearinitjupyterfolderrtail_from_commandline)__name__ __module__ __qualname____doc__rlistr$tpListstr__annotations__r%boolr'floatr(Optionalintr)r*r+r,r-r.r/r0rr1r2rr rr"r")sK.#U4888Hbgcl888GTHe!D"+c !!!ItFDGTFDE4#D"+d ###GT $FBK ### CS    !D"+c !!!#t#####r r"cp|jj}| |jdz}t|}|j:|jdddz}t |jj|z }|j| sCg}t |jj}tj |D]\}} } | D]} t || z |} | j dr_| j dsE| j| jz } |t%| dd|j*| st'd|d|dt'd d |t+jd |dz|jz} t| } |j}n"#t0$rt3| d YnwxYwt5|t6st3|d |S)Nz.grids./z.py_z No grid file z in package z. Maybe you made a typo?zPotential grids are: z, rz( has no exploration function `explorer`.z' must be an instance of `dora.Explorer`)dora grid_packagepackagergridreplacer__file__parentexistsoswalk relative_tonameendswith startswithstemappendr:rjoinsysexitexplorerAttributeErrorr isinstancer )rmainrEgrids grid_filename grid_file candidatespkg_rootrootfoldersfilesfilefullpath grid_namerGrWs r _get_explorerf[so9)L|h. L ) )E y ))#s33e; ((/-?  y 0 0 2 2 ''.$&GH$5$5 G G D'5 G G JJ-::8DD=))%00G9Q9QRU9V9VG'>H%%c(mm&;&;C&E&EFFF  G 9 )9)9);); ) ))<))) * * * ;DIIj$9$9 ; ;<<<  s"TY.I 9 % %DF= FFF DDDEEEEEF h ) )D BBBCCC Os%G--H  H rrZc&t||}|}t||t}t||t }d|_t||t |||j|||dS)NT)rfget_slurm_configr r r"r2run_gridrG)rrZrWslurmrules grid_argss r grid_actionrmsD$''H  ! ! # #EUD!!! MMEUD!!! I"&IY%%% T8TYui@@@@@r rWrerkrjreturnc "t|tsJ||}|jj|jjz |z }|ddt}t|t}|j rZtd5} t|||| } || | dddn #1swxYwYnt|||} || |t|j} t%|j|| } |jr|jrt-dtdt/| d |d |jr6t3d } | d krt-d td| D]#} | j|| j$|td| D]@} | jj rtC| jjd| _Ag}g}|"D]}|j#|jvr|$| |%|j#}|J|$|Z#tL$r}td|j#d|||jj'j(z }tS|}|=|d}td|j*d|j#d||Yd}~d}~wwxYw||j+sd| D"|j,D]}"fd|D}|s|d}|j-|jj.}t/|dkr|/|||]|0|5|D]} |/| || dddn #1swxYwY|D]}|1si|jJ||j|2|j}td|jj*d|jj.d||j+r| D]} | 1si| jJ|2| j}td| jj*d| jj.d||| j|js| D]} || jj.z }| s|3rK|3r4|4| jj4ksJ|5| jj||D]}|6|j7r| D]} |8| j|j+r| S| std| Sd|j|j |j9fD}|rh|jstudt/|dksJ|d} | |} n'#tv$rt-d|jYnwxYw|2| j}|jty| jjn|j9Q| j st-d | j d!t{j>d"d"d#d$d%| j gno| j st-d&|d tj@t| j tjCn#t$rYnwxYw| S|jErt}ntx}|d'| |jGr#|jEsdd(lHmI}|Jd)|t|||| |rn|jKsnt|jMd*z}|t|D]F} d+|| z d,d-}!|| z d.krt|!d/}!||!d01tjQdG|d2| S)3a Run a grid search, this is the API underlying the `dora grid` command, so that it can be used from a notebook. You can also provide patterns to filter out XPs to be displayed. Args: main (DecoratedMain): main training function, decorated with Dora. explorer (Explorer): explorer instance that will define the XPs to launch. grid_name (str): this must be a unique name for the grid. rules (SubmitRules): see `dora.conf.SubmitRules`, those defines the rules for rescheduling failed XP etc. slurm (SlurmConfig or None): if provided, this will override the default Slurm config defined my the `main` argument. Returns: A list of `dora.shep.Sheep`. NT)exist_okparents)r)poolz'--dry_run is incompatible with --clear.zYou are about to restart z experiments from the grid z' from scratch. This cannot be reverted.zConfirm [yN]: yzAbort...zCanceling all current jobs...zDeleting XP folders...z$Error when trying to load old sheep z: rzCanceling job z from unloadable sheep rAc(i|]}|jj|Sr)xpsig.0sheeps r zrun_grid..s===UUX\5===r c(g|]}|v|Srr)ryrw sheep_maps r zrun_grid..s'TTTs3)CSCSIcNCSCSCSr rz for no longer required sheep rBz for sheep zNo sheep to handle.cg|]}||Srr)ryactions rr~zrun_grid..s]]]&&J\vJ\J\J\r zIThe folder, log, and tail flags are only supported from the command line.zInvalid index zLog z does not existr1z-n200z-fz"Log file does not exist for sheep zMonitoring Grid )display)wait<zNext update in z.0fz seconds 31 )endz< )RrYr rhrDdir_gridsmkdirrrr_slowrr completeupdater7sheepsvalues_filter_grid_sheepsr$r-r+rlenr2inputlowerjob cancel_lazycommitrvr0rKriterdirrOrSget_sheep_from_sig Exceptionshepjob_filerjob_idr, job_arrays slurm_configsrwmaybe_submit_lazy job_arrayis_doneget_name is_symlinkresolve symlink_tounlinkr.init_xpr1 RuntimeError IndexErrorprintrLexecvpshutil copyfileobjopenrUstdoutBrokenPipeErrorr*rr/IPythonr clear_outputr%r?r'rangertimesleep)#rZrWrerkrjr grid_folderherdshepherdrslauncherrreplrz to_unlink old_sheepschild old_sheeperrorrjobsrr array_sheepsfirstrOlinkactionsindex maybe_printrrelaoutr}s# @rriris * h ) ))) ) }%%'')-$)"22Y>KtT222 66D#&&&H z  # # t%DAAAH HX    MMOOO                HeT22 OO $+$$&& ' 'F f = =F z < = ; < < <  5F  5 5PY 5 5 5 6 6 6  ! ")**Dzz||s""j!!! +,,, 0 0Ey$$$UY/// $%%%  Ex%%'' 1000EIIIJ$$&&-- :T[ ( (   U # # # -$77 CC !,,,!!),,,, . . .P5:PPPPQQQ 49>#::))#q'CYYYEJYYYZZZ((--- . )" OO ; H==f===  H HITTTTiTTTL  OE&ux|4E<  A%%**5%????''..HH!-HH 225%GGGGHHHHHHHHHHHHHHHH 44   "" 4=,,,   / / /==..D 3!533")33,033 4 4 4 {0 0 0E==?? 0y,,,}}UX..WUY%5WW%(,WWQUWWXXX$$UY/// <  1 1E%(,.D{{}} 1 1 1 1((XT\\^^ux?V?V?X?X-X-X-XX-X0000  E LLNNNN y# # #E LL " " " " {  !""" ]]T[$(DI$F]]]G% R QRR R7||q      25MEE 2 2 2 04;00 1 1 1 1 1 2}}UX&& ; " %(/ " " " " Y "9##%% 97UY777888 IfvtUD%)D E E E E9##%% DB4BBBCCC "4 ??CJ????"      {  K.9../// < ,  , ' ' ' ' ' '  d + + + 4x = =  |  DMB&'' <<  CDECKDDDDCs{RsD)) K & & & & JqMMMM H%& Msa3CC"C*K M.(A.qs+++Cs3x+++r )r7rSr?remove ValueErrorrrvr)r$rZrindexesprzrOrs @rrrasG (^^  NN3q66 " " " OOA        H  C}}UX&& tX & &  JJu   ,++++7+++ Js"A AArrc$d|D\}}fd|D}d|jt||jn |jrt d|Dfd|D}g}d} t t |||D]\} \} } } | }| sd} |d}n |dd }| | | j r | j j nd | j j |d }i}||d < | | | }n%#t$r|| }YnwxYw|||||r |d |t%jdt%jd |g|z}|t%j|||| S)zSingle iteration of monitoring of the jobs in a Grid. Returns `True` if all jobs are done or failed, and `False` otherwise. cg|] }|j Sr)rvrxs rr~zmonitor..zs&B&B&BEux&B&B&Br cDg|]}|jSr)get_xp_historyrv)ryrzrZs rr~zmonitor..{s)AAA5$$UX..AAAr Nc34K|]}t|VdSr)r)rymetricss r zmonitor..s(99G3w<<999999r c$g|] }|d Srr)ryrr(s rr~zmonitor..s!===WUdU^===r TFzN/A)rOrsidrwstateMetaz Base name: )shortengroups)colors) get_namesr(rr)min enumerateziprrrrrvrw process_sheepNotImplementedErrorprocess_historyrrStttablegroup get_grid_metaget_grid_metrics treetable get_colors)rrZrWrrnames base_name historieslinesfinishedrrzhistoryrOrmetalineotherrr(s ` @rr%r%usp ~~&B&BT&B&B&BCCE9AAAADAAAI D y9TY'(( :99y99999 ====9=== EH)23tY3N3N)O)O%%w }} H =EE"1"IE',y859##b8<   V  6**5'::EE" 6 6 6,,W55EEE 6 E T. M9--- H HVX3355 6 6  % % ' '(   E K UE(2E2E2G2GHHHIII OsD//EE)7r6concurrent.futuresr dataclassesrrr functoolsrrLpathlibrtypingr8rrUrconfr r r explorer r rrZrrrrrrrrutilsrrrrrCallabler:r;rr"rfAnyrmr>r9rirrr<r%rr rr s322222((((((((  <<<<<<<<<<----------,,,,,,,,,,!!!!!!!!========== ' G < <R[#<<<    .$.$.$.$.$.$.$ .$b###L Abf AM A A A A#.+--SW!,@@=@H@@@8: K8P@@3575>@@@@F""'#,m "3575>(4"&4 44QV44)-444444r