VhddlmZddlZddlZddlZddlZddlZddlZddl m Z ej e Z eddZdZdZdd Zd Zd Zd ZdS)) namedtupleN)get_xp DistribSpecz5rank world_size local_rank node_rank num_nodes sourcect}|jdkrdSdtjvrdtjvs Jdtjd}t jd|gzddj}|d }|tjd<d tjvrt}|j } tj }||j z }n#t$rYnwxYwtj|}|d d }t%|tjd <d tjvret%|jtjd <t%|jtjd<t%|jtjd<dSdS)aCalling this function will set the distributed environement including the master addr, master port etc. You shouldn't call this if you call `dora.distrib.init`, but it can be useful if you need to let some other framework handle the distributed initialization. rN MASTER_ADDRSLURM_JOB_NODELISTzcase not handledzscontrol show hostnamesT)capture_outputcheckr MASTER_PORTi Ni` WORLD_SIZERANK LOCAL_RANK)get_distrib_spec world_sizeosenvironsprunsplitstdoutdecodersigsubmititJobEnvironmentjob_id RuntimeErrorrandomRandomrandintstrrank local_rank) specnodelistnodes master_nodexpseedenvrng master_ports R/var/www/html/movieo_spanner_bot/venv/lib/python3.11/site-packages/dora/distrib.pyset_distrib_envr.s   D !BJ&&#rz1113E111:2306688H:E&*$8888>vvxx Ah $/ =!BJ&& XXv )++C CJ DD    D mD!!kk%// $' $4$4 =!2:%%#&t#7#7 <  ^^ 6#&t#7#7 <   &%s%D DDcdtjvruttjd}ttjd}dtjvr ttjd}n|}d}d}d}nV tj}|j}|j}|j}|j}|j }d}n#t$rd}d}d}d}d}d}YnwxYwt||||||S) zReturn information on the distributed setup, i.e. world size, rank etc. This can be used even before distributed training is initialized, which is useful for PytorchLightning for instance. r rrrrr*rempty) rrintrr global_rank num_tasksr#node num_nodesrr)r"rr# node_rankr5sourcer*s r-rr>s  rz!!2:f%&&L122 2: % %RZ 566JJJ   )++C?DJJI IFF   DJJIIFFF   tZY 6 R RRsB>>CCncclcZtjrdSt}|jdkrt ddSt}tj r%tj |j n|dksJ|j j r(dtj|jz}nt%d}tj|||j|jt d|j|j|j |j|j j rKtjt)d kr|jdSdSdS) z Initialize DDP. Nrzworld_size is 1, skipping init.r8zfile://zenv://)backend init_methodrr"z*Distributed init: %d/%d (local %d) from %sr)torch distributedis_initializedrrloggerinforcuda is_available set_devicer#dorause_rendezvousrpathabspathrendezvous_filer.init_process_groupr"r7barrierunlink)r:r$r(r;s r-initrLas ''))   D ! 5666 B z  ! do....&     w"'//"2D"E"EE   ((? Y )  KK4 4?DOT[BBB w( !!### 66Q;;   % % ' ' ' ' ' (( ;c&tdkSNr)r"rMr- is_masterrQs 66Q;rMc~tjrtjSdSrO)r<r=r>get_rankrPrMr-r"r"s3 '')) ))+++qrMc~tjrtjSdS)Nr)r<r=r>get_world_sizerPrMr-rrs3 '')) //111qrM)r8) collectionsrloggingrr subprocessrrr<r(r getLogger__name__r?rr.rrLrQr"rrPrMr-r[s#"""""    8 $ $jJLL "8"8"8J S S SF!(!(!(!(HrM