$hr)ddlmZmZmZmZddlZddlmZddlmZm Z m Z m Z m Z m Z mZmZmZmZmZmZmZddlmZddgZGd deZd d ed ed ed ed ed ze_ d(deedeedeedeedeedeedeedededeedeedeedededededeeefd ed!ed"ef(d#Zdeedeedeedeedeedeedeedeedededededeeefd ed!ed"ededef$d$Zdeedeedeedeedeedeedeedeedededededeeefd ed!ed"ededef$d%Zdeedeedeedeedeedeedeedeedededededeeefd ed!ed"ededed&df&d'ZdS)))ListOptionalUnionTupleN)Tensor) OptimizerParamsT_use_grad_for_differentiable _get_value_stack_if_compiling_dispatch_sqrt_default_to_fused_or_foreach_capturable_doc_differentiable_doc _foreach_doc _fused_doc _maximize_doc _view_as_real)$_get_fused_kernels_supported_devicesAdamadamceZdZ dddddddded eeefd eeefd ed ed ede edededede effdZ fdZ dZ e ddZxZS)rMbP?g?g+?:0yE>rFN)foreachmaximize capturabledifferentiablefusedparamslrbetaseps weight_decayamsgradrrrr r!c  d|kstd|t|tr|r| stdd|kstd|d|dcxkrdksntd|dd|dcxkrdksntd |dd|kstd |t|||||||| | | } t || | rj| rt d d |_t t fd|j Dst d d|rt ddSdS)NzInvalid learning rate: Elr as a Tensor is not supported for capturable=False and foreach=TruezInvalid epsilon value: r?z#Invalid beta parameter at index 0: rz#Invalid beta parameter at index 1: zInvalid weight_decay value: ) r#r$r%r&r'rrrr r!z)`fused` does not support `differentiable`Tc3rK|]1}|dD]&}|jjvotj|V'2dS)r"N)devicetypetorchis_floating_point).0pgpfused_supported_devicess P/var/www/html/auto_sub_bot/venv/lib/python3.11/site-packages/torch/optim/adam.py z Adam.__init__..8so/1PRS[P\KL !88+'**zX`fused=True` requires all the params to be floating point Tensors of supported devices: .z0`fused` and `foreach` cannot be `True` together.) ValueError isinstancerdictsuper__init__ RuntimeError_step_supports_amp_scalingrall param_groups)selfr"r#r$r%r&r'rrrr r!defaultsr4 __class__s @r5r=z Adam.__init__s6byy;r;;<< < b& ! ! fg fj fdee eczzrLrNzerosrRr-rP zeros_likepreserve_format requires_gradrO) rBrSparams_with_gradgradsexp_avgs exp_avg_sqsmax_exp_avg_sqs state_steps has_complexr3rLs r5 _init_groupzAdam._init_groupPs< x' 2' 2Av!u/222  ''***6#u&'sttt QV$$$ 1 u::?? !.D27.D BemAHMMMM"\#U]CCC&M (-'7I^'_'_'_E)$*/*:1ELa*b*b*bE,'Y'k272B1TYTi2j2j2j./i 0111""5#6777#D#**51A+BCCC)*muV}/Jm&'klll#pd (D(DpUS_M`p&'nooo""5=111r7c|d}|5tj5|}dddn #1swxYwY|jD]}g}g}g}g}g}g} |d\} } |||||||| } t |||||| f|d| | | |d|d|d|d|d|d |d |d t |d dt |d dd|S)zPerform a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nr$r'r#r&r%rrrr r! grad_scale found_inf)r'rkbeta1beta2r#r&r%rrrr r!rnro) _cuda_graph_capture_health_checkr/ enable_gradrArlrgetattr) rBclosurelossrSrerfrgrhrirjrprqrks r5rFz Adam.steps --///  "$$ ! !wyy ! ! ! ! ! ! ! ! ! ! ! ! ! ! !&' ' E! EHK OK >LE5** K    i(';">2%Lz*i( .$%56Gn"4t<<!$ T::)     . s AA A)rrrrFN)__name__ __module__ __qualname__r rrQrrboolrr=rIrlr rF __classcell__)rDs@r5rrsV-1.:"'(!& 0W,0"'$)(-)-0W0W0W 0W5&=)0WeUl+0W 0W % 0W  0W#4.0W 0W"0W"&0W!0W0W0W0W0W0Wd P P P P P333j"777"!77777r7a Implements Adam algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{ (lr)}, \beta_1, \beta_2 \text{ (betas)},\theta_0 \text{ (params)},f(\theta) \text{ (objective)} \\ &\hspace{13mm} \lambda \text{ (weight decay)}, \: \textit{amsgrad}, \:\textit{maximize} \\ &\textbf{initialize} : m_0 \leftarrow 0 \text{ ( first moment)}, v_0\leftarrow 0 \text{ (second moment)},\: \widehat{v_0}^{max}\leftarrow 0\\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}\textbf{if} \: \textit{maximize}: \\ &\hspace{10mm}g_t \leftarrow -\nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{if} \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}m_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ &\hspace{5mm}v_t \leftarrow \beta_2 v_{t-1} + (1-\beta_2) g^2_t \\ &\hspace{5mm}\widehat{m_t} \leftarrow m_t/\big(1-\beta_1^t \big) \\ &\hspace{5mm}\widehat{v_t} \leftarrow v_t/\big(1-\beta_2^t \big) \\ &\hspace{5mm}\textbf{if} \: amsgrad \\ &\hspace{10mm}\widehat{v_t}^{max} \leftarrow \mathrm{max}(\widehat{v_t}^{max}, \widehat{v_t}) \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}^{max}} + \epsilon \big) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}} + \epsilon \big) \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Adam: A Method for Stochastic Optimization`_. a Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, Tensor, optional): learning rate (default: 1e-3). A tensor LR is not yet supported for all our implementations. Please use a float LR if you are not also specifying fused=True or capturable=True. betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay (L2 penalty) (default: 0) amsgrad (bool, optional): whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`_ (default: False) z z .. _Adam\: A Method for Stochastic Optimization: https://arxiv.org/abs/1412.6980 .. _On the Convergence of Adam and Beyond: https://openreview.net/forum?id=ryQu7f-RZ Fr"rfrgrhrirjrrr r!rnrorkr'rprqr#r&r%rc | 2|0t||d\}}|rt|tr|sd}| d} |d}tjs(t d|Dstd|r-tj rtd| r-tj rtd| r&tj st}n/|r&tj st}nt}|||||||| | ||||||||| | dS) znFunctional API that performs Adam algorithm computation. See :class:`~torch.optim.Adam` for details. NF) use_fusedc3JK|]}t|tjVdSrw)r:r/r)r1ts r5r6zadam..-s/2d2dST:a3N3N2d2d2d2d2d2dr7zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsz6torch.jit.script not supported with foreach optimizersz4torch.jit.script not supported with fused optimizers) r'rkrprqr#r&r%rrr rnro) rr:rr/_utils is_compilingr@r>jit is_scripting _fused_adam_multi_tensor_adam_single_tensor_adam)r"rfrgrhrirjrrr r!rnrorkr'rprqr#r&r%r_funcs r5rrs> }1&.TYZZZ 7  z"f-- j G } < $ $ & &os2d2dXc2d2d2d/d/domnnnU59))++USTTT S''))SQRRR #UY++--# #//11#!"D       " &#r7c ||Jtjrt| tsJt |D]\}}|s||n|| }||}||}||}tjs(|r&|jr|js|j r|j s Jd|dz }| dkr| || }tj |rotj |}tj |}tj |}|rtj ||||<tj |}| |d| z || ||d| z |s|r|}d| |zz }d| |zz }| |z }|}|}|r|r||}n||}||tj||||||zz ||z }n0|||zz ||z }|||nt1|}d| |zz }d| |zz }| |z }t3|}|rTtj||||||||z |}n*||z |}|||| |r7tj ||rtj||||<dS)NzGIf capturable=True, params and state_steps must be CUDA or XLA tensors.rralpha)value)out)r/rrr:rQ enumeraterris_cudais_xlaaddr^ view_as_reallerp_mul_addcmul_conjnegsqrtclonecopy_maximumadd_addcdiv_r rview_as_complex) r"rfrgrhrirjrnror'rkrprqr#r&r%rrr iparamr]rZr[step_trFbias_correction1bias_correction2 step_size step_size_negbias_correction2_sqrtr\denoms r5rrPsD(  )"3"3 3 y%"e$$$$$f%%QKQK5'6uQxxeAhY1+ ^ Q|((** Yz Y Y#)> Y7<| YHN  Y YX Y YV !  1  88E866D  E " " .%d++D(11G+J77J L%*%78J%K%K"&u--E  dAI&&&''diikkU'KKK / =/ =D 5D=0  5D=0 --I%MMOOM$4$9$9$;$; ! p!8%4Q%7%=%=%?%?NN%4Q%7N"((~z)R)RSSS )+00226Km6[\bbcfivcvww#**.Cm.STZZ[^an[noo NN7E * * * *f%%D 5D=0  5D=0 --I$23C$D$D ! N oa0*/RSBTUUUU)+00225JJPPQTUU#**-BBHHMM NN7E)N < < <  Ku'q 22 K!&!6q7I!J!JOA cQKQKr7c  t|dkrdSt tr|stdtjs3|r1tdt||Ds Jd||J|r Jdtj ||||||g}| D]"\\}}}}}}}|rt j |}| r(|rt|||||nt|||||djr,t j|t jddd nt j|d | dkr1|rt j||| nt j||| }t j||d z t j| t j|||d z ~|r@t j |}t j |}t j|d t j|d t j|t j| t j|t j||}|}|r*t j||t j|}nt j|}t j||t j||t j||t j|||^ fd |D} fd |D}t= fd |D}d|D}|r*t j||t j|}nt j|}t j||t j||t j||||$dS)Nrr*c38K|]\}}|jo|jVdSrw)r)r1r3rFs r5r6z%_multi_tensor_adam..s/VV'!T19-VVVVVVr7z@If capturable=True, params and state_steps must be CUDA tensors.z#_foreach ops don't support autogradr+cpu)r-rrc:g|]}dt|zz Srr )r1rFrps r5 z&_multi_tensor_adam..5+]]]$EZ-=-=$= =]]]r7c:g|]}dt|zz Srr)r1rFrqs r5rz&_multi_tensor_adam..6rr7c g|] }|z dz S)rX)r1bcr#s r5rz&_multi_tensor_adam..8s!,W,W,Wb2g^,W,W,Wr7c,g|]}t|SrX)r)r1rs r5rz&_multi_tensor_adam..:s $S$S$SB^B%7%7$S$S$Sr7)rNr:rr>r/rrr@zipr "_group_tensors_by_device_and_dtyperM _foreach_negris_cpu _foreach_add_rP _foreach_add_foreach_lerp_ _foreach_mul__foreach_addcmul_ _foreach_pow _foreach_sub_ _foreach_neg_ _foreach_div__foreach_reciprocal__foreach_sqrt__foreach_maximum_ _foreach_sqrt_foreach_addcdiv_r )r"rfrgrhrirjrnror'rkrprqr#r&r%rrr grouped_tensors device_params device_gradsdevice_exp_avgsdevice_exp_avg_sqsdevice_max_exp_avg_sqsdevice_state_stepsrrrrrexp_avg_sq_sqrts ``` r5rrs& 6{{a"fdjdbccc < $ $ & &O:OVVS=U=UVVVVV O O N O OV  )"3"3 3DDDDD B + LNNO # # % %c`c`    < -l;;L  ` `m\?L^`vwwwwm\?L^___ a ' 7   2ELU4S4S4S[^ _ _ _ _ _   2A 6 6 6 1   c#L-|TTTTT$1, Uabbb  _lAIFFF .666  2L,PQTYPYZZZ  7 `$1%9KLL $1%9KLL    0! 4 4 4   0! 4 4 4   0 1 1 1   0" 5 5 5  &'7 8 8 8  !1 2 2 2 )I$4 ! J'(>@RSSS#("56L"M"M"'"56H"I"I  1F G G G   5 5 5   ; ; ;  #M?O T T T T]]]]J\]]] ]]]]J\]]] +,W,W,W,WFV,W,W,WXXI$S$SBR$S$S$S ! J'(>@RSSS#("56L"M"M"'"56H"I"I  1F G G G   5 5 5  #M?OU^ _ _ _ _Gc`c`r7returnc |sdS|rtd| |j|ind}| |j|ind}t| tr!t | jdkr | j| ind}t j||||||g}|D]\\}}\\}}}}}}}d\}}|&||vr||d||<||}|&||vr||d||<||}|&||vr"| |d||<||} tj |dtj |||||||| | | | |||||&tj ||gt|zdS) Nz9Adam with fused=True does not support differentiable=Truer)NNT) non_blocking)r-rr) r'r#rprqr&r%rrnro)r>r-r:rstrr ritemstor/r _fused_adam_rrN) r"rfrgrhrirjrnror'rkrprqr#r&r%rrr grad_scale_dictfound_inf_dictlr_dictrr-rrrrrrrdevice_grad_scaledevice_found_infs r5rrJs7* XVWWW9C9Oz(*55UYO6?6Ki& 22QUN",B!7!7]C NNers7//////////// 666666666666666666666666666666LKKKKK 6 ooooo9oood&L     !  " #  $ %  & '   M? T$(! %!%(,'+"KKfKV K<K6lKv, K 6l K4.KKKKf%KV$KK !K"#K$%K&5&= !'K()K*+K,-KKKK\mKV mK#F|mK"&v,mK&*&\mK*.f mK &*&\ mK %-V$4 mK$,F#3mK"&mK&*mK %mK %mK"%-0mK',mK#mK #'!mK"%)#mK$)-%mKmKmKmK`G`tF|G`"6lG`!%fG`%)LG`)-V G` %)L G` $,F#3 G`#+6"2G`!%G`%)G`$G`$G`!/G`&+G`"G` "&!G`"$(#G`$(,%G`G`G`G`THb LHb <Hb6lHbf Hb &\ Hb f Hb HbHbHbHb Hb Hb eVmHbHb !Hb"#Hb$%Hb&'Hb( )HbHbHbHbHbHbr7