$hjn)ddlZddlmZddlmZmZmZmZmZmZm Z m Z m Z m Z m Z mZmZddlmZmZmZmZddlmZddgZGd deZd d e d e d ed e d e d ze_ d(deedeedeedeedeedeedeedededeedeedeedededededeeefd ed!ed"ef(d#Zdeedeedeedeedeedeedeedeededededeeefd ed!ed"edededef$d$Zdeedeedeedeedeedeedeedeededededeeefd ed!ed"edededef$d%Zdeedeedeedeedeedeedeedeededededeeefd ed!ed"edededed&df&d'ZdS))N)Tensor) Optimizer_use_grad_for_differentiable _get_value_dispatch_sqrt_stack_if_compiling_capturable_doc_differentiable_doc _foreach_doc _fused_doc _maximize_doc_default_to_fused_or_foreachParamsT _view_as_real)ListOptionalTupleUnion)$_get_fused_kernels_supported_devicesAdamWadamwceZdZ dddddddded eeefd eeefd ed ed edede ededede effdZ fdZ dZ e ddZxZS)rMbP?g?g+?:0yE>{Gz?FN)maximizeforeach capturabledifferentiablefusedparamslrbetaseps weight_decayamsgradrrr r!r"c  d|kstd|t|tr|r| stdd|kstd|d|dcxkrdksntd|dd|dcxkrdksntd |dd|kstd |t|||||||| | | } t || | rj| rt d d |_t t fd|j Dst d d|rt ddSdS)NzInvalid learning rate: Elr as a Tensor is not supported for capturable=False and foreach=TruezInvalid epsilon value: r?z#Invalid beta parameter at index 0: rz#Invalid beta parameter at index 1: zInvalid weight_decay value: ) r$r%r&r'r(rrr r!r"z)`fused` does not support `differentiable`Tc3rK|]1}|dD]&}|jjvotj|V'2dS)r#N)devicetypetorchis_floating_point).0pgpfused_supported_devicess Q/var/www/html/auto_sub_bot/venv/lib/python3.11/site-packages/torch/optim/adamw.py z!AdamW.__init__..?spR\12 !88+'**zX`fused=True` requires all the params to be floating point Tensors of supported devices: .z0`fused` and `foreach` cannot be `True` together.) ValueError isinstancerdictsuper__init__ RuntimeError_step_supports_amp_scalingrall param_groups)selfr#r$r%r&r'r(rrr r!r"defaultsr5 __class__s @r6r>zAdamW.__init__ s<byy;r;;<< < b& ! ! fg fj fdee eczz)>///E*+ OOE), - - -   u\2 3 3 3Y @&&u-='>???%& i5=+F i"#ghhhY lJuT{F$C$C lER^L_ l"#jkkk   uV} - - - -r8c|d}|5tj5|}dddn #1swxYwY|jD]}g}g}g}g}g}g} |d} |d\} } ||||| |||| } t |||||| f| | | |d|d|d|d|d|d |d |d t |d dt |d d| d|S)zPerform a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nr(r%r$r'r&rrr r!r" grad_scale found_inf)r(beta1beta2r$r'r&rrr r!r"rorprl) _cuda_graph_capture_health_checkr0 enable_gradrBrmrgetattr)rCclosurelossrTrfrgrhrirjrkr(rqrrrls r6rGz AdamW.steps --///  "$$ ! !wyy ! ! ! ! ! ! ! ! ! ! ! ! ! ! !&* * E! EHK OKI&G >LE5**   K     ;">2%Lz*i( .$%56Gn"4t<<!$ T::')     . s AA A)rrrrFN)__name__ __module__ __qualname__rrrRrrboolrr>rJrmrrG __classcell__)rEs@r6rr sX$(%1":W"& $ $:W:W:W:W %- :WUE\" :W  :W  :W:W:W$:W:W:W~:W:W:W:W:W:WxPPPPP";;;z":::"!:::::r8a Implements AdamW algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \gamma \text{(lr)}, \: \beta_1, \beta_2 \text{(betas)}, \: \theta_0 \text{(params)}, \: f(\theta) \text{(objective)}, \: \epsilon \text{ (epsilon)} \\ &\hspace{13mm} \lambda \text{(weight decay)}, \: \textit{amsgrad}, \: \textit{maximize} \\ &\textbf{initialize} : m_0 \leftarrow 0 \text{ (first moment)}, v_0 \leftarrow 0 \text{ ( second moment)}, \: \widehat{v_0}^{max}\leftarrow 0 \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}\textbf{if} \: \textit{maximize}: \\ &\hspace{10mm}g_t \leftarrow -\nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm} \theta_t \leftarrow \theta_{t-1} - \gamma \lambda \theta_{t-1} \\ &\hspace{5mm}m_t \leftarrow \beta_1 m_{t-1} + (1 - \beta_1) g_t \\ &\hspace{5mm}v_t \leftarrow \beta_2 v_{t-1} + (1-\beta_2) g^2_t \\ &\hspace{5mm}\widehat{m_t} \leftarrow m_t/\big(1-\beta_1^t \big) \\ &\hspace{5mm}\widehat{v_t} \leftarrow v_t/\big(1-\beta_2^t \big) \\ &\hspace{5mm}\textbf{if} \: amsgrad \\ &\hspace{10mm}\widehat{v_t}^{max} \leftarrow \mathrm{max}(\widehat{v_t}^{max}, \widehat{v_t}) \\ &\hspace{10mm}\theta_t \leftarrow \theta_t - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}^{max}} + \epsilon \big) \\ &\hspace{5mm}\textbf{else} \\ &\hspace{10mm}\theta_t \leftarrow \theta_t - \gamma \widehat{m_t}/ \big(\sqrt{\widehat{v_t}} + \epsilon \big) \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `Decoupled Weight Decay Regularization`_. a Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, Tensor, optional): learning rate (default: 1e-3). A tensor LR is not yet supported for all our implementations. Please use a float LR if you are not also specifying fused=True or capturable=True. betas (Tuple[float, float], optional): coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999)) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) weight_decay (float, optional): weight decay coefficient (default: 1e-2) amsgrad (bool, optional): whether to use the AMSGrad variant of this algorithm from the paper `On the Convergence of Adam and Beyond`_ (default: False) z z .. _Decoupled Weight Decay Regularization: https://arxiv.org/abs/1711.05101 .. _On the Convergence of Adam and Beyond: https://openreview.net/forum?id=ryQu7f-RZ Fr#rgrhrirjrkrr r!r"rorprlr(rqrrr$r'r&rc tjs(td|Dst d| 2|0t ||d\}}|rt |tr|sd}| d} |d}|r-tj rt d| r-tj rt d| r&tj st}n/|r&tj st}nt}|||||||| ||||||||| | | dS) zpFunctional API that performs AdamW algorithm computation. See :class:`~torch.optim.AdamW` for details. c3JK|]}t|tjVdSrx)r;r0r)r2ts r6r7zadamw..4s/2d2dST:a3N3N2d2d2d2d2d2dr8zPAPI has changed, `state_steps` argument must contain a list of singleton tensorsNF) use_fusedz6torch.jit.script not supported with foreach optimizersz4torch.jit.script not supported with fused optimizers) r(rqrrr$r'r&rr r!rorprl) r0_utils is_compilingrAr?rr;rjit is_scripting _fused_adamw_multi_tensor_adamw_single_tensor_adamw)r#rgrhrirjrkrr r!r"rorprlr(rqrrr$r'r&r_funcs r6rrs: < $ $ & & s2d2dXc2d2d2d/d/d  ^    }1&.TYZZZ 7  z"f-- j G }U59))++USTTT S''))SQRRR $UY++--$ $//11$"#D  ! %%r8c ||Jtjrt| tsJt |D]}\}}|s||n|| }||}||}||}tjs(|r&|jr|js|j r|j s Jdtj |rotj |}tj |}tj |}|rtj ||||<tj |}|dz }| d| | zz | |d| z | | ||d| z |s|r|}d| |zz }d| |zz }| |z }|}|}|r|r||}n||}||tj||||||zz | |z }n0|||zz | |z }|||nt-|}d| |zz }d| |zz }| |z }t/|}|rTtj||||||||z | }n*||z | }|||| |r7tj ||rtj||||<dS)NzGIf capturable=True, params and state_steps must be CUDA or XLA tensors.r)value)out)r0rrr;rR enumeraterris_cudais_xlar_ view_as_realmul_lerp_addcmul_negsqrtclonecopy_maximumadd_addcdiv_rrview_as_complex) r#rgrhrirjrkrorpr(rqrrr$r'r&rr r!rliparamr^r[r\step_trGbias_correction1bias_correction2 step_size step_size_negbias_correction2_sqrtr]denoms r6rris?,  )"3"3 3 y%"e$$$$$f%%UKUK5'6uQxxeAhY1+ ^ Q|((** Yz Y Y#)> Y7<| YHN  Y YX Y YV  E " " .%d++D(11G+J77J L%*%78J%K%K"&u--E !   1rL(()))  dAI&&&''d!e)'DDD 3 =3 =D 5D=0  5D=0 --I%MMOOM$4$9$9$;$; ! ,!8%4Q%7%=%=%?%?NN%4Q%7N"((~z)R)RSSS $A&++--1F1VW$s]*++ OO%%)>)NO$s]*++ NN7E * * * *f%%D 5D=0  5D=0 --I$23C$D$D ! N oa0*/RSBTUUUU)+00225JJPPQTUU#**-BBHHMM NN7E)N < < <  Ku'q 22 K!&!6q7I!J!JOA kUKUKr8c  t|dkrdSt tr|stdtjs3|r1tdt||Ds Jd|r Jd||Jtj ||||||g}| D] \\}}}}}}}|rt j |}|r(|rt|||||nt|||||djr,t j|t jddd nt j|d | dkrt j|d | zz t j||d z t j| t j|||d z ~|r@t j |}t j |}t j|d t j|d t j|t j| t j|t j||}|}|r*t j||t j|}nt j|}t j||t j|| t j||t j|||H fd |D} fd |D}t; fd |D}d|D}|r*t j||t j|}nt j|}t j||t j|| t j||||dS)Nrr+c38K|]\}}|jo|jVdSrx)r)r2r4rGs r6r7z&_multi_tensor_adamw..s@  +21dAI &$,      r8z@If capturable=True, params and state_steps must be CUDA tensors.z#_foreach ops don't support autogradr,cpu)r.)alpharc:g|]}dt|zz Srr)r2rGrqs r6 z'_multi_tensor_adamw..R+]]]$EZ-=-=$= =]]]r8c:g|]}dt|zz Srr)r2rGrrs r6rz'_multi_tensor_adamw..Srr8c g|] }|z dz S)rY)r2bcr$s r6rz'_multi_tensor_adamw..Us!,W,W,Wb2g^,W,W,Wr8c,g|]}t|SrY)r)r2rs r6rz'_multi_tensor_adamw..Ws $S$S$SB^B%7%7$S$S$Sr8)rOr;rr?r0rrrAzipr"_group_tensors_by_device_and_dtyperN _foreach_negris_cpu _foreach_add_rQ _foreach_mul__foreach_lerp__foreach_addcmul_ _foreach_pow _foreach_sub_ _foreach_neg_ _foreach_div__foreach_reciprocal__foreach_sqrt__foreach_maximum_ _foreach_sqrt_foreach_addcdiv_r )r#rgrhrirjrkrorpr(rqrrr$r'r&rr r!rlgrouped_tensors device_params device_gradsdevice_exp_avgsdevice_exp_avg_sqsdevice_max_exp_avg_sqsdevice_state_stepsrrrrrexp_avg_sq_sqrts ``` r6rrs* 6{{a"fdjdbccc < $ $ & &N:N  69&+6N6N      N N M N N DDDDD   )"3"3 3Bxo{DLMMO # # % %^`^`    < -l;;L  ` `m\?L^`vwwwwm\?L^___ a ' 7   2ELU4S4S4S[^ _ _ _ _ _   2A 6 6 6 1     q2 3D/D E E E _lAIFFF .666  2L,PQTYPYZZZ  7 `$1%9KLL $1%9KLL    0! 4 4 4   0! 4 4 4   0 1 1 1   0" 5 5 5  &'7 8 8 8  !1 2 2 2 )I$4 ! J'(>@RSSS#("56L"M"M"'"56H"I"I  1F G G G   5 5 5   ; ; ;  #M?O T T T T]]]]J\]]] ]]]]J\]]] +,W,W,W,WFV,W,W,WXXI$S$SBR$S$S$S ! J'(>@RSSS#("56L"M"M"'"56H"I"I  1F G G G   5 5 5  #M?OU^ _ _ _ _}^`^`r8returnc |sdS|rtd| |j|ind}| |j|ind}t| tr!t | jdkr | j| ind}t j||||||g}|D]\\}}\\}}}}}}}d\}}|&||vr||d||<||}|&||vr||d||<||}|&||vr"| |d||<||} tj |dtj |||||||| | | | | ||||&tj ||gt|zdS) Nz9Adam with fused=True does not support differentiable=Truer)NNT) non_blocking)r.rr) r(r$rqrrr'r&rrorp)r?r.r;rstrrritemstor0r _fused_adamw_rrO) r#rgrhrirjrkrorpr(rqrrr$r'r&rr r!rlgrad_scale_dictfound_inf_dictlr_dictrr.rrrrrrrdevice_grad_scaledevice_found_infs r6rrgs7* XVWWW9C9Oz(*55UYO6?6Ki& 22QUN",B!7!7]C NNers] iiiiiiiiiiiiiiiiiiiiiiiiiiiiii0///////////KKKKKK G FFFFFIFFFR&L     !  " #  $ %  & '   M? V#  #'"&OO LO <O6lOf O &\ O f Od^OOO D>O OOO"#O$ %O& 'O( eVm)O*+O, -O./OOOOdsK LsK <sK6lsKf sK &\ sK f sK sKsKsK sK sK femsKsK sK !sK"#sK$%sK&'sKsKsKsKlE` LE` <E`6lE`f E` &\ E` f E` E`E`E` E` E` femE`E` E` !E`"#E`$%E`&'E`E`E`E`PHb LHb <Hb6lHbf Hb &\ Hb f Hb HbHbHb Hb Hb eVmHbHb Hb !Hb"#Hb$%Hb&'Hb( )HbHbHbHbHbHbr8