$hX8(ddlZddlmZddlmZmZmZmZmZmZm Z ddl m Z m Z ddgZ GddeZd d ed ed ed ze_ d de ede ede ede ede ede edededededededededefdZde ede ede ede ede edededededededededefdZde ede ede ede ede edededededededededefdZdS)!N)Tensor) Optimizer_default_to_fused_or_foreach_use_grad_for_differentiable_differentiable_doc _foreach_doc _maximize_doc _view_as_real)ListOptionalRMSproprmspropcteZdZ ddeeded effd Zfd Zd Zedd Z xZ S)r{Gz?Gz?:0yE>rFNforeachmaximizedifferentiablec ld|kstd|d|kstd|d|kstd|d|kstd|d|kstd|t|||||||| |  } t|| dS)NgzInvalid learning rate: zInvalid epsilon value: zInvalid momentum value: zInvalid weight_decay value: zInvalid alpha value: ) lrmomentumalphaepscentered weight_decayrrr) ValueErrordictsuper__init__) selfparamsrrrrrrrrrdefaults __class__s S/var/www/html/auto_sub_bot/venv/lib/python3.11/site-packages/torch/optim/rmsprop.pyr!zRMSprop.__init__ sbyy;r;;<< <czz000E+,$(-(8)>)))E*%   u\2 3 3 3Z 1$$$++E2C,DEEEZ  4  z!2333%& ?:eFmV+L+L ?"#=>>> &MMMQ MMMMr'c|d}|5tj5|}dddn #1swxYwY|jD]z}g}g}g}g}g}|||||||} t ||||||d|d|d|d|d|d|d|d |d | {|S) zPerforms a single optimization step. Args: closure (Callable, optional): A closure that reevaluates the model and returns the loss. Nrrrrrrrrr) rrrrrrrrrrC)r5 enable_gradr*rEr) r"closurelossr-r>r?r@rBrArCs r&r/z RMSprop.stepbsH  "$$ ! !wyy ! ! ! ! ! ! ! ! ! ! ! ! ! ! !&  E! EKI#% **52BE;XlnwxxK  $;Gn%L">2z*z*i(z*$%56'     $ s /33) rrrrrFNFF)N) __name__ __module__ __qualname__r boolr!r)rErr/ __classcell__)r%s@r&rr s  "&$#+#+$#+#+#+#+#+#+#+#+J66666'''R"'''"!'''''r'a Implements RMSprop algorithm. .. math:: \begin{aligned} &\rule{110mm}{0.4pt} \\ &\textbf{input} : \alpha \text{ (alpha)},\: \gamma \text{ (lr)}, \: \theta_0 \text{ (params)}, \: f(\theta) \text{ (objective)} \\ &\hspace{13mm} \lambda \text{ (weight decay)},\: \mu \text{ (momentum)},\: centered\\ &\textbf{initialize} : v_0 \leftarrow 0 \text{ (square average)}, \: \textbf{b}_0 \leftarrow 0 \text{ (buffer)}, \: g^{ave}_0 \leftarrow 0 \\[-1.ex] &\rule{110mm}{0.4pt} \\ &\textbf{for} \: t=1 \: \textbf{to} \: \ldots \: \textbf{do} \\ &\hspace{5mm}g_t \leftarrow \nabla_{\theta} f_t (\theta_{t-1}) \\ &\hspace{5mm}if \: \lambda \neq 0 \\ &\hspace{10mm} g_t \leftarrow g_t + \lambda \theta_{t-1} \\ &\hspace{5mm}v_t \leftarrow \alpha v_{t-1} + (1 - \alpha) g^2_t \hspace{8mm} \\ &\hspace{5mm} \tilde{v_t} \leftarrow v_t \\ &\hspace{5mm}if \: centered \\ &\hspace{10mm} g^{ave}_t \leftarrow g^{ave}_{t-1} \alpha + (1-\alpha) g_t \\ &\hspace{10mm} \tilde{v_t} \leftarrow \tilde{v_t} - \big(g^{ave}_{t} \big)^2 \\ &\hspace{5mm}if \: \mu > 0 \\ &\hspace{10mm} \textbf{b}_t\leftarrow \mu \textbf{b}_{t-1} + g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \\ &\hspace{10mm} \theta_t \leftarrow \theta_{t-1} - \gamma \textbf{b}_t \\ &\hspace{5mm} else \\ &\hspace{10mm}\theta_t \leftarrow \theta_{t-1} - \gamma g_t/ \big(\sqrt{\tilde{v_t}} + \epsilon \big) \hspace{3mm} \\ &\rule{110mm}{0.4pt} \\[-1.ex] &\bf{return} \: \theta_t \\[-1.ex] &\rule{110mm}{0.4pt} \\[-1.ex] \end{aligned} For further details regarding the algorithm we refer to `lecture notes `_ by G. Hinton. and centered version `Generating Sequences With Recurrent Neural Networks `_. The implementation here takes the square root of the gradient average before adding epsilon (note that TensorFlow interchanges these two operations). The effective learning rate is thus :math:`\gamma/(\sqrt{v} + \epsilon)` where :math:`\gamma` is the scheduled learning rate and :math:`v` is the weighted moving average of the squared gradient. a Args: params (iterable): iterable of parameters to optimize or dicts defining parameter groups lr (float, optional): learning rate (default: 1e-2) momentum (float, optional): momentum factor (default: 0) alpha (float, optional): smoothing constant (default: 0.99) eps (float, optional): term added to the denominator to improve numerical stability (default: 1e-8) centered (bool, optional) : if ``True``, compute the centered RMSProp, the gradient is normalized by an estimation of its variance weight_decay (float, optional): weight decay (L2 penalty) (default: 0) z z Fr#r?r@rBrArrrrCrrrrrrc "|t||d\}}|r-tjrt d|r&tjst }nt }||||||| | | | | ||||dS)zsFunctional API that performs rmsprop algorithm computation. See :class:`~torch.optim.RMSProp` for details. NF) use_fusedz6torch.jit.script not supported with foreach optimizers) rrrrrrrrrC)rr5jit is_scriptingr9_multi_tensor_rmsprop_single_tensor_rmsprop)r#r?r@rBrArrrrCrrrrrr_funcs r&rrs01&.TYZZZ 7U59))++USTTT&uy--//&$%D   !%r'c t|D]\}}||}| s|n| }||}|dkr|||}tj|}|r/E/E/G/G'S'S , '>=*=?P %  >!.}==M 1   f#M>VVVVV % 2=.Xd e e e ]++  <,.ABO!||&&'CDDD :&&'8999 . ;? ; ; ; ; /777  3]MYZ]bYbcccc  *  !2M1u9 M M M()<>OQbjlmmmC   % % %  S ) ) ) )%&9::C  S ) ) ) a<<   rsz YYYYYYYYYYYYYYYYYY!!!!!!!! i @@@@@i@@@F*T           U:J# 22 L2 <2f2F| 2 v, 2d^2222 2 2 2 !2"#2$%2222j71 L71 <71f71F| 71 v, 71 71 71 71717171717171717171t?S L?S <?Sf?SF| ?S v, ?S ?S ?S ?S?S?S?S?S?S?S?S?S?S?S?Sr'