VhgdZddlZddlmZddlZddlmZddlmZddl m Z ddl m Z dd l mZdd lmZdd lmZdd lmZmZdd lmZmZmZmZmZGddejZdS)zB This code contains the spectrogram and Hybrid version of Demucs. N)wiener)nn) functional)Fraction) rearrange)CrossTransformerEncoder)rescale_module) capture_init)spectroispectro)pad1dScaledEmbedding HEncLayer MultiWrap HDecLayerc?eZdZdZedddddddddd ddd d d dd dd ddd d ddd dddddd ddddddddddddddddgddddddddddddd df>fd Zd Zd(d!Zd"Zd#Z d$Z d%e fd&Z d'Z xZS))HTDemucsaV Spectrogram and hybrid Demucs model. The spectrogram model has the same structure as Demucs, except the first few layers are over the frequency axis, until there is only 1 frequency, and then it moves to time convolutions. Frequency layers can still access information across time steps thanks to the DConv residual. Hybrid model have a parallel time branch. At some layer, the time branch has the same stride as the frequency branch and then the two are combined. The opposite happens in the decoder. Models can either use naive iSTFT from masking, Wiener filtering ([Ulhih et al. 2017]), or complex as channels (CaC) [Choi et al. 2020]. Wiener filtering is based on Open Unmix implementation [Stoter et al. 2019]. The loss is always on the temporal domain, by backpropagating through the above output methods and iSTFT. This allows to define hybrid models nicely. However, this breaks a bit Wiener filtering, as doing more iteration at test time will change the spectrogram contribution, without changing the one from the waveform, which will lead to worse performance. I tried using the residual option in OpenUnmix Wiener implementation, but it didn't improve. CaC on the other hand provides similar performance for hybrid, and works naturally with hybrid models. This model also uses frequency embeddings are used to improve efficiency on convolutions over the freq. axis, following [Isik et al. 2020] (https://arxiv.org/pdf/2008.04470.pdf). Unlike classic Demucs, there is no resampling here, and normalization is always applied. 0NirFTg? rgMbP?sing@gi'g@?)g@rgffffff?diag*idgffffff?g?iDc@ t| |_| |_||_||_||_||_||_| |_ ||_ ||_ |=|_ |>|_ |?|_||_|dz|_||_||_d|_||ksJt)j|_t)j|_t)j|_t)j|_|}@|@}A|jr|Adz}A|p|}B|}C|dz}Dt5| D]}E|E|k}F|Ddk}G|}H|}I|Gs|DdksJ|dz}I|}Hd}Jd}K|Gr |D|kr|D}Id}Jd}K|I|H|G|J|F| ||||ddd}Lt7|L}Md |Md <||Md <||Md <d|Md <t7|L}Nd}O| r |E|krd}Od|Nd<|Krt9|B|C}C|C}Bt;|A|Cf|dz|d|L}P|Gr/t;|@|Bf|dz||Kd|M}Q|j|Q|Ort?|P| }P|j|P|Ed kr*|jtA|jz}@|@}A|jr|Adz}AtC|C|Af|dz|Ed k|d|N}R|Ort?|R| }R|Gr5tC|B|@f|dz|K|Ed k|d|M}S|j"d |S|j"d |R|B}@|C}AtG||Bz}BtG||Cz}C|Gr|D|krd}Dn|D|z}D|Ed kr!|rtI|D|A|||_||_%|