rhddZddlmZddlmZmZddlZddlZddlmZddl m Z ddl m Z dd l mZmZd d lmZej$eZeed Gdde ZGddej,ZGddej,ZGddej,ZGddej,ZeGdde ZdgZy)zPyTorch UnivNetModel model.) dataclass)OptionalUnionN)nn) ModelOutput)PreTrainedModel)auto_docstringlogging) UnivNetConfigz Output class for the [`UnivNetModel`], which includes the generated audio waveforms and the original unpadded lengths of those waveforms (so that the padding can be removed by [`UnivNetModel.batch_decode`]). ) custom_introcbeZdZUdZdZeejed<dZ eejed<y)UnivNetModelOutputa" waveforms (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Batched 1D (mono-channel) output audio waveforms. waveform_lengths (`torch.FloatTensor` of shape `(batch_size,)`): The batched length in samples of each unpadded waveform in `waveforms`. N waveformswaveform_lengths) __name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__r/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/univnet/modeling_univnet.pyrr s4.2Ix))*148hu0018rrcTeZdZdZdeffd ZdejfdZdZ dZ xZ S)#UnivNetKernelPredictorResidualBlockz Implementation of the residual block for the kernel predictor network inside each location variable convolution block (LVCBlock). Parameters: config: (`UnivNetConfig`): Config for the `UnivNetModel` model. configc t||j|_|j|_|j |_|j|_|j dz dz}tj|j|_ tj|j|j|j |d|_ tj|j|j|j |d|_y)Nr Tpaddingbias)super__init__model_in_channelschannelskernel_predictor_conv_size kernel_sizekernel_predictor_dropout dropout_probleaky_relu_sloperDropoutdropoutConv1dconv1conv2)selfrr# __class__s rr&z,UnivNetKernelPredictorResidualBlock.__init__=s 00 !<<";; & 7 7##a'A-zz$"3"34 YYt}}dmmT=M=MW^eij YYt}}dmmT=M=MW^eij r hidden_statesc|}|j|}|j|}tjj ||j }|j |}tjj ||j }||zSN)r/r1r functional leaky_relur-r2)r3r5residuals rforwardz+UnivNetKernelPredictorResidualBlock.forwardMst  ]3  =1  00@U@UV  =1  00@U@UV x''rctjj}ttjjdr$tjjj}||j ||j yN weight_norm)rutilsr>hasattrparametrizationsr1r2r3r>s rapply_weight_normz5UnivNetKernelPredictorResidualBlock.apply_weight_normWsRhh** 288,,m <((33??KDJJDJJrctjj|jtjj|jyr7)rr?remove_weight_normr1r2r3s rrEz6UnivNetKernelPredictorResidualBlock.remove_weight_norm_s. ##DJJ/ ##DJJ/r) rrrrr r&rrr;rCrE __classcell__r4s@rrr3s3kk (U%6%6( 0rrcbeZdZdZ d dededeffd ZdejfdZ dZ d Z xZ S) UnivNetKernelPredictora  Implementation of the kernel predictor network which supplies the kernel and bias for the location variable convolutional layers (LVCs) in each UnivNet LVCBlock. Based on the KernelPredictor implementation in [maum-ai/univnet](https://github.com/maum-ai/univnet/blob/9bb2b54838bb6d7ce767131cc7b8b61198bc7558/model/lvcnet.py#L7). Parameters: config: (`UnivNetConfig`): Config for the `UnivNetModel` model. conv_kernel_size (`int`, *optional*, defaults to 3): The kernel size for the location variable convolutional layer kernels (convolutional weight tensor). conv_layers (`int`, *optional*, defaults to 4): The number of location variable convolutional layers to output kernels and biases for. rconv_kernel_size conv_layersct||j|_d|jz|_||_||_|j|jz|j z|j z|_|j|j z|_|j|_ |j|_ |j|_|j|_|j"|_|jdz dz}t%j&|j|jddd|_t%j*t-|j Dcgc] }t/|c}|_t%j&|j|j|j|d|_t%j&|j|j|j|d|_ycc}w)Nr!r Tr")r%r&model_hidden_channelsconv_in_channelsconv_out_channelsrKrLkernel_channels bias_channels num_mel_binsresnet_in_channels kernel_predictor_hidden_channelsresnet_hidden_channelsr)resnet_kernel_sizekernel_predictor_num_blocks num_blocksr-rr0 input_conv ModuleListranger resblocks kernel_conv bias_conv)r3rrKrLr#_r4s rr&zUnivNetKernelPredictor.__init__us  & < <!"V%A%A!A 0&  ! !D$:$: :T=R=R RUYUeUe e "33d6F6FF"("5"5&,&M&M#"("C"C << & 7 7**Q.14))D$;$;T=X=XZ[efmqr]bcgcrcr]s'tXY(KF(S'tu99  ' ')=)=t?V?V`gnr   ' '););T=T=T^elp  (usG2 spectrogramc|j\}}}|j|}tjj ||j }|j D] }||} |j|}|j|}|j||j|j|j|j|j} |j||j|j|j} | | fS)a Maps a conditioning log-mel spectrogram to a tensor of convolutional kernels and biases, for use in location variable convolutional layers. Note that the input spectrogram should have shape (batch_size, input_channels, seq_length). Args: spectrogram (`torch.FloatTensor` of shape `(batch_size, input_channels, seq_length)`): Tensor containing the log-mel spectrograms. Returns: tuple[`torch.FloatTensor, `torch.FloatTensor`]: tuple of tensors where the first element is the tensor of location variable convolution kernels of shape `(batch_size, self.conv_layers, self.conv_in_channels, self.conv_out_channels, self.conv_kernel_size, seq_length)` and the second element is the tensor of location variable convolution biases of shape `(batch_size, self.conv_layers. self.conv_out_channels, seq_length)`. )shaper[rr8r9r-r^r_r`viewrLrPrQrK contiguous) r3rb batch_sizera seq_lengthr5resblockkernel_hidden_statesbias_hidden_stateskernelsbiasess rr;zUnivNetKernelPredictor.forwards "%0$5$5! Az 4  00@U@UV  4H$]3M 4 $// >!^^M:'++      ! !  " "  ! !    *, $((      " "   *, rcvtjj}ttjjdr$tjjj}||j |j D]}|j||j||jyr=) rr?r>r@rAr[r^rCr_r`r3r>layers rrCz(UnivNetKernelPredictor.apply_weight_normshh** 288,,m <((33??KDOO$^^ &E  # # % &D$$%DNN#rc<tjj|j|jD]}|jtjj|j tjj|j yr7)rr?rEr[r^r_r`r3rps rrEz)UnivNetKernelPredictor.remove_weight_normse ##DOO4^^ 'E  $ $ & ' ##D$4$45 ##DNN3r)r rrrrr intr&rrr;rCrErGrHs@rrJrJdsO&!" $ $ $  $ L,5#4#4,\ $4rrJc eZdZdZdededeffd ZddZ ddejdejd ejded ef d Z d Z d Z xZ S)UnivNetLvcResidualBlocka Implementation of the location variable convolution (LVC) residual block for the UnivNet residual network. Parameters: config: (`UnivNetConfig`): Config for the `UnivNetModel` model. kernel_size (`int`): The kernel size for the dilated 1D convolutional layer. dilation (`int`): The dilation for the dilated 1D convolutional layer. rr*dilationcNt||j|_||_||_|j |_|j |jdz zdz}tj|j|j|j||j |_ y)Nr r!)r#rx) r%r&rOhidden_channelsr*rxr-rr0conv)r3rr*rxr#r4s rr&z UnivNetLvcResidualBlock.__init__s %;;&  & 7 7--4#3#3a#78A=II     ]]   rc|}tjj||j}|j |}tjj||j}|j ||||}t j|ddd|jddft j|dd|jdddfz}||z}|SN)hop_size) rr8r9r-r{location_variable_convolutionrsigmoidrztanh)r3r5kernelr$r~r:s rr;zUnivNetLvcResidualBlock.forwards  00@U@UV  -0  00@U@UV ::=&RVai:j  mA7M9M9M7Mq4P&QRUZU_U_ !T113Q6 7V  !=0 rr5rr$r~cF|j\}}}|j\}}} } } || |zk7rtd| |zd|d|t| dz dz z} tjj || | fdd}|j d|d| zz|}||kr$tjj |d|fdd}|j d||}|d d d d d d d d d |f}|jdd }|j d | d}tjd ||} | jtj } |jd jd jtj }| |z} | jj|| d } | S)u Performs location-variable convolution operation on the input sequence (hidden_states) using the local convolution kernel. This was introduced in [LVCNet: Efficient Condition-Dependent Modeling Network for Waveform Generation](https://huggingface.co/papers/2102.10815) by Zhen Zheng, Jianzong Wang, Ning Cheng, and Jing Xiao. Time: 414 μs ± 309 ns per loop (mean ± std. dev. of 7 runs, 1000 loops each), test on NVIDIA V100. Args: hidden_states (`torch.FloatTensor` of shape `(batch_size, in_channels, in_length)`): The input sequence of shape (batch, in_channels, in_length). kernel (`torch.FloatTensor` of shape `(batch_size, in_channels, out_channels, kernel_size, kernel_length)`): The local convolution kernel of shape (batch, in_channels, out_channels, kernel_size, kernel_length). bias (`torch.FloatTensor` of shape `(batch_size, out_channels, kernel_length)`): The bias for the local convolution of shape (batch, out_channels, kernel_length). dilation (`int`, *optional*, defaults to 1): The dilation of convolution. hop_size (`int`, *optional*, defaults to 256): The hop_size of the conditioning sequence. Returns: `torch.FloatTensor`: the output sequence after performing local convolution with shape (batch_size, out_channels, in_length). z#Dim 2 of `hidden_states` should be z ) but got zX. Please check `hidden_states` or `kernel` and `hop_size` to make sure they are correct.r r!constantrrNrszbildsk,biokl->bolsd) memory_format)rd ValueErrorrurr8padunfold transposereinsumtochannels_last_3d unsqueezerfre)r3r5rr$rxr~batchra in_length out_channelsr* kernel_lengthr#output_hidden_statess rrz5UnivNetLvcResidualBlock.location_variable_convolutions<,11q)=C\\:q, ] 1 25mh6N5OzZcYde]]  S+/Q!677 ))-'79KZYZ[ %,,Q1w;0FQ h MM--ma]JXYZM%,,Q(C %aAq)8)&;< %//15 %,,Q Q?  %||,A=RXY366UE[E[6\~~b!++B/22AWAW2X3d:3>>@EEe\[]^##rctjj}ttjjdr$tjjj}||j yr=)rr?r>r@rAr{rBs rrCz)UnivNetLvcResidualBlock.apply_weight_normPsFhh** 288,,m <((33??KDIIrcVtjj|jyr7)rr?rEr{rFs rrEz*UnivNetLvcResidualBlock.remove_weight_normWs ##DII.r)r r)rrrrr rur&r;rrrrCrErGrHs@rrwrws      , * ?$((?$!!?$ ?$  ?$  ?$B/rrwcxeZdZdZ d dededeffd ZdejdejfdZ d Z d Z xZ S) UnivNetLvcBlocka# Implementation of the location variable convolution (LVC) residual block of the UnivNet residual block. Includes a `UnivNetKernelPredictor` inside to predict the kernels and biases of the LVC layers. Based on LVCBlock in [maum-ai/univnet](https://github.com/maum-ai/univnet/blob/9bb2b54838bb6d7ce767131cc7b8b61198bc7558/model/lvcnet.py#L98) Parameters: config (`UnivNetConfig`): Config for the `UnivNetModel` model. layer_id (`int`): An integer corresponding to the index of the current LVC resnet block layer. This should be between 0 and `len(config.resblock_stride_sizes) - 1)` inclusive. lvc_hop_size (`int`, *optional*, defaults to 256): The hop size for the location variable convolutional layers. rlayer_id lvc_hop_sizec t||j|_|j||_|j ||_|j||_ ||_ |j|_ t|j|_ tj|j|jd|jz|j|jdz|jdzz|jdz|_t#||j |j|_tj&t)|jDcgc]&}t+||j |j|(c}|_ycc}w)Nr!)strider#output_padding)r%r&rOrzresblock_kernel_sizesr*resblock_stride_sizesrresblock_dilation_sizes dilationscond_hop_lengthr-lenrZrConvTranspose1d convt_prerJkernel_predictorr\r]rwr^)r3rrrir4s rr&zUnivNetLvcBlock.__init__ms< %;;!77A228< 77A+ & 7 7dnn-++    O;;KK1$t{{Q6;;?  !7vt?O?OQUQ`Q` a[`aeapap[q rVW $VT-=-=t~~a?P Q r  rs+Fr5rbc Rtjj||j}|j |}|j |\}}t |jD]?\}}|dd|ddddddddf}|dd|ddddf}|||||j}A|Sr}) rr8r9r-rr enumerater^r) r3r5rbrlrmrrirr$s rr;zUnivNetLvcBlock.forwards 00@U@UV }5 // <$T^^4 aKAxQ1aA-.F!Q1*%D$]FD4K_K_`M a rcbtjj}ttjjdr$tjjj}||j |j j|jD]}|jyr=) rr?r>r@rArrrCr^ros rrCz!UnivNetLvcBlock.apply_weight_normswhh** 288,,m <((33??KDNN# //1^^ &E  # # % &rctjj|j|jj|j D]}|jyr7)rr?rErrr^rrs rrEz"UnivNetLvcBlock.remove_weight_normsI ##DNN3 002^^ 'E  $ $ & 'rrrtrHs@rrr[sW*      < U%6%6 UEVEV &'rrceZdZUeed<dZdeffd Ze d dejde ejde ejde ejde e de eejeff d Zd Zd Zd ZxZS) UnivNetModelrinput_featuresc ^t||t|j|_|j |_t j|j|jdddd|_ t|j}d}g}|jD]}||z}|j|t jt|Dcgc]}t||||c}|_t j|jdddd|_|j%ycc}w)Nr rreflect)r*rr# padding_mode)rr)r#r)r%r&rr num_kernelsr-rr0r'rOconv_prerappendr\r]rr^ conv_post post_init)r3r num_layers hop_length hop_lengthsrrr4s rr&zUnivNetModel.__init__s  v;;< & 7 7  $ $  ( ("   556   22 +F#f,J   z * +z*     !,Q   6#?#?Aq_hi  s D*noise_sequence padding_mask generator return_dictreturnc||n|jj}|jdk(}|s|jd}|j\}}} |'|jdk(} | sX|jd}nF|||jj f} t j| ||j|j}|jd} |dkDr| dk(r|j|dd}n| dkDr|dk(r|j| dd}| |k7rtd| d|d|J|jdk(r|jd}|jd} | |k7rtd | d|d|jd d}|jd d}|j|}|jD] }|||} tj j#||j$}|j'|}t j(|}|j+d}d}|t j,|d }|s||f}|St/|| S) a noise_sequence (`torch.FloatTensor`, *optional*): Tensor containing a noise sequence of standard Gaussian noise. Can be batched and of shape `(batch_size, sequence_length, config.model_in_channels)`, or un-batched and of shape (sequence_length, config.model_in_channels)`. If not supplied, will be randomly generated. padding_mask (`torch.BoolTensor`, *optional*): Mask indicating which parts of each sequence are padded. Mask values are selected in `[0, 1]`: - 1 for tokens that are **not masked** - 0 for tokens that are **masked** The mask can be batched and of shape `(batch_size, sequence_length)` or un-batched and of shape `(sequence_length,)`. generator (`torch.Generator`, *optional*): A [torch generator](https://pytorch.org/docs/stable/generated/torch.Generator.html) to make generation deterministic. return_dict: Whether to return a [`~utils.ModelOutput`] subclass instead of a plain tuple. Example: ```python >>> from transformers import UnivNetFeatureExtractor, UnivNetModel >>> from datasets import load_dataset, Audio >>> model = UnivNetModel.from_pretrained("dg845/univnet-dev") >>> feature_extractor = UnivNetFeatureExtractor.from_pretrained("dg845/univnet-dev") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> # Resample the audio to the feature extractor's sampling rate. >>> ds = ds.cast_column("audio", Audio(sampling_rate=feature_extractor.sampling_rate)) >>> inputs = feature_extractor( ... ds[0]["audio"]["array"], sampling_rate=ds[0]["audio"]["sampling_rate"], return_tensors="pt" ... ) >>> audio = model(**inputs).waveforms >>> list(audio.shape) [1, 140288] ``` Nrr)rdtypedevicer z&The batch size of `noise_sequence` is z+ and the batch size of `input_features` is z', but the two are expected to be equal.z$The batch size of `padding_mask` is r!)dim)rr)ruse_return_dictrrrdr'rrandnrrrepeatrrrr^rr8r9r-rrsqueezesumr)r3rrrrrspectrogram_batchedspectrogram_batch_sizespectrogram_lengthranoise_sequence_batchednoise_sequence_shapenoise_sequence_batch_sizepadding_mask_batch_sizer5riwaveformroutputss rr;zUnivNetModel.forwards`&1% ")!/!9!9!!<%; >89R8ST((>'??fh   #!Q&+55a8 &2&8&8&; #&*@@ :;R:ST,,B+CCjl '00A6 '11!Q7 m4  DH$]NCM D 00@U@UV }5  =1 !((+   #$yy1= !12GN!-  rcPt|tjtjtjfrm|j j jd|jj|j%|jj jyyy)zInitialize the weights.g)meanstdN) isinstancerLinearr0rweightdatanormal_rinitializer_ranger$zero_)r3modules r _init_weightszUnivNetModel._init_weightsRss fryy"))R5G5GH I MM   & &CT[[5R5R & S{{&   &&(' JrcRtjj}ttjjdr$tjjj}||j |j D]}|j||jyr=) rr?r>r@rArr^rCrros rrCzUnivNetModel.apply_weight_normYsqhh** 288,,m <((33??KDMM"^^ &E  # # % &DNN#rctjj|j|jD]}|jtjj|j yr7)rr?rErr^rrrs rrEzUnivNetModel.remove_weight_normcsM ##DMM2^^ 'E  $ $ & ' ##DNN3r)NNNN)rrrr rmain_input_namer&r rrr Generatorboolrtuplerr;rrCrErGrHs@rrrs &O%}%N7;48/3&* x ))x !!2!23x u001 x EOO, x d^ x  uU&&');; <x x t)$4rr)r dataclassesrtypingrrrtorch.utils.checkpointrmodeling_outputsrmodeling_utilsr r?r r configuration_univnetr get_loggerrloggerrModulerrJrwrr__all__rrrrs"!" +-,0   H %  9 9   9.0")).0bu4RYYu4p|/bii|/~M'biiM'`{4?{4{4|  r