rhQ:ddlZddlZddlmZmZmZddlZddlZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lmZddlm Z m!Z!ddl"m#Z#e!jHe%Z&GddeZ'GddeZ(GddeZ)Gdde jTZ+Gdde jTZ,Gdde jTZ-Gdde jTZ. dBde jTd ej^d!ej^d"ej^d#eej^d$ee0d%e0d&eej^fd'Z1Gd(d)e jTZ2Gd*d+e jTZ3Gd,d-eZ4Gd.d/e jTZ5e Gd0d1eZ6 dCd2e7e8e8fd3e0d4e8d#eejrd5e8d6ejtf d7Z;e Gd8d9e6Ze d>;Gd?d@e6Z?gdAZ@y)DN)CallableOptionalUnion)nn)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutputSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringlogging) SEWConfigc&eZdZdfd ZdZxZS)SEWNoLayerNormConvLayercdt||dkDr|j|dz nd|_|j||_t j |j|j|j||j||j|_ t|j|_ y)Nrr kernel_sizestridebias)super__init__conv_dim in_conv_dim out_conv_dimrConv1d conv_kernel conv_stride conv_biasconvr feat_extract_activation activationselfconfiglayer_id __class__s w/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/sew/modeling_sew.pyr z SEWNoLayerNormConvLayer.__init__/s !#Car1cV|jdkDr|ddddd|j f}|SNr)ror4s r0r6zSEWSamePadLayer.forwards6    ")!Q0F43F3F2F0F*FGMr1r8r=s@r0rkrksKr1rkc$eZdZfdZdZxZS) SEWUpsamplingct|tj|j|j|j z|_t|j|_ |j |_yr3) rr rLinearr]r` projectionr r)r*r,r-r/s r0r zSEWUpsampling.__init__sW ))F$6$68J8JVMbMb8bc !?!?@$33r1c.|j|}|j|}|jdkDrc|j\}}}||jz}||jz}|j |||j|}|j |||}|S)Nr)rvr*r`sizereshape)r,r5bszsrc_len src_embed_dimtgt_len tgt_embed_dims r0r6zSEWUpsampling.forwards 6  6    "*7*<*<*> 'C- 3 33G)T-@-@@M)11#w@S@SUbcM)11#w NMr1r8r=s@r0rsrss 4 r1rsc.eZdZdZfdZdZdZxZS)SEWFeatureEncoderz.Construct the features from raw audio waveformc t||jdk(rDt|dgt |j dz Dcgc]}t ||dzc}z}nV|jdk(r.t |j Dcgc]}t||}}ntd|jdtj||_ d|_ d |_ ycc}wcc}w) Ngroupr)r.rlayerz`config.feat_extract_norm` is z), but has to be one of ['group', 'layer']FT)rr feat_extract_normrIrangenum_feat_extract_layersrr? ValueErrorr ModuleList conv_layersgradient_checkpointing_requires_grad)r,r-irr/s r0r zSEWFeatureEncoder.__init__s   # #w .0!DEINvOmOmpqOqIrIDE'Q?IK % % 0NSTZTrTrNst0!DtKt01I1I0JJst ==5&+#"Ius C" C'cJ|jD] }d|_ d|_yNF) parameters requires_gradrr,params r0_freeze_parametersz$SEWFeatureEncoder._freeze_parameterss(__& (E"'E  (#r1c|dddf}|jr|jrd|_|jD] }||} |S)NT)rtrainingrr)r, input_valuesr5 conv_layers r0r6zSEWFeatureEncoder.forwardsP$QW-    4==*.M '** 6J&}5M 6r1)r9r:r;__doc__r rr6r<r=s@r0rrs8#"$ r1rmodulequerykeyvalueattention_maskscalingdropout head_maskc ||jddz}tj||jdd|z} || |z} tj j | d} || |jddddz} tj j| ||j} tj| |} | jddj} | | fS)NrFrSrr[r)pr) rytorchmatmulrGr functionalsoftmaxviewrr contiguous) rrrrrrrrkwargs attn_weights attn_outputs r0eager_attention_forwardrs**R.D(<<s}}Q':;gEL!#n4 ==((2(>L#innQAq&AA ==((6??([L,,|U3K''1-88:K  $$r1cHeZdZdZ ddededededededeeffd Z dd e jd ee jd ee jd ee jdeede e dee jee jeee jffdZxZS) SEWAttentionz=Multi-headed attention from 'Attention Is All You Need' paper embed_dim num_headsr is_decoderr is_causalr-c t|||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ tj||||_ tj||||_tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)r)rr rrrhead_dimr-rrrrrruk_projv_projq_projout_proj) r,rrrrrrr-r/s r0r zSEWAttention.__init__ s "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 94@ ii 94@ ii 94@  )YTB r1r5key_value_statesrlayer_head_maskoutput_attentionsrreturnc |du}|jdd\}} |r|jdn| } || d|jf} || d|jf} |j|j| j dd} |r|n|}|j |j| j dd}|j |j| j dd}t}|jjdk7rt|jj}||| |||f|jsdn |j|j||d|\}}|j|| dj}|j!|}||dfS)z#Input shape: Batch x Time x ChannelNrFrrSeager)rrrr)shaperrrrGrrrr-_attn_implementationrrrrrzrr)r,r5rrrrris_cross_attentionr{r~r| q_input_shapekv_input_shape query_statescurrent_states key_states value_statesattention_interfacerrs r0r6zSEWAttention.forward(s.T9%**3B/ W/A"((+wgr4==9 wDMM:7t{{=166 FPPQRTUV -?)]5T[[055~FPPQRTUV 7t{{>277HRRSTVWX (? ;; + +w 6"9$++:Z:Z"[ $7      % $}}C$,,LL/% %  % ! \"))#w;FFH mmK0 L$..r1)rFTFN)NNNF)r9r:r;rintfloatboolrrr rTensorrr tupler6r<r=s@r0rrsG  &*CCC C  C  CC#CD481526,1 3/||3/#5<<03/!. 3/ "%,,/ 3/ $D> 3/-.3/ u||Xell3XeELL>Q5RR S3/r1rc$eZdZfdZdZxZS)SEWFeedForwardct|tj|j|_tj |j|j|_ t|jtrt|j|_n|j|_tj |j|j|_tj|j |_yr3)rr rDropoutactivation_dropoutintermediate_dropoutrur]intermediate_sizeintermediate_dense isinstance hidden_actstrr intermediate_act_fn output_densehidden_dropoutoutput_dropoutrws r0r zSEWFeedForward.__init___s $&JJv/H/H$I!"$))F,>,>@X@X"Y f'' -'-f.?.?'@D $'-'8'8D $IIf&>&>@R@RS jj)>)>?r1c|j|}|j|}|j|}|j|}|j |}|Sr3)rrrrrr4s r0r6zSEWFeedForward.forwardlsX// > 00? 11-@ ))-8 ++M: r1r8r=s@r0rr^s @r1rc&eZdZfdZddZxZS)SEWEncoderLayerct|t|j|j|j d||_tj|j|_ tj|j|j|_ t||_tj|j|j|_y)NF)rrrrr-eps)rr rr]num_attention_headsattention_dropout attentionrrrrrBlayer_norm_epsrCr feed_forwardfinal_layer_normrws r0r zSEWEncoderLayer.__init__ws %((00,,  zz&"7"78 ,,v'9'9v?T?TU*62 " V-?-?VEZEZ [r1c|}|j|||\}}}|j|}||z}|j|}||j|z}|j |}|f}|r||fz }|S)Nrr)rrrCrr)r,r5rr attn_residualr_outputss r0r6zSEWEncoderLayer.forwards% )- .L]*8* & |Q ]3 % 5  6 %(9(9-(HH --m<  "    &Gr1rr8r=s@r0rrvs \r1rc.eZdZfdZ ddZxZS) SEWEncoderct|||_t||_t j |j|j|_t j|j|j|_ t j|j|_t j t#|j$Dcgc] }t'|c}|_t+||_d|_ycc}w)NrF)rr r-rQpos_conv_embedr AvgPool1dr`poolrBr]rrCrrrrrnum_hidden_layersrlayersrsupsampler)r,r-rr/s r0r zSEWEncoder.__init__s  8@LL!6!68M8MN ,,v'9'9v?T?TUzz&"7"78 mmeFLdLdFe$f_V%<$fg %f- &+#%gsDc 2|rdnd}|rdnd}||jdjdd|jd}|jjdk(rd||<|d|vr|nd}ngd||<|j j d} | |jjz} |jd|jjz} tjd| | jjddj| jdd} | | jddkj }d |ddddddfj|j z }|tj|jj z}|j|jdd|jd|jd}|jd} |j#dd}|j%|}|j'|}t!|j)d|j)d}|d d|f|d d|fz}|j#dd}|j+|}|j-|}t/xs t1|}|j2D]j}|r||fz}tj4g}|j6xr||jj8k}|r|r|||| }|d}|rd }|sb|dfz}l|r||fz}|j;|}|jd| kr4t<j>jA|ddd| |jdz f}|stCd|||fDStE|||S)NrFrrSflash_attention_2rrdevice?dtype.rNNc3&K|] }|| ywr3r).0vs r0 z%SEWEncoder.forward..smq_`_lmslast_hidden_stater5 attentions)# unsqueezerepeatrr-rlongsumr`rarangerrexpandtorfinfominrGrrryrCrr r rrandr layerdroprrrpadrr)r,r5rroutput_hidden_states return_dictall_hidden_statesall_self_attentionsexpand_attention_mask input_lengthsoutput_lengthsmax_encoder_length attention_idsn_input_timestepsposition_embeddingspooled_hidden_states min_length synced_gpusrdropout_probabilityskip_the_layer layer_outputss r0r6zSEWEncoder.forwards#7BD$5b4  %$2$<$Q>Q2R2V2V!V!/!6!6"((+Q0D0DR0H.J^J^_aJb"*//2%//15 "11-@#yy7,11"57K7P7PQS7TU ,S+:+-=>ATUXZe[eZeUeAff %//15  6  ]3 02R6LT6R [[ PE#$58H$H!#(**R. !]]Z/BT[[EZEZ/ZN![ %!.Te! !.a 0  ,  &9]1=M'>q'AFKKD[D['["\]]   GG  fkk.. 2  * MM   & &CT[[5R5R & S r|| < = KK   " " $ MM   $ $S )  *)+ 6:.76:3N"::FOOV__;]mn:oD// 0B0BCDD#::6==XY:ZD// 0B0BCDD'' (:(:; fryy"))4 5&++:Q KK   " " $;R 5DDDDs4L5(4M5L>M rcd}t|jj|jjD]\}}||||}|S)zH Computes the output length of the convolutional layers c>tj||z |ddzS)Nfloor) rounding_moder)rdiv) input_lengthrrs r0_conv_out_lengthzMSEWPreTrainedModel._get_feat_extract_output_lengths.._conv_out_length/s"99\K7wWZ[[ [r1)zipr-r%r&)r,rr@rrs r0 _get_feat_extract_output_lengthsz3SEWPreTrainedModel._get_feat_extract_output_lengths*sQ  \ $'t{{'>'> @W@W#X Q K,]KPM Qr1feature_vector_lengthrc|j|jdjtj}|j d}tj ||f|j|j}d|tj|j d|j|dz f<|jdgjdjdgj}|S)NrFr)rrrr) rBrrrr rzerosrrrflipcumsumr)r,rCrr batch_sizes r0"_get_feature_vector_attention_maskz5SEWPreTrainedModel._get_feature_vector_attention_mask9s>>~?Q?QRT?UVYYZ_ZdZde#))!,  . /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOr1N)r9r:r;r__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnr9rr LongTensorrrBrIrr1r0r)r)sg $O&*#N%@ eEDTDTVYDY>Z   ]b]m]m r1r)r mask_prob mask_length min_masksrc|\}dkr tdkDrtdddtjjdj fd}|-|j j djnt|Dcgc]}c}}tj|ft } g} |} | d k(r| S|D]} || } tjjtj| dz z | d }t|d k(rdz }n|d }tj|tj| | z tj |zg}| j#|tj$| } tj&| dddddf|| f} | j)|| z} tjddddf}tj&||| fj)|| z}| |z} | j+dz kDrdz | | dz kD<tj,| | dd| Scc}w) an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `ct|zz z}t|}|zkDrz}|dz z |krt|dz z d}|S)z;Given input length, compute how many spans should be maskedrr)rmax)r?num_masked_spanepsilonrSrRrTsequence_lengths r0compute_num_masked_spanz6_compute_mask_indices..compute_num_masked_spanlsoi,6DwNOoy9 [ (? :-}G[G[chcmcm n /3/E/E/H/HI\I\/]M+ , ;; ( (1 ,#8[)++77 KK;;++<< $ $)<<0D]MaMainisis#t #74#@#G#GO]_#` 23M. /r1rrrrrcZ||n|jj}||n|jj}||n|jj}|j |}|j dd}|j |}|jr|j|}|j|}||j|jd|}|j||}|j|||||} | d}|s |f| ddzSt|| j| j S)a/ mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. NrrS)rrrrrrr)r-rruse_return_dictrvrGrCrwrxrzrIrrrrr5r ) r,rrrrrrextract_featuresr5encoder_outputss r0r6zSEWModel.forwardsU2C1N-TXT_T_TqTq$8$D $++JjJj &1%  %!DD]EXEXYZE[]klN00Rc0d ,, )/!5# ' (* !#oab&99 9+)77&11  r1rNNNNN)r9r:r;rr r FloatTensorrrQrrrrrrrr6r<r=s@r0rtrtsy.:>59 ,((,$E$5$56,!!1!12 ,\269=,0/3&*3 u||,3 !.3 $E$5$56 3 $D> 3 'tn 3 d^3  uo% &3 3 r1rtzk SEW Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). ) custom_introceZdZddeeffd ZdZdZdZdZ e ddee jdee jd ee d ee d ee d ee jd eeeffdZxZS) SEWForCTC target_langct||t||_t j |j |_||_|jtd|jdt|dr|jr |jn |j}t j ||j|_|j%y)a- target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`SEWForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `SEWForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration. add_adapter)rr rtr*rr final_dropoutrr vocab_sizerr/rbroutput_hidden_sizer]rulm_headr)r,r-rrr/s r0r zSEWForCTC.__init__As  F#zz&"6"67 &    $00@AHH *1)GFL^L^F % %djdvdv yy!3V5F5FG  r1c|j}|&t|jddtd|d|-t|jddtj dy||j |dyy)a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. Nadapter_attn_dimzCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T) force_load)rrr-rloggerinfo load_adapter)r,rs r0 tie_weightszSEWForCTC.tie_weights^s&&  "wt{{ D 'tn D d^ D &D  un$ %D D r1rz SEW Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. ceZdZfdZdZdZdZe d dee jdee jdee dee d ee d ee jd e e effd ZxZS)SEWForSequenceClassificationct||t|dr|jr t dt ||_|jdz}|jr0tjtj||z |_ tj|j|j |_tj|j |j$|_|j)y)NrzZSequence classification does not support the use of SEW adapters (config.add_adapter=True)r)rr rbrrrtr*ruse_weighted_layer_sumrr}rrf layer_weightsrur]classifier_proj_size projector num_labels classifierr)r,r- num_layersr/s r0r z%SEWForSequenceClassification.__init__s  6= )f.@.@l F#--1  ( (!#ejj.Dz.Q!RD 6#5#5v7R7RS))F$?$?ARARS r1cXtjdt|jy)z Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. rNrrs r0rz5SEWForSequenceClassification.freeze_feature_extractorrr1cL|jjjyrrrs r0rz3SEWForSequenceClassification.freeze_feature_encoderrr1cP|jjD] }d|_ yrrrs r0rz.SEWForSequenceClassification.freeze_base_modelrr1rrrrrrrc<||n|jj}|jjrdn|}|j|||||}|jjrr|t}t j |d}tjj|jd} || jdddzjd}n|d}|j|}||jd} n|j|j d|} | j#dj%dd|j d} d || <|jd| jdjddz } |j'| } d}|Ft)}|| jd|jj*|jd}|s| f|tdz}||f|zS|St-|| |j.|j0 S) a input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`SEWProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrrrrFrrSrr)r-rrr*rrstackrrrrrrrr,rIrr r rrrrr5r )r,rrrrrrrr5 norm_weights pooled_output padding_maskexpand_padding_maskrrloss_fctrs r0r6z$SEWForSequenceClassification.forward s.&1%->r1a-HHMMRSMTM#AJM}5  !)..1.5MBB=CVCVWXCY[ijL"."8"8"<"C"CAq-J]J]^_J`"a 25M.. /)--!-4|7G7GA7G7N7S7STVXY7ZZM/  ')HFKKDKK,B,BCV[[QS_UDY)F)G!HHF)-)9TGf$ Ev E'!//))   r1r)r9r:r;r rrrrrrrrrrrr6r<r=s@r0rrs" &8(26,0/3&*)-B u||,B !.B $D> B 'tn B d^ B &B  u.. /B B r1r)rrrtr))NrNrq)Ar0rtypingrrrnumpyr^rrtorch.nnr activationsr integrations.deepspeedr integrations.fsdpr modeling_flash_attention_utilsr modeling_layersr modeling_outputsrrrmodeling_utilsrrprocessing_utilsrrarrconfiguration_sewr get_loggerr9rrr?rIModulerQrkrsrrrrrrrrr)rrrQndarrayrrrtrrr__all__rr1r0rs, ,, %!@7B9YYF&,(   H %8*6660((VbiiBII,# #X $(,% II% <<% % << % U\\* % e_ %% %%