rh_ddlZddlmZmZmZddlZddlZddlm Z ddlm Z ddl m Z ddl mZddlmZddlmZmZdd lmZdd lmZdd lmZmZmZdd lmZmZdd lm Z ddl!m"Z"m#Z#m$Z$ddl%m&Z&e#rddl'm(Z(e$jRe*Z+Gdde jXZ-Gdde jXZ.GddeZ/GddeZ0GddeZ1Gdde jXZ2Gdde jXZ3 dId e jXd!ejhd"ejhd#ejhd$eejhd%ee5d&e5d'eejhfd(Z6Gd)d*e jXZ7Gd+d,e jXZ8Gd-d.eZ9Gd/d0e jXZ:Gd1d2e jXZ;Gd3d4eZ<Gd5d6e jXZ=e"Gd7d8eZ> dJd9e?e@e@fd:e5d;e@d$eejdZCe"Gd?d@e>ZDdZEe"dABGdCdDe>ZFe"dEBGdFdGe>ZGgdHZHy)KN)CallableOptionalUnion)CrossEntropyLoss)ACT2FN)is_deepspeed_zero3_enabled)is_fsdp_managed_module)_prepare_4d_attention_mask#_prepare_4d_attention_mask_for_sdpa)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputCausalLMOutputSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringis_torch_flex_attn_availablelogging) HubertConfig)make_flex_block_causal_maskc$eZdZfdZdZxZS)HubertPositionalConvEmbeddingct|tj|j|j|j |j dz|j |_d|_|jr&tj|j|_ntjj}ttjjdr$tjjj}tr(ddl}|j"j%|jj&d5||jdd|_dddt|jdrU|jjj&j(}|jjj&j*}n,|jj,}|jj.}|j"j1|||j"j1||n||jdd|_t3|j |_t6|j8|_y#1swY'xYw) N) kernel_sizepaddinggroups weight_normr modifier_rankweight)namedimparametrizations)super__init__nnConv1d hidden_sizenum_conv_pos_embeddingsnum_conv_pos_embedding_groupsconv batch_normconv_pos_batch_norm BatchNorm1dutilsr"hasattrr(r deepspeedzeroGatheredParametersr% original0 original1weight_gweight_vregister_external_parameterHubertSamePadLayerr rfeat_extract_activation activation)selfconfigr"r6r;r< __class__s }/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/hubert/modeling_hubert.pyr*z&HubertPositionalConvEmbedding.__init__3s II      6622a777     % % nnV-?-?@DO((..Krxx00-@ hh77CC )+ ^^66tyy7G7GWX6YM +DIIH! LDIM499&89#yy99@@JJH#yy99@@JJH#yy11H#yy11H::4J::4J' aH )&*H*HI  !?!?@MMs ?I??J c|jdd}|j|j|}|j|}|j|}|j |}|jdd}|S)Nrr) transposer1r0r r@rA hidden_statess rDforwardz%HubertPositionalConvEmbedding.forwardXsn%//15 ?? & OOM:M -0  ]3  6 %//15 __name__ __module__ __qualname__r*rI __classcell__rCs@rDrr2s#AJ rJrc$eZdZfdZdZxZS)r>cPt||dzdk(rd|_yd|_y)Nrrr)r)r*num_pad_remove)rAr.rCs rDr*zHubertSamePadLayer.__init__es) #:Q#>!#CarJcV|jdkDr|ddddd|j f}|SNr)rSrGs rDrIzHubertSamePadLayer.forwardis6    ")!Q0F43F3F2F0F*FGMrJrKrPs@rDr>r>dsKrJr>c&eZdZdfd ZdZxZS)HubertNoLayerNormConvLayercdt||dkDr|j|dz nd|_|j||_t j |j|j|j||j||j|_ t|j|_ y)Nrrrstridebias)r)r*conv_dim in_conv_dim out_conv_dimr+r, conv_kernel conv_stride conv_biasr0rr?r@rArBlayer_idrCs rDr*z#HubertNoLayerNormConvLayer.__init__ps FDYDYZDO))FOOB$79K9KLzz&":":; rJc|jr|j|}|j|}|j|}|Sre)rrlrrrGs rDrIzHubertFeatureProjection.forwards;  $ $ OOM:M 6  ]3 rJrKrPs@rDrrs <rJrmodulequerykeyvalueattention_maskscalingr head_maskc ||jddz}tj||jdd|z} || |z} tj j | d} || |jddddz} tj j| ||j} tj| |} | jddj} | | fS)Nrorrr'r)pr) sizetorchmatmulrFr+ functionalsoftmaxviewrr contiguous) rrrrrrrrkwargs attn_weights attn_outputs rDeager_attention_forwardrs**R.D(<<s}}Q':;gEL!#n4 ==((2(>L#innQAq&AA ==((6??([L,,|U3K''1-88:K  $$rJcHeZdZdZ ddededededededeeffd Z dd e jd ee jd ee jd ee jdeede e dee jee jeee jffdZxZS)HubertAttentionz=Multi-headed attention from 'Attention Is All You Need' paper embed_dim num_headsr is_decoderr[ is_causalrBc t|||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ tj||||_ tj||||_tj||||_tj||||_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r)r[)r)r*rrrhead_dimrBrrrrr+rk_projv_projq_projout_proj) rArrrrr[rrBrCs rDr*zHubertAttention.__init__s "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $"ii 94@ ii 94@ ii 94@  )YTB rJrHkey_value_statesrlayer_head_maskoutput_attentionsrreturnc |du}|jdd\}} |r|jdn| } || d|jf} || d|jf} |j|j| j dd} |r|n|}|j |j| j dd}|j |j| j dd}t}|jjdk7rt|jj}||| |||f|jsdn |j|j||d|\}}|j|| dj}|j!|}||dfS)z#Input shape: Batch x Time x ChannelNrorreager)rrrr)shaperrrrFrrrrB_attn_implementationrrrrreshaperr)rArHrrrrris_cross_attentionbsztgt_lensrc_len q_input_shapekv_input_shape query_statescurrent_states key_states value_statesattention_interfacerrs rDrIzHubertAttention.forward/s.T9%**3B/ W/A"((+wgr4==9 wDMM:7t{{=166 FPPQRTUV -?)]5T[[055~FPPQRTUV 7t{{>277HRRSTVWX (? ;; + +w 6"9$++:Z:Z"[ $7      % $}}C$,,LL/% %  % ! \"))#w;FFH mmK0 L$..rJ)rFTFN)NNNF)rLrMrNrintfloatboolrrr*rTensorrr tuplerIrOrPs@rDrr sG  )-CCC C  C  CC&CD481526,1 3/||3/#5<<03/!. 3/ "%,,/ 3/ $D> 3/-.3/ u||Xell3XeELL>Q5RR S3/rJrc$eZdZfdZdZxZS)HubertFeedForwardct|tj|j|_tj |j|j|_ t|jtrt|j|_n|j|_tj |j|j|_tj|j |_yre)r)r*r+ractivation_dropoutintermediate_dropoutrr-intermediate_sizeintermediate_dense isinstance hidden_actstrrintermediate_act_fn output_densehidden_dropoutoutput_dropoutrs rDr*zHubertFeedForward.__init__fs $&JJv/H/H$I!"$))F,>,>@X@X"Y f'' -'-f.?.?'@D $'-'8'8D $IIf&>&>@R@RS jj)>)>?rJc|j|}|j|}|j|}|j|}|j |}|Sre)rrrrrrGs rDrIzHubertFeedForward.forwardssX// > 00? 11-@ ))-8 ++M: rJrKrPs@rDrres @rJrc&eZdZfdZddZxZS)HubertEncoderLayerct|t|j|j|j d||_tj|j|_ tj|j|j|_ t||_tj|j|j|_y)NFrrrrrBr)r)r*rr-num_attention_headsattention_dropout attentionr+rrrrkrrlr feed_forwardfinal_layer_normrs rDr*zHubertEncoderLayer.__init__~s (((00,,  zz&"7"78 ,,v'9'9v?T?TU-f5 " V-?-?VEZEZ [rJc|}|j|||\}}}|j|}||z}|j|}||j|z}|j |}|f}|r||fz }|SNrr)rrrlrrrArHrr attn_residualr_outputss rDrIzHubertEncoderLayer.forwards% )- .L]*8* & |Q ]3 % 5  6 %(9(9-(HH --m<  "    &GrJrrKrPs@rDrr}s \rJrc eZdZfdZ d dej deejdededef dZ de ejdfd ejfd Z xZ S) HubertEncoderct|||_t||_t j |j|j|_ t j|j|_ t jt|jDcgc] }t!|c}|_d|_ycc}wNrF)r)r*rBrpos_conv_embedr+rkr-rrlrrrrr~num_hidden_layersrlayersrrArBrrCs rDr*zHubertEncoder.__init__s  ;FC,,v'9'9v?T?TUzz&"7"78 mmvOgOgIh$iA%7%?$ij &+#%j!CNrHrroutput_hidden_states return_dictc|rdnd}|rdnd}|5|jdjdd|jd}d||<|j||}|j |} || z}|j |}|j |}txs t|} |jD]j} |r||fz}tjg} |jxr| |jjk} | r| r| |||}|d}| rd}|sb|dfz}l|r||fz}|std|||fDSt!||| S) NrorrrrNNc3&K|] }|| ywrer.0vs rD z(HubertEncoder.forward..mq_`_lmlast_hidden_staterH attentions) unsqueezerepeatr_update_full_maskrrlrr r rrrandrrB layerdroprrrArHrrrrall_hidden_statesall_self_attentionsexpand_attention_maskposition_embeddings synced_gpusr|dropout_probabilityskip_the_layer layer_outputss rDrIzHubertEncoder.forwards#7BD$5b4  %$2$<$eZdZfdZdej fdZxZS)HubertAttnAdapterLayerct||j|_|j|_t j|j |_t j|j |j|_ t j|_ t j|j|j |_ y)z Implements adapter modules directly with 3D tensor weight as parameters and without using ModuleList to speed up training throughput. N)r)r*adapter_attn_dim input_dimr- hidden_dimr+rknormrlinear_1ReLUact_fnlinear_2rs rDr*zHubertAttnAdapterLayer.__init__s 00 ,,LL1  $//4>>B ggi  $..$//B rJrHc|j|}|j|}|j|}|j|}|Sre)r3r4r6r7rGs rDrIzHubertAttnAdapterLayer.forward s@ -0  m4  M2  m4 rJ)rLrMrNr*r FloatTensorrIrOrPs@rDr.r.s CU%6%6rJr.cfeZdZfdZ ddej deej defdZxZ S)!HubertEncoderLayerStableLayerNormct|t|j|j|j d||_tj|j|_ tj|j|j|_ t||_tj|j|j|_t#|ddt%||_yd|_y)NFrrr0)r)r*rr-rrrr+rrrrkrrlrrrgetattrr. adapter_layerrs rDr*z*HubertEncoderLayerStableLayerNorm.__init__s (((00,,  zz&"7"78 ,,v'9'9v?T?TU-f5 " V-?-?VEZEZ [ 6-t 4 @!7!?D !%D rJrHrrc$|}|j|}|j|||\}}}|j|}||z}||j|j |z}|j ||j |z}|f}|r||fz }|Sr)rlrrrrr>rs rDrIz)HubertEncoderLayerStableLayerNorm.forward+s &  6 )- .L]*8* & |Q ]3 % 5 %(9(9$:O:OP]:^(__    ))D,>,>},MMM "    &GrJr) rLrMrNr*rrrrrIrOrPs@rDr;r;s>&,26"' ||!. rJr;cpeZdZfdZ ddZdeejdfdejfdZxZ S)HubertEncoderStableLayerNormct|||_t||_t j |j|j|_ t j|j|_ t jt|jDcgc] }t!|c}|_d|_ycc}wr)r)r*rBrrr+rkr-rrlrrrrr~rr;rrrs rDr*z%HubertEncoderStableLayerNorm.__init__Fs  ;FC,,v'9'9v?T?TUzz&"7"78 mm@EfF^F^@_ `1 .v 6 `  ',# arNc|rdnd}|rdnd}|5|jdjdd|jd}d||<|j||}|j |} || z}|j |}t xs t|} |jD]j} |r||fz}tjg} |jxr| |jjk} | r| r| |||}|d}| rd}|sb|dfz}l|j|}|r||fz}|std|||fDSt!||| S) Nrrorrrrrc3&K|] }|| ywrerr s rDr z7HubertEncoderStableLayerNorm.forward..rrr)rrrrrrr r rrrrrBrrlrrrs rDrIz$HubertEncoderStableLayerNorm.forwardQs#7BD$5b4  %$2$<$tj||z |ddzS)Nfloor) rounding_moder)rdiv) input_lengthrrZs rD_conv_out_lengthzPHubertPreTrainedModel._get_feat_extract_output_lengths.._conv_out_lengths"99\K7wWZ[[ [rJ)ziprBr_r`)rArYr`rrZs rD _get_feat_extract_output_lengthsz6HubertPreTrainedModel._get_feat_extract_output_lengthssQ  \ $'t{{'>'> @W@W#X Q K,]KPM QrJfeature_vector_lengthrc|j|jdjtj}|j d}tj ||f|j|j}d|tj|j d|j|dz f<|jdgjdjdgj}|S)Nror)r(devicer)re) rbsumtorlongrzerosr(rearangeflipcumsumr)rArcroutput_lengths batch_sizes rD"_get_feature_vector_attention_maskz8HubertPreTrainedModel._get_feature_vector_attention_masks>>~?Q?QRT?UVYYZ_ZdZde#))!,  . /~7K7KTbTiTi uv ^%9%9!%<^EZEZ[]kno]opq',,bT299"=BBB4HMMOrJN)rLrMrNr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointing_supports_flash_attn_supports_sdpa_supports_flex_attnrXrr LongTensorrrbrorrJrDrGrGsh  $O&*#N[B eEDTDTVYDY>Z   ]b]m]m rJrGr mask_prob mask_length min_masksrc|\}dkr tdkDrtdddtjjdj fd}|-|j j djnt|Dcgc]}c}}tj|ft } g} |} | d k(r| S|D]} || } tjjtj| dz z | d }t|d k(rdz }n|d }tj|tj| | z tj |zg}| j#|tj$| } tj&| dddddf|| f} | j)|| z} tjddddf}tj&||| fj)|| z}| |z} | j+dz kDrdz | | dz kD<tj,| | dd| Scc}w) an Computes random mask spans for a given shape. Used to implement [SpecAugment: A Simple Data Augmentation Method for ASR](https://huggingface.co/papers/1904.08779). Note that this method is not optimized to run on TPU and should be run on CPU as part of the preprocessing during training. Args: shape: The shape for which to compute masks. This should be of a tuple of size 2 where the first element is the batch size and the second element is the length of the axis to span. mask_prob: The percentage of the whole axis (between 0 and 1) which will be masked. The number of independently generated mask spans of length `mask_length` is computed by `mask_prob*shape[1]/mask_length`. Note that due to overlaps, `mask_prob` is an upper bound and the actual percentage will be smaller. mask_length: size of the mask min_masks: minimum number of masked spans attention_mask: A (right-padded) attention mask which independently shortens the feature axis of each batch dimension. rz&`mask_length` has to be bigger than 0.zO`mask_length` has to be smaller than `sequence_length`, but got `mask_length`: z and `sequence_length`: `ct|zz z}t|}|zkDrz}|dz z |krt|dz z d}|S)z;Given input length, compute how many spans should be maskedrr)rmax)r_num_masked_spanepsilonryrxrzsequence_lengths rDcompute_num_masked_spanz6_compute_mask_indices..compute_num_masked_spansoi,6DwNOoy9 [ (? :-}G[G[chcmcm n /3/E/E/H/HI\I\/]M+ , ;; ( (1 ,#8[)++77 KK;;++<< $ $)<<0D]MaMainisis#t #74#@#G#GO]_#` 23M. /rJrrrrrc||n|jj}||n|jj}||n|jj}|j |}|j dd}||j |jd|}|j|}|j||}|j|||||} | d}|s |f| ddzSt|| j| jS)a1 mask_time_indices (`torch.BoolTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices to mask extracted features for contrastive loss. When in training mode, model learns to predict masked extracted features in *config.proj_codevector_dim* space. Example: ```python >>> from transformers import AutoProcessor, HubertModel >>> from datasets import load_dataset >>> processor = AutoProcessor.from_pretrained("facebook/hubert-large-ls960-ft") >>> model = HubertModel.from_pretrained("facebook/hubert-large-ls960-ft") >>> def map_to_array(example): ... example["speech"] = example["audio"]["array"] ... return example >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> ds = ds.map(map_to_array) >>> input_values = processor(ds["speech"][0], return_tensors="pt").input_values # Batch size 1 >>> hidden_states = model(input_values).last_hidden_state ```Nrr)rrrrrrr)rBrruse_return_dictrrFrorrrrrrHr) rArrrrrrextract_featuresrHencoder_outputss rDrIzHubertModel.forwards,H2C1N-TXT_T_TqTq$8$D $++JjJj &1%59 ,((,$E$5$56,!!1!12 ,\269=,0/3&*D u||,D !.D $E$5$56 D $D> D 'tn D d^D  uo% &D D rJrUzn Hubert Model with a `language modeling` head on top for Connectionist Temporal Classification (CTC). ) custom_introceZdZddeeffd ZdZdZdZdZ e ddee jdee jd ee d ee d ee d ee jd eeeffdZxZS) HubertForCTC target_langct||t||_t j |j |_||_|jtd|jdt|dr|jr |jn |j}t j ||j|_|j%y)a0 target_lang (`str`, *optional*): Language id of adapter weights. Adapter weights are stored in the format adapter..safetensors or adapter..bin. Only relevant when using an instance of [`HubertForCTC`] with adapters. Uses 'eng' by default. NzYou are trying to instantiate z with a configuration that does not define the vocabulary size of the language model head. Please instantiate the model as follows: `HubertForCTC.from_pretrained(..., vocab_size=vocab_size)`. or define `vocab_size` of your model's configuration. add_adapter)r)r*rUrHr+r final_dropoutrr vocab_sizerrCr5routput_hidden_sizer-rlm_headr)rArBrrrCs rDr*zHubertForCTC.__init__s  !&) zz&"6"67 &    $00@AHH *1)GFL^L^F % %djdvdv yy!3V5F5FG  rJc|j}|&t|jddtd|d|-t|jddtj dy||j |dyy)a' This method overwrites [`~PreTrainedModel.tie_weights`] so that adapter weights can be correctly loaded when passing `target_lang=...` to `from_pretrained(...)`. This method is **not** supposed to be called by the user and is prone to be changed in the future. Nr0zCannot pass `target_lang`: z- if `config.adapter_attn_dim` is not defined.z)By default `target_lang` is set to 'eng'.T) force_load)rr=rBrloggerinfo load_adapter)rArs rD tie_weightszHubertForCTC.tie_weightss&&  "wt{{ D 'tn D d^ D &D  un$ %D D rJrz Hubert Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. ceZdZfdZdZdZdZe d dee jdee jdee dee d ee d ee jd e e effd ZxZS)rWct||t|dr|jr t dt ||_|jdz}|jr0tjtj||z |_ tj|j|j |_tj|j |j$|_|j)y)Nrz]Sequence classification does not support the use of Hubert adapters (config.add_adapter=True)r)r)r*r5rrrUrHruse_weighted_layer_sumr+rrrrMrr-classifier_proj_size projector num_labels classifierr)rArB num_layersrCs rDr*z(HubertForSequenceClassification.__init__s  6= )f.@.@o "&) --1  ( (!#ejj.Dz.Q!RD 6#5#5v7R7RS))F$?$?ARARS rJcXtjdt|jy)z Calling this function will disable the gradient computation for the feature encoder so that its parameters will not be updated during training. rNrrs rDrz8HubertForSequenceClassification.freeze_feature_extractorrrJcL|jjjyrrrs rDrz6HubertForSequenceClassification.freeze_feature_encoderrrJcP|jjD] }d|_ yrrrs rDrz1HubertForSequenceClassification.freeze_base_modelrrJrrrrrrrc<||n|jj}|jjrdn|}|j|||||}|jjrr|t}t j |d}tjj|jd} || jdddzjd}n|d}|j|}||jd} n|j|j d|} | j#dj%dd|j d} d || <|jd| jdjddz } |j'| } d}|Ft)}|| jd|jj*|jd}|s| f|tdz}||f|zS|St-|| |j.|j0 S) a  input_values (`torch.FloatTensor` of shape `(batch_size, sequence_length)`): Float values of input raw speech waveform. Values can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoProcessor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. See [`HubertProcessor.__call__`] for details. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NTrrrrorrrr)rBrrrHrrstackr+rrrMrrfrrJrorrrrrrrrHr)rArrrrrrrrH norm_weights pooled_output padding_maskexpand_padding_maskrrloss_fctrs rDrIz'HubertForSequenceClassification.forwards.&1%->r1a-HHMMRSMTM#AJM}5  !)..1.5MBB=CVCVWXCY[ijL"."8"8"<"C"CAq-J]J]^_J`"a 25M.. /)--!-4|7G7GA7G7N7S7STVXY7ZZM/  ')HFKKDKK,B,BCV[[QS_UDY)F)G!HHF)-)9TGf$ Ev E'!//))   rJr)rLrMrNr*rrrrrrrrrrrrIrOrPs@rDrWrWs" &;(26,0/3&*)-B u||,B !.B $D> B 'tn B d^ B &B  u.. /B B rJrW)rrWrUrG)NrNrU)Irtypingrrrnumpyrrtorch.nnr+r activationsrintegrations.deepspeedr integrations.fsdpr modeling_attn_mask_utilsr r modeling_flash_attention_utilsr modeling_layersrmodeling_outputsrrrmodeling_utilsrrprocessing_utilsrr4rrrconfiguration_hubertrintegrations.flex_attentionr get_loggerrLrModulerr>rWrhrqryrrrrrrrrr.r;rArGrrrwndarrayrrUrrrW__all__rrJrDrs,,, %!@7gB9YYF&JJ. !J   H %/BII/d!;*9690#299#Lbii0 $(,% II% <<% % << % U\\* % e_ %% %%