rh3"6dZddlZddlmZmZmZddlZddlZddl Zddlm Z ddl m Z ddl mZddlmZmZmZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZmZm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)ddl*m+Z+ddl,m-Z-e)j\e/Z0dZ1dHde2de2de3dejhfdZ5dejhde2de2fdZ6 dIde7e2e2fde3de2d eejpd!e2dejrf d"Z:Gd#d$e jvZ< dJd%e jzd&ejhd'ejhd(ejhd eejhd)ee3d*e3d+eejhfd,Z>Gd-d.e jzZ?Gd/d0eZ@Gd1d2eZAe(Gd3d4e$ZBGd5d6eBZCGd7d8eBZDe(Gd9d:eBZEe(d;<Gd=d>e-eBZFGd?d@eBZGe(dA<GdBdCeBeZHe(dD<GdEdFeBZIgdGZJy)KzPyTorch Whisper model.N)CallableOptionalUnion)nn)CrossEntropyLoss)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentions!CausalLMOutputWithCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutputSequenceClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)auto_docstringlogging) WhisperConfig)WhisperGenerationMixinlengthchannels max_timescalereturnc|dzdk7rtd|dtj||dzdz z }tj| tj |dzz}tj |j dd|j ddz}tj|j|jgdS)z*Returns sinusoids for positional embeddingrzVNumber of channels has to be divisible by 2 for sinusoidal positional embeddings, got z channels.rdim) ValueErrormathlogtorchexparangeviewcatsincos)rr r!log_timescale_incrementinv_timescales scaled_times /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/whisper/modeling_whisper.py sinusoidsr66s!|qdemdnnx y  #hh}5Q9JKYY 77%,,xST}:UUVN,,v&++B2^5H5HB5OOK 99koo'):; CC input_ids pad_token_iddecoder_start_token_idc|j|j}|ddddfj|ddddf<||dddf<| td|j |dk(||S)z1 Shift input ids one token to the right. Nr%rrz1self.model.config.pad_token_id has to be defined.i) new_zerosshapecloner( masked_fill_)r8r9r:shifted_input_idss r5shift_tokens_rightrACs}"++IOO<(CRC0668ae4adLMM""#4#.compute_num_masked_spanzsoi,6DwNOoy9 [ (? :-#nQ1o " o5M&NN ==((2(>L#innQAq&AA ==((6??([L,,|U3K''1-88:K  $$r7czeZdZdZ ddededededededeed eeffd Z dd e jd ee jd ee dee jdee jdedee jde edee jee jeee jffdZxZS)WhisperAttentionz=Multi-headed attention from 'Attention Is All You Need' paper embed_dim num_headsr is_decoderbias is_causal layer_idxconfigc zt |||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ |/|r-tjd|jjd||_t!j"||d|_t!j"||||_t!j"||||_t!j"||||_y) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).rzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.Fr)rwrxrrrhead_dimrr(rrrlogger warning_oncerzrrrLineark_projv_projq_projout_proj) ryrrrrrrrrrzs r5rxzWhisperAttention.__init__s  "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $"     *4>>+B+B*CD,,  #ii 95A ii 94@ ii 94@  )YTB r7 hidden_stateskey_value_statespast_key_valuerDlayer_head_maskoutput_attentionscache_positionrr"c T|du} |jdd\} } | | d|jf} |j||jz} | j| } | j ddj } |it|trY|jj|j}| r&d|j|j<|j}n |j}||n|}| rK|rIrG|j|jj}|j|jj }n|j#|j | d|j$|j}|j'|j | d|j$|j}|j ddj }|j ddj }|)| s|nd}|j)|||jd|i\}}t*}|j,j.dk7rt0|j,j.}||| |||f|j2sdn |j4d ||d |\}}|j7| | dj }|j9|}||fS) z#Input shape: Batch x Time x ChannelNr%rr$Treager?)rrrr)r=rrrr.rr isinstancer is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesrrrupdaterr_attn_implementationrrrrdr)ryrrrrDrrrris_cross_attentionbsztgt_len q_input_shape query_statesrcurrent_states key_states value_statesattention_interfacerrs r5rzWhisperAttention.forwards".T9%**3B/ Wgr4==9 {{=1DLL@ (|((-8 #--a3>>@   %*^EX*Y'2266t~~FJ!<@))$..9!/!E!E!/!D!D.>-I)} .Z'..t~~>CCJ)00@GGL^499#r4>>SWS`S`aJ;;~6;;CT^^UYUbUbcL#--a3>>@J'11!Q7BBDL)7It+9+@+@ dnn?OQ_>`,( L)@ ;; + +w 6"9$++:Z:Z"[ $7      % $}}C$,,/% %  % ! \"))#w;FFH mmK0 L((r7)rFTFNN)NNNNFN)rrr__doc__rIfloatr[rrrxr+Tensorr rrtuplerrrs@r5rrsTG  #'*.&C&C&C &C  &C  &C&CC=&C'&CV48*.1526"'15P)||P)#5<<0P)! P) !. P) "%,,/ P) P)!.P)-.P) u||Xell3XeELL>Q5RR SP)r7rc eZdZdeffd Z d dej dej dej dedej f dZxZ S) WhisperEncoderLayerrcht||j|_t |j|j |j ||_tj|j|_ |j|_ t|j|_|j|_tj |j|j"|_tj |j"|j|_tj|j|_y)N)rrrr)rwrxd_modelrrencoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normrr activation_function activation_fnactivation_dropoutrencoder_ffn_dimfc1fc2final_layer_normryrrzs r5rxzWhisperEncoderLayer.__init__ts )nn44,,   %'LL$@!~~ #F$>$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r7rrDrrr"c|}|j|}|j||||\}}tjj ||j|j }||z}|}|j |}|j|j|}tjj ||j|j }|j|}tjj ||j|j }||z}|jtjk(rEtj|jjdz }tj || |}||fS)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrDrrri)minrJ)rrrrrrrrrrrrPr+float16finforJclamp)ryrrDrrresidualr clamp_values r5rzWhisperEncoderLayer.forwardsS$!11-@ &*nn')+/ '5' # |  --mt||VZVcVc-d  =0  --m< **488M+BC  --mt?V?Vaeanan-o /  --mt||VZVcVc-d  =0   %-- /++m&9&9:>>EK!KK K<[YMl**r7)F) rrrrrxr+rr[rrrs@r5rrssY=}=.#( )+||)+ )+ )+  )+  )+r7rcNeZdZddedeeffd Z ddejdeejdeejdeejdeejd eejd ee d ee d ee d eejdejfdZ xZ S)WhisperDecoderLayerrrc t||j|_t |j|j |j dd|||_|j|_t|j|_ |j|_ tj|j|_t |j|j |j d|||_tj|j|_tj$|j|j&|_tj$|j&|j|_tj|j|_y)NT)rrrrrrr)rrrr)rwrxrrrdecoder_attention_headsrrrr rrrrrr encoder_attnencoder_attn_layer_normrdecoder_ffn_dimrrrryrrrzs r5rxzWhisperDecoderLayer.__init__s  )nn44,, ~~ #F$>$>?"(";";$&LL$@!, NN  * *,,  (*||DNN'C$99T^^V-C-CD99V33T^^D " T^^ <r7rrDencoder_hidden_statesencoder_attention_maskrcross_attn_layer_head_maskrr use_cacherr"c 0|} |j|}|j|||||| \}} tjj ||j|j }| |z}d} |h|} |j |}|j||||||\}} tjj ||j|j }| |z}|} |j|}|j|j|}tjj ||j|j }|j|}tjj ||j|j }| |z}|f}|r|| | fz }|S)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. encoder_hidden_states (`torch.FloatTensor`): cross attention input to the layer of shape `(batch, seq_len, embed_dim)` encoder_attention_mask (`torch.FloatTensor`): encoder attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. cross_attn_layer_head_mask (`torch.FloatTensor`): mask for cross-attention heads in a given layer of size `(decoder_attention_heads,)`. past_key_value (`Tuple(torch.FloatTensor)`): cached past key and value projection states output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )rrrDrrrrN)rrrDrrr) rrrrrrrrrrrrr)ryrrDrrrrrrrrrself_attn_weightscross_attn_weightsoutputss r5rzWhisperDecoderLayer.forwards>!11-@ ,0>>'))+/) ,:, ( ( --mt||VZVcVc-d  =0 " ,$H 88GM040A0A+!65 :-"3 1B1 -M-MM11-4< >Gr7rv) NNNNNNFTN)rrrrrrIrxr+rr r[ LongTensorrrrs@r5rrs=}=#=D268<9=26=A8<,1$(59L||L!.L ( 5 L !) 6 L "%,,/ L%-U\\$:L!!45L$D>LD>L!!1!12L Lr7rcbeZdZUeed<dZdZdZddgZdZ dZ dZ dZ dZ dejfd Zy ) WhisperPreTrainedModelrmodelinput_featuresTrrc|jj}t|tjtj frY|j jjd||j%|jjjyyt|tjrf|j jjd||j2|j j|jjyyt|tjrJ|j jjd|jjjyt|trJ|j j j#t%|j j j&yt|t(rW|jj*r@|j,jjd|jj.dzz yyy)Nr)meanstdrr)rinit_stdrrrConv1dr}datanormal_rzero_ Embeddingrtrfill_WhisperEncoderembed_positionscopy_r6r=WhisperForAudioClassificationuse_weighted_layer_sum layer_weightsnum_hidden_layers)ryrrs r5 _init_weightsz$WhisperPreTrainedModel._init_weights.skk"" fryy"))4 5 MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>.  - MM   $ $S ) KK   " " $  /  " " ) ) / / 6;Q;Q;X;X;^;^0_ `  = >{{11$$))//t{{7T7TWX7X0YZ2?r7rhc|dz dzdz}|S)zH Computes the output length of the convolutional layers rr$)ryrhs r5 _get_feat_extract_output_lengthsz7WhisperPreTrainedModel._get_feat_extract_output_lengthsAs'*q014 r7N)rrrr__annotations__base_model_prefixmain_input_namesupports_gradient_checkpointing_no_split_modules_supports_flash_attn_supports_sdpa_supports_flex_attn_can_compile_fullgraphrr+rrrr7r5rr!sT &O&*#.0EFN![&e>N>Nr7rceZdZdZdeffd ZdZdejfdZ dejfdZ d d Z xZ S) r z Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`WhisperEncoderLayer`]. Args: config: WhisperConfig rcLt|||j|_|j|_|j }|j |_|j|_|j|_ |jrtj|nd|_ tj|j |dd|_tj||ddd|_tj$|j||_|j&j)dtj*t-|j.Dcgc] }t1|c}|_tj4|j |_d|_|j;ycc}w)Nrrr) kernel_sizepaddingr$)r strider!F)rwrxrencoder_layerdrop layerdropr num_mel_binsr9rtmax_source_positionsscale_embeddingr)sqrt embed_scalerrconv1conv2rr requires_grad_ ModuleListrYencoder_layersrrr layer_normgradient_checkpointing post_init)ryrrrgrzs r5rxzWhisperEncoder.__init__Ss2  ~~ 11NN "//!..$*$?$?!393I3I499Y/sYYt00)TUV YYy)1VWX !||D,E,EyQ ++E2mm%PVPePeJf$gQ%8%@$gh ,,v~~6&+#  %hsF!cJ|jD] }d|_ d|_yNF) parameters requires_grad_requires_grad)ryparams r5_freeze_parametersz!WhisperEncoder._freeze_parametersks(__& (E"'E  (#r7r"c|jSrvr*rys r5get_input_embeddingsz#WhisperEncoder.get_input_embeddingsps zzr7rc||_yrvr:ryrs r5set_input_embeddingsz#WhisperEncoder.set_input_embeddingsss  r7c |jj|jjdz|jjdz}|j d|k7r"t d|d|j dd|d||n|jj}||n|jj}||n|jj}tjj|j|}tjj|j |}|jddd }tj|j j"|j$ } ||j!| z} tjj'| |j&|j( } |rd nd} |rd nd} |[|j+dt-|j.k(s2Jd t-|j.d|j+ddt1|j.D]j\} }|r| | fz} d}|j(r&tj2g}||j4krd}|rd}n|| d||| nd|}|d} |sb| |d fz} l|j7| } |r| | fz} |st9d| | | fDSt;| | | S)a Args: input_features (`torch.LongTensor` of shape `(batch_size, feature_size, sequence_length)`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec libary (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] attention_mask (`torch.Tensor`)`, *optional*): Whisper does not support masking of the `input_features`, this argument is preserved for compatibility, but it is not used. By default the silence in the input log mel spectrogram are ignored. head_mask (`torch.Tensor` of shape `(encoder_layers, encoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. rr%z7Whisper expects the mel input features to be of length z , but found z-. Make sure to pad the input mel features to .Nr$rdevicerrz&The head_mask should be specified for  layers, but it is for FT)NN)rrc3&K|] }|| ywrvr.0vs r5 z)WhisperEncoder.forward..seqWXWdeslast_hidden_stater attentions)rr&r*r"r+r=r(routput_hidden_statesuse_return_dictrrgelupermuter+r-r num_embeddingsrCrrrr]r enumeraterTr$r/rr)ryrrDrrrM return_dictexpected_seq_length inputs_embeds all_positionsrencoder_statesall_attentionsidx encoder_layerto_dropdropout_probability layer_outputss r5rzWhisperEncoder.forwardvsH#kk>>ARARSTAUUX\XbXbXiXijkXll    #': :IJ]I^^jkykkAClDkEErsFrGGHI 2C1N-TXT_T_TqTq$8$D $++JjJj &1%>#A&3t{{+;< 8T[[9I8JJabkbpbpbrstbuavvwx <#,DKK"8 F C#!/=2B!BG}}&+jjn#&7"G , -!7@7LYs^RV&7 ! !.a 0  !/=3C2E!E/ F2 6 +}.>>Ne]NN$See e+>Vd  r7)NNNNN) rrrrrrxr8rModuler<r?rrrs@r5r r JsP}0$ bii")) !b r7r cNeZdZdZdZdeffd Z ddZxZS)WhisperDecoderz Transformer decoder consisting of *config.decoder_layers* layers. Each layer is a [`WhisperDecoderLayer`] Args: config: WhisperConfig r8rc Jt|||j|_|j|_|j |_|j|_|j|_|jrtj|jnd|_ tj|j |j|j |_t%|j|j|_tj(t+|j,Dcgc]}t/||c}|_|j2dk(|_|j2dk(|_tj8|j|_d|_|j?ycc}w)Nrflash_attention_2sdpaF) rwrxrdecoder_layerdropr$r9rtmax_target_positionsr&r'r)r(rr)rr vocab_size embed_tokensrqr r-rYdecoder_layersrrr_use_flash_attention_2 _use_sdparr/r0r1rs r5rxzWhisperDecoder.__init__s8  ~~ 11!..$*$?$?!$*$?$?!8>8N8N499V^^4TWLL):):FNNDL\L\]9$:S:SU[UcUcdmmEJ6K`K`Ea b  3 b  '-&A&AEX&X#44>,,v~~6&+#  csF c n| | n|jj} | | n|jj} | | n|jj} | | n|jj} | | t d|&|j }|jd|d}n!||j dd}n t d||j|}| r?|=|jjrttt}n t}d}| | d}n||j}| (tj|||dz|j} |$| j!dj#|dd}||j%|||}n|j%|||}||j'|jz}t(j*j-||j,|j. }t1|j||| || }|j2r%|j.r| rt4j7d d } | rd nd}| rd nd}| r|d nd}t9||gddgD]f\}}| |j dt;|j<k(r3Jd|dt;|j<d|j ddt?|j<D]~\}}| r||fz }|j.r%tj@g}||jBkr?|||||||nd|||nd| r|nd| | |  }|d}| sj||dfz }|v||dfz }|jE|}| r||fz }| r|nd}| stGd|||||fDStI|||||S)ac Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. Indices can be obtained using [`WhisperTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the attention modules in encoder to avoid performing cross-attention on hidden heads. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. past_key_values (`EncoderDecoderCache` or `tuple(tuple(torch.FloatTensor))`, *optional*): Pre-computed hidden-states that can be used to speed up auto-regressive (sequential) decoding. There are four sets of pre-computed hidden-states: key and values states in the self-attention blocks (2) and in the cross-attention blocks (2). The `past_key_values` are returned when `use_cache=True` is passed or when `config.use_cache=True` Two formats are allowed: - An [`~cache_utils.EncoderDecoderCache`] instance; - Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. If `past_key_values` are used, the user can optionally input only the last `decoder_input_ids` (those that don't have their past key value states given to this model) of shape `(batch_size, 1)` instead of all `decoder_input_ids` of shape `(batch_size, sequence_length)`. inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. cache_position (`torch.LongTensor` of shape `(sequence_length)`, *optional*): Indices depicting the position of the input sequence tokens in the sequence. It is used to update the cache in the correct position and to infer the complete sequence length. NzTYou cannot specify both decoder_input_ids and decoder_inputs_embeds at the same timer%zEYou have to specify either decoder_input_ids or decoder_inputs_embedsrrrB)r~rr)r input_embedsrDrpast_key_valuesrz^`use_cache = True` is incompatible with gradient checkpointing. Setting `use_cache = False`...Frrcross_attn_head_maskzThe `z` should be specified for rDrA)rDrrrrrrrr$c3$K|]}|| ywrvrrFs r5rIz)WhisperDecoder.forward..s=s)rKrmrrLcross_attentions)%rrrMrrNr(rr.rgis_encoder_decoderr r get_seq_lengthr+r-rC unsqueezerepeatr torrrrrr0rrzipr]rrRrTr$r/rr)ryr8rDrrrnrmrUrrrrMrSr input_shaper~ positionsr causal_maskall_hidden_statesall_self_attnsall_cross_attentions attn_mask mask_namerY decoder_layerr\r] next_caches r5rzWhisperDecoder.forwards]b2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKst t  "#..*K!r;r?;I  &',,.s3Kde e   --i8M 0{{--"5lnln"U"..!"  %%3A%6 "  (%4%C%C%E "  !"\\&(>Q(OXeXlXlN  )33A6==k!naPL  ,,2HWc-I,,6L[g-I& ]5I5I(JJ  --mt||VZVcVc-d (;;&))+%    & &4==##t" "6BD0d&7s r5r?z!WhisperModel.set_input_embeddings$) !r7c|jSrv)rr;s r5 get_encoderzWhisperModel.get_encoder ||r7c|jSrvrr;s r5 get_decoderzWhisperModel.get_decoderrr7c8|jjyz Calling this function will disable the gradient computation for the Whisper encoder so that its parameters will not be updated during training. Nrr8r;s r5freeze_encoderzWhisperModel.freeze_encoder '')r7rrDc2t|jdds|S|j\}}}|jjdkDr|jrt ||f|jj|jj ||jj}tj||jtj}|dddfjd|d}d||<|jjdkDr|jrt ||f|jj|jj|jj}tj||jtj}d||<|S) z Masks extracted features along time axis and/or along feature axis according to [SpecAugment](https://huggingface.co/papers/1904.08779). apply_spec_augmentTr)rBrCrDrE)rCrPNr%)rBrCrE)getattrrrmask_time_probrromask_time_lengthmask_time_min_masksr+tensorrCr[expandmask_feature_probmask_feature_lengthmask_feature_min_masks)ryrrDrf hidden_sizerNmask_time_indicesmask_feature_indicess r5_mask_input_featuresz!WhisperModel._mask_input_featuress[t{{$8$?! !4B3F3F3H0 K ;; % % )dmm 5_-++44 KK88-++99 ! !& ->~G\G\didndn o  1!T' : A A"kSU V 01N, - ;; ( (1 ,#8[)++77 KK;;++<< $ $)<<0D^MbMbjojtjt#u 34N/ 0r7decoder_input_idsdecoder_attention_maskrdecoder_head_maskrnencoder_outputsrmdecoder_inputs_embedsdecoder_position_idsrrrMrSrr"c| | n|jj} ||n|jj}| | n|jj} ||n|jj}|*|j ||}|j ||| ||}nI|rGt|ts7t|dt|dkDr|dndt|dkDr|dnd}|j|||d||| | | | | ||| }|s||zSt|j|j|j|j|j |j|j|j S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`WhisperTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Whisper uses the `decoder_start_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_whisper._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the BART paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, WhisperModel >>> from datasets import load_dataset >>> model = WhisperModel.from_pretrained("openai/whisper-base") >>> feature_extractor = AutoFeatureExtractor.from_pretrained("openai/whisper-base") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_features = inputs.input_features >>> decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id >>> last_hidden_state = model(input_features, decoder_input_ids=decoder_input_ids).last_hidden_state >>> list(last_hidden_state.shape) [1, 2, 512] ```N)rDrrrMrSrrr$rJ) r8rDrrrnrmrUrrrrMrSr)rKrmdecoder_hidden_statesdecoder_attentionsrpencoder_last_hidden_staterencoder_attentions)rrrMrrNrrrrr]rrrKrmrrLrp)ryrrDrrrrrnrrmrrrrrMrSrdecoder_outputss r5rzWhisperModel.forwardsD2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKK?w D>w $D>w 'tnw d^!w "!!1!12#w $ uU\\"$66 7%w w r7rzh The Whisper Model with a language modeling head. Can be used for automatic speech recognition. ) custom_introc(eZdZdZdgZdeffd ZdZdZdZ dZ d e jfd Z d Ze dd eej$d eej&deej&deej&deej(deej(deej(deeeej$deeedeeej$deeej&deej&deedeedeedeedeej&d eeej(eff$dZxZS)WhisperForConditionalGenerationrproj_out.weightrct||t||_t j |j |jd|_|j|_ |jyNFr) rwrxrrrrrrfproj_outrer1rs r5rxz(WhisperForConditionalGeneration.__init__sT  !&)  &..&2C2C%P $*$?$?! r7c6|jjSrv)rrr;s r5rz+WhisperForConditionalGeneration.get_encoderzz%%''r7c6|jjSrv)rrr;s r5rz+WhisperForConditionalGeneration.get_decoderrr7c|jSrvrr;s r5get_output_embeddingsz5WhisperForConditionalGeneration.get_output_embeddings }}r7c||_yrvrrynew_embeddingss r5set_output_embeddingsz5WhisperForConditionalGeneration.set_output_embeddings & r7r"c6|jjSrvrr<r;s r5r<z4WhisperForConditionalGeneration.get_input_embeddingszz..00r7cL|jjjyr)rrr8r;s r5rz.WhisperForConditionalGeneration.freeze_encoders --/r7rrDrrrrrnrrmrrlabelsrrrMrSrcH||n|jj}| ~| jd|jkDr)t d| jdd|jd|7| 5t | |jj |jj}|j||||||||| | | | ||||}|j|d}d}| at}| j|j} ||jd|jj| jd}|s|f|ddz}||f|zS|St!|||j"|j$|j&|j(|j*|j,|j. S) a decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`WhisperTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) Whisper uses the `decoder_start_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). decoder_attention_mask (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. If you want to change padding behavior, you should read [`modeling_whisper._prepare_decoder_attention_mask`] and modify to your needs. See diagram 1 in [the BART paper](https://huggingface.co/papers/1910.13461) for more information on the default strategy. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. decoder_position_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Indices of positions of each input sequence tokens in the position embeddings. Selected in the range `[0, config.n_positions - 1]`. [What are position IDs?](../glossary#position-ids) labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. `sequence_length` should be smaller than or equal to `config.max_target_positions`. Example: ```python >>> import torch >>> from transformers import AutoProcessor, WhisperForConditionalGeneration >>> from datasets import load_dataset >>> processor = AutoProcessor.from_pretrained("openai/whisper-tiny.en") >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = processor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_features = inputs.input_features >>> generated_ids = model.generate(inputs=input_features) >>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> transcription ' Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.' ```NrzLabels' sequence length z- cannot exceed the maximum allowed length of z tokens.)rDrrrrrrnrmrrrrrMrSrrr%) losslogitsrmrrrprrr)rrNr=rer(rAr9r:rrrrurCr.rfrdrrmrrrprrr)ryrrDrrrrrnrrmrrrrrrMrSrr lm_logitsrloss_fctoutputs r5rz'WhisperForConditionalGeneration.forwardsX&1%?@ ))*@ D>@ $D>@ 'tn!@ "d^#@ $!!1!12%@ & uU\\"O3 4'@ @ r7rc:eZdZdZfdZdZdZdZdZxZ S)WhisperDecoderWrapperz This wrapper class is a helper class to correctly load pretrained checkpoints when the causal language model is used in combination with the [`EncoderDecoderModel`] framework. cRt||d|_t||_yr3)rwrxrqr`rrs r5rxzWhisperDecoderWrapper.__init__Es$  $)!%f- r7c.|jjSrvrr;s r5r<z*WhisperDecoderWrapper.get_input_embeddingsJrr7c&||j_yrvrr>s r5r?z*WhisperDecoderWrapper.set_input_embeddingsMrr7c|jSrvrr;s r5rz!WhisperDecoderWrapper.get_decoderPrr7c&|j|i|Srvr)ryargsrs r5rzWhisperDecoderWrapper.forwardSst||T,V,,r7) rrrrrxr<r?rrrrs@r5rr?s! . )*-r7rzx Whisper decoder with a language modeling head on top (linear layer with weights tied to the input embeddings). c eZdZdgZdZfdZdZdZdejfdZ dZ d Z d Z e ddeej"d eej$d eeej(d eej$deej$deedeej(deej"deedeedeedeedeej"deeeffdZxZS)WhisperForCausalLMrr8ct||d|_t||_t j |j|jd|_ |jyr) rwrxrqrrrrrrfrr1rs r5rxzWhisperForCausalLM.__init__`sP  $)!*62  &"4"4f6G6GeT  r7c|jSrvrr;s r5rz(WhisperForCausalLM.get_output_embeddingsjrr7c||_yrvrrs r5rz(WhisperForCausalLM.set_output_embeddingsmrr7r"c6|jjSrvrr;s r5r<z'WhisperForCausalLM.get_input_embeddingsprr7c:|jj|yrv)rr?r>s r5r?z'WhisperForCausalLM.set_input_embeddingsss ''.r7c&||j_yrvrr)ryrs r5 set_decoderzWhisperForCausalLM.set_decodervs$ r7c.|jjSrvrr;s r5rzWhisperForCausalLM.get_decoderyszz!!!r7rDrrrnrmrUrrrrMrSrc| | n|jj} | | n|jj} | | n|jj} t |t t tfr|d}|jj|||||||| | | | |  }|j|d}d}|a|j|j}t}||jd|jj|jd}| s|f|ddz}||f|zS|St!|||j"|j$|j&|j(S)a encoder_outputs (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention if the model is configured as a decoder. cross_attn_head_mask (`torch.Tensor` of shape `(decoder_layers, decoder_attention_heads)`, *optional*): Mask to nullify selected heads of the cross-attention modules. Mask values selected in `[0, 1]`: - 1 indicates the head is **not masked**, - 0 indicates the head is **masked**. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import WhisperForCausalLM, WhisperForConditionalGeneration, WhisperProcessor >>> import torch >>> from datasets import load_dataset >>> processor = WhisperProcessor.from_pretrained("openai/whisper-large-v2") >>> model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v2") >>> assistant_model = WhisperForCausalLM.from_pretrained("distil-whisper/distil-large-v2") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> sample = ds[0]["audio"] >>> input_features = processor( ... sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt" ... ).input_features >>> predicted_ids = model.generate(input_features, assistant_model=assistant_model) >>> # decode token ids to text >>> transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0] >>> transcription ' Mr. Quilter is the apostle of the middle classes and we are glad to welcome his gospel.' ```Nr) r8rDrrrnrmrUrrrMrSrr%r)rrrmrrLrp)rrrMrNrrrlistrrrrurCrr.rfrrmrrLrp)ryr8rDrrrnrmrUrrrrMrSrrrrrrs r5rzWhisperForCausalLM.forward|sun2C1N-TXT_T_TqTq$8$D $++JjJj &1%B,07;+/59-1$(,0/3&*59c E,,-c !.c "%(9(9":; c ELL) c 'u||4 c "%c   1 12c ))*c D>c $D>c 'tnc d^c !!1!12c  u77 8c c r7rz Whisper Encoder Model with a sequence classification head on top (a linear layer over the pooled output) for tasks like SUPERB Keyword Spotting. cPeZdZfdZdZdej fdZdej fdZe dde e jde e jd e eee jd e e jd e ed e ed e edeee jeffdZxZS)r ct||t||_|jdz}|j r0t jtj||z |_ t j|j|j|_t j|j|j|_|j#yr|)rwrxr rrrr Parameterr+r_rrrclassifier_proj_size projector num_labels classifierr1)ryr num_layersrzs r5rxz&WhisperForAudioClassification.__init__s  %f- --1  ( (!#ejj.Dz.Q!RD 6#5#5v7R7RS))F$?$?ARARS r7c8|jjy)z Calling this function will disable the gradient computation for the Whisper encoder so that its parameters will not be updated during training. Only the projection layers and classification head will be updated. Nrr;s r5rz,WhisperForAudioClassification.freeze_encoderrr7r"c6|jjSrv)rr<r;s r5r<z2WhisperForAudioClassification.get_input_embeddingss||0022r7rc:|jj|yrv)rr?r>s r5r?z2WhisperForAudioClassification.set_input_embeddingss ))%0r7rrrrrrMrSc||n|jj}||n|jj}|jjrd}n||jj}||n|jj}||j |||||}|jjrr|t }tj|d}tjj|jd} || jdddzjd}n|d}|j|}|j!d} |j#| } d} |at%} |j'| j(}| | jd|jj*|jd} |s| f|ddz}| | f|zS|St-| | |j.|j0S) a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, WhisperForAudioClassification >>> from datasets import load_dataset >>> feature_extractor = AutoFeatureExtractor.from_pretrained("sanchit-gandhi/whisper-medium-fleurs-lang-id") >>> model = WhisperForAudioClassification.from_pretrained("sanchit-gandhi/whisper-medium-fleurs-lang-id") >>> ds = load_dataset("google/fleurs", "all", split="validation", streaming=True) >>> sample = next(iter(ds)) >>> inputs = feature_extractor( ... sample["audio"]["array"], sampling_rate=sample["audio"]["sampling_rate"], return_tensors="pt" ... ) >>> input_features = inputs.input_features >>> with torch.no_grad(): ... logits = model(input_features).logits >>> predicted_class_ids = torch.argmax(logits).item() >>> predicted_label = model.config.id2label[predicted_class_ids] >>> predicted_label 'Afrikaans' ```NTrrr&r%r)rrrrL)rrrMrrNr_HIDDEN_STATES_START_POSITIONr+stackrrrrr.rWrrrrrurCrrrrL)ryrrrrrrMrSr norm_weights pooled_outputrrrrs r5rz%WhisperForAudioClassification.forwardsX2C1N-TXT_T_TqTq$8$D $++JjJj  ;; - -#' ! )#';;#C#C %0%->r1a-HHMMRSMTM+A.M}5 %**q*1 /  ')HYYv}}-FFKKDKK,B,BCV[[QS_UDY!44F)-)9TGf$ Ev E')77&11   r7)NNNNNNN)rrrrxrrr^r<r?rrr+rrrrr[rrrrrs@r5r r s *3bii31"))16:,0EI-1,0/3&*]  !1!12] ELL)] "%e.?.?(@"AB ] ))* ] $D> ] 'tn] d^]  uU\\"$<< =] ] r7r )rrrrr )i')Nr)NrN)Krr)typingrrrnumpyrRr+torch.utils.checkpointrtorch.nnr activationsr cache_utilsr r r generationr masking_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrrmodeling_utilsrrprocessing_utilsrutilsrrconfiguration_whisperrgeneration_whisperr get_loggerrrrrIrrr6rArrndarrayrorrqr^rrrrrr r`rrrrr __all__rr7r5r s ,, %!CC)/:G&,06   H % ! Dc DS D D5<< D %,, c [^ *26 t c?tttU--. t  t ZZ tn--" $(,% II% <<% % << % U\\* % e_ %% %%<{)ryy{)~<+4<+~l4l^%_%%PN +N br +r j )  D d &<>Td  d N-2-0 D /D  D N y $:y  y x r7