rh;ZddlZddlmZmZmZddlZddlmZddlmZddl m Z ddl m Z ddl mZdd lmZmZmZdd lmZmZdd lmZdd lmZmZmZmZdd lmZddlm Z m!Z!ddl"m#Z#m$Z$ejJe&Z' d+dejPdejRdejRdejRdeejRdee*de*deejRfdZ+GddejPZ,GddeZ-eGdd eZ.ed!"Gd#d$e.Z/Gd%d&ejPZ0ed'"Gd(d)e.e Z1gd*Z2y),N)CallableOptionalUnion)nn)ACT2FN)Cache)GenerationMixin)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPastCausalLMOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)check_model_inputs) AutoModelAutoModelForCausalLM) VoxtralConfigVoxtralEncoderConfigmodulequerykeyvalueattention_maskscalingdropout head_maskc $||jddz}tj||jdd|z} |0|jdk(r!| |ddddddd|j dfz} t jj| d} || |jddddz} t jj| ||j } tj| |} | jddj} | | fS) Nrr)dimrptraining) sizetorchmatmul transposendimshaper functionalsoftmaxviewr#r- contiguous) rrrr r!r"r#r$kwargs attn_weights attn_outputs /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/voxtral/modeling_voxtral.pyeager_attention_forwardr<,s**R.D(<<s}}Q':;gEL!n&9&9Q&>#nQ1o " o5M&NN ==((2(>L#innQAq&AA ==((6??([L,,|U3K''1-88:K  $$cLeZdZdZ ddededededededeed eeffd Z d e jd ed efdZ dde jdee jdee jdede e jee jee e jff dZxZS)VoxtralAttentionz=Multi-headed attention from 'Attention Is All You Need' paper embed_dim num_headsr# is_decoderbias is_causal layer_idxconfigc zt |||_||_||_||z|_||_|j |z|jk7rtd|jd|d|j dz|_||_ ||_ |/|r-tjd|jjd||_t!j"||d|_t!j"||||_t!j"||||_t!j"||||_y) Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).r'zInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrC)super__init__r@rAr#head_dimrF ValueErrorr"rBrDlogger warning_once __class____name__rErLineark_projv_projq_projout_proj) selfr@rAr#rBrCrDrErFrOs r;rJzVoxtralAttention.__init__Ms  "" !Y.  MMI %$.. 8MdnnM]$YKr3 }}d* $"     *4>>+B+B*CD,,  #ii 95A ii 94@ ii 94@  )YTB r=tensorseq_lenbszc|j|||j|jjddj S)Nrr)r6rArKr1r7)rVrWrXrYs r;_shapezVoxtralAttention._shapeus7{{3GQQRSUVWbbddr= hidden_statesr!layer_head_maskoutput_attentionsreturnc N|j\}}}|j|j||jz||} |j|j |d|} |j|j |d|} t } |jjdk7rt|jj} | || | | |f|jsdn |jd||d|\} }| j||dj} |j| } | |fS)z#Input shape: Batch x Time x Channelr&eager?)r#r"r^r$)r.r[rTr"rRrSr<rF_attn_implementationrr-r#reshaper7rU)rVr\r!r]r^r8rYtgt_len_ query_states key_states value_statesattention_interfacer:r9s r;forwardzVoxtralAttention.forwardxs!(,,.Wa{{4;;}#= #LgWZ[ [[]!;RE {{4;;}#=r3G (? ;; + +w 6"9$++:Z:Z"[ $7      % $}}C$,,/% %  % ! \"))#w;FFH mmK0 L((r=)rbFTFNN)NNF)rP __module__ __qualname____doc__intfloatboolrrrJr/Tensorr[tuplerl __classcell__rOs@r;r?r?JsG  #'*.&C&C&C &C  &C  &C&CC=&C'&CPeU\\eCece 2626"' ))||))!.))"%,,/ ))  )) u||Xell3XeELL>Q5RR S))r=r?c eZdZdeffd Z d dej dej dej dedej f dZxZ S) VoxtralEncoderLayerrFcht||j|_t |j|j |j ||_tj|j|_ |j|_ t|j|_|j|_tj |j|j"|_tj |j"|j|_tj|j|_y)N)r@rAr#rF)rIrJd_modelr@r?encoder_attention_headsattention_dropout self_attnr LayerNormself_attn_layer_normr#ractivation_function activation_fnactivation_dropoutrQencoder_ffn_dimfc1fc2final_layer_normrVrFrOs r;rJzVoxtralEncoderLayer.__init__s )nn44,,   %'LL$@!~~ #F$>$>?"(";";99T^^V-C-CD99V33T^^D " T^^ <r=r\r!r]r^r_c|}|j|}|j||||\}}tjj ||j|j }||z}|}|j |}|j|j|}tjj ||j|j }|j|}tjj ||j|j }||z}|jtjk(rEtj|jjdz }tj || |}||fS)a Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`): attention mask of size `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values. layer_head_mask (`torch.FloatTensor`): mask for attention heads in a given layer of size `(encoder_attention_heads,)`. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. )r\r!r]r^r+i)minmax)rr}rr4r#r-rrrrrdtyper/float16finforclamp)rVr\r!r]r^residualr9 clamp_values r;rlzVoxtralEncoderLayer.forwardsS$!11-@ &*nn')+/ '5' # |  --mt||VZVcVc-d  =0  --m< **488M+BC  --mt?V?Vaeanan-o /  --mt||VZVcVc-d  =0   %-- /++m&9&9:>>EK!KK K<[YMl**r=)F) rPrmrnrrJr/rsrrrlrurvs@r;rxrxsY=}=.#( )+||)+ )+ )+  )+  )+r=rxcFeZdZUeed<dZdZdZdZdZ dZ dZ dZ dZ dZdZy)VoxtralPreTrainedModelrFmodelTNpast_key_valuescjt|jdr|jjn|jjj}t |t j t jfrY|jjjd||j%|jjjyyt |t jrJ|jjjd|jjjyt |t jrf|jjjd||j 2|jj|j jyyy)Ninitializer_rangerb)meanstdrc)hasattrrFr audio_config isinstancerrQConv1dweightdatanormal_rCzero_r~fill_ Embedding padding_idx)rVrrs r; _init_weightsz$VoxtralPreTrainedModel._init_weightss3 t{{$78 KK ) )));; fryy"))4 5 MM   & &CS & 9{{&   &&('  - MM   $ $S ) KK   " " $  - MM   & &CS & 9!!- ""6#5#56<<>..r=)rPrmrnr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_cache_class_supports_attention_backend_can_compile_fullgraphrr=r;rrsH &*#"3N "&!?r=rz: The Voxtral encoder, which is a Whisper encoder. ) custom_introceZdZUdZeed<dZdgZee dZ deffd Z dZ de jfd Zd e jfd Ze dd eefd Zdej,fdZxZS)VoxtralEncoderz Transformer encoder consisting of *config.encoder_layers* self attention layers. Each layer is a [`VoxtralEncoderLayer`]. Args: config: VoxtralEncoderConfig rFinput_featuresrx) attentionsr\ct|||j|_|j|_|j }|j |_|j|_|j|_ |jrtj|nd|_ tj|j |dd|_tj||ddd|_tj$|j||_|j&j)dtj*t-|j.Dcgc] }t1|c}|_tj4|j |_tj8dd|_d|_|j?ycc}w) Nrcrr) kernel_sizepaddingr)rstriderF)r) rIrJr#encoder_layerdrop layerdroprz num_mel_bins pad_token_idrmax_source_positionsscale_embeddingmathsqrt embed_scalerrconv1conv2rembed_positionsrequires_grad_ ModuleListrangeencoder_layersrxlayersr~ layer_norm AvgPool1d avg_poolergradient_checkpointing post_init)rVrFr@rgrOs r;rJzVoxtralEncoder.__init__sD  ~~ 11NN "//!..$*$?$?!393I3I499Y/sYYt00)TUV YYy)1VWX !||D,E,EyQ ++E2mm%PVPePeJf$gQ%8%@$gh ,,v~~6,,q3&+# %hsF=cJ|jD] }d|_ d|_y)NF) parameters requires_grad_requires_grad)rVparams r;_freeze_parametersz!VoxtralEncoder._freeze_parameters8s(__& (E"'E  (#r=r_c|jSNrrVs r;get_input_embeddingsz#VoxtralEncoder.get_input_embeddings=s zzr=r c||_yrrrVr s r;set_input_embeddingsz#VoxtralEncoder.set_input_embeddings@s  r=r8c |jj|jjdz|jjdz}|j d|k7r"t d|d|j dd|d|j|jjj|jjj}tjj|j|}tjj|j |}|jddd }|jj}||zj|j}tjj!||j |j" }t%|j&D]\}} | ||d } | d}|j)|}t+| S)a Args: input_features (`torch.LongTensor` of shape `(batch_size, feature_size, sequence_length)`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] attention_mask (`torch.Tensor`)`, *optional*): Voxtral does not support masking of the `input_features`, this argument is preserved for compatibility, but it is not used. By default the silence in the input log mel spectrogram are ignored. rr&z:Qwen2Audio expects the mel input features to be of length z , but found z-. Make sure to pad the input mel features to .)rdevicerrr+N)r!r])last_hidden_state)rFrrrrr3rLtorrrrr4gelupermuterr#r- enumeraterrr ) rVrr!r8expected_seq_length inputs_embeds embed_posr\idx encoder_layer layer_outputss r;rlzVoxtralEncoder.forwardCs&#kk>>ARARSTAUUX\XbXbXiXijkXll    #': :LM`Laamn|oCoCDFoGnHHuvIuJJKL (**1B1B1H1HQUQ[Q[QbQbQiQi*j **4::n+EF  **4::m+DE %--aA6 ((// &266}7J7JK  --mt||VZVcVc-d "+DKK"8 - C)- $M *!,M  - 6 +  r= input_lengthsc6|dz dzdz}|dz dzdz}||fS)zs Computes the output length of the convolutional layers and the output length of the audio encoder rrr)rVroutput_lengthss r; _get_feat_extract_output_lengthsz/VoxtralEncoder._get_feat_extract_output_lengthsts7'*q014 '!+1A5n,,r=r)rPrmrnrorrmain_input_namerr?rx_can_record_outputsrJrrModulerrrrrrlr/ LongTensorrrurvs@r;rrs  ! &O./&, 34$ bii"))- +, - - `-e>N>N-r=rc*eZdZdeffd ZdZxZS)VoxtralMultiModalProjectorrFcft|tj|jj |j jd|_t|j|_ tj|j j|j jd|_ y)NFrH) rIrJrrQrintermediate_size text_config hidden_sizelinear_1rprojector_hidden_actactlinear_2rs r;rJz#VoxtralMultiModalProjector.__init__~sz  &"5"5"G"GI[I[IgIgnst &556 &"4"4"@"@&BTBTB`B`glm r=cl|j|}|j|}|j|}|Sr)rrr)rVaudio_featuresr\s r;rlz"VoxtralMultiModalProjector.forwards2 n5 /  m4 r=)rPrmrnrrJrlrurvs@r;rr}sn}n r=rzs The Voxtral model, which consists of Whisper encoder, a multi-modal projector and a LLama language model. ceZdZdgZddiZddgdgfiZdgZfdZdZd Z d Z d Z d Z d Z dejfdZee ddeej(deejdeej*deej(deedeejdeej(deedeej(deeej*fdeedefdZfdZxZS)VoxtralForConditionalGenerationzlm_head.weightlm_head colwise_repr\logitsrc*t|||jj|_t j |j |_tj |j|_ t||_ |jyr) rIrJr vocab_sizer from_configr audio_towerrlanguage_modelrmulti_modal_projectorrrs r;rJz(VoxtralForConditionalGeneration.__init__sn   ,,77$001D1DE2>>v?Q?QR%?%G" r=c6|jjSr)r rrs r;rz4VoxtralForConditionalGeneration.get_input_embeddingss""7799r=c:|jj|yr)r rrs r;rz4VoxtralForConditionalGeneration.set_input_embeddingss 007r=c6|jjSr)r get_output_embeddingsrs r;rz5VoxtralForConditionalGeneration.get_output_embeddingss""88::r=c:|jj|yr)r set_output_embeddings)rVnew_embeddingss r;rz5VoxtralForConditionalGeneration.set_output_embeddingss 11.Ar=c:|jj|yr)r  set_decoder)rVdecoders r;rz+VoxtralForConditionalGeneration.set_decoders ''0r=c6|jjSr)r  get_decoderrs r;rz+VoxtralForConditionalGeneration.get_decoders""..00r=rc|j|}|j}|jd|jjj }|j |}|S)a This method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector. Args: input_features (`torch.FloatTensor`): Float values of mel features extracted from the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]` or a `numpy.ndarray`, *e.g.* via the soundfile library (`pip install soundfile`). To prepare the array into `input_features`, the [`AutoFeatureExtractor`] should be used for extracting the mel features, padding and conversion into a tensor of type `torch.FloatTensor`. See [`~WhisperFeatureExtractor.__call__`] Returns: `torch.FloatTensor`: The audio embeddings. r&)rrrerFrrr )rVr audio_outputsaudio_hidden_states audio_embedss r;get_audio_embedsz0VoxtralForConditionalGeneration.get_audio_embedssZ((8 +==199"dkk>V>V>h>hi112EF r= input_idsr! position_idsrrlabels use_cachecache_positionlogits_to_keepr8r_c ||j|}|/|j|} ||jjk(} | || <|jd||||||| | d| }|S)ap Example: ```python >>> from transformers import VoxtralForConditionalGeneration, AutoProcessor >>> import torch >>> device = "cuda" if torch.cuda.is_available() else "cpu" >>> repo_id = "mistralai/Voxtral-Mini-3B-2507" >>> processor = AutoProcessor.from_pretrained(repo_id) >>> model = VoxtralForConditionalGeneration.from_pretrained(repo_id, torch_dtype=torch.bfloat16, device_map=device) >>> conversation = [ { "role": "user", "content": [ { "type": "audio", "url": "https://huggingface.co/datasets/hf-internal-testing/dummy-audio-samples/resolve/main/dude_where_is_my_car.wav", }, {"type": "text", "text": "What can you tell me about this audio?"}, ], } ] >>> inputs = processor.apply_chat_template(conversation) >>> inputs = inputs.to(device, dtype=torch.bfloat16) >>> outputs = model.generate(**inputs, max_new_tokens=30) >>> processor.batch_decode(outputs[:, inputs.input_ids.shape[1]:], skip_special_tokens=True) ["This audio is a humorous conversation between two friends, likely in English, where one of them is trying to figure out what the other's tattoo says."] ```)r!rrrrrr r!r)rrrFaudio_token_idr )rVrrr!rrrrrr r!r8raudio_token_maskoutputss r;rlz'VoxtralForConditionalGeneration.forwardsb  7D557 BM  %00@L )DKK,F,FF .:M* ++>4+>+> , )%+')) ,  , r=c|jdd}|jd}t| |i|}| |ddk(r||d<|S)Nrr r)popgetrIprepare_inputs_for_generation)rVargsr8rr  model_inputsrOs r;r)z=VoxtralForConditionalGeneration.prepare_inputs_for_generations\ $4d;$45wD!!1!12Dc5<</0D+,D DDL  r=r)rrr)NrbN)3rtypingrrrr/r activationsr cache_utilsr generationr modeling_layersr modeling_outputsr r rmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrautorrconfiguration_voxtralrr get_loggerrPrMrrsrqr<r?rxrrrr__all__rr=r;r?s~, ,, ! )9``F&RR/2F   H % $(,% II% <<% % << % U\\* % e_ %% %%