rh:ddlZddlZddlmZmZddlmZddlmZm Z ddl Z ddl m Z ddl m cmZddlmZddlmZmZmZddlmZdd lmZmZdd lmZdd lmZdd lm Z m!Z!dd l"m#Z#m$Z$ddl%m&Z&m'Z'ddl(m)Z)ddl*m+Z+m,Z,m-Z-m.Z.m/Z/ddl0m1Z1ddl2m3Z3m4Z4m5Z5m6Z6e/jne8Z9ee-dGdde Z:ee-dGdde+Z;Gdde jxZ=Gdde jxZ>Gd d!e jxZ?Gd"d#e jxZ@Gd$d%e jxZAGd&d'e jxZBGd(d)e jxZCGd*d+e jxZDGd,d-e jxZEGd.d/e jxZFGd0d1e'ZGGd2d3e jZIGd4d5e jxZJGd6d7e jxZKGd8d9e jxZLGd:d;e jxZMd<ZNd=e jd>ePd?e jfd@ZQ dedAe jxdBe jdCe jdDe jdEee jdFeRdGeeRdHeeRd?eSe je jffdIZT dfdJe jdKe jdLe jdMee jdNePf dOZUGdPdQe jxZVGdRdSeZWe-GdTdUe'ZXe-dVGdWdXeXZYe-dYGdZd[eXeZZGd\d]e jxZ[e-d^Gd_d`eXZ\e-daGdbdceXeZ]gddZ^y)gN)CallableSequence) dataclass)OptionalUnion)ACT2FN)Cache DynamicCacheSlidingWindowLayer)GenerationMixin)create_causal_mask!create_sliding_window_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tuplelogging) AutoModel)Gemma3nAudioConfig Gemma3nConfigGemma3nTextConfigGemma3nVisionConfigzL Base class for Gemma3n outputs, with hidden states and attentions. ) custom_introcbeZdZUdZdZeejed<dZ eejed<y)Gemma3nModelOutputWithPasta  past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. audio_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. audio_hidden_states of the model produced by the audio encoder and after projecting the last hidden state. Nimage_hidden_statesaudio_hidden_states) __name__ __module__ __qualname____doc__r(rtorch FloatTensor__annotations__r)/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/gemma3n/modeling_gemma3n.pyr'r'2s5 8<%"3"34;7;%"3"34;r2r'zS Base class for Gemma3n causal language model (or autoregressive) outputs. cFeZdZUdZdZeejed<dZ eejed<dZ ee e eje fed<dZeeejed<dZeeejed<dZeejed<dZeejed <y) Gemma3nCausalLMOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`tuple(tuple(torch.FloatTensor))`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. audio_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. audio_hidden_states of the model produced by the audio encoder and after projecting the last hidden state. Nlosslogitspast_key_values hidden_states attentionsr(r))r*r+r,r-r6rr.r/r0r7r8rlistr r9tupler:r(r)r1r2r3r5r5Ms&)-D(5$$ %,*.FHU&& '.GKOXeD):):$;U$BCDK8T) r*r+r,intr[boolrHrWr.Tensorr^rb __classcell__rOs@r3r>r>qsGPCPePPJ!!%,,! =r2r>ceZdZdeffd Zdej dejdej fdZdej de d e d e d e d e d e dej fdZ dej dej dej fdZ xZ S)%Gemma3nAudioRelativePositionEmbeddingconfigcRt|||_|jj|_|jj |_|j |jz|_td|jjdz |_ |jj|_ tj|j |j|jzd|_d}d}|j dz}t!j"t%|t%|z t|dz dz }|t'j(t'j*|| zz}|j-d|j%j/dj/dd y) Nrr FbiasrDg@rinv_timescalesrE)rGrHrjconf_num_attention_heads num_heads hidden_sizechannelshead_dimmaxconf_attention_context_left max_backwardconf_attention_context_right max_forwardrILinearpos_projmathlogr[r.exparangerL unsqueeze)rNrj min_timescale max_timescalenum_timescaleslog_timescale_incrementrnrOs r3rHz.Gemma3nAudioRelativePositionEmbedding.__init__sL  == //  7 4;;#J#JQ#NO;;CC $--$--1OV[\   !+"&((5+?% BV+V"WZ]^lop^prsZt"t&5<<3OSjRj3j)kk    " , ,Q / 9 9! <  r2positiondtyperXcP|jjd}||jj|jt j z}t jt j|t j|gd}|j|S)NrQ)devicerr?) r[rrntorr.float32catsincostype)rNrr scaled_time timing_signals r3_get_timing_signal_1d_posz?Gemma3nAudioRelativePositionEmbedding._get_timing_signal_1d_poss}>>#--b1!4!4!7!7xV[VcVc!7!dd  599[#9599[;Q"RXZ[ !!%((r2term_bd_before_shift batch_sizerpnum_query_blocksquery_block_sizekey_context_sizemax_span_plus_1c|dz|z }d|f} tjj|| } | j|||||dzzf} | ddddddd||zf} | j|||||f} | S)aZPerforms the relative shift. Args: term_bd_before_shift: Tensor of shape [B, N, U, W, F_span]. batch_size (B), num_heads (N), num_query_blocks (U), query_block_size (W), key_context_size (C = W+L+R), max_span_plus_1 (F_span = L+R+1). Returns: Tensor of shape [B, N, U, W, C]. r rN)rI functionalpadreshape)rNrrrprrrrpad_amount_last_dim padding_tupleterm_bd_paddedterm_bd_reshapedterm_bd_slicedterm_bd_shifteds r3_relative_shiftz5Gemma3nAudioRelativePositionEmbedding._relative_shifts4 0!3F/0 **+?O *11  $4q$89   *!Q3X5EHX5X3X*XY)00     r2querieskeysc R|j\}}}}}|j\}}} }}tj|j|j dz d|j j d} | jd} |j| |j} |j| } | jd| |j|jjd}|jddddd}|jddddd}tj||}|jddddd}|jddd}|j||||z|}tj||}|j||||| }|j!|||||| | }||zS) Nr rQrrrrr)r`r.r~rvrxrrrrrzrrprssqueezepermutematmulr)rNrrrrrrprs_r pos_indicesrsin_emb_timing_signalprojected_sin_embsin_emb queries_pkeys_p_tterm_ac q_permuted s_permuted q_reshapedterm_bd_unshifed_matmulterm_bd_unshifedrs r3r^z-Gemma3nAudioRelativePositionEmbedding.forwards OVmmK $&6 8'+zz$11ll4#4#4t7G7G6G!6KRX_XfXfgqq  &++A. $ > > w}}!?!  !MM*?@#++APTP]P]^ff  OOAq!Q2 <<1aA.,,y(3__Q1a3 __Q1-  '' I?ORb?bdlm #(,,z:"F3::       ..        ((r2) r*r+r,r!rHr.rerrrcrr^rfrgs@r3riris 1 .)%,,)u{{)W\WcWc) ;#ll;; ;  ;  ;;; ;zL)u||L)5<<L)ELLL)r2ric$eZdZdeffd Zdej dededej fdZdej dej fd Z dej dej fd Z dej d ejdej fd Z xZ S) Gemma3nAudioAttentionrjct|||_|jj|_|jj |_|j |jz|_|jj|_|jj|_ td|jjdz |_ |jj|_|j|jz|jz|_t#||_t'j(t+j,|j f|_t'j0|j |j|j zd|_t'j0|j |j|j zd|_t'j0|j |j|j zd|_|j dz}dt*j&j8j;t+j<dz }|j?d||zjAjCd t+jDt+jF|j |jft*jH d jJ}t+jDt+jF|j|j ft*jH |j|jz }t+jF|j|j ft*jH }||z|z}|j?d |d |j?d t+j<|jjMd y)Nrr FrlrDq_scalerEr)diagonallocal_causal_valid_masksoftcap)'rGrHrjrorprqrsconf_attention_chunk_size chunk_sizerwmax_future_horizonrtrumax_past_horizonconf_attention_logit_capattention_logits_soft_cap context_sizerirelative_position_embeddingrIrJr.zeros per_dim_scaleryq_projk_projv_projrsoftplusrMrLclonedetachtrilrKrdTr[)rNrjr r_softplus_0lower_causal_maskupper_causal_maskrrOs r3rHzGemma3nAudioAttention.__init__7s  ==;;22((DNN: ++??"&++"J"J #At{{'N'NQR'R S)-)M)M& OOd.C.CCdF]F]]+PQW+X(\\%++t}}6F*GHii 0 0$..4==2PW\] ii 0 0$..4==2PW\] ii 0 0$..4==2PW\] --%UXX0099%,,s:KLL  Y<)?(F(F(H(O(O(Q^cd!JJ JJ))4??;5:: N  ! "JJ JJ):):;5:: N**T-D-DD #(**doot?P?P-QY^YcYc"d"9 > @  r2rVpad_left pad_rightrXc|j^}}}|j||g|}|j||g|}tj|||gd}|S)Nr r)r` new_zerosr.r) rNrVrrbatchr tail_shapeleftrights r3 _pad_dim1zGemma3nAudioAttention._pad_dim1bs^ !q:{{E89j9: UI; ;< IItQ&A .r2r9c(|j}|dd\}}||jzdz |jz}||jz|z x}dkDr|j|d|}|||jf|ddz}|j|j }|S)aETurns a sequence to non overlapping blocks. Args: hidden_states: a tensor of [batch, time, ...]. Returns: A tensor of [batch, num_blocks, block_size, ...], with necessary paddings, where output[:, i, ...] are x[:, i*block_size:(i+1)*block_size, ...]. Nrr r)r`rrr contiguous)rNr9r`bt num_blocks padding_len permute_dimss r3_convert_to_blockz'Gemma3nAudioAttention._convert_to_blockis##Ray1$//)A-$//A %7!; ;Kq @ NN=![IM:t7%)C %--l;FFH r2c\|j}|j|jzdz }|j|||}|j}|j}|j d||}|j dkDr'|j dkDrtj|dd}|jS)aExtracts temporal context for every block. Args: hidden_states: a tensor of [batch, time, ...]. Returns: A tensor of [batch, num_blocks, context_size, ...], with necessary paddings, where context_size = block_size + left_context + right_context, and output[:, i, ...] are x[:, start-left_context:end+right_context, ...], start = i * block_size, end = (i + 1) * block_size. r ) dimensionsizesteprrrQ)source destination) rrrrrunfoldndimr.movedimr)rNr9rr frame_len frame_step x_unfoldeds r3_extract_block_contextz,Gemma3nAudioAttention._extract_block_contexts((++doo=A }h J %% __ #))AIJ)W     !joo&9z"!LJ$$&&r2maskc  g|jdd|j|j}|j|j |j }|j |j |j }|j|j |j }tjjj|j}ddd|jf}|j|} ||jz| z}|jdd\} } |j|} |j!|} |j!|}| jd}|}|j!|}|j"dk(rI|jd|jdz|j$k(r|j | ||j$}|j| ||j$fk7r,t'd|jd| d|d|j$d |j)dj)d }|j*j)d j)d j)d }tj,||j/|j0}|j3| | }|j4j/|j0}||z }tj6|}||z}tj8||tj:|j<j>}tjjjA|dtjB j/|j< }|j\}}}}}|jd}|jEd ddddj d||}|jEd ddddj d||}tjF||} | j |||||jEd dddd}!|!j | ||jHz|j|jf}!|!ddd| f}!|!S)NrQr rrrz%Shape of extracted_valid_mask_blocks z is not (z, z) after potential reshape.rr?rr)%r`rprsrrrrrr.rIrrrviewrrrrr ValueErrorrr logical_andrrrrtanhwherefinforminsoftmaxrrbmmr)"rNr9r qkv_shape query_states key_states value_statesper_dim_scale_spbroadcast_shapeper_dim_scale_sp_broadcastrq_time query_blocks key_blocks value_blocksroriginal_valid_maskextracted_valid_mask_blockscondition_from_input_validitycondition_from_causalityfinal_condition_for_wherer7 softcap_val probabilitiesb_dimn_dimu_dimw_dimc_dimh_dimprob_bunv_bun result_bmmcontext_vectorss" r3r^zGemma3nAudioAttention.forwardsTNm))#2.NN N {{=199)DOOQ [[/77 BMMO {{=199)DOOQ  88..778J8JKaDMM2%5%:%:?%K"#dll25OO )//3 F--l; 00< 22<@ '--a0 $e'+&A&ABU&V# ( , , 1+11!47R7X7XYZ7[[_c_p_pp*E*M*M,d.?.?+ ' ' , ,     1  /556i |L$%R(9(9'::TV )D(M(Ma(P(Z(Z[](^%$(#?#?#I#I!#L#V#VWX#Y#c#cde#f %*$5$5 ) $ ' '(E(L(L M% ! 11, Klloofmm4 +%F#+%6 FLL@Y@]@]^++33F%--3X[[bnbtbt[u -:,?,?)ueUE""2& ((Aq!Q7??E5Q$$Q1a3;;BuMYYx/ $,,UE5%OWWXY[\^_abdef)11 4??2   *!WfW*5r2)r*r+r,r!rHr.rercrrr BoolTensorr^rfrgs@r3rr6s) 1) V5<<335<<u|| ,.'ELL.'U\\.'`dU\\d9I9Idelldr2rcreZdZdZ d dedeedeffd ZdejdejfdZ xZ S) Gemma3nAudioCumulativeGroupNormaApplies Group Normalization cumulatively over the time dimension. This layer normalizes the input by calculating the mean and variance cumulatively over the time dimension (dim 1). The statistics are computed over all feature dimensions (specified by `feature_dims` and `num_channels`) for elements marked as valid by the optional `mask`. If a `mask` is provided (True for valid, False for invalid/padded), invalid time steps do not contribute to the statistics calculation, and their corresponding output values are zeroed out. Scale and bias, if enabled, are applied per-channel (last dimension). This behavior is similar to JAX's `GroupNormalization` with `num_groups=1` and `cumulative=True`. num_channels feature_dimsr@c t|||_t||_||_t jtj||_ ttddt|jzdz|_ y)Nrr )rGrHr!r<r"r@rIrJr.rKrCrangelenreduction_axes)rNr!r"r@rOs r3rHz(Gemma3nAudioCumulativeGroupNorm.__init__'sr (!,/ll5::l#;< $E!QT5F5F1G-G!-K$LMr2r9rXc|j|jfz}|jdd|k7rtd|jddd||j}t j }|j|}t j||}t j||jd}t j|d }t j||jd} t j| d } t j| d } || z } || z jd} t j| |jd}t j|d }|| z }|| z t j||jzz}|j j|}dg|j#dz z|jgz}||j%|z}||z}|j|S) zApplies cumulative group norm, optionally using a mask. Args: hidden_states: Input tensor, shape [B, T, *feature_dims, C]. Returns: Normalized tensor with the same shape as x. rNzInput tensor shape suffix z> does not match expected suffix (feature_dims + num_channels) rTr?rRr rrD)r)r"r!r`rrr.rr ones_likesumr&cumsumclamprTrsqrtr@rCr?r)rNr9expected_input_suffix input_dtype calc_dtypex_calc mask_calcsum_values_at_tcum_sum_valueselements_in_group_at_tcum_count_elementssafe_cum_count_elementscum_meansquared_diff_from_meansum_sq_diff_at_tcum_sum_sq_diff cum_variance normalized_xscalescale_view_shape final_outputs r3r^z'Gemma3nAudioCumulativeGroupNorm.forward9s!% 1 1T5F5F4H H   qr "&; ;,]-@-@-D,EF99N8OQ  $)) ]] !!*- OOF*=  ))F0C0CTRo1="'9$:M:MW[!\"\\*@aH"'++.@c"J"$;; #)8"3!8!8!; 99%;ATAT^bc ,,'7Q?')@@ )U[[9P-QQ  z*3-"3"3"5"9:d>O>O=PP#ejj1A&BB $i/ {++r2)gMbP?) r*r+r,r-rcrr[rHr.rer^rfrgs@r3r r sT( NNsmN N$G,U\\G,ellG,r2r c ~eZdZdZ d dedededeeeeefffd Zdejdejfd Z xZ S) Gemma3nAudioSSCPConvBlockzA single convolution block for the SubSampleConvProjection. This block consists of a 2D convolution, followed by CumulativeGroupNorm, and a ReLU activation. It handles manual padding for the convolution. rjidxinput_freq_dimmanual_paddingcJt |||_||_|dk(rdn|jj|dz }|jj|}|jj |\}}|jj |\} } tj||||f| | fdd|_ ||jdz|jdz} | |z | zdz} t|| f|jj|_ tj|_y)Nrr )rrF) in_channels out_channels kernel_sizestridepaddingrm)r!r"r@)rGrHrjrEsscp_conv_channel_sizesscp_conv_kernel_sizesscp_conv_stride_sizerIConv2dconvr sscp_conv_group_norm_epsnormReLU activation)rNrjrCrDrErGrHkernel_hkernel_wstride_hstride_w f_in_padded f_out_convrOs r3rHz"Gemma3nAudioSSCPConvBlock.__init__s%  ,!8a)K)KCRSG)T {{99#> ![[>>sC(![[>>sC(II#%h'   %t':':1'==@S@STU@VV !H,9A= 3%$ 44  '')r2audio_encodingsrXc6tj||jdd}|j|}|j ddddj }|j |}|j ddddj }|j|S)Nconstantr)modevaluerrrr )FrrErPrrrRrT)rNr[audio_encodings_paddedaudio_encodings_conv x_for_normx_normedaudio_encodings_normeds r3r^z!Gemma3nAudioSSCPConvBlock.forwards"#8K8KR\dg!h $yy)?@*11!Q1=HHJ 99Z(!)!1!1!Q1!=!H!H!J566r2))rrrr) r*r+r,r-r!rcr<rHr.rer^rfrgs@r3rBrBsc5A )$")$)$ )$ c3S01 )$V7u||7 7r2rBc\eZdZdeffd Zdej dej fdZxZS)#Gemma3nAudioSubSampleConvProjectionrjcpt|||_|j}g}g}t dD]n}|j |\}}|j |\}} d} |dz } d} d} | | | | f}|j||| z| z}||z | zdz}|j||}ptd|j||d|_ td|d||d|_ |jd}|d}||z|_ tj|j|jjd|_y)Nrrr )rCrDrjrErQFrl)rGrHrjinput_feat_sizer$rMrNappendrBconv_0conv_1rLinput_proj_in_featuresrIryrqinput_proj_linear)rNrjcurrent_f_for_block_inputcalculated_block_paddingcalculated_f_out_dimsirUrVrWrX pad_t_top pad_t_bottom pad_f_left pad_f_rightmanual_padding_tuplerYf_out_after_conv final_c_out final_f_outrOs r3rHz,Gemma3nAudioSubSampleConvProjection.__init__s  $*$:$:!#% "q 9A!'!=!=a!@ Hh!'!=!=a!@ HhI#a pre_attn_normrattnrIrypost post_normrNrjrOs r3rHz'Gemma3nAudioConformerAttention.__init__s  $ 7 7 0%,,t{{?\?\2]jop+DKK,C,CD)&1 IId33T[[5L5LSXY ' (?(?@r2r[audio_mel_maskrXc|}tj||j |j}|j|}|j ||}|j \}}}} |j |||| z} |j| }tj||j |j}||j|zSrZ) r.r,rrrr`rrr) rNr[raudio_encodings_input_to_attnaudio_encodings_normaudio_encodings_attn_outrrrprsr|s r3r^z&Gemma3nAudioConformerAttention.forwards(7%++o8N8N7NPTPfPfg#11/B#'99-A>#R %=$B$B!1i#;#C#CAq)V^J^#_ ))$<=++o8N8N7NPTPfPfg,t~~o/NNNr2 r*r+r,r!rHr.rerr^rfrgs@r3rrsAA1AOu||OUEUEUOZ_ZfZfOr2rc\eZdZdeffd Zdej dej fdZxZS) Gemma3nAudioConformerFeedForwardrjct|||_|jdt j |jj dt|jj|_ tj|jj|jjdzd|_ tj|jjdz|jjd|_ t|jj|_t j |jj|_y)NrFrErrl)rGrHrjrLr.rMrr>rqpre_layer_normrIry ffw_layer_1 ffw_layer_2post_layer_normconf_residual_weightpost_layer_scalers r3rHz)Gemma3nAudioConformerFeedForward.__init__,s   0%,,t{{?\?\2]jop,T[[-D-DE99T[[%<%U>UXY>Y`ef99T[[%<%(,(8(8(I--,,_=(,(8(8(I++o8N8N7NPTPfPfg..??T-B-BBCCr2rrgs@r3rr+s0 O1 O Du|| D Dr2rc\eZdZdeffd Zdej dej fdZxZS) Gemma3nAudioConformerLightConv1drjc t|||_t|jj|jj |_tj|jj|jjdzd|_ tj|jj|jj|jjdd|jjd|_ |jdtj|jj d t|jj|jj |_tj|jj|jjd|_|jjdz |_y) Nr@rFrlr r)rGrHrIrJrKgroupsrmrrE)rGrHrjr>rq rms_norm_epsrrIry linear_startConv1dconf_conv_kernel_sizedepthwise_conv1drLr.rMr conv_norm linear_endcausal_paddingrs r3rHz)Gemma3nAudioConformerLightConv1d.__init__EsD  ,T[[-D-D$++JbJbcIIdkk&=&=t{{?V?VYZ?Zafg "  //00 99;;**!  0%,,t{{?\?\2]jop' (?(?T[[E]E]^))DKK$;$;T[[=T=T[`a"kk??!Cr2r[rXc:|}|j|}|j|}tjjj |d}|j ddd}tj||jdf}|j|}|j ddd}tj||j |j}|j|}tjj|}|j|}||z}|S)NrQrrrr )rrr.rIrglurr`rrrr,rrrr)rNr[audio_encodings_residualaudio_encodings_permutedaudio_encodings_permuted_paddedr]s r3r^z(Gemma3nAudioConformerLightConv1d.forwardZs#2 --o>++O<((--11/r1J#2#:#:1a#C *+%%0H4K^K^`aJb*c'//0OP)11!Q:++o8N8N7NPTPfPfg..9--,,_=///: #;; r2rrgs@r3rrDs-D1D*u|| r2rcteZdZdeffd Zdej dejdej fdZxZ S)Gemma3nAudioConformerBlockrjct|||_t|j|_t |j|_t|j|_t|j|_ |jdtj|jjdt|jj|_y)NrFrE)rGrHrjrffw_layer_startr attentionrlconv1d ffw_layer_endrLr.rMrr>rqrRrs r3rHz#Gemma3nAudioConformerBlock.__init__ps  ? L7 D7 D =dkkJ 0%,,t{{?\?\2]jop"4;;#:#:; r2r[rrXcj|j|}|j||}|}||jdj|jz}|j |}|j |}tj||j |j}|j|}|S)NrQ) rrrrrrrr.r,rrR)rNr[rvalidity_mask_for_lconvaudio_encodings_for_lconv_inputr]s r3r^z"Gemma3nAudioConformerBlock.forward{s..?...I#1/*9v?\?\9] ^A ' / ^  ^sA-rrXc,|j|}|jd}d}tt|jj D]!}||jj |dz}#t j||j|z}t j||jddz }|jdkDr>|jdk(r/|jdj|jdd}n`|j|jk(rG|jddk(r5|jddk7r#||jdk(r|jd}t j|d|}|jD] } | ||} |jjdkDr@|dddd|jjf}|dddd|jjf}|j!|jdd}||fS)aEncodes a batch of MELs. Args: audio_mel: a torch.Tensor of shape [batch, num_frames, num_channels, mel_bins]. Returns: audio_encodings: a torch.Tensor of shape `[batch_size, self.config.audio_soft_tokens_per_image, self.config.audio_config.hidden_size]` audio_mel_mask: a torch.BoolTensor of shape [batch, num_frames]. r rr)rtrQNr)rr`r$r%rjrNr.r~rr,rrexpandgatherrconf_reduction_factor masked_fill) rNrrr[t_subtime_stride_productstride_pair_idxindices current_maskblocks r3r^zGemma3nAudioEncoder.forwards88C %%a($S)J)J%KL YO 4;;#D#D_#UVW#X X  Y ,,u^-B-BCFYY++g>+?+?+BQ+FG    "w||q'8''*11.2F2Fq2I2NG   7<< /$$Q'1, a A%q))''*G||NAw? ^^ CE#O\BO C ;; , ,q 0-a1UDKK4U4U1U.UVO'+Odkk.O.O+O(OPL)55l6L6LR6PRUV ,,r2)r*r+r,r-r!r0main_input_namerHr.rerr<r^rfrgs@r3rrsYp !O 1 5-5-7<7G7G5- u||U--- .5-r2rc ZeZdZdZd dedededeffd Zdejffd Z xZ S) Gemma3nTextScaledWordEmbeddingz\ This module overrides nn.Embeddings' forward by multiplying with embeddings scale. num_embeddings embedding_dim padding_idx embed_scalecvt|||||jdtj|dy)NrFrE)rGrHrLr.rM)rNrrrrrOs r3rHz'Gemma3nTextScaledWordEmbedding.__init__s3  D ]ELL,ERWXr2 input_idsct|||jj|jj zSrZ)rGr^rrrCr)rNrrOs r3r^z&Gemma3nTextScaledWordEmbedding.forwards2wy)D,<,<,?,? @Q@Q,RRRr2)rD) r*r+r,r-rcr[rHr.rer^rfrgs@r3rrsGYsY3YSY_dYSSSr2rc`eZdZdZdeffd ZdejdejfdZxZ S)Gemma3nTextLaurelBlockz Learned Augmented Residual Layerrjct|||_tj|jj |jj d|_tj|jj |jj d|_t|jj |jj|_ y)NFrlr) rGrHrjrIryrq laurel_rank linear_left linear_rightr>rpost_laurel_normrs r3rHzGemma3nTextLaurelBlock.__init__s  99T[[%<%U>U\abIIdkk&=&=t{{?V?V]bc .t{{/F/FDKKLdLd er2r9rXcr|j|}|j|}|j|}||zSrZ)rrr)rNr9laurel_hidden_statesnormed_laurel_hidden_statess r3r^zGemma3nTextLaurelBlock.forwardsC-1-=-=m-L-1->->?S-T&*&;&; YYv2teL n!<<}}!!&8"344r2)r) r*r+r,r#rcrHr.rer^rrfrgs@r3rrsP Q0 QS QU\\ell5U\\5ell5r2rcXeZdZdZdeffd ZdejdejfdZdejdejfdZ d ejd ejdejfd Z d ejdejfd Z d ejdejfdZ xZ S)Gemma3nTextAltUpaAlternating Updates (AltUp) The AltUp module wraps transformer layers. The `predict` step modifies the input to the transformer layer, and the `correct` step propagates the output of the transformer layer to the sparsely updated dimensions. See more in the research paper: https://proceedings.neurips.cc/paper_files/paper/2023/file/f2059277ac6ce66e7e5543001afa8bb5-Paper-Conference.pdf rjcFt|||_tjt j |jj|_tj|jj|jjd|_ tj|jj|jjdzd|_ tj|jj|jjd|_ t|jj|jj|_|j#dt j$|jjdzdy)NFrlrrrouter_input_scalegrE)rGrHrjrIrJr.rrqcorrect_output_scaleryaltup_num_inputscorrection_coefsprediction_coefsmodality_routerr>r router_normrLrMrs r3rHzGemma3nTextAltUp.__init__#s  $&LLT[[=T=T1U$V! " $++*F*F HdHdkp q " $++*F*F HdHdfgHgns t!yy)@)@$++B^B^ejk)$++*A*At{{G_G_` 15<< @W@WY]@]3^kpqr2rVrXc|j||jz}|j|}tj|j j |SrZ)r rr r.rr[r\)rNrV router_inputsrouteds r3compute_router_modalitiesz*Gemma3nTextAltUp.compute_router_modalities-sM((+d.E.EE %%m4zz&,,.)11!44r2r9c|j||jj}|jro|jjY|j j jj|jj |jj|j |jg|jdd|jj|jjjdddd}tj|jdddd|}|jdddd}||z }|jj!|S)aPredicts the output of a layer using a trainable map. Args: hidden_states: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` derived by stacking the input embeddings and preprocessing the last `num_altup_inputs - 1` matrices. Returns: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` containing the predictions. NrQrr rr)rrjaltup_active_idxtrainingaltup_coef_clipr rCdataclamp_rr`rrr.rrr\)rNr9 modalities all_coefs predictionss r3predictzGemma3nTextAltUp.predict2s@33M$++B^B^4_` ==T[[88D  ! ! ( ( - - 4 4dkk6Q6Q5QSWS^S^SnSn o D ! !* - W i &&s+ i-1[[-I-I iKO;;KgKg i WQ1a  ll=#8#8Aq!#DiP !))!Q15 }$ %%'// >>r2r activatedcj|j|}|||jjz }|j|jjddd}|jj Y|j jjj|jj |jj |j |dz}|jdddjd}tj||}||z }|jj|S)a_Corrects the predictions relative to the Args: predictions: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` derived by stacking the input embeddings and preprocessing the last `num_altup_inputs - 1` matrices. activated: A 3D tensor of shape `[batch_size, num_tokens, hidden_size]` containing the activated inputs. Returns: A 4D tensor of shape `[num_altup_inputs, batch_size, num_tokens, hidden_size]` correcting the original predictions relative to the activated input embeddings. r rDrrrQ)rrjrrepeatrrr rCrrrrr.mulrr\)rNrrr innovationr correcteds r3correctzGemma3nTextAltUp.correctNs33I> T[[-I-I!JJ &&t{{'C'CQ1M ;; & & 2  ! ! ( ( - - 4 4dkk6Q6Q5QSWS^S^SnSn o #'"7"7 "Cc"I %%aA.88< IIj)4 [ ##%--i88r2r cp|j|j|jzj|S)a  This is only defined as the `forward` so that accelerate hooks can move correctly `correct_output_scale` (which is a nn.Parameter, not a Module) between devices when offloading. It is otherwise only used in `scale_corrected_output` )r\rrNr s r3r^zGemma3nTextAltUp.forwardks2 !!$";"; *z&:M:Mt/T#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+>@GG HZHZ[\H]_acdehhijiqiqr ,QaZ 8 > > @'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BF%%F.rZ) r*r+r,r#rHr.no_gradrr^rfrgs@r3r'r'xs4/0/"U]]_ <