rh4ddlZddlmZddlmZmZmZmZddlZddl m Z ddl m cm Z ddl mZddlmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZm Z ddl!m"Z"m#Z#ddl$m%Z%m&Z&ddl'm(Z(ddl)m*Z*m+Z+m,Z,m-Z-m.Z.ddl/m0Z0m1Z1m2Z2e.jfe4Z5edGdde jlZ7Gdde jlZ8Gdde jlZ9Gdde jlZ:Gdde jlZ;Gdd e jlZZJdUd?ZKGd@dAe jlZLGdBdCe jlZMGdDdEeZNee+dFGGdHdIe ZOe+GdJdKeGZPe+GdLdMeGZQee+dNGGdOdPe ZRGdQdReGeZSgdSZTy)VN) dataclass)AnyCallableOptionalUnion) LayerNorm)ACT2FN)Cache DynamicCache)GenerationMixin)use_kernel_forward_from_hub)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPast ModelOutput)ROPE_INIT_FUNCTIONSdynamic_rope_update)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleis_torchdynamo_compilinglogging) Glm4vConfigGlm4vTextConfigGlm4vVisionConfigRMSNormc,eZdZdfd ZdZdZxZS) Glm4vRMSNormct|tjt j ||_||_y)z; Glm4vRMSNorm is equivalent to T5LayerNorm N)super__init__nn Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s {/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/glm4v/modeling_glm4v.pyr'zGlm4vRMSNorm.__init__2s1 ll5::k#:; #c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NTkeepdim) dtypetor*float32powmeanrsqrtr-r,)r. hidden_states input_dtypevariances r2forwardzGlm4vRMSNorm.forward:sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r3c^t|jjd|jS)Nz, eps=)tupler,shaper-r.s r2 extra_reprzGlm4vRMSNorm.extra_reprAs* ))*+6$2G2G1HIIr3)gư>)__name__ __module__ __qualname__r'rBrG __classcell__r1s@r2r$r$0s$;Jr3r$c,eZdZddeffd ZdZxZS) Glm4VisionMlpbiasct||j|_|j|_t j |j|j||_t j |j|j||_t j |j|j||_ t|j|_ yNrO) r&r'r/out_hidden_sizeintermediate_sizer(Linear gate_projup_proj down_projr hidden_actact_fn)r.configrOr1s r2r'zGlm4VisionMlp.__init__Fs !--!'!7!74#3#3T5K5KRVWyy!1!143I3IPTU 4#9#94;K;KRVWV../ r3c|j|j|j||j|zSN)rXrZrVrWr. hidden_states r2rBzGlm4VisionMlp.forwardOs2~~dkk$..*FG$,,WcJddeer3F)rHrIrJboolr'rBrKrLs@r2rNrNEs0T0fr3rNc`eZdZdeddffd Zdej dej fdZxZS)Glm4vVisionPatchEmbedr[returnNcTt||j|_|j|_|j|_|j |_|j|j|jg}tj|j|j |||_ y)N) kernel_sizestride) r&r' patch_sizetemporal_patch_size in_channelsr/ embed_dimr(Conv3dproj)r.r[rfr1s r2r'zGlm4vVisionPatchEmbed.__init__Ts  ++#)#=#= !--++//$//R IId..K`kl r3r?c6|jjj}|jd|j|j |j |j }|j|j|jd|j}|S)Nr6r9) rmr,r9viewrjrirhr:rk)r.r? target_dtypes r2rBzGlm4vVisionPatchEmbed.forward^s~yy''-- %**   $":":DOOT__  -"2"2"2"FGLLRQUQ_Q_` r3 rHrIrJr!r'r*TensorrBrKrLs@r2rcrcSs5m0mTmU\\ellr3rccReZdZddededdffd ZdedejfdZxZ S) Glm4vVisionRotaryEmbeddingdimthetardNct|d|tjd|dtj|z zz }|j d|dy)N?rr5roinv_freqF persistent)r&r'r*arangefloatregister_buffer)r.rvrwrzr1s r2r'z#Glm4vVisionRotaryEmbedding.__init__hsK %ELLC%++$NQT$TUV ZeDr3seqlenctj||jj|jj}tj ||j}|S)Ndevicer9)r*r}rzrr9outer)r.rseqfreqss r2rBz"Glm4vVisionRotaryEmbedding.forwardmsAll6$--*>*>dmmFYFYZ C/ r3)g@) rHrIrJintr~r'r*rsrBrKrLs@r2rurugs8ECEEDE cellr3ruc neZdZd dededededdf fd Zdejdejfd Z xZ S) Glm4vVisionPatchMergerrv context_dimrYrOrdNcxt|tj||||_t ||_tj||||_tj||||_tj||||_ tj|_ t||_ yrQ)r&r'r(rUrmrpost_projection_normrVrWrXGELUact1r rZ)r.rvrrYrOr1s r2r'zGlm4vVisionPatchMerger.__init__ts IIc3T2 $-cN!3 $?yyk= ;$?GGI Z( r3r_c|j|}|j|j|}|j|j |j ||j |zSr])rmrrrXrZrVrWr^s r2rBzGlm4vVisionPatchMerger.forward~sYyy. yy!:!:1D!--"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr3rdc |jj}|jd}|jd}|j} |j | |j | }}|dk(r%t j d|| |j} nt|tr&t j|| t j}t|t js&t j|| t j}|jd} t| dz} |j| | |jdddj!dj | t j"} t j$t't)|Dcgc]}||dfj+||c}j | t j"}t j$t't)|Dcgc]}||dfj+||c}j | t j"}|j | t j"}|j | t j"}|dz|z dzdz }|dz|z dzdz }t j,||fdj!dj!d}t/j0| |dd d }|j3dj3djdd}|j |jj |j} || z}|Scc}wcc}w) a Forward pass with integrated position encoding adaptation using 2D interpolation. Args: embeddings: Input embeddings tensor lengths (torch.Tensor): Sequence lengths for each image in the batch. image_shapes (torch.Tensor): Tensor of shape [batch_size, 3] representing the image shapes (t, h, w). h_coords (torch.Tensor): Tensor of shape [total_seq] representing the h coordinate for each patch. w_coords (torch.Tensor): Tensor of shape [total_seq] representing the w coordinate for each patch. Returns: torch.Tensor: Embeddings with adapted position encoding added. rrrg?r5r6rvbicubicFborder)mode align_corners padding_mode)rr,rErr:r*emptyr9 isinstancelisttensorlongrsrrppermute unsqueezer;catrangelenrepeatstackF grid_samplesqueeze)r. embeddingslengths image_shapesh_coordsw_coordspos_embed_weightr/ total_seqradapted_pos_embed orig_size_sq orig_size pos_embed_2ditarget_htarget_wnorm_wnorm_hgridinterpolated_embed_fp32adapted_pos_embed_fp32s r2rBzGlm4vVisionEmbeddings.forwards 2299&,,Q/ NN1% !((&[[0(++f2E( > % A{6QaQgQg h '4(,,wvUZZPlELL9$||LuzzZ ,11!4LL#-.I %%iKHAq!167 yyQVWZ[bWcQd!eA,q!t"4";";GAJ"G!efiiU]]jHyyQVWZ[bWcQd!eA,q!t"4";";GAJ"G!efiiU]]jH  {{& {FH{{& {FH#~1Q6:F#~1Q6:F;;/R8BB1EOOPQRD'(mmd%V^' # &=%D%DQ%G%O%OPR%S%[%[\]_`%a " 6 9 9:J:P:P Q T TU_UfUf g  "33 7"f"fs  M" M'rrrLs@r2rrs' q0 qGPUP\P\Gr3rc|dd|jddzf}|d|jddzdf}tj| |fdS)*Rotates half the hidden dims of the input..Nr6r5r)rEr*rxx1x2s r2 rotate_halfrsZ 3"!''"+"" " #B 3 q " " #B 99rc2YB ''r3qkcossinrdc|j}|j}|j|j}}|jdj|jdj}}||zt||zz}||zt||zz}|j |}|j |}||fS)N)r9r~rrr:)rrrr orig_q_dtype orig_k_dtypeq_embedk_embeds r2apply_rotary_pos_emb_visionrs77L77L 779aggiqA}}R &&(#--*;*A*A*CC3w;q>C/0G3w;q>C/0Gjj&Gjj&G G r3r?n_repc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rErreshape)r?rbatchnum_key_value_headsslenhead_dims r2 repeat_kvrso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr3modulequerykeyvalueattention_maskscalingdropoutkwargsc Tt||j}t||j} tj||j dd|z} |#|ddddddd|j df} | | z} t jj| dtjj|j} t jj| ||j} tj| | } | j ddj} | | fS)Nr5r rr6rvr9)ptrainingr)rnum_key_value_groupsr*matmul transposerEr( functionalsoftmaxr;r:r9rr contiguous) rrrrrrrr key_states value_states attn_weights causal_mask attn_outputs r2eager_attention_forwardrs3 ; ;}}}}tj|d}|j|dj5}|j7|}|Scc}wcc}}}w)Nrr r6rr5aSThe attention layers in this model are transitioning from computing the RoPE embeddings internally through `rotary_pos_emb` (2D tensor of RoPE theta values), to using externally computed `position_embeddings` (Tuple of tensors, containing cos and sin). In v4.54 `rotary_pos_emb` will be removed and `position_embeddings` will be mandatory.reagerflash_attention_2F)rrr cu_seq_lens_q cu_seq_lens_k max_length_q max_length_kr)rrrr)rErrrrunbindlogger warning_oncer*rrrrrrrr[_attn_implementationrmaxrrrsplittolistziprrm)r.r?rrrr seq_length query_statesrrembrrattention_interface max_seqlenr_rrsplitsrrv attn_outputss r2rBzGlm4vVisionAttention.forward$s#((+ HH] # + +J4>>2 N V VWXZ[]^`a b i ijk l / j,  &   G  ))^^<"EC'')C'')C*HC#>|ZY\^a#b j#--a3==a@ ))!Q/99!< #--a3==a@ (? ;; + +w 6"9$++:Z:Z"[  ;; + +/B B$QR.:cr?:??AJ0  $ #'==d6L6L((''NK"!nz#26GLXZdfrKsAG FGNN$4!<F #F|  Aq!$  $( LL'+}}C$:P:P#    L  ))La8K!))*b9DDF ii , - s '-J1"AJ6NN) rHrIrJr!r'r*rsrrDrBrKrLs@r2rrs 0 T "26KO M||MLLM!. M &eELL%,,,F&GH M Mr3rc eZdZdfd Z d dej dej deej deeej ej fdej f dZxZ S) Glm4vVisionBlockrdct|t|j|j|_t|j|j|_t||_t|d|_ y)Nr0FrR) r&r'r$r/ rms_norm_epsnorm1norm2rattnrNmlprs r2r'zGlm4vVisionBlock.__init__us\ !&"4"4&:M:MN !&"4"4&:M:MN (0  e4r3r?rrrc ||j|j|f|||d|z}||j|j|z}|S)N)rrr)r$r"r%r#)r.r?rrrrs r2rBzGlm4vVisionBlock.forward|sc&  JJ} %) !) 3 )  )  &M1J(KK r3rdNr) rHrIrJr'r*rsrrDrBrKrLs@r2rrtsq526KO ||LL!.  &eELL%,,,F&GH  r3rc<eZdZUeed<dZdZddgZdZdZ dZ dZ dZ y)Glm4vPreTrainedModelr[modelTGlm4vTextDecoderLayerrpast_key_valuesN) rHrIrJr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_can_compile_fullgraph_supports_attention_backendr3r2r)r)s= &*#02DE"3N!"&r3r)ceZdZUeed<dgZd fd ZdZdejdejdejfdZ xZ S) Glm4vVisionModelr[rrdcFt|||j|_|j|_t ||_t ||_|j|jz}t|dz|_ tjt|jDcgc] }t!|c}|_t%|j&|j(|j*|_t/|j|j0|_tj4|j|j&|j|j|_t/|j|j0|_d|_|j=ycc}w)Nr5)rvrrYr )rj out_channelsrfrgF)r&r'spatial_merge_sizerhrrrc patch_embedr/rrurr( ModuleListrdepthrblocksrrSrTrYmergerr$r!post_conv_layernormConv2d downsamplepost_layernormgradient_checkpointing post_init)r.r[rrr1s r2r'zGlm4vVisionModel.__init__sA  "(";"; ++/708%%)9)998QGmmuV\\GZ$[!%5f%=$[\ ,&&F4L4LY_YjYj  $00B0BH[H[#\ ))**//11,,   +6+=+=6CVCVW&+# %\s%Fcg}|D]s\}}}tj|jdjd|}|j ||j z|j ||j z|j }|j dddd}|j}tj|jdj|d}|j ||j z|j ||j z|j }|j dddd}|j}|jtj||gdj|dvtj|d}|ddddfj}|j|} | |jd} | |fS)Nrr6rr5r r)r*r}rrrr;rflattenappendrrrrr) r.grid_thwpos_idsthwhpos_idswpos_ids max_grid_sizerotary_pos_emb_fullrs r2 rot_pos_embzGlm4vVisionModel.rot_pos_embs SGAq!||A003::2qAH''T,,,''T,,,'' H  ''1a3H'')H||A003::1bAH''T,,,''T,,,'' H  ''1a3H'')H NN5;;(';DKKAqQ R) S*))G+ AB++- "11-@,W5==a@w&&r3r?rJc |j|}|j|}|j|\}}tj||fd}|j |j f}tj|dddf|dddfz|dddfjdtjjr |jntj}tj|dd }|dd|ddz j}|j!||||dddf|dddf}|j"D] } | ||| }|j%|}|j'd|j(|j(|j*d}|j-dd dd}|j/|j'd|j0j2}|j5|}|S) az Args: hidden_states (`torch.Tensor` of shape `(seq_len, hidden_size)`): The final hidden states of the model. grid_thw (`torch.Tensor` of shape `(num_images_or_videos, 3)`): The temporal, height and width of feature shape of each image in LLM. Returns: `torch.Tensor`: hidden_states. r6rNrr5rr)rr)r)rrr )r<rArSr*rrrrepeat_interleavecumsumjit is_tracingr9int32rpadrrr?rDrpr;rErrCr[rSr@) r.r?rJrimage_type_idsrrrseqlensblks r2rBzGlm4vVisionModel.forwards((7 00? )-)9)9()C&ii8bA"wwy#'')4,,Xad^hq!tn-LhWXZ[W[n]dd %*II$8$8$:(.. e UU:vQ7 ab>JsO3;;= w.YZ\]Y]J^`noprsos`tu ;; C%$7M ++M: %** '')@)@-BUBUVXBY &--aAq9  6;;B @[@[\  M2 r3r') rHrIrJr!r-r0r'rSr*rsrBrKrLs@r2r8r8sB +,8':.U\\.U\\.ell.r3r8c^eZdZddeffd Zej edZxZ S)Glm4vTextRotaryEmbeddingr[ct|t|drG|j;|jj d|jj d|_nd|_|j |_|j |_||_ t|j |_ |j|j|\}|_ |jd|d|j|_y)N rope_scaling rope_typetypedefaultrzFr{)r&r'hasattrragetrbmax_position_embeddingsmax_seq_len_cachedoriginal_max_seq_lenr[r rope_init_fnattention_scalingrrzoriginal_inv_freq)r.r[rrzr1s r2r'z!Glm4vTextRotaryEmbedding.__init__ s  6> *v/B/B/N#0044[&BUBUBYBYZ`BabDN&DN"("@"@$*$B$B! /?+/+<+DDFMMaQ]QcQcdeQfhjlmn ,Q4] ; A A C'1!((--'E!((--[`J`ahhmmfk ^^ U C 5&,,.1F1L1L1NNYYZ[]^_E))UEN3C'')d444C'')d444C  5 vvAGGv$cff177f&;;;  5 5s BFFr]) rHrIrJr r'r*no_gradrrBrKrLs@r2r_r_ s3//"U]]_ < @P@PW\] r3r?rrrpast_key_valueoutput_attentions use_cachecache_positionrrdc  |j\} } } |j|} |j|}|j|}| j | | d|j j dd} |j | | d|j j dd}|j | | d|j j dd}|\}}t| ||||jd\} }|'|||d}|j|||j|\}}t}|jjdk7rt|jj}||| |||f|jsdn |j |j"d| \}}|j%| | dj'}|j)|}|||fS) Nr6rr5rz)rrrrr)rr)sizerrrrprrrraupdaterrr[rrrrrrrr)r.r?rrrrrrrrbszq_lenrrrrrr cache_kwargsrrrs r2rBzGlm4vTextAttention.forwards&**, UA{{=1 [[/ {{=1 #((eRGQQRSUVW __S%T]]CMMaQRS #((eRGQQRSUVW &S#B *c30A0A/0R$  j  %#&snUL'5'<'09!. 09 u//0 09 ! 09 0909!!1!1209-.09 u||Xell3XeELL>Q5RR S09r3rcVeZdZfdZdej dej fdZxZS) Glm4vTextMLPc*t|||_tj|j d|j zd|_tj|j |j d|_t|j|_ y)Nr5FrR) r&r'r[r(rUr/rT gate_up_projrXr rY activation_fnrs r2r'zGlm4vTextMLP.__init__sp  IIf&8&8!f>V>V:V]bc6#;#;V=O=OV[\#F$5$56r3r?rdc|j|}|jdd\}}||j|z}|j|S)Nr5r6r)rchunkrrX)r.r? up_statesgates r2rBzGlm4vTextMLP.forwardsL%%m4 #//!/4i 2 24 88 ~~i((r3)rHrIrJr'r* FloatTensorrBrKrLs@r2rrs'7)U%6%6)5;L;L)r3rcxeZdZdedeffd Z ddejdeejejfde ejde ejde eejd e e d e e d e ejd eeje eejejfffd Z xZS)r+r[rct||j|_t|||_t ||_t|j|j|_ t|j|j|_ t|j|j|_ t|j|j|_ y)Nr ) r&r'r/r self_attnrr%r$r!input_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernormrs r2r'zGlm4vTextDecoderLayer.__init__s !--+FI>'+F,>,>FDWDWX(4V5G5GVM`M`(a%(4V5G5GVM`M`(a%".v/A/AvGZGZ"[r3r?rrrrrrrrdc 0|} |j|}|jd||||||||d| \}} } |j|}| |z}|} |j|}|j |}|j |}| |z}|f} |r| | fz } |r| | fz } | S)N)r?rrrrrrrr6)rrrrr%r)r.r?rrrrrrrrresidualself_attn_weightspresent_key_valueoutputss r2rBzGlm4vTextDecoderLayer.forwards!,,]; ?Mdnn ? ' 3)%)/) ?  ? ; (*;55mD  =0 !55mD / // >  =0  "  )+ +G  )+ +Gr3r)rHrIrJr rr'r*rsrDrrrarrBrKrLs@r2r+r+s\\3\26378<,1$)59/||/#5<<#=>/!. / u//0 / !u||!45 /$D>/D>/!!1!12/ u  (51B1BEDUDU1U+V"WW X/r3r+zJ Base class for Llava outputs, with hidden states and attentions. ) custom_introceZdZUdZdZej ed<dZe e ej ed<dZ e e ej ed<dZ e e ej ed<dZe ejed<y)Glm4vModelOutputWithPasta past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Nlast_hidden_stater,r? attentions rope_deltas)rHrIrJrrr*rr-r,rrr?rDrrrr6r3r2rr s} ,0u((/9=OXd5#4#456=8b $D>b 'tnb !!1!12b -.b  u-- .b b r3rc%eZdZUdZiZeed<ddgZfdZdZ dZ dZ d Z d!d e ejd e ejd e ejd e ej deej ej ff dZ d"dej&d e ejfdZd"dej&d e ejfdZ d#d ejdej&dej&dej&fdZee d$d ejd e ej de ejde eej&de ej&de ede ede ede ej de ej&d e ejd e ejde ejde ejdeedeeeff d ZxZ S)% Glm4vModelr[r+rct||tj|j|_t j|j|_d|_ |jyr]) r&r'r8 _from_config vision_configvisualr text_configlanguage_modelrrFrs r2r'zGlm4vModel.__init__sU  &33F4H4HI ,99&:L:LM r3c6|jjSr])rget_input_embeddingsrFs r2rzGlm4vModel.get_input_embeddingss""7799r3c:|jj|yr])rset_input_embeddingsr.rs r2rzGlm4vModel.set_input_embeddingss 007r3c||_yr]rr.decoders r2 set_decoderzGlm4vModel.set_decoders %r3c|jSr]rrFs r2 get_decoderzGlm4vModel.get_decoders"""r3rimage_grid_thwvideo_grid_thwrrdc F|jjj}|jj}|jj}|jj }g} |h||c|} |t j| }t jd|jd|jd|j|j} d\} } d}|j| j}t| D]\}}|||dk(}|j}g}d}|D]T}||k(rd}n||k(rd}||k(r|s|jd+||k(r|r|jd D|jd Vg}t!j"t|d D]7\}}t%|}|dd}|d ddz}|j|||f9g}d}|D]:\}}}t'|dkDr|d j)dznd}|dk(rQ|| d|| d|| d }!} }|j+| j+|z|!j+|z}$}#}"t j,|"j/d dj1d |#|$zj3}%t j,|#j/dd dj1|"d |$j3}&t j,|$j/ddd j1|"|#d j3}'|jt j4|%|&|'g|z| dz } d}|d k(rb||| d|| d }!} }|| j+|z|!j+|z}$}#}"t7|"D]}(t j8|(j/d dj1d |#|$zj3}%t j,|#j/dd dj1dd |$j3}&t j,|$j/ddd j1d|#d j3}'|jt j4|%|&|'g|z|dz }||| dk\r| dz } d}|dz }||z })|jt j,|)j/dd j1dd |zd}=t j:|dj=dd }*|*j| j| d|||dk(f<| j|*j)dzt'| |z t j8| |jj?d} | | fS||jAjCd dz } | jE|dk(d| j?dj1dd d j|j} | j)dddj)d dd}+|+dz|jd z } | | fSt j,|jd|jj/ddd j1d|jdd } t jF|jddg|j|j} | | fS)aU Calculate the 3D rope index based on image and video's temporal, height and width in LLM. Explanation: Each embedding sequence contains vision embedding and text embedding or just contains text embedding. For pure text embedding sequence, the rotary position embedding has no difference with modern LLMs. Examples: input_ids: [T T T T T], here T is for text. temporal position_ids: [0, 1, 2, 3, 4] height position_ids: [0, 1, 2, 3, 4] width position_ids: [0, 1, 2, 3, 4] For vision and text embedding sequence, we calculate 3D rotary position embedding for vision part and 1D rotary position embedding for text part. Examples: Temporal (Time): 3 patches, representing different segments of the video in time. Height: 2 patches, dividing each frame vertically. Width: 2 patches, dividing each frame horizontally. We also have some important parameters: fps (Frames Per Second): The video's frame rate, set to 1. This means one frame is processed each second. tokens_per_second: This is a crucial parameter. It dictates how many "time-steps" or "temporal tokens" are conceptually packed into a one-second interval of the video. In this case, we have 25 tokens per second. So each second of the video will be represented with 25 separate time points. It essentially defines the temporal granularity. temporal_patch_size: The number of frames that compose one temporal patch. Here, it's 2 frames. interval: The step size for the temporal position IDs, calculated as tokens_per_second * temporal_patch_size / fps. In this case, 25 * 2 / 1 = 50. This means that each temporal patch will be have a difference of 50 in the temporal position IDs. input_ids: [V V V V V V V V V V V V T T T T T], here V is for vision. vision temporal position_ids: [0, 0, 0, 0, 50, 50, 50, 50, 100, 100, 100, 100] vision height position_ids: [0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1] vision width position_ids: [0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1] text temporal position_ids: [101, 102, 103, 104, 105] text height position_ids: [101, 102, 103, 104, 105] text width position_ids: [101, 102, 103, 104, 105] Here we calculate the text start position_ids as the max vision position_ids plus 1. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Returns: position_ids (`torch.LongTensor` of shape `(3, batch_size, sequence_length)`) mrope_position_deltas (`torch.Tensor` of shape `(batch_size)`) r rrr9r)rrFTimagevideotextc |dS)Nrr6)rs r2z+Glm4vModel.get_rope_index..s [\]^[_r3r6r5r.rr7r)$r[rr;image_token_idvideo_start_token_idvideo_end_token_idr* ones_liker+rEr9rr:ryrrI itertoolsgroupbyrrritemr}rprrHrrrrrrrrV masked_fill_zeros),r.rrrrr;rrrmrope_position_deltastotal_input_idsr image_index video_indexvideo_group_indexr input_tokensinput_token_typevideo_check_flgtokeninput_type_grouprgroup start_index end_indexllm_pos_ids_listvideo_frame_num modality_type start_idxend_idxst_idxrLrMrN llm_grid_t llm_grid_h llm_grid_wt_indexh_indexw_indext_idxtext_len llm_positionsmax_position_idss, r2get_rope_indexzGlm4vModel.get_rope_indexsLv"[[66II33#{{??![[;; "  n&@ND^'O%!&!A ::""oo '' L(, $K ! +../E/EFN )/ :W ` 9%nQ&71&<= (//1 #% "') 8E 44*."44*/.(//8.0_(//8(//7 8$& "+"3"3I>N4OQ_"`KJC KE"'(1+K %b ! q 0I$++S+y,IJ K $& "#9I7,5M9g?BCS?TWX?X-b1557!;^_F$/*;7:*;7:*;7: 1 FFHFFH(::FFH(::1;J #(,,z":"?"?A"F"M"MbR\_iRi"j"r"r"t"',,z":"?"?2q"I"P"PQ[]_ak"l"t"t"v"',,z":"?"?1b"I"P"PQ[]gik"l"t"t"v(// Wgw&>r1&E&L&LRQ[^hQh&i&q&q&sG&+ll:&>&C&CAr1&M&T&TUVXZ\f&g&o&o&qG&+ll:&>&C&CAq"&M&T&TUVXbdf&g&o&o&qG,33EKK'SZ@[4\_e4ef g*Q.),{0KA0NN'1,K01-'1,$+Y#6(// X0F0K0KAr0R0Y0YZ[]_0`ci0ij*+o7,r!& *: B J J1b Q ?L?O?OP\PcPc?d S!^A%6!%;;<%,,]->->-@1-Ds?[\K]G^-^_oW `p%*LL1FyO_O_$`$j$jkl$m !!66 6)-224;;B?!C )).A*=qA+55a8??2rJMMnNcNcd #/#3#3Au#3#Ea#H#L#LRY]#L#^_`#a (81(<~?S?STV?W(W% !66 6LL!3I%?&$&B9??1+=+BO!-M-2E2Ea2HA2M #-E,F(F)t+Fq0AQ0FV#t+T/M/M/OST/T &'*CHXHXH`,0,?,?""#8 -@-) k $/ -:,?,?) J&1$A&)9)99==m>R>RS %||J}?S?ST +00B7>>z2N !-!33J%++a.4PVW3XE+//6 +55a8??2rJ %$%%  %)+'/!5)    (%77#33!//))((   r3)NNNNr]r)NNNNNNNNNNNNNN)!rHrIrJr._checkpoint_conversion_mappingrr-r0r'rrrrrr*rrsrDrrrr!r.rrrrarrrrrBrKrLs@r2rrs%'" 02DE:8&# 15595915 |7E,,-|7!!1!12|7!!1!12 |7 !. |7 u||U\\) * |7~dh#(#4#4FNuO_O_F`0u/@/@RZ[`[k[kRl(-1,0 &6##&6((&6)) &6 )) &6P'+1537=A59$(,0/3/3;?59592659t ##t !.t u//0 t "$u'8'8"9: t   1 12 t D>t $D>t 'tnt u||,t &e&7&78t !!1!12t !!1!12t e../t !!1!12t +,!t " u.. /#t t r3rzQ Base class for Glm4v causal language model (or autoregressive) outputs. ceZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZeej ed<y) Glm4vCausalLMOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Nlosslogitsr,r?rr)rHrIrJrrGrr*rr-rHr,rr?rDrrrr6r3r2rFrFBs )-D(5$$ %,*.FHU&& '.9=OXd5#4#456=8JJr3c.|jjSr])r*rrFs r2rz,Glm4vForConditionalGeneration.language_modelszz(((r3c.|jjSr])r*rrFs r2rz$Glm4vForConditionalGeneration.visualszz   r3rrrr,rlabelsrrrr4rrlogits_to_keeprrdc ||n|jj}| | n|jj} |jd|| | | |||||||| |d |}|d}t |t r t | dn|}|j|dd|ddf}d}|2|j|||jjj}t|||j|j|j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Glm4vForConditionalGeneration >>> model = Glm4vForConditionalGeneration.from_pretrained("THUDM/GLM-4.1V-9B-Thinking") >>> processor = AutoProcessor.from_pretrained("THUDM/GLM-4.1V-9B-Thinking") >>> messages = [ { "role": "user", "content": [ {"type": "image"}, {"type": "text", "text": "What is shown in this image?"}, ], }, ] >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) >>> inputs = processor(text=[text], images=[image], vision_infos=[vision_infos]) >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "The image shows a street scene with a red stop sign in the foreground. In the background, there is a large red gate with Chinese characters ..." ```N) rrrrrrrr,rrrrrr)rHrUr)rGrHr,r?rrr6)r[rrr*rrslicerL loss_functionrrrFr,r?rr)r.rrrr,rrUrrrr4rrrrrrrVrrr? slice_indicesrHrGs r2rBz%Glm4vForConditionalGeneration.forwards/D2C1N-TXT_T_TqTq$8$D $++JjJj $** % 3))%)+'/!5)  "  9C>SV8W~ot4]k mA}a,?@A  %%VFt{{OfOfOqOq%rD*#33!//))++   r3c  lt||f||||||| | | |d | } d| d<|ddk7r d| d<d| d<| S)N) r,rrrrrrrrrrrrr)r&prepare_inputs_for_generation)r.rr,rrrrrrrrrr model_inputsr1s r2r\z;Glm4vForConditionalGeneration.prepare_inputs_for_generationsx"w<   +)')%% 3))    (, ^$ !  !+/L (26L. /r3c|||jtj|jjtj |j k(d}||jtj|jjtj |j k(d}||jtj|jjtj |j k(d}nK||jjk(}||jjk(}||jjk(}tj|j|jz d}|dkD}||z}|jd} |jd} | | fS)aa Get the number of images and videos for each sample to calculate the separation length of the sample tensor. These parameters are not passed through the processor to avoid unpredictable impacts from interface modifications. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Returns: image_nums (`torch.LongTensor` of shape `(batch_size, num_images_sample)`) video_nums (`torch.LongTensor` of shape `(batch_size, num_videos_sample)`) r).rrrr) rr*rr[image_start_token_idrrrrrVrr') r.rris_imageis_video_start is_video_end video_level inside_videostandalone_images image_counts video_countss r2_get_image_nums_and_video_numsz#5#5#7,:J:J:L#LRST "Q % 6),,,3 %))a)0 \))r3 expand_sizeis_encoder_decoderc dk(rfSgdfd}fd}|jd||r*jd td|dd<fS) Nr)rrrrsecond_per_grid_tsc  jdd} jdd}j jdd\}}d}|D]}|dk(rctj|t |}|Dcgc]'}tj |dj )} }|||| ||<l|dk(rt |} |||| ||<|d k(rctj|t |}|Dcgc]'}tj |dj )} }|||| ||<|dk(rt |} |||| ||<|d k(s t||tstd |d t||dtj||} t |} || | } | j||<|Scc}wcc}w)Nrrr)rctj||}|gdg|jdz zz}tj|Dcgc]}|j|c}d}|Scc}w)Nrrr)r*rrvrr)rr repeat_timessamples repeat_argssampleresults r2_repeat_interleave_sampleszGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visual.._repeat_interleave_sampleslsa++a1+nsaeegk/BB g#VFMFMM;$?#V\]^ $WsA&rrr)rrorrlzExpected value for key 'z' to be a list, but got z instead.) rfrhr*rrrr'r TypeErrorrcrr)dict_to_expandrr image_nums video_numsrtrrprrrrrir model_kwargsr.s r2"_expand_dict_for_generation_visualzgGlm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generation_visuales*)--.>EN)--.>EN%)%H%H)9)9/4)P&I& "J   &" :.(#kk.$z:JKGMTU6uzz&a8<<>UGU*D&s+W;+N3',,":.G*D&s+W;+N3'11#kk.$z:JKGMTU6uzz&a8<<>UGU*D&s+W;+N3',,":.G*D&s+W;+N3'00%nS&94@'6se;STXYghkYlTmSnnwx#\\.*=>F":.G7^ijF*0--/N3'E" :F" !=VVs =,G(,G-c|D]J}|dk7s ||t||tjs-|vs2||jd||<L|S)Nrrr)rr*rsrU)rvrri visual_keyss r2_expand_dict_for_generationz`Glm4vForConditionalGeneration._expand_inputs_for_generation.._expand_dict_for_generationsl% d++&s+7">##6 E;.*8*=*O*OP[ab*O*cN3' d" !r3rrencoder_outputszMIf `is_encoder_decoder` is True, make sure that `encoder_outputs` is defined.)rUrfr)r.rirjrryrzr}r|s`` `` @r2_expand_inputs_for_generationz;Glm4vForConditionalGeneration._expand_inputs_for_generationTs ! l* *w 0 "d ":,G  !33KQ3GI2<@  12: !pqq.I,WhJi.jL* +,&&r3r])NNNNNNNNNNNNNNNNr) NNNNNTNNNN)rFN)&rHrIrJrD_tied_weights_keysr'rrrrr*rrrrr!propertyrrrrrsrrarrrrrDrFrBr\rhr6rrrrKrLs@r2rJrJ`sO%'"*+1/((dhR#(#4#4RFNuO_O_F`R Ku/@/@KRZ[`[k[kRlK))!!'+1537=A59-1$(,0/3&*/3;?5959265934%g ##g !.g u//0 g "$u'8'8"9: g   1 12 g ))*g D>g $D>g 'tng d^g u||,g &e&7&78g !!1!12g !!1!12g e../!g "!!1!12#g $c5<</0%g &+,'g ( u11 2)g g X 'X156*E,,-6*  -6* u||U\\) * 6*t#(04 Z'Z'!Z'E,,- Z' uc3h/ 0 Z'r3rJ)rJrr)r)r)r)Ur dataclassesrtypingrrrrr*torch.nnr(torch.nn.functionalrrr activationsr cache_utilsr r generationr integrationsr masking_utilsrmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrmodeling_rope_utilsrrmodeling_utilsrrprocessing_utilsrutilsrrrrrconfiguration_glm4vrr r! get_loggerrHr Moduler$rNrcrurrrrsrDrrrr~rrrr)r8r_rwrrrr+rrrrFrJ__all__r6r3r2rs*!11 !.)7/B9DKF&llPP   H %Y'J299J(J( fBII fBII(  fRYYf"TBIITn(  ||   +0<< >Cll  5<< %&  UU\\ U# U%,, U&% II% <<% % << % U\\* %  %%'(%4[299[|16 '? ' 'k+k\!