rh6PddlZddlmZmZmZddlZddlZddlm Z ddl m cm Z ddl ZddlmZddlmZddlmZmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd l m!Z!ddl"m#Z#ddl$m%Z%ddl&m'Z'm(Z(ddl)m*Z*m+Z+ddl,m-Z-m.Z.m/Z/m0Z0m1Z1ddl2m3Z3ddl4m5Z5m6Z6m7Z7ddl8m9Z9ddl:m;Z;mZ>m?Z?m@Z@mAZAmBZBmCZCmDZDmEZEmFZFddlGmHZHmIZImJZJe1jeLZMGddeZNGddeZOGdde9ZPGdd e6ZQGd!d"e?ZRGd#d$e;ZSGd%d&e<ZTGd'd(e jZVGd)d*e jZWGd+d,eEZXGd-d.eFZYGd/d0eBZZGd1d2eZZ[Gd3d4eCZ\d5Z]dPd6Z^Gd7d8e jZ_Gd9d:e5Z`Gd;deAZbGd?d@eDZcGdAdBe@ZdGdCdDe=ZeGdEdFe>ZfGdGdHeJZgGdIdJe'ZhGdKdLeIZiGdMdNeHZjgdOZky)QN)CallableOptionalUnion) LayerNorm)ACT2FN)Cache DynamicCache)PretrainedConfig) BatchFeature) ImageInput)create_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputWithPast)rope_config_validation)ALL_ATTENTION_FUNCTIONS) ImagesKwargsUnpack)PreTokenizedInput TextInput)TransformersKwargsauto_docstringcan_return_tupleis_torchdynamo_compilinglogging) VideoInput)Glm4MLP Glm4RMSNormeager_attention_forward)Qwen2_5_VLConfig) Qwen2_5_VisionPatchEmbedQwen2_5_VisionRotaryEmbedding Qwen2_5_VLCausalLMOutputWithPast"Qwen2_5_VLForConditionalGeneration Qwen2_5_VLMLPQwen2_5_VLModelQwen2_5_VLModelOutputWithPastQwen2_5_VLPreTrainedModelQwen2_5_VLRotaryEmbeddingQwen2_5_VLTextModelQwen2_5_VLVisionAttentionQwen2_5_VLVisionBlock)Qwen2_5_VLProcessorQwen2_5_VLProcessorKwargsQwen2_5_VLVideosProcessorKwargscJeZdZdZdZdZ dfd ZxZS)Glm4vVisionConfiga This is the configuration class to store the configuration of a [`Glm4vVisionModel`]. It is used to instantiate an Glm4vVisionModel model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of GLM-4.1V-9B-Thinking [THUDM/GLM-4.1V-9B-Thinking](https://huggingface.co/THUDM/GLM-4.1V-9B-Thinking). Args: hidden_size (`int`, *optional*, defaults to 1536): Dimensionality of the encoder layers and the pooler layer. depth (`int`, *optional*, defaults to 24): Number of layers (depth) in the model. attention_bias (`bool`, *optional*, defaults to `False`): Whether to add a bias to the queries, keys and values. intermediate_size (`int`, *optional*, defaults to 13696): Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder. hidden_act (`str` or `function`, *optional*, defaults to `"selu"`): The non-linear activation function (function or string) in the encoder and pooler. If string, `"gelu"`, `"relu"`, `"selu"` and `"gelu_new"` are supported. hidden_dropout_prob (`float`, *optional*, defaults to 0.0): The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. attention_dropout (`float`, *optional*, defaults to 0.0): Dropout probability for attention weights. projection_dropout (`float`, *optional*, defaults to 0.0): Dropout probability for the projection layer. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. image_size (`int` or `list[int]`, *optional*, defaults to `[336, 336]`): The size (resolution) of each image. patch_size (`int`, *optional*, defaults to `14`): The size (resolution) of each patch. num_channels (`int`, *optional*, defaults to 3): The number of input channels. out_hidden_size (`int`, *optional*, defaults to 4096): The output hidden size of the vision model. rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. spatial_merge_size (`int`, *optional*, defaults to 2): The size used for merging spatial dimensions. temporal_patch_size (`int`, *optional*, defaults to 2): The size used for patches along the temporal dimension. Example: ```python >>> from transformers import Glm4vVisionConfig, Glm4vVisionModel >>> # Initializing a Glm4vVisionConfig GLM-4.1V-9B style configuration >>> configuration = Glm4vVisionConfig() >>> # Initializing a model (with random weights) from the GLM-4.1V-9B configuration >>> model = Glm4vVisionModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```glm4v vision_configc t|di|||_||_||_||_||_||_| |_| |_ | |_ | |_ ||_ ||_ | |_||_||_y)N)super__init__depth hidden_size hidden_act num_heads in_channels image_size patch_sizespatial_merge_sizetemporal_patch_sizeout_hidden_sizeintermediate_sizeinitializer_range rms_norm_epsattention_biasattention_dropout)selfr:r;r<rGrHr=r>r?r@rFrArBrCrDrEkwargs __class__s z/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/glm4v/modular_glm4v.pyr9zGlm4vVisionConfig.__init__}s& "6" &$"&$$"4#6 .!2!2(,!2)isiluF riPh㈵>r5{Gz?)__name__ __module__ __qualname____doc__ model_typebase_config_keyr9 __classcell__rKs@rLr3r3BsN5 nJ%O!#3#3rMr3ceZdZdZdZdZdgZddddddd Zd gd gfd d gd gfd gd gfdZ dfd Z xZ S)Glm4vTextConfiga This is the configuration class to store the configuration of a [`Glm4vModel`]. It is used to instantiate a GLM-4.1V model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of GLM-4.1V-9B-Thinking [THUDM/GLM-4.1V-9B-Thinking](https://huggingface.co/THUDM/GLM-4.1V-9B-Thinking). Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 151552): Vocabulary size of the Glm4v model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`Glm4vModel`] hidden_size (`int`, *optional*, defaults to 4096): Dimension of the hidden representations. intermediate_size (`int`, *optional*, defaults to 13696): Dimension of the MLP representations. num_hidden_layers (`int`, *optional*, defaults to 40): Number of hidden layers in the Transformer encoder. num_attention_heads (`int`, *optional*, defaults to 32): Number of attention heads for each attention layer in the Transformer encoder. num_key_value_heads (`int`, *optional*, defaults to 2): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details checkout [this paper](https://arxiv.org/pdf/2305.13245.pdf). If it is not specified, will default to `32`. hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. max_position_embeddings (`int`, *optional*, defaults to 32768): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-05): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. tie_word_embeddings (`bool`, *optional*, defaults to `False`): Whether the model's input and output word embeddings should be tied. rope_theta (`float`, *optional*, defaults to 10000.0): The base period of the RoPE embeddings. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. rope_scaling (`Dict`, *optional*): Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value accordingly. Expected contents: `rope_type` (`str`): The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope', 'llama3'], with 'default' being the original RoPE implementation. `factor` (`float`, *optional*): Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In most scaling types, a `factor` of x will enable the model to handle sequences of length x * original maximum pre-trained length. `original_max_position_embeddings` (`int`, *optional*): Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during pretraining. `attention_factor` (`float`, *optional*): Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, using the `factor` field to infer the suggested value. image_token_id (`int`, *optional*): Token index used as placeholder for image embeddings. video_token_id (`int`, *optional*): Token index used as placeholder for video embeddings. ```python >>> from transformers import Glm4vTextModel, Glm4vConfig >>> # Initializing a GLM-4.1V style configuration >>> configuration = Glm4vConfig() >>> # Initializing a model from the GLM-4.1V style configuration >>> model = Glm4vTextModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` glm4v_text text_configpast_key_valuescolwiserowwise colwise_rep rowwise_rep)zlayers.*.self_attn.q_projzlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.o_projzlayers.*.mlp.gate_up_projzlayers.*.mlp.down_proj input_ids inputs_embeds hidden_statesattention_mask) embed_tokenslayersnormc ||_||_||_||_||_||_||}||_||_| |_| |_ | |_ | |_ ||_ ||_ |j*d|jvr|jd|jd<t|dh||_||_t#|Hdd| i|y)Ntype rope_type mrope_section) ignore_keystie_word_embeddingsr7) vocab_sizemax_position_embeddingsr;rDnum_hidden_layersnum_attention_headsnum_key_value_headsr<rErF use_cache rope_thetarH rope_scalingrimage_token_idvideo_token_idr8r9)rIrvr;rDrxryrzr<rwrErFr{rur|rHr}r~rrJrKs rLr9zGlm4vTextConfig.__init__s*%'>$&!2!2#6   &"5 #6 $!2("$!2(    (Vt7H7H-H-1->->v-FD  k *t/1BC,, K-@KFKrM)iPrUrV( rrOirWrSTFg@rPNNN) rXrYrZr[r\r]keys_to_ignore_at_inferencebase_model_tp_planbase_model_pp_planr9r^r_s@rLrarasP dJ#O#4"5&/%.%.%.%2"/ &(9:#%568IJ!"_$56 %!%1L1LrMrac4eZdZdZ dfd ZxZS) Glm4vConfiga\ This is the configuration class to store the configuration of a [`Glm4vModel`]. It is used to instantiate a GLM-4.1V model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of GLM-4.1V-9B-Thinking [THUDM/GLM-4.1V-9B-Thinking](https://huggingface.co/THUDM/GLM-4.1V-9B-Thinking). Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the documentation from [`PretrainedConfig`] for more information. Args: text_config (`Union[PreTrainedConfig, dict]`, *optional*, defaults to `Glm4vTextConfig`): The config object or dictionary of the text backbone. vision_config (`Union[PreTrainedConfig, dict]`, *optional*, defaults to `Glm4vVisionConfig`): The config object or dictionary of the vision backbone. image_token_id (`int`, *optional*, defaults to 151343): The image token index to encode the image prompt. video_token_id (`int`, *optional*, defaults to 151344): The video token index to encode the image prompt. image_start_token_id (`int`, *optional*, defaults to 151339): The image start token index to encode the start of image. image_end_token_id (`int`, *optional*, defaults to 151340): The image end token index to encode the end of image. video_start_token_id (`int`, *optional*, defaults to 151341): The video start token index to encode the start of video. video_end_token_id (`int`, *optional*, defaults to 151342): The video end token index to encode the end of video. ```python >>> from transformers import Glm4vForConditionalGeneration, Glm4vConfig >>> # Initializing a GLM-4.1V style configuration >>> configuration = Glm4vConfig() >>> # Initializing a model from the GLM-4.1V style configuration >>> model = Glm4vForConditionalGeneration(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```c  Zt |||_||_||_||_yN)r8r9video_start_token_idvideo_end_token_idimage_start_token_idimage_end_token_id) rIrcr5r~rrrrrrJrKs rLr9zGlm4vConfig.__init__fs0 $8!"4$8!"4rM)NNi/Oi0Oi+Oi,Oi-Oi.O)rXrYrZr[r9r^r_s@rLrr<s-' V#!#!55rMrc eZdZy) Glm4vRMSNormNrXrYrZr7rMrLrrzrMrc&eZdZddeffd ZxZS) Glm4VisionMlpbiascHt||||j|_yr)r8r9rCrD)rIconfigrrKs rLr9zGlm4VisionMlp.__init__s  &!'!7!7rMF)rXrYrZboolr9r^r_s@rLrr~s8T88rMrceZdZdeddfdZy)Glm4vVisionPatchEmbedrreturnNc^tj|j|_|j|_|j|_|j |_|j|j|jg}tj|j|j |||_ y)N) kernel_sizestride) r#r9r@rBr>r; embed_dimnnConv3dproj)rIrrs rLr9zGlm4vVisionPatchEmbed.__init__s ))+ ++#)#=#= !--++//$//R IId..K`kl rM)rXrYrZr3r9r7rMrLrrsm0mTmrMrc eZdZy)Glm4vVisionRotaryEmbeddingNrr7rMrLrrrrMrc neZdZd dededededdf fd Zdejdejfd Z xZ S) Glm4vVisionPatchMergerdim context_dimr<rrNcxt|tj||||_t ||_tj||||_tj||||_tj||||_ tj|_ t||_ y)Nr)r8r9rLinearrrpost_projection_norm gate_projup_proj down_projGELUact1ract_fn)rIrrr<rrKs rLr9zGlm4vVisionPatchMerger.__init__s IIc3T2 $-cN!3 $?yyk= ;$?GGI Z( rM hidden_statec|j|}|j|j|}|j|j |j ||j |zSr)rrrrrrr)rIrs rLforwardzGlm4vVisionPatchMerger.forwardsYyy. yy!:!:1D!--"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]joprMrc |jj}|jd}|jd}|j} |j | |j | }}|dk(r%t j d|| |j} nt|tr&t j|| t j}t|t js&t j|| t j}|jd} t| dz} |j| | |jdddj!dj | t j"} t j$t't)|Dcgc]}||dfj+||c}j | t j"}t j$t't)|Dcgc]}||dfj+||c}j | t j"}|j | t j"}|j | t j"}|dz|z dzdz }|dz|z dzdz }t j,||fdj!dj!d}t/j0| |dd d }|j3dj3djdd}|j |jj |j} || z}|Scc}wcc}w) a Forward pass with integrated position encoding adaptation using 2D interpolation. Args: embeddings: Input embeddings tensor lengths (torch.Tensor): Sequence lengths for each image in the batch. image_shapes (torch.Tensor): Tensor of shape [batch_size, 3] representing the image shapes (t, h, w). h_coords (torch.Tensor): Tensor of shape [total_seq] representing the h coordinate for each patch. w_coords (torch.Tensor): Tensor of shape [total_seq] representing the w coordinate for each patch. Returns: torch.Tensor: Embeddings with adapted position encoding added. rTrdevicedtypeg?rrrbicubicFborder)mode align_corners padding_mode)rweightshapertoremptyr isinstancelisttensorlongrrviewpermute unsqueezefloat32catrangelenrepeatstackF grid_samplesqueeze)rI embeddingslengths image_shapesh_coordsw_coordspos_embed_weightr; total_seqradapted_pos_embed orig_size_sq orig_size pos_embed_2ditarget_htarget_wnorm_wnorm_hgridinterpolated_embed_fp32adapted_pos_embed_fp32s rLrzGlm4vVisionEmbeddings.forwards 2299&,,Q/ NN1% !((&[[0(++f2E( > % A{6QaQgQg h '4(,,wvUZZPlELL9$||LuzzZ ,11!4LL#-.I %%iKHAq!167 yyQVWZ[bWcQd!eA,q!t"4";";GAJ"G!efiiU]]jHyyQVWZ[bWcQd!eA,q!t"4";";GAJ"G!efiiU]]jH  {{& {FH{{& {FH#~1Q6:F#~1Q6:F;;/R8BB1EOOPQRD'(mmd%V^' # &=%D%DQ%G%O%OPR%S%[%[\]_`%a " 6 9 9:J:P:P Q T TU_UfUf g  "33 7"f"fs  M" M') rXrYrZr3r9rrrr^r_s@rLrrs' q0 qGPUP\P\GrMrc(eZdZdeddffd ZxZS)Glm4vVisionAttentionrrNc"t||j|_tj|j |j dz|j |_tj|j |j d|_y)NrrF) r8r9rHrrr;rGqkvrrs rLr9zGlm4vVisionAttention.__init__sg !'!9!999V//1C1Ca1GfNcNcdIIf00&2D2D5Q rM)rXrYrZr3r9r^r_s@rLrrs R0RTRRrMrc eZdZdfd ZxZS)Glm4vVisionBlockct|t|j|j|_t|j|j|_t||_t|d|_ y)NepsFr) r8r9rr;rFnorm1norm2rattnrmlprs rLr9zGlm4vVisionBlock.__init__s\ !&"4"4&:M:MN !&"4"4&:M:MN (0  e4rMrN)rXrYrZr9r^r_s@rLrrs 55rMrceZdZddgZy)Glm4vPreTrainedModelGlm4vTextDecoderLayerrN)rXrYrZ_no_split_modulesr7rMrLrr s02DErMrceZdZUeed<dgZd fd ZdZdejdejdejfdZ xZ S) Glm4vVisionModelrrrcFt|||j|_|j|_t ||_t ||_|j|jz}t|dz|_ tjt|jDcgc] }t!|c}|_t%|j&|j(|j*|_t/|j|j0|_tj4|j|j&|j|j|_t/|j|j0|_d|_|j=ycc}w)Nr)rrr<r)r> out_channelsrrF)r8r9rAr@rrr patch_embedr;r=rrotary_pos_embr ModuleListrr:rblocksrrCrDr<mergerrrFpost_conv_layernormConv2d downsamplepost_layernormgradient_checkpointing post_init)rIrhead_dim_rKs rLr9zGlm4vVisionModel.__init__sA  "(";"; ++/708%%)9)998QGmmuV\\GZ$[!%5f%=$[\ ,&&F4L4LY_YjYj  $00B0BH[H[#\ ))**//11,,   +6+=+=6CVCVW&+# %\s%Fcg}|D]s\}}}tj|jdjd|}|j ||j z|j ||j z|j }|j dddd}|j}tj|jdj|d}|j ||j z|j ||j z|j }|j dddd}|j}|jtj||gdj|dvtj|d}|ddddfj}|j|} | |jd} | |fS)NrTrrrrr)rrrrreshaperArflattenappendrrrmaxr) rIgrid_thwpos_idsthwhpos_idswpos_ids max_grid_sizerotary_pos_emb_fullrs rL rot_pos_embzGlm4vVisionModel.rot_pos_emb1s SGAq!||A003::2qAH''T,,,''T,,,'' H  ''1a3H'')H||A003::1bAH''T,,,''T,,,'' H  ''1a3H'')H NN5;;(';DKKAqQ R) S*))G+ AB++- "11-@,W5==a@w&&rMrkr c |j|}|j|}|j|\}}tj||fd}|j |j f}tj|dddf|dddfz|dddfjdtjjr |jntj}tj|dd }|dd|ddz j}|j!||||dddf|dddf}|j"D] } | ||| }|j%|}|j'd|j(|j(|j*d}|j-dd dd}|j/|j'd|j0j2}|j5|}|S) az Args: hidden_states (`torch.Tensor` of shape `(seq_len, hidden_size)`): The final hidden states of the model. grid_thw (`torch.Tensor` of shape `(num_images_or_videos, 3)`): The temporal, height and width of feature shape of each image in LLM. Returns: `torch.Tensor`: hidden_states. rrNrTrr)rr)rTr)value) cu_seqlensposition_embeddingsr)rrr)rrcossinrepeat_interleavecumsumjit is_tracingrint32rpadtolistrrrrrArrrrrCr) rIrkr rimage_type_idsembr-r,seqlensblks rLrzGlm4vVisionModel.forwardNs((7 00? )-)9)9()C&ii8bA"wwy#'')4,,Xad^hq!tn-LhWXZ[W[n]dd %*II$8$8$:(.. e UU:vQ7 ab>JsO3;;= w.YZ\]Y]J^`noprsos`tu ;; C%$7M ++M: %** '')@)@-BUBUVXBY &--aAq9  6;;B @[@[\  M2 rMr) rXrYrZr3__annotations__r r9r)rrrr^r_s@rLr r sB +,8':.U\\.U\\.ell.rMr c eZdZy)Glm4vTextRotaryEmbeddingNrr7rMrLr=r=rrMr=c||ddddf}|ddddf}tj| |fdjdS) z*Rotates half the hidden dims of the input..rNrrTrr)rrr)xx1x2s rLrotate_half_llmrCsJ 319B 319B ;;Ryb ) 1 1" 55rMc R|dz}tjt|j|dDcgc] \}}||dzc}}dj |}tjt|j|dDcgc] \}}||dzc}}dj |}|dd|j ddzfj dd}|dd|j ddzfj dd}|j d}|dd|f|d|df} } |dd|f|d|df} } | |zt| |zz} | |zt| |zz}tj| | gd} tj|| gd}| |fScc}}wcc}}w)aXApplies Rotary Position Embedding with Multimodal Sections to the query and key tensors (https://qwenlm.github.io/blog/qwen2-vl/). Explanation: Multimodal 3D rotary position embedding is an extension to 1D rotary position embedding. The input embedding sequence contains vision (images / videos) embedding and text embedding or just contains text embedding. For vision embedding part, we apply rotary position embedding on temporal, height and width dimension separately. Here we split the channel dimension to 3 chunks for the temporal, height and width rotary position embedding. For text embedding part, we just apply 1D rotary position embedding. The three rotary position index (temporal, height and width) of text embedding is always the same, so the text embedding rotary position embedding has no difference with modern LLMs. Args: q (`torch.Tensor`): The query tensor. k (`torch.Tensor`): The key tensor. cos (`torch.Tensor`): The cosine part of the rotary embedding. sin (`torch.Tensor`): The sine part of the rotary embedding. mrope_section(`List(int)`): Multimodal rope section is for channel dimension of temporal, height and width in rope calculation. unsqueeze_dim (`int`, *optional*, defaults to 1): The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2. Returns: `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding. rrrr.N)rr enumeratesplitrrr0rC)qkr.r/rs unsqueeze_dimrm rotary_dimq_rotq_passk_rotk_passq_embedk_embeds rLapply_multimodal_rotary_pos_embrRs:"A%M )))CIImQSI4T*UV$!QQq1uXV\^ _ i i C )))CIImQSI4T*UV$!QQq1uXV\^ _ i i C c'SYYr]a''' ( : :1" : EC c'SYYr]a''' ( : :1" : EC2Jc;J;&'3 +;)<6Ec;J;&'3 +;)<6Es{u5;@P@PW\] rMrkr-rlrpast_key_valueoutput_attentionsr{cache_positionrJrc  |j\} } } |j|} |j|}|j|}| j | | d|j j dd} |j | | d|j j dd}|j | | d|j j dd}|\}}t| ||||jd\} }|'|||d}|j|||j|\}}t}|jjdk7rt|jj}||| |||f|jsdn |j |j"d| \}}|j%| | dj'}|j)|}|||fS) NrrTrrs)r/r.raeagerrP)dropoutrY)sizerZr[r\rr transposerRr}updaterUr!r_attn_implementationrtrainingrHrYr contiguousr])rIrkr-rlrr_r`r{rarJbszq_lenr query_states key_states value_statesr.r/ cache_kwargsattention_interface attn_output attn_weightss rLrzGlm4vTextAttention.forwards&**, UA{{=1 [[/ {{=1 #((eRGQQRSUVW __S%T]]CMMaQRS #((eRGQQRSUVW &S#B *c30A0A/0R$  j  %#&snUL'5'<'09!. 09 u//0 09 ! 09 0909!!1!1209-.09 u||Xell3XeELL>Q5RR S09rMrTc eZdZy) Glm4vTextMLPNrr7rMrLrxrxrrMrxcxeZdZdedeffd Z ddejdeejejfde ejde ejde eejd e e d e e d e ejd eeje eejejfffd Z xZS)rrrUct||j|_t|||_t ||_t|j|j|_ t|j|j|_ t|j|j|_ t|j|j|_ y)Nr) r8r9r;rT self_attnrxrrrFinput_layernormpost_attention_layernormpost_self_attn_layernormpost_mlp_layernormr^s rLr9zGlm4vTextDecoderLayer.__init__s !--+FI>'+F,>,>FDWDWX(4V5G5GVM`M`(a%(4V5G5GVM`M`(a%".v/A/AvGZGZ"[rMrkr-rlrr_r`r{rarc 0|} |j|}|jd||||||||d| \}} } |j|}| |z}|} |j|}|j |}|j |}| |z}|f} |r| | fz } |r| | fz } | S)N)rkr-rlrr_r`r{rar7)r|r{r~r}rr)rIrkr-rlrr_r`r{rarJresidualself_attn_weightspresent_key_valueoutputss rLrzGlm4vTextDecoderLayer.forwards!,,]; ?Mdnn ? ' 3)%)/) ?  ? ; (*;55mD  =0 !55mD / // >  =0  "  )+ +G  )+ +GrMrt)rXrYrZrarr9rrrurrvr FloatTensorrr^r_s@rLrrs\\3\26378<,1$)59/||/#5<<#=>/!. / u//0 / !u||!45 /$D>/D>/!!1!12/ u  (51B1BEDUDU1U+V"WW X/rMrc eZdZy)Glm4vModelOutputWithPastNrr7rMrLrrQrrMrcDeZdZdeffd Zee ddeejdeejdeejdee ejdeejdee d ee d ee d eejd eed eeeffdZxZS)Glm4vTextModelrc .t||tjt |j Dcgc]}t ||c}|_t|j|j|_ t||_ |`|`ycc}w)Nr)r)r8r9rrrrxrrnrr;rFror= rotary_embrhhas_sliding_layersr^s rLr9zGlm4vTextModel.__init__Vs{  mmGLVMeMeGf g) "69 5 g  !!3!39L9LM 2&A  %  # hsBrirlrrdrjr{r`output_hidden_statesrarJrc N||n|jj}||n|jj}||n|jj}|du|duz r t d|j r%|j r|rtjdd}|r*|(tjjs t}||j|}| F||jnd} tj| | |j dz|j"} |2| j%dddj'd|j dd}n6|j)d k(r#|d j'd|j dd}t+|j||| || } |} |j-| |}|rd nd}|rd nd}|j.D],}|r|| fz }|| f|| ||||| d | }|d} |s$||dfz }.|j1| } |r|| fz }t3| |r|nd||S)N:You must specify exactly one of input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...FrrTrrrr)N.)r input_embedsrlrardrr7)r-rlrr_r`r{ra)last_hidden_staterdrk attentions)rr`rr{ ValueErrorrrilogger warning_oncerr2r3r rmget_seq_lengthrrrrrrrrrnror)rIrirlrrdrjr{r`rrarJpast_seen_tokens causal_maskrkr-all_hidden_statesall_self_attns decoder_layer layer_outputss rLrzGlm4vTextModel.forward`s2C1N-TXT_T_TqTq$8$D $++JjJj "+!6IDKKb $D>b 'tnb !!1!12b -.b  u-- .b b rMrc%eZdZiZddgZfdZ ddeejdeejdeejdeejde ejejff d Z dd ejdeejfd Z ee ddejdeejd eejd eeejdeejdeedeedeedeejd eejdeejdeejdeejdeejdeedee eff dZxZS) Glm4vModelrrclt||tj|j|_yr)r8r9r  _from_configr5visualrs rLr9zGlm4vModel.__init__s(  &33F4H4HI rMriimage_grid_thwvideo_grid_thwrlrc F|jjj}|jj}|jj}|jj }g} |h||c|} |t j| }t jd|jd|jd|j|j} d\} } d}|j| j}t| D]\}}|||dk(}|j}g}d}|D]T}||k(rd}n||k(rd}||k(r|s|jd+||k(r|r|jd D|jd Vg}t!j"t|d D]7\}}t%|}|dd}|d ddz}|j|||f9g}d}|D]:\}}}t'|dkDr|d j)dznd}|dk(rQ|| d|| d|| d }!} }|j+| j+|z|!j+|z}$}#}"t j,|"j/d dj1d |#|$zj3}%t j,|#j/dd dj1|"d |$j3}&t j,|$j/ddd j1|"|#d j3}'|jt j4|%|&|'g|z| dz } d}|d k(rb||| d|| d }!} }|| j+|z|!j+|z}$}#}"t7|"D]}(t j8|(j/d dj1d |#|$zj3}%t j,|#j/dd dj1dd |$j3}&t j,|$j/ddd j1d|#d j3}'|jt j4|%|&|'g|z|dz }||| dk\r| dz } d}|dz }||z })|jt j,|)j/dd j1dd |zd}=t j:|dj=dd }*|*j| j| d|||dk(f<| j|*j)dzt'| |z t j8| |jj?d} | | fS||jAjCd dz } | jE|dk(d| j?dj1dd d j|j} | j)dddj)d dd}+|+dz|jd z } | | fSt j,|jd|jj/ddd j1d|jdd } t jF|jddg|j|j} | | fS)aU Calculate the 3D rope index based on image and video's temporal, height and width in LLM. Explanation: Each embedding sequence contains vision embedding and text embedding or just contains text embedding. For pure text embedding sequence, the rotary position embedding has no difference with modern LLMs. Examples: input_ids: [T T T T T], here T is for text. temporal position_ids: [0, 1, 2, 3, 4] height position_ids: [0, 1, 2, 3, 4] width position_ids: [0, 1, 2, 3, 4] For vision and text embedding sequence, we calculate 3D rotary position embedding for vision part and 1D rotary position embedding for text part. Examples: Temporal (Time): 3 patches, representing different segments of the video in time. Height: 2 patches, dividing each frame vertically. Width: 2 patches, dividing each frame horizontally. We also have some important parameters: fps (Frames Per Second): The video's frame rate, set to 1. This means one frame is processed each second. tokens_per_second: This is a crucial parameter. It dictates how many "time-steps" or "temporal tokens" are conceptually packed into a one-second interval of the video. In this case, we have 25 tokens per second. So each second of the video will be represented with 25 separate time points. It essentially defines the temporal granularity. temporal_patch_size: The number of frames that compose one temporal patch. Here, it's 2 frames. interval: The step size for the temporal position IDs, calculated as tokens_per_second * temporal_patch_size / fps. In this case, 25 * 2 / 1 = 50. This means that each temporal patch will be have a difference of 50 in the temporal position IDs. input_ids: [V V V V V V V V V V V V T T T T T], here V is for vision. vision temporal position_ids: [0, 0, 0, 0, 50, 50, 50, 50, 100, 100, 100, 100] vision height position_ids: [0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1] vision width position_ids: [0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1] text temporal position_ids: [101, 102, 103, 104, 105] text height position_ids: [101, 102, 103, 104, 105] text width position_ids: [101, 102, 103, 104, 105] Here we calculate the text start position_ids as the max vision position_ids plus 1. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Padding will be ignored by default should you provide it. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. Returns: position_ids (`torch.LongTensor` of shape `(3, batch_size, sequence_length)`) mrope_position_deltas (`torch.Tensor` of shape `(batch_size)`) rrrTrr)rrFTimagevideotextc |dS)NrTr7)r@s rLz+Glm4vModel.get_rope_index..2s [\]^[_rMrrr.r)keepdimr)$rr5rAr~rrr ones_likeonesrrrrrEr6r itertoolsgroupbyrrritemrrrrrrrrrrrr1 masked_fill_zeros),rIrirrrlrAr~rrmrope_position_deltastotal_input_idsr image_index video_indexvideo_group_indexr input_tokensinput_token_typevideo_check_flgtokeninput_type_groupkeygroup start_index end_indexllm_pos_ids_listvideo_frame_num modality_type start_idxend_idxst_idxr"r#r$ llm_grid_t llm_grid_h llm_grid_wt_indexh_indexw_indext_idxtext_len llm_positionsmax_position_idss, rLget_rope_indexzGlm4vModel.get_rope_indexsLv"[[66II33#{{??![[;; "  n&@ND^'O%!&!A ::""oo '' L(, $K ! +../E/EFN )/ :W ` 9%nQ&71&<= (//1 #% "') 8E 44*."44*/.(//8.0_(//8(//7 8$& "+"3"3I>N4OQ_"`KJC KE"'(1+K %b ! q 0I$++S+y,IJ K $& "#9I7,5M9g?BCS?TWX?X-b1557!;^_F$/*;7:*;7:*;7: 1 FFHFFH(::FFH(::1;J #(,,z":"?"?A"F"M"MbR\_iRi"j"r"r"t"',,z":"?"?2q"I"P"PQ[]_ak"l"t"t"v"',,z":"?"?1b"I"P"PQ[]gik"l"t"t"v(// Wgw&>r1&E&L&LRQ[^hQh&i&q&q&sG&+ll:&>&C&CAr1&M&T&TUVXZ\f&g&o&o&qG&+ll:&>&C&CAq"&M&T&TUVXbdf&g&o&o&qG,33EKK'SZ@[4\_e4ef g*Q.),{0KA0NN'1,K01-'1,$+Y#6(// X0F0K0KAr0R0Y0YZ[]_0`ci0ij*+o7,r!& *: B J J1b Q ?L?O?OP\PcPc?d S!^A%6!%;;<%,,]->->-@1-Ds?[\K]G^-^_oW `p%*LL1FyO_O_$`$j$jkl$m !!66 6)-224;;B?!C )).A*=qA+55a8??2rJMMnNcNcd #/#3#3Au#3#Ea#H#L#LRY]#L#^_`#a (81(<~?S?STV?W(W% !66 6LL!3I%?&$&B9??1+=+BO!-M-2E2Ea2HA2M #-E,F(F)t+Fq0AQ0FV#t+T/M/M/OST/T &'*CHXHXH`,0,?,?""#8 -@-) k $/ -:,?,?) J&1$A&)9)99==m>R>RS %||J}?S?ST +00B7>>z2N !-!33J%++a.4PVW3XE+//6 +55a8??2rJ %$%%  %)+'/!5)    (%77#33!//))((   rMNNNNr)NNNNNNNNNNNNNN)rXrYrZ_checkpoint_conversion_mappingr r9rrrvrrurrrrrrrrrrrrr^r_s@rLrrsX%'"02DEJ 15595915 |7E,,-|7!!1!12|7!!1!12 |7 !. |7 u||U\\) * |7~dh#(#4#4FNuO_O_F`0'+1537=A59$(,0/3/3;?59592659t ##t !.t u//0 t "$u'8'8"9: t   1 12 t D>t $D>t 'tnt u||,t &e&7&78t !!1!12t !!1!12t e../t !!1!12t +,!t " u.. /#t t rMrc eZdZy)Glm4vCausalLMOutputWithPastNrr7rMrLrrrrMrc)eZdZiZ ddej deejdeej deeejdeejdeej dee dee d ee d ee d eejd eejd eej deej deej deej de e ejfde ede eeff&dZ dfd Z ddeej deejdeejejffdZxZS)Glm4vForConditionalGenerationrirlrrdrjlabelsr{r`rrrrrrrralogits_to_keeprJrc ||n|jj}| | n|jj} |jd|| | | |||||||| |d |}|d}t |t r t | dn|}|j|dd|ddf}d}|2|j|||jjj}t|||j|j|j|jS)a labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. image_grid_thw (`torch.LongTensor` of shape `(num_images, 3)`, *optional*): The temporal, height and width of feature shape of each image in LLM. video_grid_thw (`torch.LongTensor` of shape `(num_videos, 3)`, *optional*): The temporal, height and width of feature shape of each video in LLM. rope_deltas (`torch.LongTensor` of shape `(batch_size, )`, *optional*): The rope index difference between sequence length and multimodal rope. Example: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, Glm4vForConditionalGeneration >>> model = Glm4vForConditionalGeneration.from_pretrained("THUDM/GLM-4.1V-9B-Thinking") >>> processor = AutoProcessor.from_pretrained("THUDM/GLM-4.1V-9B-Thinking") >>> messages = [ { "role": "user", "content": [ {"type": "image"}, {"type": "text", "text": "What is shown in this image?"}, ], }, ] >>> url = "https://www.ilankelman.org/stopsigns/australia.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True) >>> inputs = processor(text=[text], images=[image], vision_infos=[vision_infos]) >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "The image shows a street scene with a red stop sign in the foreground. In the background, there is a large red gate with Chinese characters ..." ```N) rirrrrrrlrdrjr{r`rrar)logitsrrv)lossrrdrkrrr7)rr`rmodelrrslicelm_head loss_functionrcrvrrdrkrr)rIrirlrrdrjrr{r`rrrrrrrrarrJrrk slice_indicesrrs rLrz%Glm4vForConditionalGeneration.forward%s/@2C1N-TXT_T_TqTq$8$D $++JjJj $** % 3))%)+'/!5)  "  9C>SV8W~ot4]k mA}a,?@A  %%VFt{{OfOfOqOq%rD*#33!//))++   rMc  lt||f||||||| | | |d | } d| d<|ddk7r d| d<d| d<| S)N) rdrlrjrarrrrrr{rrrr)r8prepare_inputs_for_generation)rIrirdrlrjrarr{rrrrrJ model_inputsrKs rLr z;Glm4vForConditionalGeneration.prepare_inputs_for_generationsx"w<   +)')%% 3))    (, ^$ !  !+/L (26L. /rMc|||jtj|jjtj |j k(d}||jtj|jjtj |j k(d}||jtj|jjtj |j k(d}nK||jjk(}||jjk(}||jjk(}tj|j|jz d}|dkD}||z}|jd} |jd} | | fS)aa Get the number of images and videos for each sample to calculate the separation length of the sample tensor. These parameters are not passed through the processor to avoid unpredictable impacts from interface modifications. Args: input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. Returns: image_nums (`torch.LongTensor` of shape `(batch_size, num_images_sample)`) video_nums (`torch.LongTensor` of shape `(batch_size, num_videos_sample)`) r).rrTrr) rrrrrrrrrr1rsum) rIrirjis_imageis_video_start is_video_end video_level inside_videostandalone_images image_counts video_countss rL_get_image_nums_and_video_numsz#5#5#7,:J:J:L#LRST "Q % 6),,,3 %))a)0 \))rM)NNNNNNNNNNNNNNNNr) NNNNNTNNNNr)rXrYrZrrrvrrrrrrrrrrurrr rr^r_s@rLrr"s5%'"'+1537=A59-1$(,0/3&*/3;?5959265934%g ##g !.g u//0 g "$u'8'8"9: g   1 12 g ))*g D>g $D>g 'tng d^g u||,g &e&7&78g !!1!12g !!1!12g e../!g "!!1!12#g $c5<</0%g &+,'g ( u11 2)g X 'X156*E,,-6*  -6* u||U\\) * 6*rMrc eZdZy)Glm4vVideosProcessorKwargsNrr7rMrLrrrrMrc>eZdZUeeed<eeed<eeed<y)Glm4vImagesKwargsr@rB merge_sizeN)rXrYrZrrr;r7rMrLrrs  !#& rMrc0eZdZUeed<eed<ddddiZy)Glm4vProcessorKwargs images_kwargs videos_kwargs text_kwargsF)paddingreturn_mm_token_type_idsN)rXrYrZrr;r _defaultsr7rMrLrrs#$$--(- IrMrc leZdZdZdZd fd Z d dedeee e ee e fde de e def d ZxZS) Glm4vProcessora Constructs a GLM-4V processor which wraps a GLM-4V image processor and a GLM-4 tokenizer into a single processor. [`~Glm4vProcessor.__call__`] and [`~Glm4vProcessor.decode`] for more information. Args: image_processor ([`Glm4vProcessor`], *optional*): The image processor is a required input. tokenizer ([`PreTrainedTokenizerFast`], *optional*): The tokenizer is a required input. video_processor ([`Glm4vVideoProcessor`], *optional*): The video processor is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. )PreTrainedTokenizerPreTrainedTokenizerFastc t|||||t|dsdn |j|_t|dsd|_y|j|_y)N) chat_template image_tokenz <|image|> video_tokenz <|video|>)r8r9hasattrr,r-)rIimage_processor tokenizervideo_processorr+rJrKs rLr9zGlm4vProcessor.__init__sS )_Tab.5i.O;U^UjUj.5i.O;U^UjUjrMimagesrvideosrJrc |jtfd|jji|}||jdd|i|d}|d}ni}d}|.|j dd|i|d}|j d} |d } ni}g} d} t|ts|g}|j}||jjd z} d } tt|D]} |j|| vrS|| j| z}|| j|jd |zd || <| d z } |j|| vrS|| jd |j|| <| |j jd z} d }tt|D]} |j || vr|| |d }d}t#| dr| j%d }nt| d tr| d n| }g}td t|D]}|j'|||d|}t||kr'|j'|r|dnd t||kr't|D]}||}d|jd|}||z } || j|j |d || <| |j| z| |d z}t|D]:}|j|| vs|| j|jd |zd || <<|d z }|j || vr||| jd |j|| <|dj dd}|dj dd}|j|fi|d}|j)||ddg|rUt+j,|d}t+j.|d}d |||j0k(<|j%|d<t3i||||S)a^ Main method to prepare for the model one or several sequences(s) and image(s). This method forwards the `text` and `kwargs` arguments to PreTrainedTokenizerFast's [`~PreTrainedTokenizerFast.__call__`] if `text` is not `None` to encode the text. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `List[PIL.Image.Image]`, `List[np.ndarray]`, `List[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. Both channels-first and channels-last formats are supported. text (`str`, `List[str]`, `List[List[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). videos (`np.ndarray`, `torch.Tensor`, `List[np.ndarray]`, `List[torch.Tensor]`): The image or batch of videos to be prepared. Each video can be a 4D NumPy array or PyTorch tensor, or a nested list of 3D frames. Both channels-first and channels-last formats are supported. return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. Returned when `text` is not `None`. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. - **pixel_values_videos** -- Pixel values of videos to be fed to a model. Returned when `videos` is not `None`. - **image_grid_thw** -- List of image 3D grid in LLM. Returned when `images` is not `None`. - **video_grid_thw** -- List of video 3D grid in LLM. Returned when `videos` is not `None`. tokenizer_init_kwargsNr2r rr3r! timestampsrrrz<|placeholder|>rTr6rz<|begin_of_image|>z<|end_of_image|>r"return_tensorsr$Frr) modalitiesrimm_token_type_ids)data tensor_typer7) _merge_kwargsrr0 init_kwargsr/r1poprrcopyrrrr,rreplacer-r.r6r_check_special_mm_tokensnparray zeros_liker~r )rIr2rr3rJ output_kwargs image_inputsr videos_inputsr6r merge_lengthindexrnum_image_tokensr num_framesvideo_structuretimestamps_listunique_timestampsidxselected_timestamps frame_idx timestamp_secframe_structurer8r$ text_inputs array_idsr:s rL__call__zGlm4vProcessor.__call__sT+**  "&.."<"<   /4//`v`A_`L)*:;NL!N  0D00aa-P_B`aM&**<8J*+;.>@QTd@dfghDGQJE&&$q'1q'//*;T=M=MNQ  O  %//::A=LK3t9%! O&&$q'1!/ !t?O?O>PP`an`o*p'?:; #1good.>.>QRSDG&{388:lJn]hNijkNll%&+:%6q ++tAw6&*1good6F6FHY\lHlno&pDGq 1$K=&&$q'1@q'//*;T=M=MNQC! OD'}599:JDQ#0#?#C#CD^`e#f $dnnTJ]=-IJ  %%dKWgDV%W #[!9:I " k+.F G BC i4+>+>> ?/@/G/G/IK+ ,!QK!Qrus!,, !.34%/B97954Cll%NNB       H %^3(^3BVL&VLr:5":5| ; 8M8 m4 m !> fRYYf"TBIITnR4R5,5F4Fk+k\ 8 66rJ9J9Z 7 :6:z < o (o dT T n  "B K*$FK*\ !@   4Ro(Roj rM