rhddlmZddlmZmZmZddlmZmZddl m Z ddl m Z m Z mZe rddlmZmZe j$eZGdd eZGd d eZGd d eZGddeZgdZy))Sequence)AnyOptionalUnion)PretrainedConfiglayer_type_validation)rope_config_validation)is_timm_availableloggingrequires_backends) ImageNetInfoinfer_imagenet_subsetcEleZdZdZdZdgZddddddddZdgdgfd d gd gfd gd gfd Z d-d ed ededede ee efdedededede dede de de dededede deee efde d e d!e d"ed#ee e d$e d%ed&e d'e d(ed)ed*ed+ee e e e ff@fd, ZxZS).Gemma3nTextConfiga\) This is the configuration class to store the configuration of a [`Gemma3nTextModel`]. It is used to instantiate an Gemma3nTextModel model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Gemma 3n E4B, e.g. [google/gemma-3n-E4B](https://huggingface.co/google/gemma-3n-E4B). Configuration objects that inherit from [`Gemma3nTextConfig`] and can be used to control the model outputs. Read the documentation from [`Gemma3nTextConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 262400): Vocabulary size of the Gemma3nText model. Defines the number of different tokens that can be represented by the `inputs_ids` passed when calling [`Gemma3nTextModel`] vocab_size_per_layer_input (`int`, *optional*, defaults to 262144): Vocabulary size of the per-layer text embeddings that augment the standard embeddings. hidden_size (`int`, *optional*, defaults to 2048): Dimension of the hidden representations. hidden_size_per_layer_input (`int`, *optional*, defaults to 256): Dimension of the hidden representations for per-layer emebeddings. intermediate_size (`int` or `Sequence[int]`, *optional*, defaults to 16384): Dimension of the MLP representations. MatFormer configurations may wish to provide a sequence of integers to account for vairable intermediate_size values across layers. In such cases, `len(intermediate_size) == num_hidden_layers`. num_hidden_layers (`int`, *optional*, defaults to 35): Number of hidden layers in the Transformer decoder. num_attention_heads (`int`, *optional*, defaults to 8): Number of attention heads for each attention layer in the Transformer decoder. num_key_value_heads (`int`, *optional*, defaults to 2): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details checkout this [paper](https://arxiv.org/pdf/2305.13245.pdf). If not specified, will default to `num_attention_heads`. head_dim (`int`, *optional*, defaults to 256): The attention head dimension. hidden_activation (`str` or `function`, *optional*, defaults to `"gelu_pytorch_tanh"`): The non-linear activation function (function or string) in the decoder. Will default to `"gelu_pytorch_tanh"` if not specified. `"gelu_pytorch_tanh"` uses an approximation of the `"gelu"` activation function. max_position_embeddings (`int`, *optional*, defaults to 32768): The maximum sequence length that this model might ever be used with. initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. rms_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the rms normalization layers. use_cache (`bool`, *optional*, defaults to `True`): Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if `config.is_decoder=True`. pad_token_id (`int`, *optional*, defaults to 0): Padding token id. eos_token_id (`int`, *optional*, defaults to 1): End of stream token id. bos_token_id (`int`, *optional*, defaults to 2): Beginning of stream token id. rope_theta (`float`, *optional*, defaults to 1000000.0): The base period of the RoPE embeddings. rope_scaling (`Dict`, *optional*): Dictionary containing the scaling configuration for the RoPE embeddings used in gloabl attention. NOTE: if you apply new rope type and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value accordingly. Expected contents: `rope_type` (`str`): The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope', 'llama3'], with 'default' being the original RoPE implementation. `factor` (`float`, *optional*): Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In most scaling types, a `factor` of x will enable the model to handle sequences of length x * original maximum pre-trained length. `original_max_position_embeddings` (`int`, *optional*): Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during pretraining. `attention_factor` (`float`, *optional*): Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention computation. If unspecified, it defaults to value recommended by the implementation, using the `factor` field to infer the suggested value. `beta_fast` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear ramp function. If unspecified, it defaults to 32. `beta_slow` (`float`, *optional*): Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear ramp function. If unspecified, it defaults to 1. `short_factor` (`List[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to short contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `long_factor` (`List[float]`, *optional*): Only used with 'longrope'. The scaling factor to be applied to long contexts (< `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden size divided by the number of attention heads divided by 2 `low_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE `high_freq_factor` (`float`, *optional*): Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE rope_local_base_freq (float, *optional*, defaults to 10000.0): The base period of the RoPE embeddings for local attention. attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`): Whether to use a bias in the query, key, value and output projection layers during self-attention. attention_dropout (`float`, *optional*, defaults to 0.0): The dropout ratio for the attention probabilities. sliding_window (`int`, *optional*, defaults to 512): This is the size of the sliding window used by local attention layers. layer_types (`Optional`, *optional*): A sequence of strings defining the attention type for that layer as either "sliding_attention" or "full_attention". If not provided, `layer_types` will de inferred from `num_hidden_layers` using a pattern of four "sliding_attention" layers followed one "full_attention". The last layer in the model should always be a "full_attention" layer. final_logit_softcapping (`float`, *optional*, defaults to 30.0): Scaling factor when applying tanh softcapping on the logits. altup_active_idx (`int`, *optional*, defaults to 0): The index of the prediction from which AltUp will compute additional predictions or correct altup_coef_clip (`float`, *optional*, defaults to 120.0): The maximum amplitude of an AltUp prediction or correction coeficient weight. altup_correct_scale (`bool`, *optional*, defaults to `True`): If True, apply the `AltUp.correct_output_scale` to the corrected prediction at `altup_active_idx`. altup_num_inputs (`int`, *optional*, defaults to 4): The number of predictions that AltUp should be make given the input sequence. num_kv_shared_layers (`int`, *optional*, defaults to 15): The number of layer that share KV cache values. During the forward pass, the last `num_kv_shared_layers` layers in the model "share" the KV values in that each local and global layer in this range uses the KV cache values computed for the last local or global layer, respectively, before entering this range. The value should be `num_kv_shared_layers` should be a scalar of `sliding_window_pattern`. laurel_rank (int, *optional*, defaults to 64): The intermediate size for the linear projections in the Learned Augmented Residual Layer. activation_sparsity_pattern (Sequence[float], *optional*, defaults to `(0.95, 0.95, 0.95, 0.95, 0.95, 0.95, 0.95, 0.95, 0.95, 0.95, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0)`): The sparsity factor used to extract the top-k activations for a given layer. The provided Sequence must explicitly provide a sparsity value for each layer in the model. ```python >>> from transformers import Gemma3nTextModel, Gemma3nTextConfig >>> # Initializing a Gemma3nText gemma3n_text-E4B style configuration >>> configuration = Gemma3nTextConfig() >>> # Initializing a model from the gemma3n_text-E4B style configuration >>> model = Gemma3nTextModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` gemma3n_textpast_key_valuescolwiserowwise)zlayers.*.self_attn.q_projzlayers.*.self_attn.k_projzlayers.*.self_attn.v_projzlayers.*.self_attn.o_projzlayers.*.mlp.gate_projzlayers.*.mlp.up_projzlayers.*.mlp.down_proj input_ids inputs_embeds hidden_statesattention_mask) embed_tokenslayersnorm vocab_sizevocab_size_per_layer_input hidden_sizehidden_size_per_layer_inputintermediate_sizenum_hidden_layersnum_attention_headsnum_key_value_headshead_dimhidden_activationmax_position_embeddingsinitializer_range rms_norm_eps use_cache pad_token_id eos_token_id bos_token_id rope_theta rope_scalingrope_local_base_freqattention_biasattention_dropoutsliding_window layer_typesfinal_logit_softcappingaltup_active_idxaltup_coef_clipaltup_correct_scalealtup_num_inputsnum_kv_shared_layers laurel_rankactivation_sparsity_patternc! nt%|d |||d|!t|tr"t |x}"|k7rt d|d|"dt|ts|g|z}||_||_| |_||_ ||_ ||_ ||_ | |_ ||_| |_| |_||_||_||_||_| |_||_||_||_||_||_t7||6t9|jD#cgc]}#|#dzdzdk(rdnd c}#|_n||_t;|j0||_||_||_ ||_!||_"||_#||_$| d g|z} t | x}$|k7rt d |d|$d| |_%ycc}#w) N)r+r-r,zjintermediate_size must have an explicit intermediate size for every layer or one for all layers. Expected z values but got .rfull_attentionsliding_attentionzeactivation_sparsity_pattern must have an explicit activation sparsity value for every layer.Expected )&super__init__ isinstancerlen ValueErrorrrr'rr!r"r#r%r$r(r)r*r.r1r2r&r3r5r4r0r/r ranger r r:r6r7r8r9r;r<)&selfrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:r;r<kwargs intsize_lenilen_asp __class__s& /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/gemma3n/configuration_gemma3n.pyrFzGemma3nTextConfig.__init__s)H  %%%    ' 2sK\G]8] bs7s-..>{m1N -x8!2 36G G $*D''>$&!2!2#6   #6 !2("$,!2!2,'>$&$8!(t$  W\]a]s]sWt RSQUaK1$4 :MM D  +D d../+F($8! 0.#6 0& & .+.%2C*C '67 7GwiqJ ,G(5 sF2) ii@#rTgelu_pytorch_tanhi{Gz?ư>Trr?rWg.ANg@FrCiNg>@rg^@T@)#ffffff?r^r^r^r^r^r^r^r^r^rCrCrCrCrCrCrCrCrCrCrCrCrCrCrCrCrCrCrCrCrCrCrCrCrC)__name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inferencebase_model_tp_planbase_model_pp_planintrrstrfloatboolrdictrrF __classcell__rPs@rQrr%sL\ J#4"5%.%.%.%."+ )"+&(9:#%568IJ!"_$56"*1+.7=!##$#$!4'-#'"'15&.$#&!/3)- !!&$( !$&OiCfGfG%(fG fG &) fG !hsm!34 fGfG!fG!fGfGfG"%fG!fGfGfG !fG"#fG$%fG&'fG(tCH~.)fG*$+fG,-fG.!/fG01fG2hsm,3fG4"'5fG67fG89fG:";fG<=fG>"?fG@AfGB&.eE8E?4J.K%LCfGfGrc+eZdZdZdZ ddedededededed ed ed ed ed edededededeeefdedeeeefeeeffdeeeefeeefff&fd ZxZ S)Gemma3nAudioConfiga2 This is the configuration class to store the configuration of a [`Gemma3nAudioEncoder`]. It is used to instantiate an `Gemma3nAudioEncoder` model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Gemma 3n E4B, e.g., [google/gemma-3n-E4B](https://huggingface.co/google/gemma-3n-E4B). Configuration objects that inherit from [`Gemma3nAudioConfig`] and can be used to control the model outputs. Read the documentation from [`Gemma3nAudioConfig`] for more information. Args: vocab_size (`int`, *optional*, defaults to 128): Vocabulary size of the additional hard-token embeddings for audio model. These augment the embeddings included in the `Gemma3nTextModel` to provide, e.g., the end of audio and audio soft token placeholder tokens when converting `input_ids` to embeddings in the `Gemma3nForConditionalGeneration` model. vocab_offset (`int`, *optional*, defaults to 262272): Offset between the tokenizer vocab index for the token ids embedded by `Gemma3nMultimodalEmbedder` and the 0-indexed `Gemma3nMultimodalEmbedder.embedding` table. input_feat_size (`int`, *optional*, defaults to 128): The number of channels in each mel-spectrogram frame. hidden_size (`int`, *optional*, defaults to 1536): Dimension of the hidden representations. rms_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the rms normalization layers. gradient_clipping (`float`, *optional*, defaults to 10000000000.0): Clipping value used to stablize extremely large gradient values. conf_attention_chunk_size (`int`, *optional*, defaults to 12): The sub-sequence size for local attention processing inside the Conformer ("conf") section of the Universal Speech Model. conf_attention_context_left (`int`, *optional*, defaults to 13): The left context size of the local attention inside the Conformer ("conf") section of the Universal Speech Model. conf_attention_context_right (`int`, *optional*, defaults to 0): The right context size of the local attention inside the Conformer ("conf") section of the Universal Speech Model. conf_attention_logit_cap (`float`, *optional*, defaults to 50.0): Logit cap applied during local attention inside the Conformer ("conf") section of the Universal Speech Model. conf_num_attention_heads (`int`, *optional*, defaults to 8): The number of attention heads in local attention inside the Conformer ("conf") section of the Universal Speech Model. conf_num_hidden_layers (`int`, *optional*, defaults to 12): The number of layers that use local attention inside the Conformer ("conf") section of the Universal Speech Model. conf_conv_kernel_size (`int`, *optional*, defaults to 5): Convolution kernel size for the conformer block inside the Conformer ("conf") section of the Universal Speech Model. conf_reduction_factor (`int`, *optional*, defaults to 4): Reduction factor used in the conformer block inside the Conformer ("conf") section of the Universal Speech Model. conf_residual_weight (`float`, *optional*, defaults to 0.5): Residual connection weight inside the Conformer ("conf") section of the Universal Speech Model. sscp_conv_channel_size (`tuple(int, int)`, *optional*, defaults to `(128, 32)`): The channel sizes for the first and second convolutional layers in the Sub-sample Convolution Projection ("sscp") section of the Universal Speech Model. sscp_conv_group_norm_eps (`float`, *optional*, defaults to 0.001): Epsilon used in group normalization in the subsample convolution projection in the Sub-sample Convolution Projection ("sscp") section of the Universal Speech Model. sscp_conv_kernel_size (`tuple(tuple(int, int), tuple(int, int))`, *optional*, defaults to `((3, 3), (3, 3))`): Kernel sizes of the two convolutional layers in the subsample convolution projection in the Sub-sample Convolution Projection ("sscp") section of the Universal Speech Model. The kernel sizes are specified as a tuple of height and width for each layer, where the height corresponds to the time dimension and the width corresponds to the frequency dimension. sscp_conv_stride_size (`tuple(tuple(int, int), tuple(int, int))`, *optional*, defaults to `((2, 2), (2, 2))`): Stride sizes of the two convolutional layers in the subsample convolution projection in the Sub-sample Convolution Projection ("sscp") section of the Universal Speech Model. The stride sizes are specified as a tuple of height and width for each layer, where the height corresponds to the time dimension and the width corresponds to the frequency dimension. Example: ```python >>> from transformers import Gemma3nAudioConfig, Gemma3nAudioEncoder >>> # Initializing a Gemma3nAudioEncoder gemma3n_audio-E4B-style configuration >>> configuration = Gemma3nAudioConfig() >>> # Initializing a model from the gemma3n_audio-E4B style configuration >>> model = Gemma3nAudioEncoder(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` gemma3n_audior vocab_offsetinput_feat_sizerr)gradient_clippingconf_attention_chunk_sizeconf_attention_context_leftconf_attention_context_rightconf_attention_logit_capconf_num_attention_headsconf_num_hidden_layersconf_conv_kernel_sizeconf_reduction_factorconf_residual_weightsscp_conv_channel_sizesscp_conv_group_norm_epssscp_conv_kernel_sizesscp_conv_stride_sizec .t|di|||_||_||_||_||_||_||_||_ | |_ | |_ | |_ | |_ | |_||_||_||_||_||_||_yNrD)rErFrsrr)rrrrtrurvrwrxryrzr{r|r}r~rrr)rKrrrrsrr)rtrurvrwrxryrzr{r|r}r~rrrrLrPs rQrFzGemma3nAudioConfig.__init__s: "6".&($(!2)B&+F(,H)(@%(@%&<#%:"%:"$8!&<#(@%%:"%:"rn)rirZg _B rgI@rVrr@r[g?)r gMbP?)rrr)rWrWr) r_r`rarbrcrgrituplerFrlrms@rQrprp.sTSj!J)""#3)++-,-*.()&(%&%&&)2;*.J J /0;0;0; 0;  0;  0;!0;$'0;&)0;'*0;#(0;#&0;!$0; #0; #0; $!0;"!&c3h#0;$#(%0;& %U38_eCHo%EF'0;. %U38_eCHo%EF/0;0;rnrpceZdZdZdZ ddedededededed ed e e ffd Z e d e ee fffd Zde ee fffd ZxZS)Gemma3nVisionConfiga7 This is the configuration class to store the configuration for a timm backbone [`TimmWrapper`]. It is used to instantiate an timm model model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the Gemma 3n E4B vision tower, e.g. [google/gemma-3n-E4B](https://huggingface.co/google/gemma-3n-E4B). Configuration objects inherit from [`Gemma3nVisionConfig`] and can be used to control the model outputs. Read the documentation from [`Gemma3nVisionConfig`] for more information. Config loads imagenet label descriptions and stores them in `id2label` attribute, `label2id` attribute for default imagenet models is set to `None` due to occlusions in the label descriptions. Args: initializer_range (`float`, *optional*, defaults to 0.02): The standard deviation of the truncated_normal_initializer for initializing all weight matrices. do_pooling (`bool`, *optional*, defaults to `False`): Whether to do pooling for the last_hidden_state in `TimmWrapper` or not. architecture (`str`, *optional*, defaults to `"mobilenetv5_300m_enc"`): Determines vision architecture for TimmWrapper. hidden_size (`int`, *optional*, defaults to 2048): Dimension of the hidden representations. vocab_size (`int`, *optional*, defaults to 128): Vocabulary size of the additional hard-token embeddings for vision model. vocab_offset (`int`, *optional*, defaults to 262144): Offset between the tokenizer vocab index for the token ids embedded by `Gemma3nMultimodalEmbedder` and the 0-indexed `Gemma3nMultimodalEmbedder.embedding` table. rms_norm_eps (`float`, *optional*, defaults to 1e-06): The epsilon used by the rms normalization layers. Example: ```python >>> from transformers import Gemma3nVisionConfig, TimmWrapper >>> # Initializing a TimmWrapper gemma3n_vision-E4B-style configuration >>> configuration = Gemma3nVisionConfig() >>> # Initializing a gemma3n_vision-E4B-style TimmWrapper from the configuration >>> model = TimmWrapper(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` gemma3n_visionr( do_pooling architecturerrrrr) model_argsc  t |di| ||_||_||_||_||_||_||_||_ yr) rErFr(rrrrrrrr)) rKr(rrrrrrr)rrLrPs rQrFzGemma3nVisionConfig.__init__sR "6"!2$$(&$((rn config_dictc |jd}d|vxsd|v}|[|sYt|dgt|}|r?t|}|j }|j d}|D cgc]} ||  }} |a|s_t t||d<tt|t|k(r#t|D cic]\} } | |  c} } |d<nd|d<|jdd} |jdd} | xs| |d<d |vrd|d vr|d jddt|0|fi|Scc} wcc} } w) N label_names num_labelsid2labeltimmT)as_dictlabel2id num_classespretrained_cfg) getr rrrlabel_descriptionsrk enumeraterHsetpoprE from_dict)clsrrLris_custom_modelimagenet_subset dataset_infosynsetsrsynsetrNnamenum_labels_in_kwargsnum_labels_in_dictrPs rQrzGemma3nVisionConfig.from_dictsr!oom4 &&0HJ&4H   cF8 ,3K@O+O< &224%1%D%DT%D%R"HOPf1&9P P  "?!%i &>> from transformers import Gemma3nForConditionalGeneration, Gemma3nConfig, Gemma3nTextConfig >>> # Initializing a MobileNet vision config, which is loaded from TIMM >>> vision_config = Gemma3nVisionConfig() >>> # Initializing a Gemma3n Audio config >>> audio_config = Gemma3nAudioConfig() >>> # Initializing a Gemma3n Text config >>> text_config = Gemma3nTextConfig() >>> # Initializing a Gemma3n gemma-3-4b style configuration >>> configuration = Gemma3nConfig(text_config, vision_config, audio_config) >>> # Initializing a model from the gemma-3-4b style configuration >>> model = Gemma3nTextConfig(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```gemma3n) text_config vision_config audio_configrrraudio_soft_tokens_per_imagevision_soft_tokens_per_image boi_token_id eoi_token_idimage_token_id boa_token_id eoa_token_idaudio_token_idr(c  :t|di| t|tr t di|}n!|t }t j dt|tr tdi|}n!|t}t j dt|tr tdi|}n!|t}t j d||_ ||_ ||_ ||_ ||_ ||_||_||_| |_| |_| |_| |_y)Nz5text_config is None. Using default Gemma3nTextConfig.z9vision_config is None. Using default Gemma3nVisionConfig.z7audio_config is None. Using default Gemma3nAudioConfig.rD)rErFrGrkrloggerinforrprrrrrrrrrrrr()rKrrrrrrrrrrrr(rLrPs rQrFzGemma3nConfig.__init__us  "6" k4 (+:k:K  +-K KKO P mT */@-@M  "/1M KKS T lD )-= =L  !-/L KKQ R&*(+F(,H)((,((,!2rn) NNNrTirRiirirY)r_r`rarbrcrrrp sub_configsrrrkrhrrgrirFrlrms@rQrr/s< |J(,*KKONRLP+.,/##%##%#'03e$5tCH~$EFG03 &94S>&I JK03u%7c3h%GHI 03 &) 03 '* 03030303030303!0303rnr)rprrrN)collections.abcrtypingrrrconfiguration_utilsrr modeling_rope_utilsr utilsr r r timm.datarr get_loggerr_rrrprr__all__rDrnrQrs,%''J9BB=   H %FG(FGRH;)H;Vs*slv3$v3r ^rn