rh7dZddlmZddlmZmZmZddlZddlZddlm Z ddl m Z ddl m Z mZdd lmZdd lmZdd lmZdd lmZdd lmZmZddlmZmZddlmZddlm Z m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)e#jTe+Z,ee!dGddeZ-ee!dGddeZ.Gdde j^Z0 dMde j^dejbd ejbd!ejbd"eejbd#e2d$e2fd%Z3Gd&d'e j^Z4Gd(d)e j^Z5Gd*d+e j^Z6Gd,d-e j^Z7Gd.d/eZ8Gd0d1e j^Z9e!Gd2d3eZ:e!d4Gd5d6e:Z;d7ejbd8eGd=d>e j^Z?Gd?d@e j^Z@e!dAGdBdCe:ZAGdDdEe j^ZBe!dFGdGdHe:ZCe!dIGdJdKe:eZDgdLZEy)NzPyTorch Idefics2 model.) dataclass)CallableOptionalUnionN)nn)ACT2FN)Cache DynamicCache)GenerationMixin)_prepare_4d_attention_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutput ModelOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging) AutoModel)Idefics2ConfigIdefics2PerceiverConfigIdefics2VisionConfigz| Base class for Idefics2 model's outputs that may also contain a past key/values (to speed up sequential decoding).  custom_introceZdZUdZdZeejed<dZ ee e ejed<dZ ee ejed<dZ ee ejed<dZ ee ejed<y)Idefics2BaseModelOutputWithPastai last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. If `past_key_values` is used only the last hidden-state of the sequences of shape `(batch_size, 1, hidden_size)` is output. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) and optionally if `config.is_encoder_decoder=True` 2 additional tensors of shape `(batch_size, num_heads, encoder_sequence_length, embed_size_per_head)`. Contains pre-computed hidden-states (key and values in the self-attention blocks and optionally if `config.is_encoder_decoder=True` in the cross-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder, and optionally by the perceiver Nlast_hidden_statepast_key_values hidden_states attentionsimage_hidden_states)__name__ __module__ __qualname____doc__r#rtorch FloatTensor__annotations__r$tupler%r&r'/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/idefics2/modeling_idefics2.pyr"r")s&6:x 1 129AEOXeE%*;*;$<=>E8B%(9(9":;Br1r"zT Base class for Idefics2 causal language model (or autoregressive) outputs. ceZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZee ejed<y) Idefics2CausalLMOutputWithPasta loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder, and optionally by the perceiver Nlosslogitsr$r%r&r')r(r)r*r+r5rr,r-r.r6r$listr%r/r&r'r0r1r2r4r4Js )-D(5$$ %,*.FHU&& '.9=OXd5#4#456=8B%(9(9":;Br1r4cxeZdZdZdeffd ZdejdejdejfdZ xZ S)Idefics2VisionEmbeddingsaX This is a modified version of `siglip.modelign_siglip.SiglipVisionEmbeddings` to enable images of variable resolution. The modifications are adapted from [Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution](https://huggingface.co/papers/2307.06304) which allows treating images in their native aspect ratio and without the need to resize them to the same fixed size. In particular, we start from the original pre-trained SigLIP model (which uses images of fixed-size square images) and adapt it by training on images of variable resolutions. configct||j|_|j|_|j |_t j|j|j|j |j d|_ |j|j z|_ |jdz|_ |j|_ t j|j|j|_y)Nvalid) in_channels out_channels kernel_sizestridepaddingr)super__init__ hidden_size embed_dim image_size patch_sizerConv2d num_channelspatch_embeddingnum_patches_per_side num_patches num_positions Embeddingposition_embeddingselfr: __class__s r2rCz!Idefics2VisionEmbeddings.__init__us ++ ++ ++!yy++??  %)OOt$F!44a7!--"$,,t/A/A4>>"Rr1 pixel_valuespatch_attention_maskreturnc.|j\}}}}|j|}|jdjdd}||jz||jz} } t j d|jz dd|jz } t j|| | zfd} t|D]\} }|dddfj}|dj}t j ||j|j}t j ||j|j}||z dz}||z dz}t j|| d }t j|| d }|dddf|jz|zj}|| | |jd j<| j!|j"j$j} ||j#| z}|S) Nrr?r)size fill_value)devicedtypeg !?T)right)shaperJflatten transposerGr,arangerKfull enumeratesumrZr[ bucketizeviewcputorOweight)rQrSrT batch_size_max_im_hmax_im_w patch_embeds embeddingsmax_nb_patches_hmax_nb_patches_w boundaries position_ids batch_idx p_attn_mask nb_patches_h nb_patches_w h_indices w_indicesfractional_coords_hfractional_coords_wbucket_coords_hbucket_coords_wpos_idss r2forwardz Idefics2VisionEmbeddings.forwards,8,>,>) Ax++L9 !))!,66q!< -5-H(VZVeVeJe*\\!d&?&?"?a$JcJcFcd zz 4DGW4W'Xefg &/0D&E J "I{&q!t,002L&q>--/L \,:M:MUaUgUghI \,:M:MUaUgUghI"+l":h"G "+l":h"G #oo.A:UYZO#oo.A:UYZO&q$w/$2K2KKo]ffhGBIL #K$4$4R$8$<$<$> ? J $t'>'>'E'E'L'LM $"9"9,"GG r1) r(r)r*r+rrCr,r- BoolTensorTensorr __classcell__rRs@r2r9r9jsDS3S&E$5$5UM]M]bgbnbnr1r9modulequerykeyvalueattention_maskscalingdropoutc lt|dr,t||j}t||j}tj||j dd|z}|#|ddddddd|j df} || z}tjj|dtjj|j}tjj|||j}tj||} | j ddj} | |fS) Nnum_key_value_groupsrrr])dimr[)ptrainingr)hasattr repeat_kvrr,matmulr`r^r functionalsoftmaxfloat32rhr[rr contiguous) rrrrrrrkwargs attn_weights causal_mask attn_outputs r2eager_attention_forwardrsv-.V889%! #k1 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$r1c eZdZdZfdZ ddej deej deej eej ffdZ xZ S)Idefics2VisionAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_d|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).F)rBrCr:rDrEnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutrrLineark_projv_projq_projout_proj is_causalrPs r2rCz Idefics2VisionAttention.__init__s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r1r%rrUc :|j\}}}|j|}|j|}|j|} |j |||j |j jdd}|j |||j |j jdd}| j |||j |j jdd} t} |jjdk7rt|jj} | |||| ||j|j|jsdn |j\} } | j!|||j#} |j%| } | | fS)z#Input shape: Batch x Time x Channelrreagerrrr)r^rrrrfrrr`rr:_attn_implementationrrrrrreshaperr) rQr%rrrj seq_lengthrEquerieskeysvaluesattention_interfacerrs r2rzIdefics2VisionAttention.forwardsa-:,?,?) J ++m,{{=)]+,,z:t~~t}}U__`acdeyyZOYYZ[]^_ZT^^T]]S]]^_abc(? ;; + +w 6"9$++:Z:Z"[ $7     nnJJ#}}C$,, % ! \"))*j)LWWY mmK0 L((r1N) r(r)r*r+rCr,rrr/rrrs@r2rrsUG226$)||$)!.$) u||Xell33 4 $)r1rcVeZdZfdZdej dej fdZxZS)Idefics2VisionMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yr) rBrCr:r hidden_act activation_fnrrrDintermediate_sizefc1fc2rPs r2rCzIdefics2VisionMLP.__init__sd  #F$5$5699V//1I1IJ99V55v7I7IJr1r%rUcl|j|}|j|}|j|}|Sr)rrr)rQr%s r2rzIdefics2VisionMLP.forward s4/ **=9 / r1)r(r)r*rCr,rrrrs@r2rrs$KU\\ellr1rc6eZdZdedededeffd ZdZxZS) Idefics2MLPrDr output_sizerct|tj||d|_tj||d|_tj||d|_t||_yNFbias) rBrCrr gate_projup_proj down_projr act_fn)rQrDrrrrRs r2rCzIdefics2MLP.__init__s[ ;0ANyy.?eL #4kNZ( r1c|j|j|j||j|zSr)rrrr)rQxs r2rzIdefics2MLP.forward"s0~~dkk$..*;S>ST**$66((**  r1c|jd}|jj|dd}|j|||d}|}|j |}||j |z}|dddfS)Nrr)r^rrepeatrrr)rQ hidden_staterjrresiduals r2rz-Idefics2MultiheadAttentionPoolingHead.forward8sv!''*  !!*a3~~e\<HK ~~l3 $((<"88 AqD!!r1)r(r)r*r+rrCrrrs@r2rr's&  3   "r1rc eZdZdeffd Z ddej dej deede ejfdZ xZ S) Idefics2EncoderLayerr:cDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ y)Nr) rBrCrDrEr self_attnrrr layer_norm1rr layer_norm2rPs r2rCzIdefics2EncoderLayer.__init__Fsm ++08<<F $ u  ! $r1rc eZdZdZdeffd Z d deejdee dee dee de e e ff d Z xZS) Idefics2Encoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`Idefics2EncoderLayer`]. Args: config: Idefics2Config r:ct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) rBrCr:r ModuleListrangenum_hidden_layersrlayersgradient_checkpointing)rQr:rkrRs r2rCzIdefics2Encoder.__init__sP  mm5QWQiQiKj$ka%9&%A$kl &+#%lsA#rroutput_hidden_states return_dictrUc~||n|jj}||n|jj}||n|jj}|rdnd}|rdnd}|}|jD]&} |r||fz}| |||} | d}|s|| dfz}(|r||fz}|st d|||fDSt |||S)ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr0)rrrc3&K|] }|| ywrr0).0vs r2 z*Idefics2Encoder.forward..seqWXWdesr#r%r&)r:rruse_return_dictrr/r) rQ inputs_embedsrrrrencoder_statesall_attentionsr% encoder_layer layer_outputss r2rzIdefics2Encoder.forwards<2C1N-TXT_T_TqTq$8$D $++JjJj &1%>Ne]NN$See e+>Vd  r1NNNN)r(r)r*r+rrCrr,rrrr/rrrrs@r2rrwsx,~,26,0/3&* = !.= $D> = 'tn = d^ =  uo% &= r1rcBeZdZUeed<dZdZgdZdZdZ dZ dZ dZ dZ y)Idefics2PreTrainedModelr:modelT)rrIdefics2PerceiverLayerIdefics2DecoderLayerr$ct|jd|jjj}t |t j t jfrY|jjjd||j%|jjjyyt |t jrf|jjjd||j2|jj|jjyyt |t jrJ|jjj!d|jjjyt |t"r&|jjj!dyt |t j$r|j'yt |t(r%|j*jjyt |t,r&|j.jj!dyy)Ninitializer_ranger)meanstdrW)getattrr:get_text_configr  isinstancerrrHridatanormal_rzero_rN padding_idxrfill_Idefics2RMSNormr_reset_parametersrrIdefics2PerceiverResamplerlatents)rQrr s r2 _init_weightsz%Idefics2PreTrainedModel._init_weightssdkk#6 8S8S8U8g8gh fryy"))4 5 MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>.  - MM   $ $S ) KK   " " $  0 MM   $ $S )  5 5 6  $ $ &  E F LL   % % '  : ; NN   % %c *RTaTgTg#h ,,'//!5# ' ,A. //0AB%'/!"*== =/)77&11  r1r)r(r)r*rr.r rr!rCr.r1rrr,rrrr/rrrrs@r2r$r$s ! NY3Y <@,0/3&* ; 'u'7'78; $D> ; 'tn ; d^ ;  uo% &; ; r1r$r%n_reprUc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r^expandr)r%r8batchnum_key_value_headsslenrs r2rrGso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr1c,eZdZdfd ZdZdZxZS)rct|tjt j ||_||_y)z> Idefics2RMSNorm is equivalent to T5LayerNorm N)rBrCrrr,r5rivariance_epsilon)rQrDrrRs r2rCzIdefics2RMSNorm.__init__Us1 ll5::k#:; #r1c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)Nrr]T)keepdim) r[rhr,rpowr rsqrtr@ri)rQr% input_dtypevariances r2rzIdefics2RMSNorm.forward]sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r1c^t|jjd|jS)Nz, eps=)r/rir^r@r-s r2 extra_reprzIdefics2RMSNorm.extra_reprds* ))*+6$2G2G1HIIr1)gư>)r(r)r*rCrrHrrs@r2rrTs$;Jr1rc eZdZd deeddffd Z ddejdejdeejdeejd ee d e d e de ejeejee ejffd Z xZ S)Idefics2PerceiverAttentionN layer_idxrUcJt|||_d|_|j|_|j |_|j|_|j|_ |j |jz|_ |j|_ |jdz|_ tj|j|j |jzd|_tj|j|j|jzd|_tj|j|j|jzd|_tj|j |jz|jd|_d|_y)ziPerceiver Cross-Attention Module --> let long-form inputs be `context`, resampled embeddings be `latents`NrFr)rBrCr:rKrDresampler_n_headsrresampler_head_dimrr<rrrrrrrro_projrrQr:rKrRs r2rCz#Idefics2PerceiverAttention.__init__is2  !--1111 #)#=#= $(NNd6N6N$N!!'!9!9}}d* ii 0 0$..4==2PW\] ii 0 0$2J2JT]]2Zafg ii 0 0$2J2JT]]2Zafg ii >@P@PW\] r1rcontextrrspast_key_valuer use_cachec ||j\}} } | |jdz} tj||gd} |j|} |j | }|j | }| j || |j|jjdd} |j || |j|jjdd}|j || |j|jjdd}t|d|}| |j|||j\}}t}|jj dk7rN|jj dk(r|rt"j%d nt&|jj }||| ||||j(|j*|j,sd n |j. \}}|j1|| |j|jz}|j3|}|sd}|||fS) a Runs Perceiver Self-Attention, with special (context, latents) appended along the `seq` dimension! Args: latents (`torch.Tensor`): Tensor of shape [bsz, n_latents, embed_dim] representing fixed length latents to compress to. context (`torch.Tensor`): Tensor of shape [bsz, seq, embed_dim] representing long-form context to resample. attention_mask (`torch.Tensor`, *optional*): Tensor of shape [bsz, 1, seq, n_latents] representing attention mask. position_ids (`torch.LongTensor`, *optional*): Tensor of shape [bsz, seq] representing position indices of each input token. past_key_value (`tuple[torch.Tensor]`, *optional*): Tuple of tensors containing cached key and value states. output_attentions (`bool`, *optional*, defaults to `False`): Whether to return attention weights. use_cache (`bool`, *optional*, defaults to `False`): Whether to use past_key_value for caching. rrrrrRNrsdpaz`torch.nn.functional.scaled_dot_product_attention` does not support `output_attentions=True`. Falling back to eager attention. This warning can be removed using the argument `attn_implementation="eager"` when loading the model.rr)rXr,concatrrrrfrrr`r<rupdaterKrr:rlogger warning_oncerrrrrrrO)rQrrQrrsrRrrSbszq_lenrk kv_seq_lenr%rrrrrrs r2rz"Idefics2PerceiverAttention.forward}s,  UAW\\^A..  gw%7R@ ++g&{{=)]+,,sE4>>4==ISSTUWXYyyj$*B*BDMMR\\]^`abS*d.F.F V``abdef '7H  %)00vt~~NLD&(? ;; + +w 6{{//69>O##L '>dkk>^>^&_#$7     nnLL#}}C$2H2H % ! \"))#udnnt}}6TU kk+.  LL.88r1rNNNFF)r(r)r*rrrCr,r LongTensorr rr/rrrs@r2rJrJhs(3-402637*."'C9C9C9!. C9 u//0 C9 ! C9 C9C9 u||Xell3XeELL>Q5RR SC9r1rJceZdZdeffd Z d dej dej deej deejdee dee d ee d e ejee ejejfffd Z xZS) rrKc0t||j|_|j|_|j |_|j|_t|j|j|_ t|j|j|_ t|||_ t|j|j|_ t|j|jdz|j|j|_y)Nr)rKrDrrr)rBrCrDresampler_n_latents n_latentsresampler_depthdepth rms_norm_epsrinput_latents_norminput_context_normrJrpost_attention_layernormrrrrPs r2rCzIdefics2PerceiverLayer.__init__s !--33++ "//"1$2B2BHYHY"Z"1$2B2BHYHY"Z3FiP(78H8HdN_N_(`%**$0014**((  r1rrQrrsrRrrSrUc |} |j|}|j|}|j|||\}} } | |z}|} |j|}|j |}| |z}|f} |r| | fz } |r| | fz } | S)a Args: latents (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` context (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states )rrQr)rirjrrkr) rQrrQrrsrRrrSrrself_attn_weightspresent_key_valuers r2rzIdefics2PerceiverLayer.forwards2))'2))'28<)9G9 5"$5 W$//8((7#W$*  )+ +G  )+ +Gr1r^)r(r)r*rrCr,rrr_r rr/r-rrrs@r2rrs # ,2637*.,1$)222!. 2 u//0 2 ! 2$D>2D>2 u  (51B1BEDUDU1U+V"WW X2r1rzi Idefics2 perceiver resampler model that performs `depth` blocks of cross-attention with a fixed ceZdZUeed<dZdZdZdfd Ze de jde jde jfdZ xZ S) rr:TrUc Zt|||j|_|j|_|j|_|j |_|j|_tjtj|j |j|_ tjt|jDcgc]}t!||c}|_t%|j|j|_|j(dk(|_ycc}wr&)rBrCrDrrdrerfrgrhrrr,r5rrrrrrnormrr*)rQr:idxrRs r2rCz#Idefics2PerceiverResampler.__init__s  !-- ++33++ "//||EJJt~~t?O?O$PQ mmTYZ^ZdZdTe$fS%;FC%H$fg #D$4$4$:K:KL &,&A&AEX&X#%gsD(rQrc F|jjdj|jdg|jj }t j |j d|j df|j|j}t j||gd}|js"t||j|jn|}|}|jD]}||||dddd }|d}|j|}|S) zw context (`torch.FloatTensor` of shape `(batch, seq_len, embed_dim)`): Input to the layer. rrr3r]rU)tgt_lenNF)rrsrRrrS)r unsqueezer:r^rXr,r5r[rZcatr*r rerrq)rQrQrrlatent_attention_maskcompressed_contextperceiver_layerrs r2rz"Idefics2PerceiverResampler.forward's,,((+22GMM!4D3[t||GXGXGZ3[\ %   #W\\!_ 5^=Q=QZhZoZo! N4I#JPRS.. '~w}}dnn ]  %#{{ 2O+"-!#"'M"/q!1  2"YY'9:!!r1)rUN)r(r)r*rr.r _supports_flash_attention_2r!rCrr,rrrrs@r2rr s^ $#N"&Y"&"&" &"  &"&"r1rc$eZdZfdZdZxZS)Idefics2Connectorc2t|t|jj|j j |j j|j j|_tj|j|_ y)Nrc) rBrCr vision_configrD text_configrrmodality_projectionr _from_configperceiver_configperceiver_resamplerrPs r2rCzIdefics2Connector.__init__Rsr #.,,88$00BB**66))44 $  $>#J#J6KbKb#c r1cN|j|}|j||}|S)N)rQr)rr)rQr'rs r2rzIdefics2Connector.forward\s2"667JK"66?Rcq6r""r1)r(r)r*rCrrrs@r2r|r|Qsd#r1r|z[ Idefics2 model consisting of a SIGLIP vision encoder and Mistral language decoder c#ReZdZdeffd ZdZdZdZdZde jde e jd e e jfd Z dd e jd e jfd Zeed dde e jde e jde e jde ede e jd e e jd e e j&d e e jde ede ede ede e jde edeedeeeffdZxZS) Idefics2Modelr:c0t|||jjj|_|jjj |_tj|j|_ t||_ tj|j|_|j j"|_|jj&|_|jj(dk(|_|j-y)Nr')rBrCr:r pad_token_idr vocab_sizer$rr~ vision_modelr| connectorr from_config text_modelrrd image_seq_lenimage_token_idrr* post_initrPs r2rCzIdefics2Model.__init__hs  ;;22??++11<<5BB6CWCWX*62#//0B0BC#44HH"kk88&,&8&8&M&MQd&d# r1cfdd}|jj||_|jj||_y)aE Enables the gradients for the input embeddings. This is useful for lora when using gradient checkpointing. c.f. https://github.com/huggingface/peft/issues/1402#issuecomment-1913675032 Override to set output.requires_grad = True for both the decoder's and vision model's embeddings. ctt|jdk(r|St|jdS)Nr)lenr7children)rget_lowest_modules r2rzCIdefics2Model.enable_input_require_grads..get_lowest_modules?4)*+q0 )foo.?)@)CDDr1c&|jdyNTrequires_grad_rinputoutputs r2make_inputs_require_gradszKIdefics2Model.enable_input_require_grads..make_inputs_require_grads  ! !$ 'r1N)r.register_forward_hook_text_require_grads_hookr_vision_require_grads_hook)rQrrs @r2enable_input_require_gradsz(Idefics2Model.enable_input_require_gradsxsN E ()-(A(A(C(Y(YZs(t%*;D