rhrBdZddlmZddlmZmZmZddlZddlZddlm Z ddl m Z ddl m Z mZdd lmZdd lmZdd lmZdd lmZdd lmZmZddlmZmZddlmZddlm Z m!Z!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(e#jRe*Z+ee!dGddeZ,ee!dGddeZ-Gdde j\Z/ dDde j\dej`d ej`d!ej`d"eej`d#e1d$e1fd%Z2Gd&d'e j\Z3Gd(d)e j\Z4Gd*d+e j\Z5Gd,d-eZ6Gd.d/e j\Z7d0ej`d1e8d2ej`fd3Z9Gd4d5e j\Z:Gd6d7e j\Z;e!Gd8d9eZ<e!d:Gd;dd?e<Z>e!d@GdAdBeE8B%(9(9":;Br0r!zS Base class for Idefics causal language model (or autoregressive) outputs. ceZdZUdZdZeejed<dZ eejed<dZ ee ejed<dZ ee ejed<dZee ejed<dZee ejed<y) Idefics3CausalLMOutputWithPastak loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`tuple(torch.FloatTensor)`, *optional*): Tuple of `torch.FloatTensor` (one for the output of the image embeddings, `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder Nlosslogitsr#r$r%r&)r'r(r)r*r4rr+r,r-r5r#listr$r.r%r&r/r0r1r3r3Js )-D(5$$ %,*.FHU&& '.9=OXd5#4#456=8B%(9(9":;Br0r3cxeZdZdZdeffd ZdejdejdejfdZ xZ S)Idefics3VisionEmbeddingsaX This is a modified version of `siglip.modelign_siglip.SiglipVisionEmbeddings` to enable images of variable resolution. The modifications are adapted from [Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution](https://huggingface.co/papers/2307.06304) which allows treating images in their native aspect ratio and without the need to resize them to the same fixed size. In particular, we start from the original pre-trained SigLIP model (which uses images of fixed-size square images) and adapt it by training on images of variable resolutions. configct||j|_|j|_|j |_t j|j|j|j |j d|_ |j|j z|_ |jdz|_ |j|_ t j|j|j|_y)Nvalid) in_channels out_channels kernel_sizestridepaddingr)super__init__ hidden_size embed_dim image_size patch_sizerConv2d num_channelspatch_embeddingnum_patches_per_side num_patches num_positions Embeddingposition_embeddingselfr9 __class__s r1rBz!Idefics3VisionEmbeddings.__init__us ++ ++ ++!yy++??  %)OOt$F!44a7!--"$,,t/A/A4>>"Rr0 pixel_valuespatch_attention_maskreturnc.|j\}}}}|j|}|jdjdd}||jz||jz} } t j d|jz dd|jz } t j|| | zfd} t|D]\} }|dddfj}|dj}t j ||j|j}t j ||j|j}||z dz}||z dz}t j|| d }t j|| d }|dddf|jz|zj}|| | |jd j<| j!|j"j$j} ||j#| z}|S) Nrr?r)size fill_value)devicedtypeg !?T)right)shaperIflatten transposerFr+arangerJfull enumeratesumrYrZ bucketizeviewcputorNweight)rPrRrS batch_size_max_im_hmax_im_w patch_embeds embeddingsmax_nb_patches_hmax_nb_patches_w boundaries position_ids batch_idx p_attn_mask nb_patches_h nb_patches_w h_indices w_indicesfractional_coords_hfractional_coords_wbucket_coords_hbucket_coords_wpos_idss r1forwardz Idefics3VisionEmbeddings.forwards,8,>,>) Ax++L9 !))!,66q!< -5-H(VZVeVeJe*\\!d&?&?"?a$JcJcFcd zz 4DGW4W'Xefg &/0D&E J "I{&q!t,002L&q>--/L \,:M:MUaUgUghI \,:M:MUaUgUghI"+l":h"G "+l":h"G #oo.A:UYZO#oo.A:UYZO&q$w/$2K2KKo]ffhGBIL #K$4$4R$8$<$<$> ? J $t'>'>'E'E'L'LM $"9"9,"GG r0) r'r(r)r*rrBr+r, BoolTensorTensorr~ __classcell__rQs@r1r8r8jsDS3S&E$5$5UM]M]bgbnbnr0r8modulequerykeyvalueattention_maskscalingdropoutc tj||jdd|z}|||z}tjj |dtj j|j}tjj|||j}tj||} | jddj} | |fS)Nr\)dimrZ)ptrainingrr) r+matmulr_r functionalsoftmaxfloat32rgrZrr contiguous) rrrrrrrkwargs attn_weights attn_outputs r1eager_attention_forwardrs<<s}}R'<=GL!#n4 ==((2U]](SVVW\WbWbcL==((6??([L,,|U3K''1-88:K  $$r0c eZdZdZfdZ ddej deej deej eej ffdZ xZ S)Idefics3VisionAttentionz=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_d|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).gF)rArBr9rCrDnum_attention_heads num_headshead_dim ValueErrorscaleattention_dropoutrrLineark_projv_projq_projout_proj is_causalrOs r1rBz Idefics3VisionAttention.__init__s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r0r$rrTc :|j\}}}|j|}|j|}|j|} |j |||j |j jdd}|j |||j |j jdd}| j |||j |j jdd} t} |jjdk7rt|jj} | |||| ||j|j|jsdn |j\} } | j!|||j#} |j%| } | | fS)z#Input shape: Batch x Time x Channelrreager)rrr)r]rrrrerrr_rr9_attn_implementationrrrrrreshaperr) rPr$rrri seq_lengthrDquerieskeysvaluesattention_interfacerrs r1r~zIdefics3VisionAttention.forwardsa-:,?,?) J ++m,{{=)]+,,z:t~~t}}U__`acdeyyZOYYZ[]^_ZT^^T]]S]]^_abc(? ;; + +w 6"9$++:Z:Z"[ $7     nnJJ#}}C$,, % ! \"))*j)LWWY mmK0 L((r0N) r'r(r)r*rBr+rrr.r~rrs@r1rrsUG226$)||$)!.$) u||Xell33 4 $)r0rcVeZdZfdZdej dej fdZxZS)Idefics3VisionMLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yr) rArBr9r hidden_act activation_fnrrrCintermediate_sizefc1fc2rOs r1rBzIdefics3VisionMLP.__init__sd  #F$5$5699V//1I1IJ99V55v7I7IJr0r$rTcl|j|}|j|}|j|}|Sr)rrr)rPr$s r1r~zIdefics3VisionMLP.forward s4/ **=9 / r0)r'r(r)rBr+rr~rrs@r1rrs$KU\\ellr0rc$eZdZfdZdZxZS)Idefics3SimpleMLPct||jj|jdzz}|j j}t j||d|_y)NrFbias) rArB vision_configrC scale_factor text_configrrproj)rPr9 input_size output_sizerQs r1rBzIdefics3SimpleMLP.__init__sT ))559L9La9OP ((44 IIj+EB r0c$|j|Sr)r)rPxs r1r~zIdefics3SimpleMLP.forwardsyy|r0)r'r(r)rBr~rrs@r1rrsC r0rc eZdZdeffd Z ddej dej deede ejfdZ xZ S) Idefics3EncoderLayerr9cDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ y)Neps) rArBrCrDr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rOs r1rBzIdefics3EncoderLayer.__init__sm ++08<<F $ u  ! $r0rc eZdZdZdeffd Z d deejdee dee dee de e e ff d Z xZS) Idefics3Encoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`Idefics3EncoderLayer`]. Args: config: Idefics3Config r9ct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}w)NF) rArBr9r ModuleListrangenum_hidden_layersrlayersgradient_checkpointing)rPr9rjrQs r1rBzIdefics3Encoder.__init__XsP  mm5QWQiQiKj$ka%9&%A$kl &+#%lsA#rroutput_hidden_states return_dictrTc~||n|jj}||n|jj}||n|jj}|rdnd}|rdnd}|}|jD]&} |r||fz}| |||} | d}|s|| dfz}(|r||fz}|st d|||fDSt |||S)ad Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) output_attentions (`bool`, *optional*): Whether or not to return the attentions tensors of all attention layers. See `attentions` under returned tensors for more detail. output_hidden_states (`bool`, *optional*): Whether or not to return the hidden states of all layers. See `hidden_states` under returned tensors for more detail. return_dict (`bool`, *optional*): Whether or not to return a [`~utils.ModelOutput`] instead of a plain tuple. Nr/)rrrc3&K|] }|| ywrr/).0vs r1 z*Idefics3Encoder.forward..seqWXWdesr"r$r%)r9rruse_return_dictrr.r) rP inputs_embedsrrrrencoder_statesall_attentionsr$ encoder_layer layer_outputss r1r~zIdefics3Encoder.forward_s<2C1N-TXT_T_TqTq$8$D $++JjJj &1%>Ne]NN$See e+>Vd  r0NNNN)r'r(r)r*rrBrr+rrrr.rr~rrs@r1rrOsx,~,26,0/3&* = !.= $D> = 'tn = d^ =  uo% &= r0rr$n_reprTc|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)r]expandr)r$rbatchnum_key_value_headsslenrs r1 repeat_kvrso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr0c,eZdZdfd ZdZdZxZS)Idefics3RMSNormct|tjt j ||_||_y)z> Idefics3RMSNorm is equivalent to T5LayerNorm N)rArBr Parameterr+onesrhvariance_epsilon)rPrCrrQs r1rBzIdefics3RMSNorm.__init__s1 ll5::k#:; #r0c"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)Nrr\T)keepdim) rZrgr+rpowmeanrsqrtrrh)rPr$ input_dtypevariances r1r~zIdefics3RMSNorm.forwardsy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r0c^t|jjd|jS)Nz, eps=)r.rhr]rrPs r1 extra_reprzIdefics3RMSNorm.extra_reprs* ))*+6$2G2G1HIIr0)gư>)r'r(r)rBr~rrrs@r1rrs$;Jr0rc,eZdZfdZddZdZxZS)Idefics3Connectorcdt||j|_t||_yr)rArBrrmodality_projectionrOs r1rBzIdefics3Connector.__init__s) "//#4V#< r0c|j\}}}t|dzx}}|j||||}|j||t||z ||z}|jdddd}|j |t||z t||z ||dzz}|jdddd}|j |t||dzz ||dzz}|S)Ng?rrrr)rWintrepermuter)rPrrbszseqrDheightwidths r1 pixel_shufflezIdefics3Connector.pixel_shufflesffhS)S#X& FF3y 1 FF3EL$8 99|;S T IIaAq ! IIc3u|34c&<:O6PR[_kmn_nRo p IIaAq ! IIc3slAo67lTUo9V Wr0c`|j||j}|j|}|Sr)rrr )rPr&s r1r~zIdefics3Connector.forwards4"001DdFWFWX"667JK""r0)r)r'r(r)rBrr~rrs@r1rrs= #r0rcBeZdZUeed<dZdZddgZdZdZ dZ dZ dZ dZ y) Idefics3PreTrainedModelr9modelTrIdefics3DecoderLayerr#ct|jd|jjj}t |t j t jfrY|jjjd||j%|jjjyyt |t jrf|jjjd||j2|jj|jjyyt |t jrJ|jjj!d|jjjyt |t"r&|jjj!dyy)Ninitializer_ranger)rstdrV)getattrr9get_text_configr isinstancerrrGrhdatanormal_rzero_rM padding_idxrfill_r)rPrrs r1 _init_weightsz%Idefics3PreTrainedModel._init_weightssAdkk#6 8S8S8U8g8gh fryy"))4 5 MM   & &CS & 9{{&   &&('  - MM   & &CS & 9!!- ""6#5#56<<>.  - MM   $ $S ) KK   " " $  0 MM   $ $S )1r0N)r'r(r)rr-base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendr"r/r0r1rrsB &*#24JK"3N"&*r0rzO The Idefics3 Vision Transformer Model outputting raw image embedding. c eZdZUeed<dZdZdZdeffd ZdZ dZ d de e jde ede ed e ed eeeff d ZxZS) Idefics3VisionTransformerr9Tct|||j}t||_t ||_|j|_tj||j|_ |jdk(|_ y)Nrflash_attention_2)rArBrCr8rnrencoderrFrrrpost_layernormr_use_flash_attention_2)rPr9rDrQs r1rBz"Idefics3VisionTransformer.__init__sm  && 26:&v.  ++ ll9&:O:OP&,&A&AEX&X#r0c|jSrrnrs r1get_input_embeddingsz.Idefics3VisionTransformer.get_input_embeddings s r0c||_yrr3rPrs r1set_input_embeddingsz.Idefics3VisionTransformer.set_input_embeddingss r0rSrrrrTc||n|jj}||n|jj}||n|jj}|j d}|s|j }t j||j d|z|j d|zf}|jt j|j}|j||}|j|d}|jst||j}nt j |sd}|j#|||||} | d} |j%| } |s | f| ddzSt'| | j(| j* S) NrrrrZrYrRrSr\)rrrrrrr)r9rrrrWrFr+rrgrrYrnrer1r rZanyr/r0rr$r%) rPrRrSrrrrirFr$encoder_outputsr"s r1r~z!Idefics3VisionTransformer.forwards2C1N-TXT_T_TqTq$8$D $++JjJj &1%RTaTgTg#h 001#' ,,'//!5# ' ,A. //0AB%'/!"*== =/)77&11  r0r)r'r(r)rr-r(r'r)rBr4r7rr+rrrr.rr~rrs@r1r,r,s ! NY3Y <@,0/3&* 7 'u'7'787 $D> 7 'tn 7 d^ 7  uo% &7 r0r,zZ Idefics3 model consisting of a SIGLIP vision encoder and Llama3 language decoder c#ReZdZdeffd ZdZdZdZdZde jde e jd e e jfd Z dd e jd e jfd Zeed dde e jde e jde e jde ede e jd e e jd e e j&d e e jde ede ede ede e jde edeedeeeffdZxZS) Idefics3Modelr9ct|||jjj|_|jjj |_tj|j|_ t||_ tj|j|_t!|jj"|jj$zdz|j&dzz |_|jj*|_|jj,dk(|_|j1y)Nrr.)rArBr9r pad_token_idr  vocab_sizer, _from_configr vision_modelr connectorr from_config text_modelr rErFr image_seq_lenimage_token_idrr1 post_initrOs r1rBzIdefics3Model.__init__Ts  ;;22??++11<<5BB6CWCWX*62#//0B0BC ""--1E1E1P1PPUV V[a[n[npq[q r #kk88&,&8&8&M&MQd&d# r0cfdd}|jj||_|jj||_y)aE Enables the gradients for the input embeddings. This is useful for lora when using gradient checkpointing. c.f. https://github.com/huggingface/peft/issues/1402#issuecomment-1913675032 Override to set output.requires_grad = True for both the decoder's and vision model's embeddings. ctt|jdk(r|St|jdS)Nr)lenr6children)rget_lowest_modules r1rNzCIdefics3Model.enable_input_require_grads..get_lowest_moduleqs?4)*+q0 )foo.?)@)CDDr0c&|jdyNTrequires_grad_rinputoutputs r1make_inputs_require_gradszKIdefics3Model.enable_input_require_grads..make_inputs_require_gradsy  ! !$ 'r0N)r4register_forward_hook_text_require_grads_hookrC_vision_require_grads_hook)rPrVrNs @r1enable_input_require_gradsz(Idefics3Model.enable_input_require_gradsgsN E ()-(A(A(C(Y(YZs(t%*;D