rhGd=d>e jZZ?Gd?d@e jZZ@edAGdBdCe9ZAGdDdEe jZZBGdFdGe jZZCGdHdIe jZZDGdJdKe jZZEedLGdMdNe9ZFgdOZGy)PzPyTorch FLAVA model.N) OrderedDict) dataclass)AnyOptionalUnion)nn)ACT2FN)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)PreTrainedModel find_pruneable_heads_and_indicesprune_linear_layer) ModelOutputauto_docstringlogging torch_int) FlavaConfigFlavaImageCodebookConfigFlavaImageConfigFlavaMultimodalConfigFlavaTextConfigzfacebook/flava-image-codebookg$(~k@a Output from FlavaModel containing embeddings and outputs from individual encoders. Note that `image_embeddings` and `text_embeddigns` returned are similar to pooled output returned from a transformer. If you want embeddings for contrastive loss or retrieval use a FLAVA model's `image_projection` and `text_projection` layers on `image_embeddings` and `text_embeddings` respectively. ) custom_introceZdZUdZdZeejed<dZ ee ed<dZ eejed<dZ ee ed<dZ eejed<dZee ed<d eefd Zy) FlavaModelOutputa image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. image_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. text_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids` are present): The output of the [`FlavaTextModel`]. multimodal_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present and `skip_multimodal_encoder` is `None` or `False`): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_output (`BaseModelOutputWithPooling`, returned when `input_ids` and `pixel_values` are present and `skip_multimodal_encoder` is `None` or `False`): The output of the [`FlavaMultimodalModel`]. Nimage_embeddings image_outputtext_embeddings text_outputmultimodal_embeddingsmultimodal_outputreturncHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw))r!rr#Ngetattrto_tuple).0kselfs {/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/flava/modeling_flava.py z,FlavaModelOutput.to_tuple..Us=  TTDGZabfhiZjZsZsZu u -0tuplekeysr,s`r-r)zFlavaModelOutput.to_tupleTs# YY[   )__name__ __module__ __qualname____doc__rrtorch FloatTensor__annotations__rr r r!r"r#r1rr)r4r-rr3s 59hu00189=L(56=37OXe//078Bx :;B %* r4rz@ Class representing pretraining losses from FLAVA model ceZdZUdZdZeejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<dZ eejed<d efd Zy) FlavaLossesa mim (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mim_labels` and `pixel_values` are present, `input_ids_masked` is absent and `mim_weight` > 0.): Masked Image Modeling loss as used in BeIT calculated only for unimodal image data. mlm (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mlm_labels` and `input_ids_masked` are present, `pixel_values` is absent and `mlm_weight` > 0.): Masked Language Modeling loss as used in BERT calculated only for unimodal text data. itm (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `itm_labels`, `input_ids_masked`, `pixel_values` are present and `itm_weight` > 0.): Image Text Matching (ITM) loss calculated for paired image-text data. Note that ITM loss is calculated on masked pairs in FLAVA. global_contrastive (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `input_ids` and `pixel_values` are present and `global_contrastive_weight` > 0.): Contrastive loss for image-text similarity similar to CLIP but calculated globally for paired image-text data. This is calculated on unmasked images and texts. mmm_image (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mim_labels`, `pixel_values` and `input_ids_masked` are present and `mmm_image_weight` > 0.): Masked Multimodal Modeling loss's image component calculated on paired image-text data. mmm_text (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `mlm_labels`, `pixel_values` and `input_ids_masked` are present and `mmm_text_weight` > 0.): Masked Multimodal Modeling loss's text component calculated on paired image-text data. Nmimmlmitmglobal_contrastive mmm_imagemmm_textr$cBd}|jD] }|d}|S|S)NTF)values)r,all_nonevs r-rGzFlavaLosses.all_nonezs5 A}   r4)r5r6r7r8r?rr9r:r;r@rArBrCrDboolrGr<r4r-r>r>[s"(,C%## $+'+C%## $+'+C%## $+6:!2!23:-1Ix))*1,0Hhu(()0$r4r>a Output from FlavaForPreTraining containing embeddings, and outputs from individual encoders. Note that `image_embeddings` and `text_embeddings` returned are similar to pooled output returned from a transformer. If you want embeddings for contrastive loss or retrieval use a FLAVA model's `image_projection` and `text_projection` layers on `image_embeddings` and `text_embeddings` respectively. ceZdZUdZdZeejed<dZ e ed<dZ eejed<dZ ee ed<dZeejed<dZee ed<dZeejed <dZee ed <dZeejed <dZee ed <dZeejed <dZee ed<dZeejed<dZee ed<dZeejed<dZeejed<dZeejed<dZeejed<dZeejed<dZeejed<dZeejed<dee fdZ!y)FlavaForPreTrainingOutputay loss (`torch.FloatTensor`, *optional*, returned when `return_loss` is True): Total loss calculated for this model. loss_info (`FlavaLosses`): Detailed info for FLAVA Pretraining losses. Check `FlavaLosses` class description for the information on the keys. image_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. image_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. text_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids` are present): The output of the [`FlavaTextModel`]. multimodal_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present and `skip_unmasked_multimodal_encoder` is `None` or `False`): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_output (`BaseModelOutputWithPooling`, returned when `input_ids` and `pixel_values` are present and `skip_unmasked_multimodal_encoder` is `None` or `False`): The output of the [`FlavaMultimodalModel`]. image_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `pixel_values` are present): The image embeddings which are basically the pooled output of [`FlavaImageModel`]. Uses `bool_masked_pos` to create masked images. image_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `pixel_values` are present): The output of the [`FlavaImageModel`]. Uses `bool_masked_pos` to create masked images. text_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids_masked` are present): The text embeddings which are basically the pooled output of [`FlavaTextModel`]. text_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids_masked` are present): The output of the [`FlavaTextModel`]. multimodal_masked_embeddings (`torch.FloatTensor` of shape `(batch_size, output_dim)`, *optional*, returned when `input_ids` and `pixel_values` are present): The multimodal embeddings which are basically the pooled output of [`FlavaTextModel`]. multimodal_masked_output (`BaseModelOutputWithPooling`, *optional*, returned when `input_ids_masked` and `pixel_values` are present): The output of the [`FlavaMultimodalModel`]. mim_logits (`torch.FloatTensor` of shape `(batch_size, num_image_patches, image_vocab_size)` or of shape `(total_masked_patches, image_vocab_size)` , *optional*, returned when `pixel_values` are present and `input_ids_masked` are not): The logits for MIM unimodal loss. Uses `book_masked_pos` to get masked patches. The flattened output is returned when `bool_masked_pos` has some of the patches masked. mlm_logits (`torch.FloatTensor` of shape `(batch_size, text_seq_length, text_vocab_size)` or of shape `(total_masked_seq_length, text_vocab_size)`, *optional*, returned when `input_ids_masked` are present and `pixel_values` are not): The logits for MLM unimodal loss. The flattened output is returned when `input_ids_masked` has some of the tokens masked. itm_logits (`torch.FloatTensor` of shape `(batch_size, 2)`, *optional*, returned when `input_ids_masked` and `pixel_values` are present): The logits for ITM loss. Note that ITM loss is calculated on masked pairs in FLAVA. contrastive_logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeddings` and `text_embeddings` but passed through FLAVA's `image_projection` and `text_projection` layers respectively. This represents the image-text similarity scores. This is calculated on unmasked images and texts. contrastive_logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeddings` and `image_embeddings` but passed through FLAVA's `text_projection` and `image_projection` layers respectively. This is calculated on unmasked images and texts. mmm_image_logits (`torch.FloatTensor` of shape `(batch_size, num_image_patches, image_vocab_size)` or of shape`(total_masked_patches, image_vocab_size)`, *optional*, returned when `pixel_values` and `input_ids_masked` are present): The logits for MMM image multimodal loss. Uses `book_masked_pos` to get masked patches. The flattened output is returned when `bool_masked_pos` has some of the patches masked. mmm_text_logits (`torch.FloatTensor` of shape `(batch_size, text_seq_length, text_vocab_size)` or of shape `(`(total_masked_seq_length, text_vocab_size)`), *optional*, returned when `pixel_values` and `input_ids_masked` are present): The logits for MMM text multimodal loss. The flattened output is returned when `input_ids_masked` has some of the tokens masked. Nloss loss_inforrr r!r"r#image_masked_embeddingsimage_masked_outputtext_masked_embeddingstext_masked_outputmultimodal_masked_embeddingsmultimodal_masked_output mim_logits mlm_logits itm_logitscontrastive_logits_per_imagecontrastive_logits_per_textmmm_image_logitsmmm_text_logitsr$cTgdtfdjDS)N)r!rr#rQrOrSc3dK|]'}|vr|nt|j)ywNr')r*r+r,transformer_outputss r-r.z5FlavaForPreTrainingOutput.to_tuple..s4sbc)< rrr r r!r"r#rNrOrPrQrRrSrTrUrVrWrXrYrZr1rr)r<r4r-rKrKs5n)-D(5$$ %,!I{!48hu00189=L(56=37OXe//078Bx :;B;?Xe&7&78?@D"<=D:>HU%6%67>?C!;<C@D (5+<+<"=DEIh'ABI.2J**+2.2J**+2.2J**+2@D (5+<+<"=D?C%*;*;!<C48hu001837OXe//07 t%* tr4rKc eZdZdZddededdffd Zdejde d e dejfd Z dd ejd e ejd edejfdZ xZS)FlavaImageEmbeddingszb Construct the CLS token, position and patch embeddings. Optionally, also the mask token. configuse_mask_tokenr$Nct||xs |j}tjt j dd|j|_|r4tjt j dd|jnd|_t|j|j|j|j|_ |jj}tjt j d|dz|j|_tj |j"|_|j|_ ||_y)Nr) image_size patch_size num_channels embed_dim)super__init__ mask_tokenr Parameterr9zeros hidden_size cls_tokenPatchEmbeddingsrdrerfpatch_embeddings num_patchesposition_embeddingsDropouthidden_dropout_probdropoutra)r,rarbrq __class__s r-rizFlavaImageEmbeddings.__init__s '<6+<+<ekk!Q8J8J&KLQ_",,u{{1a9K9K'LMei /((((,,(( !  ++77 #%<< A{QPVPbPb0c#d zz&"<"<=  ++ r4 embeddingsheightwidthc|jddz }|jjddz }tjj s||k(r||k(r |jS|jddddf}|jddddf}|jd}||j z} ||j z} t |dz} |jd| | |}|jdddd}tjj|| | fdd }|jddddjdd|}tj||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rNg?rr bicubicF)sizemode align_cornersdim)shaperrr9jit is_tracingrerreshapepermuter functional interpolateviewcat) r,rwrxryrq num_positionsclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r-interpolate_pos_encodingz-FlavaImageEmbeddings.interpolate_pos_encodings`!&&q)A- 0066q9A= yy##%+*F6UZ?++ +221bqb59221ab59r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr4 pixel_valuesbool_masked_posrcV|j\}}}}|j||}|j\}} } ||jj || d} |j dk(r!|j |jdd}|jdj| } |d| z z| | zz}|jj |dd} tj| |fd}|r||j|||z}n||jz}|j|}|S)N)rr{r r?rr)rrpr~rjexpandrr unsqueezetype_asrnr9rrrrru)r,rrr batch_sizerfrxryrwseq_len_ mask_tokensmask cls_tokenss r-forwardzFlavaImageEmbeddings.forward)s4 3?2D2D/ L&%** YY J7Q?  $#d&C&CJPVX]&^^J#d&>&>>J\\*- r4FNF)r5r6r7r8rrIrir9Tensorintrr BoolTensorr __classcell__rvs@r-r`r`s/RV&&D5<<&D&DUX&D]b]i]i&DV7;). ll"%"2"23#'   r4r`c eZdZdZ d dedeeeeeffdedeffd Zd dejde d ejfd Z xZ S) roz# Image to Patch Embedding. rdrerfrgcVt|t|tjj s||f}t|tjj s||f}|d|dz|d|dzz}||_||_||_tj|||||_ y)Nrr) kernel_sizestride) rhri isinstance collectionsabcIterablerdrerqrConv2d projection)r,rdrerfrgrqrvs r-rizPatchEmbeddings.__init__Rs *koo&>&>?$j1J*koo&>&>?$j1J!!} 1 5*Q-:VW=:XY $$&))L)\fgr4rrr$c 8|j\}}}}|sV||jdk7s||jdk7r2td|d|d|jdd|jdd |j|j dj dd}|S)NrrzInput image size (*z) doesn't match model (z).r|)rrd ValueErrorrflatten transpose)r,rrrrfrxryxs r-rzPatchEmbeddings.forwardes2>2D2D/ L&%'++u8J/J (%9+,Adooa.@-AE OOL ) 1 1! 4 > >q! Dr4)r ir) r5r6r7r8rrr1rir9rrIrrrs@r-roroMs} 24 hh#uS#X./h h  h& ELL D ]b]i]i r4roceZdZdZfdZ ddeejdeejdeejfdZxZ S)FlavaTextEmbeddingszGConstruct the embeddings from word, position and token_type embeddings.c>t|tj|j|j |j |_tj|j|j |_ tj|j|j |_ tj|j |j|_ tj|j|_t#|dd|_|j'dt)j*|jj-dd|j'd t)j.|j0j3t(j4 dy) N) padding_idxepsposition_embedding_typeabsolute position_ids)rr{F) persistenttoken_type_ids)dtype)rhrir Embedding vocab_sizerm pad_token_idword_embeddingsmax_position_embeddingsrrtype_vocab_sizetoken_type_embeddings LayerNormlayer_norm_epsrsrtrur(rregister_bufferr9arangerrlrr~longr,rarvs r-rizFlavaTextEmbeddings.__init__ts/ !||F,=,=v?Q?Q_e_r_rs#%<<0N0NPVPbPb#c %'\\&2H2H&J\J\%]"f&8&8f>S>STzz&"<"<= '.v7PR\']$  ELL)G)GHOOPWXej    ekk$*;*;*@*@*B%**Ubg  r4 input_idsrrc$|j}|d}||jddd|f}|st|dr-|jddd|f}|j |d|}|}n:t j |t j|jj}|j|}|j|} || z} |jdk(r|j|} | | z } |j| } |j| } | S)Nrrr)rdevicer)r~rhasattrrrr9rlrrrrrrrrru) r,rrr input_shape seq_lengthbuffered_token_type_ids buffered_token_type_ids_expanded inputs_embedsrrwrrs r-rzFlavaTextEmbeddings.forwards  nn&  ^  ,,Q  ^ J"%::  ' ': 5"&":":<"H  - -J^^J/ \\*- r4)NNN) r5r6r7r8rirr9rrrrs@r-rrqsRQ *-115/3 ELL) !. u||, r4rceZdZdeddffd Z d dej deej deej dede e ej ej fe ej ff d Z xZ S) FlavaSelfAttentionrar$Nct||j|jzdk7r2t |ds&t d|jd|jd|j|_t |j|jz |_|j|jz|_tj|j|j|j|_ tj|j|j|j|_ tj|j|j|j|_tj|j |_y)Nrembedding_sizezThe hidden size z4 is not a multiple of the number of attention heads .bias)rhrirmnum_attention_headsrrrattention_head_size all_head_sizerLinearqkv_biasquerykeyvaluersattention_probs_dropout_probrurs r-rizFlavaSelfAttention.__init__s.     : : :a ?PVXhHi"6#5#5"67334A7  $*#=#= #&v'9'9FS"S* **,CD 6G=/2O\M]r4NNF) r5r6r7FlavaPossibleConfigsrir9rrrIrr1rrrs@r-rrsG3GG*26,0"' 3||3!.3ELL) 3  3 uU\\5<</0% 2EE F 3r4rc|eZdZdZdeddffd ZdejdejdejfdZxZ S) FlavaSelfOutputz The residual connection is defined in FlavaLayer (same as ViTLayer) instead of here (as is the case with other models), due to the layernorm applied before each block. rar$Nct|tj|j|j|_tj |j|_yr]) rhrirrrmdensersrtrurs r-rizFlavaSelfOutput.__init__sB YYv1163E3EF zz&"<"<= r4r input_tensorcJ|j|}|j|}|Sr]rrur,rrs r-rzFlavaSelfOutput.forwards$ =1  ]3 r4) r5r6r7r8rrir9rrrrs@r-rrsE >3>> U\\RWR^R^r4rceZdZdeddffd ZdeeddfdZ d dejde ejd e ejd e de e ejejfe ejff d ZxZS) FlavaAttentionrar$Nct|t||_t ||_t |_yr])rhrir attentionroutputset pruned_headsrs r-rizFlavaAttention.__init__s0 +F3%f- Er4headsc>t|dk(ryt||jj|jj|j \}}t |jj||j_t |jj||j_t |jj||j_ t |jj|d|j_ |jjt|z |j_|jj|jjz|j_ |j j||_y)Nrrr)lenrr rrrrrrrr rrunion)r,rindexs r- prune_headszFlavaAttention.prune_heads s u:? 7 4>>55t~~7Y7Y[_[l[l u 2$..2F2FN/0B0BEJ1$..2F2FN.t{{/@/@%QO .2^^-O-ORUV[R\-\*'+~~'I'IDNNLnLn'n$ --33E:r4rrrrcl|j||||}|j|d|}|f|ddz}|SN)rrrrr)r r )r,rrrr self_outputsattention_outputrs r-rzFlavaAttention.forwardsQ~~ .Iar&  ;;|A F#% QR(88r4r)r5r6r7rrirrrr9rrrIrr1rrrs@r-r r s"3"" ;S;d;*26,0"' ||!.ELL)    uU\\5<</0% 2EE F r4r c`eZdZdeddffd Zdej dej fdZxZS)FlavaIntermediaterar$Nct|tj|j|j |_t|jtrt|j|_ y|j|_ yr]) rhrirrrmintermediate_sizerr hidden_actstrr intermediate_act_fnrs r-rizFlavaIntermediate.__init__0s] YYv1163K3KL f'' -'-f.?.?'@D $'-'8'8D $r4rcJ|j|}|j|}|Sr])rr r,rs r-rzFlavaIntermediate.forward9s& =1 00? r4 r5r6r7rrir9rrrrs@r-rr/s29399U\\ellr4rcxeZdZdeddffd Zdej dej dej fdZxZS) FlavaOutputrar$Nct|tj|j|j |_tj|j|_ yr]) rhrirrrrmrrsrtrurs r-rizFlavaOutput.__init__AsB YYv779K9KL zz&"<"<= r4rrcT|j|}|j|}||z}|Sr]rrs r-rzFlavaOutput.forwardGs. =1  ]3 % 4 r4r#rs@r-r%r%@s@>3>> U\\RWR^R^r4r%ceZdZdZdeddffd Z d dejdeejdeejd e de e ejejfe ejff d Z xZ S) FlavaLayerz?This corresponds to the Block class in the timm implementation.rar$Ncrt||j|_d|_t ||_t ||_t||_ tj|j|j|_tj|j|j|_y)Nrr)rhrichunk_size_feed_forward seq_len_dimr r r intermediater%r rrrmrlayernorm_beforelayernorm_afterrs r-rizFlavaLayer.__init__Ss '-'E'E$'/-f5!&) !# V-?-?VEZEZ [!||F,>,>FDYDYZr4rrrrc|j|j||||}|d}|dd}||z}|j|}|j|}|j ||}|f|z}|Sr)r r.r/r-r ) r,rrrrself_attention_outputsrr layer_outputs r-rzFlavaLayer.forward_s"&  ! !- 0)/ "0"  2!4(,)=8 ++M: ((6 {{<? /G+r4r)r5r6r7r8rrir9rrrIrr1rrrs@r-r)r)PsI [3 [ [26,0"' ||!.ELL)    uU\\5<</0% 2EE F r4r)ceZdZdeddffd Z d dej deej deej ded ed ede e e ffd Z xZ S) FlavaEncoderrar$Nct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wr) rhrirar ModuleListrangenum_hidden_layersr)layergradient_checkpointing)r,rarrvs r-rizFlavaEncoder.__init__sN  ]]fF^F^@_#`1Jv$6#`a &+#$asA#rrrroutput_hidden_states return_dictc|rdnd}|rdnd}t|jD]2\} } |r||fz}||| nd} | ||| |} | d}|s*|| dfz}4|r||fz}|std|||fDSt|||S)Nr<rrc3&K|] }|| ywr]r<)r*rHs r-r.z'FlavaEncoder.forward..smq_`_lm)last_hidden_stater attentions) enumerater9r1r ) r,rrrrr;r<all_hidden_statesall_self_attentionsi layer_modulelayer_head_mask layer_outputss r-rzFlavaEncoder.forwards#7BD$5b4(4 POA|#$58H$H!.7.CilO(YjkM)!,M &9]1=M.  - KK   " " $ MM   $ $S )  9 : KK   " " $  4 5    ! ! ' ' )  & & + + 1 1 3  ,!!&&,,.-  4 5##  %%++-$  +    # # ) )$++*L*L M,r4) r5r6r7rr;base_model_prefixsupports_gradient_checkpointingrrrrrrer<r4r-rSrSsC &*#NE"))RYY *L$MNRVNr4rScpeZdZUeed<dZdZddedeffd Zde jfdZ de jfd Z d e eeefdd fd Ze ddeej(d eej*deedeej(deej(deedeedeedeeeffdZxZS)FlavaImageModelrazflava.image_modelradd_pooling_layerct||||_t||_t ||_tj|j|j|_ |r t|nd|_ |jyv add_pooling_layer (bool, *optional*, defaults to `True`): Whether to add a pooling layer rN)rhrirar`rwr4encoderrrrmr layernormrJpooler post_initr,rarjrvs r-rizFlavaImageModel.__init__si   .v6#F+ f&8&8f>S>ST->k&)D  r4r$c.|jjSr]rwrpr3s r-get_input_embeddingsz$FlavaImageModel.get_input_embeddingss///r4rc&||j_yr]rtr,rs r-set_input_embeddingsz$FlavaImageModel.set_input_embeddingss+0(r4heads_to_pruneNc|jD]7\}}|jj|jj |9yz Prunes heads of the model. heads_to_prune: dict of {layer_num: list of heads to prune in this layer} See base class PreTrainedModel Nitemsrnr9r rr,ryr9rs r- _prune_headszFlavaImageModel._prune_headsE +002 CLE5 LL  u % / / ; ;E B Cr4rrrrrr;r<c "||n|jj}||n|jj}||n|jj}| t d|j ||jj }|j|||} |j| |||||} | d} |j| } |j|j| nd} |s | | f| ddzSt| | | j| jS)z bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Nz You have to specify pixel_values)rrrrrr;r<rrr@ pooler_outputrrA)rarr;use_return_dictr get_head_maskr8rwrnrorpr rrA) r,rrrrrrr;r<embedding_outputencoder_outputssequence_outputrQs r-rzFlavaImageModel.forwards8 2C1N-TXT_T_TqTq$8$D $++JjJj &1%7 'tn7 d^7  u00 17 7 r4ricleZdZUeed<dZddedeffd ZdefdZ de jfdZ d e eeefdd fd Ze dd eej(d eej(deej(deej(deej(deedeedeedeeeffdZxZS)FlavaTextModelrazflava.text_modelrjct||||_t||_t ||_tj|j|j|_ |r t|nd|_ |jyrl)rhrirarrwr4rnrrrmrrorJrprqrrs r-rizFlavaTextModel.__init__Asi   -f5#F+ f&8&8f>S>ST->k&)D  r4r$c.|jjSr]rwrr3s r-ruz#FlavaTextModel.get_input_embeddingsQs...r4rc&||j_yr]rrws r-rxz#FlavaTextModel.set_input_embeddingsTs*/'r4ryNc|jD]7\}}|jj|jj |9yr{r|r~s r-rzFlavaTextModel._prune_headsWrr4rrrrrrr;r<c ||n|jj}||n|jj}||n|jj}| t d|j } |!t j| |j}|j||jj}|j|| |j} |j|||} |j| | ||||} | d} |j| } |j|j| nd}|s | |f| ddzSt!| || j"| j$S) a input_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) NzYou have to specify input_idsr)rrrrrrr)rarr;rrr~r9onesrrr8get_extended_attention_maskrwrnrorpr rrA)r,rrrrrrr;r<rextended_attention_maskrrrrQs r-rzFlavaTextModel.forward_s02C1N-TXT_T_TqTq$8$D $++JjJj &1%I 'tnI d^I  u00 1I I r4rceZdZUeed<dZdZddeffd Zdee e e fddfdZ e dde jd ee jd ee jd eed eed eedeeeffdZxZS)r`razflava.multimodal_modelrct||||_|jj|_|jr9t j t jdd|j|_ t||_ t j|j|j|_|r t|nd|_|j#y)rmrrN)rhrirararrkr9rlrmrnr4rnrrrorJrprqrrs r-rizFlavaMultimodalModel.__init__s   ![[66   \\%++aFS>ST->k&)D  r4ryr$Nc|jD]7\}}|jj|jj |9yr{r|r~s r-rz!FlavaMultimodalModel._prune_headsrr4rrrr;r<c||n|jj}||n|jj}||n|jj}|j \}}} |j r;|j j|dd} tj| |fd}|dz }|#tj||f|j}|j||jj}|j|||f|j} |j|| ||||} | d} |j!| } |j"|j#| nd}|s | |f| ddzSt%| || j&| j(S) z hidden_states (`torch.FloatTensor` of shape `(batch_size, image_num_patches + text_seq_len, hidden_size)`): The concatenated hidden states of unimodal encoders. Nr{rrrrrr)rarr;rr~rarnrr9rrrrr8rrnrorpr rrA)r,rrrrr;r<rrrrrrrrQs r-rzFlavaMultimodalModel.forwards2C1N-TXT_T_TqTq$8$D $++JjJj &1% ; 'tn ; d^;  u00 1; ; r4r`ceZdZUeed<deffd Ze ddeejdeejdeejdeejdee dee d ee d ejfd Z e dd eejd eejdee deejdeejdee dee d ee d ejfdZe ddeejd eejdeejdeejd eejdeejdeejdee dee de d ee d eeeffdZxZS)rbract||t|jts"t dt |jdt|jts"t dt |jdt|jts%t ddt |jdz|j}|j}|j}|j|_ |j|_ |j|_|j|_t!||_t%||_t)||_t-j.|j|j|_t-j.|j|j|_t-j4t7j8|j:j<|_t-j.|j|j|_ t-j.|j|j|_!|jEy)NzLconfig.text_config is expected to be of type FlavaTextConfig but is of type rzNconfig.image_config is expected to be of type FlavaImageConfig but is of type zMconfig.multimodal_config is expected to be of type FlavaMultimodalConfig but z is of type )#rhrir text_configr TypeErrortype image_configrmultimodal_configrprojection_dimrmtext_hidden_sizeimage_hidden_sizemm_hidden_sizer text_modelri image_modelr`multimodal_modelrrimage_projectiontext_projectionrkr9tensorrardrcimage_to_mm_projectiontext_to_mm_projectionrq)r,rarrrrvs r-rizFlavaModel.__init__s  &,,o>++,-Q0  &--/?@,,-.a1  &224IJ_V%=%= >?qAB  (( ** "44$33 + 7 7!-!9!9/;;(5*<8 45F G " $*@*@$BUBU V!yy)>)>@S@ST<< T[[5W5W(XY&(ii0F0FH[H[&\#%'YYt/D/DdFYFY%Z" r4rrrrrr;r<r$c b|j|||||||}|d} |j| } | S)a input_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, text_seq_length)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) Returns: text_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`FlavaTextModel`]. Examples: ```python >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("{0}") >>> processor = AutoProcessor.from_pretrained("{0}") >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], max_length=77, padding="max_length", return_tensors="pt" ... ) >>> text_features = model.get_text_features(**inputs) ``` )rrrrrr;r<r)rr) r,rrrrrr;r< text_outputsrQ text_featuress r-get_text_featureszFlavaModel.get_text_features;sNR))%/!5#' %Q ,,]; r4rrrrc d|j||||||||} | d} |j| } | S)a bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). Returns: image_features (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`FlavaImageModel`]. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("{0}") >>> processor = AutoProcessor.from_pretrained("{0}") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(images=image, return_tensors="pt") >>> image_features = model.get_image_features(**inputs) ``` )rrrrrr;rr<r)rr) r,rrrrrrr;r< image_outputsrQimage_featuress r-get_image_featureszFlavaModel.get_image_featuresssTL((%+)/!5%=#)  &a( ..}=r4image_attention_maskskip_multimodal_encoderc | | n|jj} | s tdd} d} d}d}|5|j|||| | | }|d|d} } |j | d}d}d}d}d}|6|j ||||| | | }|d|d}}|j |d}d}d}|||s|g|j\}}}|jjr|dz }tj|||j }tj||gd }nd}tj||gd }|j||| }|d}| s| |||||fSt| ||||| S) a input_ids (`torch.LongTensor` of shape `(batch_size, image_num_patches + text_seq_len)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) token_type_ids (`torch.LongTensor` of shape `(batch_size, image_num_patches + text_seq_len)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). image_attention_mask (`torch.Tensor` of shape `(batch_size, image_num_patches)`, *optional*): Mask to avoid performing attention on padding pixel values for image inputs. Mask values selected in `[0, 1]`: - 1 for pixel values that are real (i.e., **not masked**), - 0 for pixel values that are padding (i.e., **masked**). skip_multimodal_encoder (*bool*, *optional*): Skip any calculations for multimodal encoder. Useful if multimodal encoding is not going to be used. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoProcessor, FlavaModel >>> model = FlavaModel.from_pretrained("facebook/flava-full") >>> processor = AutoProcessor.from_pretrained("facebook/flava-full") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = processor(text=["a photo of a cat"], images=image, return_tensors="pt", padding=True) >>> outputs = model(**inputs) >>> image_embeddings = outputs.image_embeddings >>> text_embeddings = outputs.text_embeddings >>> multimodal_embeddings = outputs.multimodal_embeddings >>> outputs.image_embeddings.shape torch.Size([1, 197, 768]) >>> text_embeddings.shape torch.Size([1, 7, 768]) >>> multimodal_embeddings.shape torch.Size([1, 205, 768]) ``` NzRFLAVA model requires hidden states to work. Please set `output_hidden_states=True`)rrrrr;r<rr|r{)rrrrrr;r<rrr)rr<)rrr r!r"r#)rar<rrrrrrrrar9rrrr)r,rrrrrrrrrr;r<r image_statesimage_mm_projectionrr text_statestext_mm_projectionr!r"r#rrrattention_mask_imageattention_multimodalmultimodal_inputs r-rzFlavaModel.forwards F&1%;q>[O!%!;!;KO!L  $   */A/MVm))<)B)B& GQ((66qLG',zz*gNaNhNh'i$',yy2F1W]^'_$'+$$yy*=?Q)RXYZ  $ 5 5 1ES^!6! %6a$8 ! %!   -%+#"7/   r4)NNNNNNNr) NNNNNNNNNTN)r5r6r7rr;rirrr9rrIr:rrr LongTensorrr1r%rrrs@r-rbrb s ){)V-11515/3,0/3&*5ELL)5!.5!. 5 u||, 5 $D> 5'tn5d^5   55n046:3715,0,0/3&*3u||,3"%"2"233#+4. 3 !. 3 ELL) 3$D>3'tn3d^3   33j1548151526377;26,0%)&*K E,,-K u001K !. K !. K "%,,/ K u//0K 'u||4K "*$K $D>K #K d^K  uk! "K K r4rbc`eZdZdedeffd Zdej dej fdZxZS)FlavaImageCodebookResPathin_sizeout_sizec t||dz}t}tj|d<tj ||dd|d<tj|d<tj ||dd|d<tj|d <tj ||dd|d <tj|d <tj ||dd |d <tj ||_y)Nrelu_1r rrpaddingconv_1relu_2conv_2relu_3conv_3relu_4rconv_4)rhrirrReLUr Sequentialpath)r,rrkwargshid_sizerrvs r-riz"FlavaImageCodebookResPath.__init__9s q=}X7H!QOXX8X1aPXX8X1aPXX8X1aPXMM$' r4rr$c$|j|Sr])rr,rs r-rz!FlavaImageCodebookResPath.forwardIsyy|r4 r5r6r7rrir9rrrrs@r-rr8s1((s( %,,r4rcdeZdZdededeffd Zdej dej fdZxZS)FlavaImageCodebookBlockrr num_layersc t|d|dzz |_||k7rtj||dd|_ntj |_t|||_y)Nrr|rr) rhri post_gainrrid_pathIdentityrres_path)r,rrrrrvs r-riz FlavaImageCodebookBlock.__init__NsW j!m, h 99WhAqQDL;;=DL1'8D r4rr$cb|j||j|j|zzSr])rrrrs r-rzFlavaImageCodebookBlock.forwardZs'||A$--2B!BBBr4rrs@r-rrMs? E Es E ECC%,,Cr4rc neZdZd dededededef fd Zdejdejfd ZxZ S) FlavaImageCodebookLayerGroup num_blocksrrruse_poolc$t|t}t|D]4}|dk(rt ||||d|dz<t ||||d|dz<6|rt j d|d<t j||_y)Nrblock_rr|)rpool) rhrirr7rr MaxPool2drgroup) r,rrrrrblocksrErvs r-riz%FlavaImageCodebookLayerGroup.__init___s z" cAAv+B7HV`+aAw'(+B8XWa+bAw'(  c \\a8F6N]]6* r4rr$c$|j|Sr])rrs r-rz$FlavaImageCodebookLayerGroup.forwardmszz!}r4r) r5r6r7rrIrir9rrrrs@r-rr^sH +3 +C +# +QT +`d +%,,r4ra" The FLAVA's image codebook model inspired from DALL-E's original encoder. Outputs raw hidden states and can be used to generate image tokens for an image based on DALL-E's vocab. Used to generate labels for MIM. Use `get_codebook_indices` to get image tokens for an image. ceZdZUdZeed<dZdZdedeffd Z de jde jfdZ de jde jfd Z de jde jfd ZxZS) FlavaImageCodebookrarFrc t||||_|j|_|j|_|j |_|j |_|j|_|j|j z}t}tj|d<tjd|j z|jdd|d<t}tj|jd|j zdd|d <t|j |d|j zd|j z|d <t|j |d|j zd |j z|d <t|j |d |j zd |j z|d<t|j |d |j zd|j zd|d<tj||d<tj||_|j|jj r|j#D] }d|_ yy)Nrelurrrconvr inputgroup_1r|group_2rgroup_3F)rgroup_4r )rhrira num_groupsinput_channelsnum_blocks_per_grouprmrrrrrrrrrqfreeze parameters requires_grad)r,rarr output_blocksrparamrvs r-rizFlavaImageCodebook.__init__s    ++$33$*$?$?!!-- ++__t'@'@@ # " f " !d.>.>*>]^hi j f))D$7$7T=M=M9M[\fghw8  % %z1t7G7G3GTM]M]I] y9  % %z1t7G7G3GTM]M]I] y9  % %z1t7G7G3GTM]M]I] y9  % %z1t7G7G3GTM]M]I]hm y==7xmmF+   ;;  * ,&+# , r4r$ctdtdtd|j|}tj|dS)Na) Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Codebook pixel values can be obtained using [`AutoImageProcessor`] by passing `return_codebook_pixels=True`. See [`FlavaImageProcessor.__call__`] for details. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoImageProcessor, FlavaImageCodebook >>> model = FlavaImageCodebook.from_pretrained("E") >>> image_processor = AutoImageProcessor.from_pretrained("a") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = image_processor([image], return_codebook_pixels=True, return_tensors="pt") >>> inputs = dict(pixel_values=inputs.codebook_pixel_values) >>> outputs = model.get_codebook_indices(**inputs) ``` r)axis)_CHECKPOINT_FOR_CODEBOOK_DOCrr9argmaxr,rz_logitss r-get_codebook_indicesz'FlavaImageCodebook.get_codebook_indicessI 9:V8VWCC_B` a  0;;|,||H1--r4c\|j|}tjd|S)Nrr)rrSoftmaxr s r-get_codebook_probsz%FlavaImageCodebook.get_codebook_probss&;;|, rzza **r4c0dtdtdt|jdk7rtd|jd|jd|jk7r(td|jdd |j|j |S) Na* Args: pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values. Codebook pixel values can be obtained using [`AutoImageProcessor`] by passing `return_codebook_pixels=True`. See [`FlavaImageProcessor.__call__`] for details. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import AutoImageProcessor, FlavaImageCodebook >>> model = FlavaImageCodebook.from_pretrained("ra") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> inputs = image_processor([image], return_codebook_pixels=True, return_tensors="pt") >>> inputs = dict(pixel_values=inputs.codebook_pixel_values) >>> outputs = model(**inputs) >>> print(outputs.shape) (1, 196) ``` rz input shape z is not 4drz input has z channels but model built for )r rrrrr)r,rs r-rzFlavaImageCodebook.forwards 9:V8VWCC_B` a  6 |!! "a '|L,>,>+?zJK K   a D$7$7 7z,*<*S>STr4cl|j|}|j|}|j|}|Sr])rrrr"s r-rz$FlavaPredictionHeadTransform.forwards4 =1 --m< }5 r4r5r6r7rirrrs@r-rrsUr4rc,eZdZdfd ZdZdZxZS)r_c|t|||_t||_t j |j|jd|_ t jtj|j|_ |||j_|j|j_ y)NFr)rhrirar transformrrrmrdecoderrkr9rlrrY)r,rarYrvs r-riz"FlavaMaskedPredictionHead.__init__s  5f=yy!3!3V5F5FUS LLV->->!?@  "(DLL !II r4c:|j|j_yr])rrr3s r- _tie_weightsz&FlavaMaskedPredictionHead._tie_weights s II r4cJ|j|}|j|}|Sr])rrrs r-rz!FlavaMaskedPredictionHead.forwards" NN1  LLOr4r])r5r6r7rirrrrs@r-r_r_s &&r4r_c$eZdZfdZdZxZS) FlavaITMHeadct|||_t||_t j |jd|_y)Nr|) rhrirarJrprrrmseq_relationshiprs r-rizFlavaITMHead.__init__s:  !&) " &*<*#ii(;< <<(8:M:W:WXY[\:]^all,,8L8V8VWXZ[8\]`kk&88(f&es 9J$J rrs@r-r&r&"sN 9r4r&zk The FLAVA model for pretraining which outputs losses, embeddings, logits and transformer outputs. c(^eZdZgdZddedeejffd Zde jfdZ e ddee jdee jd ee jd ee jd ee jd ee jd ee jdee jdee jdeedee jdee jdee jdeededeedeedeee jeff$dZxZS)FlavaForPreTraining)zmmm_text_head.decoder.biaszmmm_image_head.decoder.biaszmlm_head.decoder.biaszmim_head.decoder.biasraimage_codebookcbt||t||_||_|j&|j rt |j|_t|j|_ t|j|_ t||_t|j|_t|j|_t#||_|jj&|_|jj&|_|j,|_|j.|_|j0|_|j2|_|j4|_|j6|_|j8|_|j:|_|j=y)z image_codebook ([`nn.Module`]): If passed, the image codebook will be set to this. Otherwise, it will be initialized using the image_codebook_config defined in the config first as the first parameter. N)rhrirbrTr< init_codebookrimage_codebook_configr_rmim_headrmlm_headr!itm_headmmm_image_head mmm_text_headr&global_contrastive_headrimage_vocab_sizetext_vocab_size mlm_weight mim_weightglobal_contrastive_weightce_ignore_index itm_weightmmm_image_weightmmm_text_weight skip_unmasked_multimodal_encoderrq)r,rar<rvs r-rizFlavaForPreTraining.__init__XsQ  ' ,    &6+?+?"4V5Q5Q"RD 2&2E2EF 1&2D2DE $V, 78K8KL6v7I7IJ'A&'I$ & 3 3 > >%11<< ++ ++)/)I)I&%55 ++ & 7 7%55060W0W- r4rcn|jdkDr!|j|jdd}|S)Nr|rr{)rrr~rs r- _resize_to_2dz!FlavaForPreTraining._resize_to_2d{s, 557Q;qvvay"%Ar4rinput_ids_maskedrcodebook_pixel_valuesrrrrrrO mlm_labels mim_labels itm_labelsrr;r< return_lossr$c||n|jj}||n|jj}| | n |j} ||tj d|}|j |||||| | ||d }|j ||||| |||d }d}|j}|j}|j}|j}|j}dx}x}x}x}x}x}} dx}!x}"x}#}$dx}%x}&}'||C| A|r?|j td| td|jj|} |jdkDr| ||}(| |j| } |j|}|j | |j#d<|(dd| j%d  dddf}(| j#|j })| |)}*|(|)ddf}(|j'|(}!|rjt(j*j-|!j/d |j0|*j/d }||jz}n|j'|(}!|j2dkDr|||}+| |j| } |+dd| j%d  dddf}+| j#|j })| |)},|+|)ddf}+|j5|+}"|rjt(j*j-|"j/d |j6|,j/d }||j2z}n|j5|+}"|j8dkDr||j;|}%| | j#d}-t=j>|-jA|-|-jCdg}|r/t(j*j-|%| } | |j8z} |||}| | |} | | |} ||}||jDdkDr|}(|j%d d z }.|(ddd d |.zddf}(| |j| } |j|}|j | |j#d<| j#|j })| |)}*|(|)ddf}(|jG|(}$|rjt(j*j-|$j/d |j0|*j/d }||jDz}n|jG|(}$||jHdkDr|}+|+dd|j%d  dddf}+| |j| } | j#|j })| |)},|+|)ddf}+|jK|+}#|rjt(j*j-|#j/d |j6|,j/d }||jHz}n|jK|+}#|l|i|jLdkDrY|j jO|dddddf}/t(j*jQ|/d }/|j jS|dddddf}0t(j*jQ|0d }0|j jTjVjYtZt\|j_|0|/|j jT\}&}'}1||&|}&|'|}'|1|}1|rWt(j*j-|&|1}2t(j*j-|'|1}3|2|3zd z }||jLz}ta||| ||| }4|r0|4jcs ted|4jgD}|s.||jh|jhjknd||jl|jljknd|j|jn|jnjknd||jh|jhjknd||jl|jljknd||jn|jnjknd|!|"|%|&|&|$|#f}5|r|4jcs||4f|5z}5tqd|5DStsd%id|d|4d|d|jhd|d|jld|jd|jnd|d|jhd|d|jld|d|jnd|!d|"d |%d!|&d"|'d#|$d$|#S)&a input_ids (`torch.LongTensor` of shape `(batch_size, text_seq_len)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) input_ids_masked (`torch.LongTensor` of shape `(batch_size, text_seq_len)`): Indices of input sequence tokens in the vocabulary. These ones are the masked version of the original task to be used with MLM. Indices can be obtained using [`AutoTokenizer`] along with [`DataCollatorForMaskedLanguageModeling`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) codebook_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_image_patches, patch_size, patch_size, 3)`, *optional*): Pixel values for image patches that are used to compute the image codebook labels for masked image modeling. token_type_ids (`torch.LongTensor` of shape `(batch_size, text_seq_len)`, *optional*): Segment token indices to indicate first and second portions of the inputs. Indices are selected in `[0, 1]`: - 0 corresponds to a *sentence A* token, - 1 corresponds to a *sentence B* token. [What are token type IDs?](../glossary#token-type-ids) bool_masked_pos (`torch.BoolTensor` of shape `(batch_size, image_num_patches)`): Boolean masked positions. Indicates which patches are masked (1) and which aren't (0). image_attention_mask (`torch.FloatTensor` of shape `(batch_size, image_num_patches)`, *optional*): Mask to avoid performing attention on padding token indices specifically for images. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) skip_unmasked_multimodal_encoder (*bool*, *optional*): Skip any calculations for multimodal encoder for unmasked inputs. FLAVA pretraining doesn't need unmasked multimodal embeddings or outputs as of now. mlm_labels (`torch.LongTensor` of shape `(batch_size, text_seq_len)`, *optional*): Labels for computing the left-to-right language and multimodal masked modeling loss (next word prediction). Indices should be in `[-100, 0, ..., text_config.vocab_size - 1]` (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., text_config.vocab_size - 1]`. mim_labels (`torch.LongTensor` of shape `(batch_size, image_num_patches)`, *optional*): Labels for computing the image and multimodal masked modeling loss. Indices should be in `[-100, 0, ..., image_config.vocab_size - 1]`. Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., image_config.vocab_size - 1]`. If not passed, they are generated automatically using the image codebook assigned to the model. By default, it uses [`FlavaImageCodebook`]. See [`FlavaImageCodebook`] to understand how to generate mim_labels. itm_labels (`torch.LongTensor` of shape `(batch_size, 1)`, *optional*): Labels for computing the image-text matching loss. 0 means the pairs don't match and 1 means they match. The pairs with 0 will be skipped for calculation of MMM and global contrastive losses as well. return_loss (`bool`, *optional*, default to None): Whether to return calculated loss or not. Examples: ```python >>> from PIL import Image >>> import requests >>> from transformers import FlavaForPreTraining, AutoProcessor >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = Image.open(requests.get(url, stream=True).raw) >>> model = FlavaForPreTraining.from_pretrained("facebook/flava-full") >>> processor = AutoProcessor.from_pretrained("facebook/flava-full") >>> text = ["a photo of a cat"] >>> inputs = processor( ... images=[image], ... text=text, ... return_masks=True, ... return_codebook_pixels=True, ... padding=True, ... max_length=77, ... return_tensors="pt", ... ) >>> output = model(**inputs) ``` Nz`input_ids_masked` isn't passed which means MLM loss won't be calculated correctlySetting it to `input_ids` so that model can work. Please pass it if this is unintentional. This is usually OKAY if you are doing inference on unmasked text...T) rrrrrrrrr;r<) rrrrrrrr;r<z`return_loss` is set to True but the image codebook is not initialized and no `mim_labels` have been passed. Reinstantiate the model with `init_codebook` set to True or pass in your custom `mim_labels`z`codebook_pixel_value` are required to generate `mim_labels` if loss is expected. Call `AutoProcessor` with `return_codebook_pixels` set to Truerrr{r|r)r?r@rArBrCrDc3(K|] }||nd ywrOr<)r*rLs r-r.z.FlavaForPreTraining.forward..s_T%5T1<_sc3&K|] }|| ywr]r<)r*rs r-r.z.FlavaForPreTraining.forward..s8qai8r?rLrMrrr r!r"r#rNrOrPrQrRrSrTrUrVrWrXrYrZr<):rarrWrOloggerwarningrTrr r"r< RuntimeErrorrrrIrQrKner~r@rr cross_entropyrrFrHrArGrLrBr9whereanynewrMrCrNrDrJr normalizerrcrZclamp_LOGIT_SCALE_CLAMP_MINLOGIT_SCALE_CLAMP_MAXrEr>rGsumrFrr)r!r#r1rK)6r,rrRrrSrrrrrrOrTrUrVrr;r<rW flava_outputflava_masked_outputpos_maskrr rNrPrR total_lossmim_lossmlm_loss mmm_text_lossmmm_image_lossgc_lossitm_lossrTrUrZrYrVr8r9sequence_for_image masked_tokensmim_labels_filteredsequence_for_textmlm_labels_filtered pos_pairs end_indextext_embeddingimage_embedding gc_labels gc_loss_image gc_loss_text flava_lossesr s6 r-rzFlavaForPreTraining.forwards ~&1%eGV^GKK KZK/4D:>> >% # .2N2Z!k&&.&; )0$Y"00EEF[\  ??Q #:#FKgKo!8 %!// ; "&"4"4_"E7;7K7K ?--d34%7JOOA !}}::"D,A,ABDWD\D\]_D` H/H!]]+=>  ??Q #9#EJfJn 6 %!// ; $5a*//!:L9L9NPQ6Q$R! * d.B.B C &0&?#$5mQ6F$G!!]]+<= !}}::"D,@,@ACVC[C[\^C_ H/H!]]+<=  ??Q #?#K'CDJ%&MM!,  ;;y}} 9==RVQWCXY!}}:::zRH/H/;3OPX3Y0)!+H!5J)!+H!5J&5h&?O ( 38M8MPQ8Q!= /44Q7!;I!3Aq1y=7H!4K!L %!// ; "&"4"4_"E7;7K7K ?--d34 * d.B.B C &0&?#%7 q8H%I"#'#6#67I#J %']]%@%@(--b$2G2GHJ]JbJbceJf&N#d&;&;;N#'#6#67I#J  ( 38L8Lq8P <  1!6L6Q6QRS6T5T5VXY2Y Z %!// ; * d.B.B C &0&?#$5mQ6F$G!"&"4"45F"G$&MM$?$?',,R1E1EFH[H`H`acHd%M"T%9%99M"&"4"45F"G  'O,GDLjLjmnLn!ZZ771a8PQN]]44^4LN"jj99:J1aQR7:STO mm55o25NO JJ " " ' ' . ./DF[ \;?;W;W1G1G< 8 oy ##3H#= "1(";%h/  " ; ;
  • Q>]>]>i#//88:os,&88D$55>>@   +F.<#8#8#:  8F88 8(  " . &22  ,  %00  #/"D"D +<< %< !4 @ @ $:  3>> *F &9%J%J " "! ""# $*:% &)8' (.) *,+  r4r])NNNNNNNNNNNNNNTNN)r5r6r7_tied_weights_keysrrrrrir9rrQrrr:rIrr1rKrrrs@r-r;r;Js!{!HRYYk #!k "d^#k $d^%k & uU\\"$== >'k k r4r;)r;rrirbr`rSr)Hr8rrr dataclassesrtypingrrrr9torch.utils.checkpointr activationsr modeling_layersr modeling_outputsr r modeling_utilsrrrutilsrrrrconfiguration_flavarrrrr get_loggerr5r[r rerfrrr>rKrr`rorrrr rr%r)r4rJrSrirr`rbrrrrrr_r!r&r;__all__r<r4r-rs2 #!'' !9KccDD   H %>_.>@UUV  {   <  +  D Wt Wt Wtx_299_H!bii!H6"))6rFFRbii$'RYY'T " "))  +++\' 299' T ")) N?NND] *] ] @m )m m `\ /\ \ ~h %h h V  *CbiiC"299(r)-r)r)j299" , 299 %9%9P ] .]  ] @  r4