rhHddlmZddlmZmZddlmZmZddlm Z ddl m Z ddl m Z ddlmZmZdd lmZmZmZdd lmZmZdd lmZmZmZmZmZd d lmZerddl Z ejBe"Z#GddedZ$Gdde Z%GddeZ&eedGddeZ'edGddeZ(gdZ)y)) dataclass)OptionalUnion)ColPaliForRetrievalColPaliPreTrainedModel)ColPaliProcessor)Cache) BatchFeature) ImageInputis_valid_image)MultiModalDataProcessingKwargsUnpack)PreTokenizedInput TextInput) ModelOutputauto_docstringcan_return_tupleis_torch_availablelogging)ColQwen2ConfigNc&eZdZddidddddidZy ) ColQwen2ProcessorKwargspaddinglongestchannels_firstT) data_formatdo_convert_rgbreturn_tensorspt) text_kwargs images_kwargs common_kwargsN)__name__ __module__ __qualname__ _defaults/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/colqwen2/modular_colqwen2.pyrr&s, y ," +D1 Ir+rF)totalc eZdZdZdZdZ ddeedeefdZ dde d e e e e e e e fd eed efd Zdd Zy)ColQwen2Processora Constructs a ColQwen2 processor which wraps a Qwen2VLProcessor and special methods to process images and queries, as well as to compute the late-interaction retrieval score. [`ColQwen2Processor`] offers all the functionalities of [`Qwen2VLProcessor`]. See the [`~Qwen2VLProcessor.__call__`] for more information. Args: image_processor ([`Qwen2VLImageProcessor`], *optional*): The image processor is a required input. tokenizer ([`Qwen2TokenizerFast`], *optional*): The tokenizer is a required input. chat_template (`str`, *optional*): A Jinja template which will be used to convert lists of messages in a chat into a tokenizable string. visual_prompt_prefix (`str`, *optional*): A string that gets tokenized and prepended to the image tokens. query_prefix (`str`, *optional*): A prefix to be used for the query. AutoImageProcessor)Qwen2TokenizerQwen2TokenizerFastNvisual_prompt_prefix query_prefixc tj|||t|dsdn |j|_t|dsdn |j|_|d}||_|d}||_y)N) chat_template image_tokenz <|image_pad|> video_tokenz <|video_pad|>zf<|im_start|>user <|vision_start|><|image_pad|><|vision_end|>Describe the image.<|im_end|><|endoftext|>zQuery: )r__init__hasattrr7r8r3r4)selfimage_processor tokenizerr6r3r4kwargss r,r9zColQwen2Processor.__init__Isz ##OYm#\29)]2S?YbYnYn29)]2S?YbYnYn  '$M $8!  $L(r+imagestextr>returnc v|jtfd|jji|}|dj dd}|du}| | t d| | t d|7t |r|g}n^t|trt |drn?t|tr$t|dtrt |dds t d|jgt|z} |jdd |i|d } | d } | |jjd z} d} tt| D]}|j| |vrQ| |j|jd | | j!| zzd| |<| dz } |j| |vrQ| |jd |j| |<|j| fddi|d}t#i|| }|d dddf|d ddd fz}tt%j&|d|j)}t$j*j,j.j1|d|d<|r.|dj3|ddk(d}|j5d|i|S|t|t6r|g}n.t|trt|dt6s t d||j8dz}g}|D]%}|j:|z|z}|j=|'|j|fddi|d}|Sy)a Main method to prepare for the model either (1) one or several texts, either (2) one or several image(s). This method is a custom wrapper around the Qwen2VLProcessor's [`~Qwen2VLProcessor.__call__`] method adapted for the ColQwen2 model. It cannot process both text and images at the same time. When preparing the the text(s), this method forwards the `text` and `kwargs` arguments to Qwen2TokenizerFast's [`~Qwen2TokenizerFast.__call__`]. When preparing the the image(s), this method forwards the `images` and `kwargs` arguments to Qwen2VLImageProcessor's [`~Qwen2VLImageProcessor.__call__`]. Please refer to the doctsring of the above two methods for more information. Args: images (`PIL.Image.Image`, `np.ndarray`, `torch.Tensor`, `list[PIL.Image.Image]`, `list[np.ndarray]`, `list[torch.Tensor]`): The image or batch of images to be prepared. Each image can be a PIL image, NumPy array or PyTorch tensor. In case of a NumPy array/PyTorch tensor, each image should be of shape (C, H, W), where C is a number of channels, H and W are image height and width. text (`str`, `list[str]`, `list[list[str]]`): The sequence or batch of sequences to be encoded. Each sequence can be a string or a list of strings (pretokenized string). If the sequences are provided as list of strings (pretokenized), you must set `is_split_into_words=True` (to lift the ambiguity with a batch of sequences). return_tensors (`str` or [`~utils.TensorType`], *optional*): If set, will return tensors of a particular framework. Acceptable values are: - `'tf'`: Return TensorFlow `tf.constant` objects. - `'pt'`: Return PyTorch `torch.Tensor` objects. - `'np'`: Return NumPy `np.ndarray` objects. - `'jax'`: Return JAX `jnp.ndarray` objects. Returns: [`BatchFeature`]: A [`BatchFeature`] with the following fields: - **input_ids** -- List of token ids to be fed to a model. - **attention_mask** -- List of indices specifying which tokens should be attended to by the model (when `return_attention_mask=True` or if *"attention_mask"* is in `self.model_input_names` and if `text` is not `None`). - **pixel_values** -- Pixel values to be fed to a model. Returned when `images` is not `None`. tokenizer_init_kwargsr#suffixNz&Either text or images must be providedz5Only one of text or images can be processed at a timerzAimages must be an image, list of images or list of list of imagesr?r$image_grid_thwz<|placeholder|>rreturn_token_type_idsF)data pixel_valuesT) batch_first input_idstoken_type_idsilabelsz*Text must be a string or a list of strings r*) _merge_kwargsrr= init_kwargspop ValueErrorr isinstancelistr3lenr< merge_sizeranger7replaceprodr torchsplittolistnnutilsrnn pad_sequence masked_fillupdatestrquery_augmentation_tokenr4append)r;r?r@audiovideosr> output_kwargsrDrG texts_doc image_inputsrE merge_lengthindexi text_inputs return_dataoffsetsrIrM texts_queryqueryaugmented_query batch_querys r,__call__zColQwen2Processor.__call__^sZ+** # "&.."<"<  }-11(DA &d 2 >A s9~.]A**il:'0|';'; ,,.?>RWCXC]C]C_coCo.prs( !   **il: $-Q<#7#78I4K[K[#\IaL ])$..&+ .K ',K{,Kl,KLK""23AqD9KHX[_`>`bfg""Hf#56   $$v t,DGS1I !MNN~66;%'K 4"&"3"3e";f"D""?3 4)$..&+ .K  +r+c i}|tjjdi}|j||jddxs|jj }|Dcgc] }|jj g||"}}|Dcgc] }||dzz } }|j| |dtdi|Scc}wcc}w)a Computes the number of placeholder tokens needed for multimodal inputs with the given sizes. Args: image_sizes (`list[list[int]]`, *optional*): The input sizes formatted as (height, width) per each image. Returns: `MultiModalData`: A `MultiModalData` object holding number of tokens per each of the provided input modalities, along with other useful data. Nr$rVrF)num_image_tokensnum_image_patchesr*)rr)getrbr<rVget_number_of_image_patchesr) r; image_sizesr> vision_datar$rV image_sizerx num_patchesrws r,_get_num_multimodal_tokensz,ColQwen2Processor._get_num_multimodal_tokenss  "3==AA/SUVM   (&**<>a$BVBVBaBaJ#.!A$$@@\*\m\! !Sdd; A !=d d   4D[lm n, ,,! es $%B?C)NNNNN)NNNN)N)r&r'r(__doc__image_processor_classtokenizer_classrrcr9r rrrrTrrr rurr*r+r,r/r/3s$1>O.2&* ) 'sm ) sm )."^b CCI0$y/4HYCZZ[C 01 C CJ-r+r/c eZdZy)ColQwen2PreTrainedModelN)r&r'r(r*r+r,rrsr+rz4 Base class for ColQwen2 embeddings output. ) custom_introceZdZUdZdZeejed<dZ eejed<dZ ee e ejefed<dZeeejed<dZeeejed<y)ColQwen2ForRetrievalOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). embeddings (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): The embeddings of the model. past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): Tuple of `tuple(torch.FloatTensor)` of length `config.n_layers`, with each tuple having 2 tensors of shape `(batch_size, num_heads, sequence_length, embed_size_per_head)`) Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. Nloss embeddingspast_key_values hidden_states attentions)r&r'r(rrrrZ FloatTensor__annotations__rTensorrrrTr rtuplerr*r+r,rrs )-D(5$$ %,)-J&-GKOXeD):):$;U$BCDK8&>r&BBJ)!&66+!,,   scsJ ) NNNNNNNNNNNNN)r&r'r(_checkpoint_conversion_mappingrr9rrrrZ LongTensorrr rboolrr __classcell__)rs@r,rrs`&("\~\ 151537+/-159$(,0/3&*/35959Z E,,-Z !.Z u//0 Z "% Z ))* Z   1 12Z D>Z $D>Z 'tnZ d^Z u||,Z !!1!12Z !!1!12Z  $Z Z r+r)rrr/)* dataclassesrtypingrr,transformers.models.colpali.modeling_colpalirr.transformers.models.colpali.processing_colpalir cache_utilsr feature_extraction_utilsr image_utilsr r processing_utilsrrrtokenization_utils_baserrr^rrrrrconfiguration_colqwen2rrZ get_loggerr&loggerrr/rrr__all__r*r+r,rs ""dK 45HHC__2   H % .e H-(H-V 4   ::  :*  d .d  d N r+