rhddlZddlZddlZddlmZddlmZmZmZddl Z ddl m Z ddl mZmZddlmZddlmZmZmZdd lmZmZdd lmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&dd l'm(Z(dd l)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0e%rdd lm1Z1e"rddl2Z2e#rddlm3Z3e$rddl4m5Z6nddl7m5Z6e&jpe9Z:dZ;ede;e(dGddeZ<eeyy)N)deepcopy)AnyOptionalUnion)custom_object_save) BatchFeature get_size_dict)BaseImageProcessorFast)ChannelDimensionSizeDictvalidate_kwargs)Unpack VideosKwargs) VIDEO_PROCESSOR_NAME TensorTypeadd_start_docstrings cached_file copy_func download_urlis_offline_mode is_remote_urlis_torch_availableis_torchvision_availableis_torchvision_v2_availableis_vision_availablelogging)requires) VideoInput VideoMetadatagroup_videos_by_shape load_videomake_batched_videosreorder_videosto_channel_dimension_format)PILImageResampling)pil_torch_interpolation_mapping) functionala Args: do_resize (`bool`, *optional*, defaults to `self.do_resize`): Whether to resize the video's (height, width) dimensions to the specified `size`. Can be overridden by the `do_resize` parameter in the `preprocess` method. size (`dict`, *optional*, defaults to `self.size`): Size of the output video after resizing. Can be overridden by the `size` parameter in the `preprocess` method. size_divisor (`int`, *optional*, defaults to `self.size_divisor`): The size by which to make sure both the height and width can be divided. default_to_square (`bool`, *optional*, defaults to `self.default_to_square`): Whether to default to a square video when resizing, if size is an int. resample (`PILImageResampling`, *optional*, defaults to `self.resample`): Resampling filter to use if resizing the video. Only has an effect if `do_resize` is set to `True`. Can be overridden by the `resample` parameter in the `preprocess` method. do_center_crop (`bool`, *optional*, defaults to `self.do_center_crop`): Whether to center crop the video to the specified `crop_size`. Can be overridden by `do_center_crop` in the `preprocess` method. do_pad (`bool`, *optional*): Whether to pad the video to the `(max_height, max_width)` of the videos in the batch. crop_size (`dict[str, int]` *optional*, defaults to `self.crop_size`): Size of the output video after applying `center_crop`. Can be overridden by `crop_size` in the `preprocess` method. do_rescale (`bool`, *optional*, defaults to `self.do_rescale`): Whether to rescale the video by the specified scale `rescale_factor`. Can be overridden by the `do_rescale` parameter in the `preprocess` method. rescale_factor (`int` or `float`, *optional*, defaults to `self.rescale_factor`): Scale factor to use if rescaling the video. Only has an effect if `do_rescale` is set to `True`. Can be overridden by the `rescale_factor` parameter in the `preprocess` method. do_normalize (`bool`, *optional*, defaults to `self.do_normalize`): Whether to normalize the video. Can be overridden by the `do_normalize` parameter in the `preprocess` method. Can be overridden by the `do_normalize` parameter in the `preprocess` method. image_mean (`float` or `list[float]`, *optional*, defaults to `self.image_mean`): Mean to use if normalizing the video. This is a float or list of floats the length of the number of channels in the video. Can be overridden by the `image_mean` parameter in the `preprocess` method. Can be overridden by the `image_mean` parameter in the `preprocess` method. image_std (`float` or `list[float]`, *optional*, defaults to `self.image_std`): Standard deviation to use if normalizing the video. This is a float or list of floats the length of the number of channels in the video. Can be overridden by the `image_std` parameter in the `preprocess` method. Can be overridden by the `image_std` parameter in the `preprocess` method. do_convert_rgb (`bool`, *optional*, defaults to `self.image_std`): Whether to convert the video to RGB. video_metadata (`VideoMetadata`, *optional*): Metadata of the video containing information about total duration, fps and total number of frames. do_sample_frames (`int`, *optional*, defaults to `self.do_sample_frames`): Whether to sample frames from the video before processing or to process the whole video. num_frames (`int`, *optional*, defaults to `self.num_frames`): Maximum number of frames to sample when `do_sample_frames=True`. fps (`int` or `float`, *optional*, defaults to `self.fps`): Target frames to sample per second when `do_sample_frames=True`. return_tensors (`str` or `TensorType`, *optional*): Returns stacked tensors if set to `pt, otherwise returns a list of tensors. data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`): The channel dimension format for the output video. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: video in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: video in (height, width, num_channels) format. - Unset: Use the channel dimension format of the input video. input_data_format (`ChannelDimension` or `str`, *optional*): The channel dimension format for the input video. If unset, the channel dimension format is inferred from the input video. Can be one of: - `"channels_first"` or `ChannelDimension.FIRST`: video in (num_channels, height, width) format. - `"channels_last"` or `ChannelDimension.LAST`: video in (height, width, num_channels) format. - `"none"` or `ChannelDimension.NONE`: video in (height, width) format. device (`torch.device`, *optional*): The device to process the videos on. If unset, the device is inferred from the input videos.z!Constructs a base VideoProcessor.)vision torchvision)backendsc+eZdZdZdZdZdZdZdZdZ dZ dZ dZ dZ dZdZdZdZdZdZdZdZeZdgZdeeddffd ZdefdZd d defd Z d>d d d ee e!e"fd ee#dee e#e$ffdZ% d?dede!dee e&e'fde(d fdZ)e*e+dedeedefdZ, d@de(d de e(e!e(e"fde-de-de.dee#dedde-de.de-de-de$d e-d!ee e$e(e$fd"ee e$e(e$fd#ee-dee e#e$fd ee#d$ee e&e/fd%ed def*d&Z0e1 dAd'e e&e2jffd(ee e&e2jffd)e-d*e-d+ee e&e-fd,e&f d-Z4dBd.e e&e2jffd/e-fd0Z5e1d'e e&e2jffde6e"e&e7fe"e&e7fffd1Z8e1d2e"e&e7ffd3Z9de"e&e7ffd4Z:de&fd5Z;d6e e&e2jfffd7Ze1dCd;Z?d  #/ h-?#EFJ ( (::,Fbcsbttwx   !ll1&68H:8UVZZ\Gll1&67;;=Gg))+ rQrOvideo_metadatainput_data_formatcdt|}||Dcgc] }|D]}| }}}ndgt|z}g}|D]k}t|tjr>t |t j|}tj|j}|j|m||fScc}}w)z: Prepare the input videos for processing. N) r#len isinstancenpndarrayr%r FIRSTrc from_numpyreappend) rGrOrirj batch_listr]batch_metadataprocessed_videosrRs rL_prepare_input_videosz(BaseVideoProcessor._prepare_input_videos$s%V,  %9Gc:XbcHhchcNc"Vc&k1N +E%,3E;K;Q;QSde((/::<  # #E * + //dsB,c t|jt|jjjdgz|jjD]}|j |t ||d!|jd}|jd}|j|||\}}|jd i|}|jd i||jd}t|ttfr t|n||d<|jd|jd |jd ||d |S) Nreturn_tensors)captured_kwargsvalid_processor_keysrjri)rOrirjresample interpolationr4 data_format)rOri)rrCr@rArB setdefaultrFr9rv_further_process_kwargs_validate_preprocess_kwargsrmr&rbr' _preprocess)rGrOr/ kwarg_namerjrir{s rLrNzBaseVideoProcessor.preprocess>sO "KKM!%d&7&7&G&G&L&L&N!OScRd!d ++;; KJ   j'$ D*I J K#JJ':;$45!%!;!;.L]"<" .--77(((262::j)9CHOacfNg9h +H 5nv   &' =!tWvnWPVWWrQdo_convert_rgb do_resizer3 size_divisorr|zF.InterpolationModedo_center_cropr5 do_rescaledo_padrescale_factor do_normalize image_mean image_stddo_sample_framesrxdevicec |r1t||Dcgc]\}}|j||||}}}||Dcgc]}|j|}}t|\}}i}|j D]4\}}|r|j |}|r|j ||||}|||<6t||}t|\}}i}|j D]4\}}|r|j|| }|j|| | | ||}|||<6t||}|rtj|dn|}td|i|Scc}}wcc}w)N)r]r^r_)r3rr|r)dimr.)data tensor_type) ziprhtor!r;r\resizer$ center_croprescale_and_normalizercstackr )rGrOrirrr3rr|rr5rrrrrrrr_r^rxrrRr]grouped_videosgrouped_videos_indexresized_videos_groupedrYstacked_videosresized_videosprocessed_videos_groupedrus rLrzBaseVideoProcessor._preprocessbs. (+6>'B#E8""58 X["\F  4:;5ehhv&;F;0EV/L,,!#%3%9%9%; ; !E>!%!4!4^!D!%"LXe"-"-; "5 ) ;((>@TU0E^/T,,#% %3%9%9%; = !E>!%!1!1.)!L!77 NL*V_N/= $U + =**BDXYCQ5;;'7Q?Wg"79I!JXfggO To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. return_unused_kwargs (`bool`, *optional*, defaults to `False`): If `False`, then this function returns just the final video processor object. If `True`, then this functions returns a `Tuple(video_processor, unused_kwargs)` where *unused_kwargs* is a dictionary consisting of the key/value pairs whose keys are not video processor attributes: i.e., the part of `kwargs` which has not been used to update `video_processor` and is otherwise ignored. subfolder (`str`, *optional*, defaults to `""`): In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. kwargs (`dict[str, Any]`, *optional*): The values in kwargs of any keys which are video processor attributes will be used to override the loaded values. Behavior concerning key/value pairs whose keys are *not* video processor attributes is controlled by the `return_unused_kwargs` keyword parameter. Returns: A video processor of type [`~video_processing_utils.ImagVideoProcessorBase`]. Examples: ```python # We can't instantiate directly the base class *VideoProcessorBase* so let's show the examples on a # derived class: *LlavaOnevisionVideoProcessor* video_processor = LlavaOnevisionVideoProcessor.from_pretrained( "llava-hf/llava-onevision-qwen2-0.5b-ov-hf" ) # Download video_processing_config from huggingface.co and cache. video_processor = LlavaOnevisionVideoProcessor.from_pretrained( "./test/saved_model/" ) # E.g. video processor (or model) was saved using *save_pretrained('./test/saved_model/')* video_processor = LlavaOnevisionVideoProcessor.from_pretrained("./test/saved_model/preprocessor_config.json") video_processor = LlavaOnevisionVideoProcessor.from_pretrained( "llava-hf/llava-onevision-qwen2-0.5b-ov-hf", do_normalize=False, foo=False ) assert video_processor.do_normalize is False video_processor, unused_kwargs = LlavaOnevisionVideoProcessor.from_pretrained( "llava-hf/llava-onevision-qwen2-0.5b-ov-hf", do_normalize=False, foo=False, return_unused_kwargs=True ) assert video_processor.do_normalize is False assert unused_kwargs == {"foo": False} ```rrrruse_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`.r)r9warningswarn FutureWarningraget_video_processor_dict from_dict) clsrrrrrrr/rvideo_processor_dicts rLfrom_pretrainedz"BaseVideoProcessor.from_pretrainedst({#1 %5!"%z$4d;  % MME   l#E  #F7O'Cs'C'CDa'lek'l$fs}}1info_upload_modified_files) rGrrr/rrrfiles_timestampsoutput_video_processor_files rLsave_pretrainedz"BaseVideoProcessor.save_pretraineds $4d;  % MME zz'". l-F7O 77>>. ) ?>2BBe!fg g NT2 #ZZ(8$?NjjN,@,@,Mb,QRG'd'':6:G#99.I     ' t^D A')ggll>CW&X# 56 /0K/LMN   ' ' -jj) ( ,,,rQc n|jdd}|jdd}|jdd}|jdd}|jdd}|jdd}|jd d} |jd d} |jd d } |jd d} |jdd} |)tjdt| t d|}d| d}| | |d<t r| st jdd} t|}tjj|}tjj|r|}d}n6t|r|}t|}n t}t!||||||| ||| |  } t)|dd5}|j+}dddt-j.}|rt jd"|||fSt jd"d#|||fS#t"$r0d}t!||||||| ||| |  }t j%dYt"$rt&$rt#d|d|dtdwxYw#1swYxYw#t,j0$rt#d |d!wxYw)$a From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a video processor of type [`~video_processing_utils.VideoProcessorBase`] using `from_dict`. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`): The identifier of the pre-trained checkpoint from which we want the dictionary of parameters. subfolder (`str`, *optional*, defaults to `""`): In case the relevant files are located inside a subfolder of the model repo on huggingface.co, you can specify the folder name here. Returns: `tuple[Dict, Dict]`: The dictionary(ies) that will be used to instantiate the video processor object. rNrFresume_downloadproxiesrrrr subfolder_from_pipeline _from_autorrvideo processor) file_typefrom_auto_classusing_pipelinez+Offline mode: forcing local_files_only=TrueT) rrrrrr user_agentrrzpreprocessor_config.jsonaAYou have video processor config saved in `preprocessor.json` file which is deprecated. Video processor configs should be saved in their own `video_preprocessor.json` file. You can rename the file or load and save the processor back which renames it automatically. Loading from `preprocessor.json` will be removed in v5.0.z Can't load video processor for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z2' is the correct path to a directory containing a z filerutf-8encodingz"It looks like the config file at 'z' is not a valid JSON file.zloading configuration file z from cache at )r9rrrrarr>rstrrrisdirrrrrrOSError warning_once ExceptionopenreadjsonloadsJSONDecodeError)rrr/rrrrrrrrr from_pipelinerris_localresolved_video_processor_filevideo_processor_filereadertextrs rLrz+BaseVideoProcessor.get_video_processor_dictSs.$JJ{D1 $4e< **%6=**Y- 7D)$4d;!::&8%@::j$/JJ{B/  #3T:  **\59  % MME   l#E#4Y  $+8J' (  %5 KKE F# (+,I(J%77==!>? 77>>7 8,I )H 8 9#@ ,89V,W )2 (<$0;1('#1#$3%5)%' 1-^ 3S7K %v{{} %#'::d#3   KK56S5TU V $V++ KK-.B-C?SpRqr $V++g 'A$0;1('#1#$3%5)%' 1-##P   67T6UV99V8WX//C.DEK  % %## 45R4SSno  s6 H&J4JJ6J+JJ J#J4rc |j}|jdd}d|vrd|vr|jd|d<d|vrd|vr|jd|d<|di|}g}|jD]0\}}t||st ||||j |2|D]}|j|dt jd||r||fS|S)a Instantiates a type of [`~video_processing_utils.VideoProcessorBase`] from a Python dictionary of parameters. Args: video_processor_dict (`dict[str, Any]`): Dictionary that will be used to instantiate the video processor object. Such a dictionary can be retrieved from a pretrained checkpoint by leveraging the [`~video_processing_utils.VideoProcessorBase.to_dict`] method. kwargs (`dict[str, Any]`): Additional parameters from which to initialize the video processor object. Returns: [`~video_processing_utils.VideoProcessorBase`]: The video processor object instantiated from those parameters. return_unused_kwargsFr3r5NzVideo processor r~)copyr9r;hasattrr<rrr>r)rrr/rvideo_processor to_removerHrIs rLrzBaseVideoProcessor.from_dicts " 488:%zz*@%H V *> >+1::f+=  ( & [4H%H06 ;0G  -5 45  ,,. &JC,e4  % & "C JJsD ! "  &&789 "F* *" "rQct|j}|jdd|jdd|jj|d<|S)z Serializes this instance to a Python dictionary. Returns: `dict[str, Any]`: Dictionary of all the attributes that make up this video processor instance. rDN_valid_kwargs_namesvideo_processor_type)r__dict__r9rK__name__)rGoutputs rLto_dictzBaseVideoProcessor.to_dictsJ$--( 0$7 ($/)-)@)@%& rQc|j}|jD]3\}}t|tjs!|j ||<5|j dd}|||d<tj|dddzS)z Serializes this instance to a JSON string. Returns: `str`: String containing all the attributes that make up this feature_extractor instance in JSON format. r:Nr2T)indent sort_keys ) rr;rmrnrotolistr9rdumps)rG dictionaryrHrIr:s rLto_json_stringz!BaseVideoProcessor.to_json_strings\\^ $**, 1JC%,"',,. 3 1 &>>* This API is experimental and may have some slight breaking changes in the next releases. Args: auto_class (`str` or `type`, *optional*, defaults to `"AutoVideoProcessor "`): The auto class to register this new video processor with. rNz is not a valid auto class.) rmrrtransformers.models.automodelsautorrar)r auto_class auto_modules rLregister_for_auto_classz*BaseVideoProcessor.register_for_auto_classFsC *c*#,,J66{J/ |+FGH H$rQvideo_url_or_urlsct|tr|Dcgc]}|j|c}St|tr t |St dt |cc}w)z Convert a single or a list of urls into the corresponding `np.array` objects. If a single url is passed, the return value will be a single object. If a list is passed a list of objects is returned. z=only a single or a list of entries is supported but got type=)rmr@ fetch_videosrr" TypeErrortype)rGrxs rLrzBaseVideoProcessor.fetch_videos`sc ' .2CDQD%%a(D D )3 //0 0[\`ar\s[tuv v EsA")NNN)NN)NNNNN)NFFNmain)F)AutoVideoProcessor)Br __module__ __qualname__rr{rrr3rr4r5rrrrrrrrr_r^rirrAmodel_input_namesrr8r rPrr\rrr dictrbfloatrhrr r@rvrBASE_VIDEO_PROCESSOR_DOCSTRINGrNboolr rr classmethodrPathLikerrtuplerrrrrrrr rr __classcell__)rKs@rLr-r-s KHJI DLIIN FJNLN CJNL./G !5G$G>1L1 8:>$(+/ 775!4567SM 7 eCJ' ( 7x)-DH 00&0$E#/?*?$@A 0 n  0489!X!X&!X  !X:!Xh,0+/$(;?+/+@h^$@hd=14:=>@h @h  @h  @hsm@h 56@h@h@h@h@h@h@hU5$u+#567@h E%e"456!@h"#4.#@h$eCJ' (%@h&SM'@h(!sJ!78)@h*(+@h, -@hD8<$!&,0o=',S"++-='>o=E#r{{"234o= o=  o= c4i() o=o=o=b;-eC4D.E;-TX;-z,,1#r{{2B,C, tCH~tCH~- .,,B*#T#s(^*#*#X c3h GG* 05bkk1A+B 0D+uS"++-='>++$%%2 weCcN.C wrQr-rrzvideo processor file)object object_class object_files)@rrrrrtypingrrrnumpyrndynamic_module_utilsrimage_processing_utilsr r image_processing_utils_fastr image_utilsr r rprocessing_utilsrrutilsrrrrrrrrrrrrrutils.import_utilsr video_utilsrr r!r"r#r$r%r&rcr'torchvision.transforms.v2r(rWtorchvision.transforms get_loggerrr>r r-r__doc__formatr~rQrLr8sH  ''4@ 3)/<"$=:   H %@"lF'" ,-T w/T w.  T wn"++=+I+I!J!!))5-?-K-K-S-S-Z-Z /CRh.[.""*6rQ