rh.CddlmZmZddlZddlmZmZddlm Z m Z m Z ddl m Z mZddlmZmZmZmZmZddlmZdd lmZdd lmZmZmZerdd lmZererdd lmZ ndd l!mZ erddl"Z"dd lm#Z#e#jHe%Z&dZ'dZ(dZ)dZ*dZ+de,dde,e-fdZ.de-de/e-e-ffdZ0GddeZ1edGddeZ2dgZ3y))OptionalUnionN) BatchFeature get_size_dict)IMAGENET_STANDARD_MEANIMAGENET_STANDARD_STDSizeDict)Unpack VideosKwargs) TensorTypeis_torch_availableis_torchvision_availableis_torchvision_v2_availableis_vision_available)requires)BaseVideoProcessor) VideoMetadatagroup_videos_by_shapereorder_videos)PILImageResampling) functional)loggingzYou are a helpful language and vision assistant. You are able to understand the visual content that the user provides, and assist the user with a variety of tasks using natural language.zgYou are provided the following series of {frame_count} frames from a {video_duration} [H:MM:SS] video. z z Frame from {timestamp}:ivideos torch.Tensorreturnctdx}}|D]0}|jdd\}}t||}t||}2||fS)zH Get the maximum height and width across all videos in a batch. z-infN)floatsizemax)r max_height max_widthvideoheightwidths /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/smolvlm/video_processing_smolvlm.pyget_max_height_widthr(Gs\#6]*J* RS) , y) *  ""resolution_max_sidec:|jdd\}}tt|}| t||n|}||z }||k\r|}t ||z }|dzdk7r(|dz }n"||kDr|}t ||z}|dzdk7r|dz }t|d}t|d}||fS)a Get the output size of the video after resizing given a dictionary specifying the max and min sizes. Args: video (`np.ndarray`): Video to resize. resolution_max_side (`int`): The longest edge of the video will be resized to this value. The shortest edge will be resized to keep the input aspect ratio. Returns: The output size of the video after resizing. rNr)r minMAX_IMAGE_SIZEr!int)r$r*r%r& aspect_ratios r'get_resize_output_image_sizer2SsJJL%MFEn.AB0C0K#fe,Qd6>L #U\)* A:? aKF %$F\)* 19> QJE ^F qME 5=r)c&eZdZUdZeeefed<y)SmolVLMVideoProcessorInitKwargsNmax_image_size)__name__ __module__ __qualname__r5dictstrr0__annotations__r)r'r4r4{s%)NDcN)r)r4) torchvision)backendsc&eZdZejZddiZddiZeZ e Z dZ dZ dZdZdZdZeZddgZdeeffd Z d*d d d ed ddedd f dZ d+d d deeefdededef dZ d,d d deeefde ede eee!fde ef dZ" d-de#d dee#ee#efdeded ed e dd ed!e!d"ed#ed$ed%e ee!e#e!fd&e ee!e#e!fde eee!fde ede ed'e ee$e%fd(e d f$d)Z&xZ'S).SmolVLMVideoProcessor longest_edgeiilTF pixel_valuespixel_attention_maskkwargsc t|di|d|vrd|vr |d|dd<d|vr=|dd|_|dd|_t |dd|j |_yy)Nr video_sampling video_size max_framesfps)default_to_squarer<)super__init__ num_framesrIrrJr )selfrD __class__s r'rLzSmolVLMVideoProcessor.__init__s "6" V  0F :5;F^F# $\ 2 v %$%56|DDO./6DH%f-=&>|&L`d`v`vwDI &r)r$rr interpolationzF.InterpolationMode antialiasrc B||ntjj}|tjjk(r/tj dtjj }|jrt||j}n@|jr%|jr|j|jf}ntd|dtj||||}|jd|jdf}tj||||}|S)a9 Resize an video to `(size["height"], size["width"])`. Args: video (`torch.Tensor`): Video to resize. size (`SizeDict`): Dictionary in the format `{"height": int, "width": int}` specifying the size of the output video. resample (`InterpolationMode`, *optional*, defaults to `InterpolationMode.BILINEAR`): `InterpolationMode` filter to use when resizing the video e.g. `InterpolationMode.BICUBIC`. Returns: `torch.Tensor`: The resized video. zYou have used fast image processor with LANCZOS resample which not yet supported for torch.Tensor. BICUBIC resample will be used as an alternative. Please fall back to image processor if you want full consistency with the original model.)r*zHSize must contain 'height' and 'width' keys, or 'longest_edge' key. Got .)rPrQrA)FInterpolationModeBILINEARLANCZOSlogger warning_onceBICUBICrAr2r%r& ValueErrorresizer5)rNr$r rPrQrDnew_sizemax_sizes r'r\zSmolVLMVideoProcessor.resizes(*7)B H[H[HdHd A//77 7   A  //77M   4$($5$5H[[TZZ TZZ0Hghlgmmnop p QZ[&&~68K8KN8[[ QZ[ r) padded_sizemax_num_framesfillreturn_pixel_maskc|jdd}|d|dz }|d|dz }||jdz } |dks|dkrtd|d|d||k7r"d|d|ddd| g} tj|| |}d} |rBt j |d dddddft j } d| d d|dd|df<|| fS) a"Pads the sample with empty video to the padded_size Args: video (`torch.Tensor`): Video to pad. padded_size (`tuple[int, int]`): Height and width to pad. max_num_frames (`int`): The maximum number of frames to which video will be padded. fill (`int`, *optional*): The value to use for the padding. return_pixel_mask (`bool`, *optional*, defaults to `True`): Whether to return a pixel mask. rNrr-zzPadding dimensions are negative. Please make sure that the padded size is larger than the original size. Got padded size: z, original size: rS)ra.dtype)r shaper[rTpadtorch zeros_likeint64) rNr$r_r`rarb original_sizepadding_height padding_width padding_framepadding pixel_masks r'rgzSmolVLMVideoProcessor.pads* RS) $Q-*::#Aq)99 &Q7 1  233>-?PQ^P__`b  K '-NAq!]SGEE%t4E ))%Q1 *=U[[QJFGJs.mA..0B-2B0BB Cj  r)metadatarMrI skip_secsc||n |j}||n |j}|jd}tt ||dz}t ||}|dkrd}d} |dz } |dkDr6|dd|zz ||zkDr%t||dz} t|||dzz } t d| } t | |dz } | | k\rd|dz } } tj| | |t} tj| } || j}g} | D]9} | |dz }t|dz}t|dz}| j||g;|| t|dfS)al Video sampling function which: - Uses `num_frames` (if provided) or calculates it from `fps` and metadata. - Applies a basic center-skip if fewer frames than available, otherwise optionally skips `skip_secs` from both the start and end. - Uniformly samples the desired number of frames between the start and end indices. Args: video (`torch.Tensor`): Video that need to be sampled. metadata (`VideoMetadata`): Metadata of the video containing information about total duration, fps and total number of frames. num_frames (`int`, *optional*): Maximum number of frames to sample. Defaults to `self.num_frames`. fps (`int` or `float`, *optional*): Target frames to sample per second. Defaults to `self.fps`. skip_secs (`float`, *optional*, defaults to `1`): Number of seconds to skip from the start and end if the video is long enough. Returns: torch.Tensor: Sampled video frames. rdurationr-r,rIrd<) rMrIrfr0roundr.r!nplinspaceunique contiguousappend)rNr$rqrMrIrrtotal_num_framesestimated_framesdesired_frames start_idxend_idxindices timestampsidxsecmmsss r' sample_framesz#SmolVLMVideoProcessor.sample_framess>$.#9Zt _c$(( ;;q>uS8J+?%?@A-z: A N "Q& q=hz2Q]BzTWGWXI78I*Y%-HHIG9% g/!34  !"$4q$8wI++i.L))G$g))+  (C'CSBYBS2XB   r2h '  ( j#hz&:";;;r)rvideo_metadatado_convert_rgb do_resize do_rescalerescale_factor do_normalizedo_paddo_sample_frames image_mean image_stdreturn_tensorsdevicec @| rw|d tdg}gg}}t||D]Q\}}|j|||||\}}}|j||j||j|Snn|}|Dcgc]D}t t |Dcgc]$}t |dz dzt |dz dzf&c}F}}}|Dcgc]}t |dz}}||Dcgc]}|j|}}t|\}}i}|jD]3\}} |r|j| } |r|j| ||} | ||<5t||}!t|!\}}i}"|jD] \}} |j| ||| | | } | |"|<"t|"|}| rt|}#td|D}$t|\}}i}%i}"|jD]&\}} |j!| |#|$\} }&| |"|<|&|%|<(t|"|}t|%|}'|rt#j$|dn|}|||d }(| r | r|t#j$'dn'|(d <t'|(| Scc}wcc}}wcc}wcc}w) NrzFrame sampling is enabled but no video metadata was found. SmolVLM requires metadata to correctly sample frames. Please pass in `VideoMetadata` object per each input video or set `do_sample_frames=False`ru)r rPc32K|]}t|yw)N)len).0r$s r' z4SmolVLMVideoProcessor._preprocess..s JU Js)r_r`)dim)rBr durationsrC) tensor_type)r[ziprr{rangerr0toritemsconvert_to_rgbr\rrescale_and_normalizer(r!rgrhstackr))rNrrrrr rPrrrrrrrrIrMrrrrrDprocessed_videostimestamps_listdurations_listr$rqrrtrgrouped_videosgrouped_videos_indexresized_videos_groupedrfstacked_videosresized_videosprocessed_videos_groupedpad_sizer`processed_padded_mask_grouped padded_masksrCdatas) r' _preprocessz!SmolVLMVideoProcessor._preprocess<sJ. a ( q " .0"^O#&v~#> /x.2.@.@R\^acl.m+z8&&z2%%h/ ''.  / & jpaf%PSTYPZJ[\3#sRxB&'cBh"_)=>\O=CC5c%jB.CNC  4:;5ehhv&;F;/DEU/V,,!#%3%9%9%; ; !E>!%!4!4^!D!%^$Vc!d,: "5 )  ; ((>@TU/D^/T,,#% %3%9%9%; = !E>!77 NL*V_N/= $U +  = **BDXY +,<=H J9I JJN3HIY3Z 0N0,. )') $)7)=)=)? D%~/3xx"080, 3A(/7C-e4  D ..FH\] #12OQe#f CQ5;;'7Q?Wg 0^lm n8 0a8) ' ( Dn==o]D ^$_>d=14:=>_> _>  _>  _> 56_>_>_>_>_>_>U5$u+#567_>E%e"456_>eCJ' (_> SM!_>"C=#_>$!sJ!78%_>&('_>r)r@)4typingrrnumpyrwimage_processing_utilsrr image_utilsrr r processing_utilsr r utilsr rrrrutils.import_utilsrvideo_processing_utilsr video_utilsrrrrtorchvision.transforms.v2rrTtorchvision.transformsrhr get_loggerr6rXDEFAULT_SYSTEM_MESSAGEDEFAULT_VIDEO_INTRODEFAULT_MEDIA_OUTTROFRAME_TIMESTAMP_MESSAGEr/rr0r(rr2r4r@__all__r<r)r'rs # 5+PO1"$=:   H %Vn5 #n!5 #$s) #%% 38_%P*l* #$[>.[>%[>| # #r)