rhcudZddlZddlZddlZddlZddlmZddlmZm Z m Z m Z m Z ddl ZddlmZddlmZmZmZmZmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"er erddl#Z#e!jHe%Z&e dZ'e d d Z(Gd d eZ)Gdd eZ*ee*jVe*_+e*jVj8e*jVjjYddde*jV_yy)zH Feature extraction saving/loading class for common feature extractors. N)UserDict) TYPE_CHECKINGAnyOptionalTypeVarUnion)custom_object_save)FEATURE_EXTRACTOR_NAMEPushToHubMixin TensorType cached_file copy_func download_urlis_flax_available is_jax_tensoris_numpy_arrayis_offline_mode is_remote_urlis_tf_availableis_torch_availableis_torch_deviceis_torch_dtypeloggingrequires_backendsSequenceFeatureExtractorSpecificFeatureExtractorTypeFeatureExtractionMixin)boundceZdZdZddeeeefdedee fffd Z dedefdZ defd Z d Z d Zddeeee ffd Zddeeee ffd ZddZxZS) BatchFeatureaU Holds the output of the [`~SequenceFeatureExtractor.pad`] and feature extractor specific `__call__` methods. This class is derived from a python dictionary and can be used as a dictionary. Args: data (`dict`, *optional*): Dictionary of lists/arrays/tensors returned by the __call__/pad methods ('input_values', 'attention_mask', etc.). tensor_type (`Union[None, str, TensorType]`, *optional*): You can give a tensor_type here to convert the lists of integers in PyTorch/TensorFlow/Numpy Tensors at initialization. Ndata tensor_typecHt|||j|y)N)r#)super__init__convert_to_tensors)selfr"r# __class__s x/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/feature_extraction_utils.pyr&zBatchFeature.__init__Ms!  K8itemreturncVt|tr|j|Std)z If the key is a string, returns the value of the dict associated to `key` ('input_values', 'attention_mask', etc.). zRIndexing with integers is not available when using Python based feature extractors) isinstancestrr"KeyErrorr(r,s r* __getitem__zBatchFeature.__getitem__Qs( dC 99T? "op pr+cH |j|S#t$rtwxYwN)r"r1AttributeErrorr2s r* __getattr__zBatchFeature.__getattr__[s* !99T? " !  !s!cd|jiSNr"r"r(s r* __getstate__zBatchFeature.__getstate__as ""r+c"d|vr |d|_yyr9r:)r(states r* __setstate__zBatchFeature.__setstate__ds U?f DI r+cP|yt|ts t|}|tjk(rJtj dt s t dddl}|j|j}|fS|tjk(r.ts t dddl fdj}|fS|tjk(rFtj dts t dddlm}|j"t$}|fSd fd t&}|fS) NNNzTensorFlow and JAX classes are deprecated and will be removed in Transformers v5. We recommend migrating to PyTorch classes or pinning your version of Transformers.zSUnable to convert output to TensorFlow tensors format, TensorFlow is not installed.rzMUnable to convert output to PyTorch tensors format, PyTorch is not installed.ct|ttfrt|dkDrt|dtj rt j |}n_t|dttfrFt|ddkDr5t|ddtj rt j |}t|tj rj|Sj|S)Nr) r/listtuplelennpndarrayarray from_numpytensor)valuetorchs r* as_tensorz5BatchFeature._get_is_as_tensor_fns..as_tensorsedE]3E Q!%(BJJ7 ""58dE];aMA-&uQx{BJJ? "eRZZ0+5++E22'5<<..r+zEUnable to convert output to JAX tensors format, JAX is not installed.ct|ttfrt|dtttjfr`|Dcgc] }t |}}t t |dkDr1|/|Dcgc]}tj|c}t}tj||Scc}wcc}w)Nrr )dtype) r/rCrDrFrGrEsetasarrayobject)rKrOval value_lensrMs r*rMz5BatchFeature._get_is_as_tensor_fns..as_tensorsedE]3 58dTY[][e[eMf8g6;!.maybe_tosb!U\\*/Fu/F/Fq/IqttT,V,,Au||,1Ctt6 tEEr+) rrLgetrErr/r0rintrfrer") r(rprqargrrkrorirjrLs `` @@@r*rnzBatchFeature.tos $ *H%zz.%8 >c$i!mq'Cc"C%)=CQTAU!#McRUhZWo!pqq  15 =1Q ^=  >s=CrAr5)r-r!)__name__ __module__ __qualname____doc__rdictr0rrr r&r3r7r<r?rbr'rn __classcell__)r)s@r*r!r!>s 9Xd38n595QUWZ\fQfKg9qqq!! #&A$%Z:P1QA$FhuS*_7M.N>-r+r!ceZdZdZdZdZdefdZe dde e de ee jfdee ee jfd ed ed ee eefd ed e fdZdde ee jfdefdZede ee jfd eeeefeeefffdZedeeefd efdZd eeeffdZede ee jfd efdZd efdZde ee jffdZdZeddZy) rz This is a feature extraction mixin used to provide saving/loading functionality for sequential and image feature extractors. Nc |jdd|_|jD]\}} t|||y#t$r%}t j d|d|d||d}~wwxYw)z'Set elements of `kwargs` as attributes.processor_classNz Can't set z with value z for )pop_processor_classresetattrr6rVerror)r(rqrgrKerrs r*r&zFeatureExtractionMixin.__init__sx!' + To test a pull request you made on the Hub, you can pass `revision="refs/pr/"`. return_unused_kwargs (`bool`, *optional*, defaults to `False`): If `False`, then this function returns just the final feature extractor object. If `True`, then this functions returns a `Tuple(feature_extractor, unused_kwargs)` where *unused_kwargs* is a dictionary consisting of the key/value pairs whose keys are not feature extractor attributes: i.e., the part of `kwargs` which has not been used to update `feature_extractor` and is otherwise ignored. kwargs (`dict[str, Any]`, *optional*): The values in kwargs of any keys which are feature extractor attributes will be used to override the loaded values. Behavior concerning key/value pairs whose keys are *not* feature extractor attributes is controlled by the `return_unused_kwargs` keyword parameter. Returns: A feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`]. Examples: ```python # We can't instantiate directly the base class *FeatureExtractionMixin* nor *SequenceFeatureExtractor* so let's show the examples on a # derived class: *Wav2Vec2FeatureExtractor* feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained( "facebook/wav2vec2-base-960h" ) # Download feature_extraction_config from huggingface.co and cache. feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained( "./test/saved_model/" ) # E.g. feature_extractor (or model) was saved using *save_pretrained('./test/saved_model/')* feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained("./test/saved_model/preprocessor_config.json") feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained( "facebook/wav2vec2-base-960h", return_attention_mask=False, foo=False ) assert feature_extractor.return_attention_mask is False feature_extractor, unused_kwargs = Wav2Vec2FeatureExtractor.from_pretrained( "facebook/wav2vec2-base-960h", return_attention_mask=False, foo=False, return_unused_kwargs=True ) assert feature_extractor.return_attention_mask is False assert unused_kwargs == {"foo": False} ```rrrruse_auth_tokenNrThe `use_auth_token` argument is deprecated and will be removed in v5 of Transformers. Please use `token` instead.V`token` and `use_auth_token` are both specified. Please set only the argument `token`.r)rwarningswarn FutureWarningrfget_feature_extractor_dict from_dict) rrrrrrrrqrfeature_extractor_dicts r*from_pretrainedz&FeatureExtractionMixin.from_pretrainedsp({#1 %5!"%z$4d;  % MME   l#E  #F7O)G)G)GHe)pio)p&s}}3>v>>r+save_directory push_to_hubc 4|jdd}|;tjdt|j d t d||d<t jj|rtd|dt j|d |rr|jd d}|jd |jt jjd }|j|fi|}|j|}|jt!||| t jj#|t$}|j'|t(j+d||r%|j-||j d|gS)az Save a feature_extractor object to the directory `save_directory`, so that it can be re-loaded using the [`~feature_extraction_utils.FeatureExtractionMixin.from_pretrained`] class method. Args: save_directory (`str` or `os.PathLike`): Directory where the feature extractor JSON file will be saved (will be created if it does not exist). push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Hugging Face model hub after saving it. You can specify the repository you want to push to with `repo_id` (will default to the name of `save_directory` in your namespace). kwargs (`dict[str, Any]`, *optional*): Additional key word arguments passed along to the [`~utils.PushToHubMixin.push_to_hub`] method. rNrrrzProvided path (z#) should be a directory, not a fileT)exist_okcommit_messagerepo_id)configzFeature extractor saved in )rr)rrrrrsrfospathisfileAssertionErrormakedirssplitsep _create_repo_get_files_timestamps _auto_classr joinr to_json_filerVinfo_upload_modified_files) r(rrrqrrrfiles_timestampsoutput_feature_extractor_files r*save_pretrainedz&FeatureExtractionMixin.save_pretraineds $4d;  % MME zz'". l-F7O 77>>. ) ?>2BBe!fg g NT2 #ZZ(8$?NjjN,@,@,Mb,QRG'd'':6:G#99.I     ' t^D A)+ ^E[(\% 78 12O1PQR   ' ' -jj) ( ...r+c |jdd}|jdd}|jdd}|jdd}|jdd}|jdd}|jd d} |jd d} |jd d} | )tjd t| t d | }|jdd} |jdd} d| d}| | |d<t r| st jdd} t|}tjj|}tjj|r$tjj|t}tjj|r|}d}n6t|r|}t!|}nt} t#||||||| ||||  } t)|d5}|j+}dddt-j.}|rt jd|||fSt jdd|||fS#t$$rt&$rt%d|d|dtdwxYw#1swYxYw#t,j0$rt%d|dwxYw) a6 From a `pretrained_model_name_or_path`, resolve to a dictionary of parameters, to be used for instantiating a feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`] using `from_dict`. Parameters: pretrained_model_name_or_path (`str` or `os.PathLike`): The identifier of the pre-trained checkpoint from which we want the dictionary of parameters. Returns: `tuple[Dict, Dict]`: The dictionary(ies) that will be used to instantiate the feature extractor object. rNrFresume_downloadproxies subfolderrrrrrr_from_pipeline _from_autofeature extractor) file_typefrom_auto_classusing_pipelinez+Offline mode: forcing local_files_only=TrueT) rrrrrrr user_agentrz"Can't load feature extractor for 'z'. If you were trying to load it from 'https://huggingface.co/models', make sure you don't have a local directory with the same name. Otherwise, make sure 'z2' is the correct path to a directory containing a z fileutf-8encodingz"It looks like the config file at 'z' is not a valid JSON file.zloading configuration file z from cache at )rrrrrfrrVrr0rrisdirrr rrrrOSError ExceptionopenreadjsonloadsJSONDecodeError)rrrqrrrrrrrrr from_pipelinerris_localfeature_extractor_fileresolved_feature_extractor_filereadertextrs r*rz1FeatureExtractionMixin.get_feature_extractor_dictsJJ{D1 $4e< **%6=**Y-JJ{D1  7D)$4d;!::&8%@::j$/  % MME   l#E #3T:  **\59#6?[  $+8J' (  %5 KKE F# (+,I(J%77==!>? 77==6 7%'WW\\2OQg%h " 77>>7 8.K +H 8 9%B ".:;X.Y +%; " 2=1*'#1#$3%5')% 3/4 5H %F{{} %%)ZZ%5 "  KK56U5VW X &v-- KK-.D-E_UtTuv &v--?  89V8WX99V8WX//E.FeM  % %## 45T4UUpq  s0I!) J6JJ!,J JJ#J?rc |jdd}g}|jD] \}}||vs |||<|j|"|D]}|j|d|di|}tj d||r||fS|S)a  Instantiates a type of [`~feature_extraction_utils.FeatureExtractionMixin`] from a Python dictionary of parameters. Args: feature_extractor_dict (`dict[str, Any]`): Dictionary that will be used to instantiate the feature extractor object. Such a dictionary can be retrieved from a pretrained checkpoint by leveraging the [`~feature_extraction_utils.FeatureExtractionMixin.to_dict`] method. kwargs (`dict[str, Any]`): Additional parameters from which to initialize the feature extractor object. Returns: [`~feature_extraction_utils.FeatureExtractionMixin`]: The feature extractor object instantiated from those parameters. return_unused_kwargsFNzFeature extractor )rreappendrVr)rrrqr to_removergrKfeature_extractors r*rz FeatureExtractionMixin.from_dict*s$ &zz*@%H  ,,. &JC,,.3&s+  % & "C JJsD ! " 9"89 ():(;<= $f, ,$ $r+ctj|j}|jj|d<d|vr|d=d|vr|d=|S)z Serializes this instance to a Python dictionary. Returns: `dict[str, Any]`: Dictionary of all the attributes that make up this configuration instance. feature_extractor_type mel_filterswindow)copydeepcopy__dict__r)rw)r(outputs r*to_dictzFeatureExtractionMixin.to_dictOsP t}}-+/>>+B+B'( F "}% v x  r+ json_filect|d5}|j}dddtj}|di|S#1swY&xYw)a Instantiates a feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`] from the path to a JSON file of parameters. Args: json_file (`str` or `os.PathLike`): Path to the JSON file containing the parameters. Returns: A feature extractor of type [`~feature_extraction_utils.FeatureExtractionMixin`]: The feature_extractor object instantiated from that JSON file. rrNr)rrrr)rrrrrs r*from_json_filez%FeatureExtractionMixin.from_json_file\sN)g . !&;;=D !!%D!1,+,, ! !s AA c|j}|jD]3\}}t|tjs!|j ||<5|j dd}|||d<tj|dddzS)z Serializes this instance to a JSON string. Returns: `str`: String containing all the attributes that make up this feature_extractor instance in JSON format. rNrT)indent sort_keys ) rrer/rFrGtolistrrdumps)r( dictionaryrgrKrs r*to_json_stringz%FeatureExtractionMixin.to_json_stringos\\^ $**, 1JC%,"',,. 3 1 &>>*-C]--$GG* 05bkk1A+B 0D%%r+rrzfeature extractor file)rR object_class object_files)-rzrrrr collectionsrtypingrrrrrr_rFdynamic_module_utilsr utilsr r r rrrrrrrrrrrrrrrL get_loggerrwrVrrr!rrformatrr+r*rs  ??4*   H %"#=> ''EMefy8yxl%^l%^ &//E/Q/Q%R"%%--91G1S1S1[1[1b1b"1GVn2c2&&.:r+