rha ddlZddlZddlZddlZddlZddlmZmZddlm Z ddl m Z ddl m Z mZmZmZddlmZmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddl m!Z!ddl"m"Z"erddl#Z$ddl#m%Z%ddl#m&Z'Gdde%Z(e jRjTZ+e e,jZjZa.t\jZjZa/de+djaZ1Gdde2Z3dee4ejjfde4de4fdZ6de3de4de4d e7e8e4e9ffd!Z:d"e4d#e7e4fd$Z;d%e4de4de4fd&Zfd*Z?d(e4de3d+e4d)e8e4e4e7ffd,Z@de3de4d e7e8e4e9fd)e4fd-ZAde3d e7e8e4e9ffd.ZBde3de4de4d e7e8e4e9fd/e9f d0ZC d=d1e4d2ee4d3eed4ee4d)e f d5ZDd6e4d)e9fd7ZEd8ZFd9efd:ZGGd;dN)ArgumentParser Namespace)date)Path)AnyCallableOptionalUnion)CONFIG_MAPPING_NAMESMODEL_NAMES_MAPPING)FEATURE_EXTRACTOR_MAPPING_NAMES)IMAGE_PROCESSOR_MAPPING_NAMES)PROCESSOR_MAPPING_NAMES)TOKENIZER_MAPPING_NAMES)VIDEO_PROCESSOR_MAPPING_NAMES)is_libcst_available)BaseTransformersCLICommand)add_fast_image_processor) CSTVisitor)matchersczeZdZdZdZdej ddfdZdej fdZdejfdZ y) ClassFinderzC A visitor to find all classes in a python module. c.g|_g|_d|_yNF)classespublic_classes is_in_classselfs {/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/commands/add_new_model_like.py__init__zClassFinder.__init__/s!#DL(*D $D nodereturnNcp|jj|jjd|_y)zwRecord class names. We assume classes always only appear at top-level (i.e. no class definition in function or similar)TN)rappendnamevaluerr!r%s r"visit_ClassDefzClassFinder.visit_ClassDef4s$ LL    0#D r$cd|_yr)rr+s r"leave_ClassDefzClassFinder.leave_ClassDef9s $D r$ctjtjtjtjgg}|j stj ||r|jdjdjj}|dk(rM|jdjj}|Dcgc]}|jjc}|_ yyyycc}w)z:Record all public classes inside the `__all__` assignment.)target)targets)bodyr__all__N) mSimpleStatementLineAssign AssignTargetNamermatchesr2r1r0r*elementsr)r!r%!simple_top_level_assign_structureassigned_variabler:elements r"visit_SimpleStatementLinez%ClassFinder.visit_SimpleStatementLine<s010E0Ehhaffh(G'HIJ1 -## $8Y(Z$(IIaL$8$8$;$B$B$H$H!$ 1#yy|11::HNV*W77==+>+>*WD'2)[#+XsC:) __name__ __module__ __qualname____doc__r#cstClassDefr,r.r5r>r$r"rr*sG  %  $s|| $ $  %s|| % X#2I2I Xr$rz # coding=utf-8 # Copyright aN the HuggingFace Team. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. # You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. ceZdZdZdefdZy) ModelInfoszJ Retrieve the basic informations about an existing model classes. lowercase_namec|jjddjdd|_|jtvr|jjdd|jtvrt |dt |j|_t|j|_|jjdd|_|jtvrCt|j\|_ |_ |jdk(rdn |j|_ nd\|_ |_ tj|jd\|_|_t!j|jd|_t%j|jd|_t)j|jd|_y) N _-z is not a valid model nameConfigPreTrainedTokenizerFast)NN)lowerreplacerHr ValueErrorr paper_name config_classcamelcase_namertokenizer_classfast_tokenizer_classrgetimage_processor_classfast_image_processor_classrvideo_processor_classrfeature_extractor_classrprocessor_class)r!rHs r"r#zModelInfos.__init__cs,224<UVZViVi>j ;D $";115NNTXTmTm  %?I ;D $";FcFgFg   G C"D$C&C%F%FtGZGZ\`%a"'F'J'J4K^K^`d'e$6::4;N;NPTUr$N)r?r@rArBstrr#rEr$r"rGrG^sVsVr$rG file_name new_content add_afterct|dd5}|j}dddj|d\}}||z|z|z}t|dd5}|j|dddy#1swYRxYw#1swYyxYw)ac A utility to add some content inside a given file. Args: file_name (`str` or `os.PathLike`): The name of the file in which we want to insert some content. new_content (`str`): The content to add. add_after (`str`): The new content is added just after the first instance matching it. rutf-8encodingNrw)openreadsplitwrite)r_r`raf old_contentbeforeafters r"add_content_to_filerps iw /1ffh  %%i3MFE9${2U:K iw /1    sA1A=1A:=Bold_model_infosnew_lowercase_namenew_model_paper_namefilenames_to_addc djd|jddjdD}|j}|j}|ddDcgc]\}}|j|d|f}}}g} |D]B\} }t j d| r| d \} } | | xs|f| d <0| j| |fDttd z dz d z d |d |ddttd z dz d z d |d |dd| D]\}}|s |jdd}ttd z dz |z 5} | j} dddt jd|d t j}|D]=}ttd z dz |z |j||j|||?ycc}}w#1swY}xYw)a Add a model to all the relevant mappings in the auto module. Args: old_model_infos (`ModelInfos`): The structure containing the class informations of the old model. new_lowercase_name (`str`): The new lowercase model name. new_model_paper_name (`str`): The fully cased name (as in the official paper name) of the new model. filenames_to_add (`list[tuple[str, bool]]`): A list of tuples of all potential filenames to add for a new model, along a boolean flag describing if we should add this file or not. For example, [(`modeling_xxx.px`, True), (`configuration_xxx.py`, True), (`tokenization_xxx.py`, False),...] rNc3<K|]}|jywNtitle.0xs r" z-add_model_to_auto_mappings..`1QWWY`rLrKrNautoz2(?:tokenization)|(?:image_processing)_auto_fast.pymodelszconfiguration_auto.pyz ("z", "z Config"), zOCONFIG_MAPPING_NAMES = OrderedDict[str, str]( [ # Add configs here r`raz"), zcMODEL_NAMES_MAPPING = OrderedDict[str, str]( [ # Add full (and cased) model names here _fast.py.pyz( {8,12}\(\s*"z",.*?\),\n)(?: {4,12}\(|\]))joinrQrjrHrUresearchr(rpTRANSFORMERS_PATHrhrifindallDOTALL)rqrrrsrtnew_cased_nameold_lowercase_nameold_cased_namefilenameto_addcorrected_filenames_to_addfile previous_fileprevious_to_addrlmatching_linesmatchs r"add_model_to_auto_mappingsrs6(WW`0B0J0J3PS0T0Z0Z[^0_``N(77$33NWghihjWkCS8V  ,f 5v>"$(> f 99JD Q-G-K *M?.;_=VPV-W &r * & - -tVn = >H$v-0GG !3 4D8H Tf H$v-0GG !3 4D9M8NeTz 7& '' E:H'(2V;hFG 1vvx  ZZ#$6#77TUW[]_]f]fN( #%069HD % .@BT U ] ]&!$  18  sG;GG new_paper_namerc"d}tjddtjdd|z}t j d|d|d}g}|D]$}d |d |}d |vr|d z }|j |&d j|}||z|zS)a4 Create a new doc file to fill for the new model. Args: new_paper_name (`str`): The fully cased name (as in the official paper name) of the new model. public_classes (`list[str]`): A list of all the public classes that the model will have in the library. u ⚠️ Note that this file is in Markdown but contain specific syntax for our doc-builder (similar to MDX) that may not be rendered properly in your Markdown viewer. --> z# ?rNz coding=utf-8 z