rhJPdZddlZddlZddlZddlmZddlmZddlZ ddl m Z m Z m Z mZmZddlmZej$eZdd d Zed Zd Zd ZdZGddZGdde ZdgZy)zTokenization classes for CLIP.N) lru_cache)Optional) AddedTokenPreTrainedTokenizer _is_control_is_punctuation_is_whitespace)loggingz vocab.jsonz merges.txt) vocab_file merges_filec tttdtddztttdtddzztttdtddzz}|dd}d }td D]1}||vs|j||jd |z|dz }3|Dcgc] }t |}}t t ||Scc}w) a8 Returns list of utf-8 byte and a mapping to unicode strings. We specifically avoids mapping to whitespace/control characters the bpe code barfs on. The reversible bpe codes work on unicode strings. This means you need a large # of unicode characters in your vocab if you want to avoid UNKs. When you're at something like a 10B token dataset you end up needing around 5K for decent coverage. This is a significant percentage of your normal, say, 32K bpe vocab. To avoid that, we want lookup tables between utf-8 bytes and unicode strings. !~¡¬®ÿNr)listrangeordappendchrdictzip)bscsnbs }/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/clip/tokenization_clip.pybytes_to_unicoder#%s U3s8SX\ *+d5TCIPQM3R.SSVZ[`adeiajloptluxyly[zV{{ AB A 4[ B; IIaL IIdQh  FA   Q#a& B  B  s C4cbt}|d}|ddD]}|j||f|}|S)z Return set of symbol pairs in a word. Word is represented as tuple of symbols (symbols being variable-length strings). rrN)setadd)wordpairs prev_charchars r" get_pairsr+>sF EEQIQR 9d#$  LcTtjdd|}|j}|S)Nz\s+ )resubstrip)texts r"whitespace_cleanr3Ls$ 66&#t $D ::r?r@s r"__init__zBasicTokenizer.__init__ts<  K*{+&<#* 0r,cJ|r$|jjt|n |j}|j|}|jr|j |}t jd|}t|}g}|D]}||vrY|jr0|j}|jdur/|j|}n|jr|j|}|j|j||tdj|}|S)aj Basic Tokenization of a piece of text. For sub-word tokenization, see WordPieceTokenizer. Args: never_split (`List[str]`, *optional*) Kept for backward compatibility purposes. Now implemented directly at the base class level (see [`PreTrainedTokenizer.tokenize`]) List of token not to split. NFCFr.)r=unionr% _clean_textr>_tokenize_chinese_chars unicodedata normalizer7r<lowerr?_run_strip_accentsextend_run_split_on_puncjoin)rAr2r=unicode_normalized_text orig_tokens split_tokenstoken output_tokenss r"tokenizezBasicTokenizer.tokenizesCNd&&,,S-=>SWScSc %  & &//5D"-"7"7t"D)*AB    MEK'%%!KKME))6 $ 7 7 >'' 33E:E    7 7{ K L M,CHH\,BC r,ctjd|}g}|D].}tj|}|dk(r|j|0dj |S)z$Strips accents from a piece of text.NFDMn)rHrIcategoryrrN)rAr2outputr*cats r"rKz!BasicTokenizer._run_strip_accentss^$$UD1 D&&t,Cd{ MM$   wwvr,cv|jr|||vr|gSt|}d}d}g}|t|krb||}t|r|j |gd}n)|r|j gd}|dj ||dz }|t|krb|Dcgc]}dj |c}Scc}w)z&Splits punctuation on a piece of text.rTFrrX)r@rlenr rrN) rAr2r=charsistart_new_wordrZr*xs r"rMz!BasicTokenizer._run_split_on_puncs$$)@T[EX6MT  #e*n8Dt$ tf%!%!MM"%!&r !!$' FA#e*n%++q +++sB6cg}|D]c}t|}|j|r4|jd|j||jdS|j|edj|S)z)Adds whitespace around any CJK character.r.rX)r_is_chinese_charrrNrAr2rZr*cps r"rGz&BasicTokenizer._tokenize_chinese_charssm $DTB$$R( c" d# c" d# $wwvr,c|dk\r|dksF|dk\r|dks<|dk\r|dks2|dk\r|dks(|d k\r|d ks|d k\r|d ks|d k\r|dks |dk\r|dkryy)z6Checks whether CP is the codepoint of a CJK character.iNii4iMiiߦii?i@ii iiiiiTF)rArfs r"rdzBasicTokenizer._is_chinese_charsr6\bFlf vg "-g "-g "-g "-f vg "-r,cg}|D]Q}t|}|dk(s|dk(s t|r$t|r|jdA|j|Sdj |S)zBPerforms invalid character removal and whitespace cleanup on text.rir.rX)rrr rrNres r"rFzBasicTokenizer._clean_textsf $DTBQw",+d*;d# c" d# $wwvr,)TNTNTr;) __name__ __module__ __qualname____doc__rBrTrKrMrGrdrFrhr,r"r9r9]s<0# 1 $L ,, 0 r,r9c (eZdZdZeZddgZ dfd ZedZ dZ dde e de e e d e e fd Z dde e de e e d ed e e ffd Z dde e de e e d e e fd ZdZdZdZdZdZddede ed eefdZxZS) CLIPTokenizera Construct a CLIP tokenizer. Based on byte-level Byte-Pair-Encoding. This tokenizer inherits from [`PreTrainedTokenizer`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`): Path to the vocabulary file. merges_file (`str`): Path to the merges file. errors (`str`, *optional*, defaults to `"replace"`): Paradigm to follow when decoding bytes to UTF-8. See [bytes.decode](https://docs.python.org/3/library/stdtypes.html#bytes.decode) for more information. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. bos_token (`str`, *optional*, defaults to `"<|startoftext|>"`): The beginning of sequence token. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc t|trt|ddn|}t|trt|ddn|}t|trt|ddn|} ddl} | j|_t|d5} tj| |_ ddd|jjD cic]\} } | |  c} } |_||_t#|_|j$jD cic]\} } | |  c} } |_t|d5} | j)j+j-dd d }dddDcgc]}t/|j-}}t1t3|t5t7||_d d d |_t=j>dt<j@|_!tE|d|||||d|y#t $r2t jdtdd|_ d|_YwxYw#1swYxYwcc} } wcc} } w#1swYxYwcc}w)NF)lstriprstriprzKftfy or spacy is not installed using custom BasicTokenizer instead of ftfy.)r?r@utf-8encoding ri<|startoftext|> <|endoftext|>)ryrzz[<\|startoftext\|>|<\|endoftext\|>|'s|'t|'re|'ve|'m|'ll|'d|[\p{L}]+|[\p{N}]|[^\s\p{L}\p{N}]+)errors unk_token bos_token eos_token pad_tokenrh)$ isinstancestrrftfyfix_text ImportErrorloggerinfor9nlpopenjsonloadencoderitemsdecoderr{r# byte_encoder byte_decoderreadr1r5tuplerrrr^ bpe_rankscacher/compile IGNORECASEpatsuperrB)rAr r r{r|r}r~rkwargsr vocab_handlekv merges_handle bpe_mergesmerge __class__s r"rBzCLIPTokenizer.__init__sJTT]_bIcJyuEir IST]_bIcJyuEir IST]_bIcJyuEir  !  MMDM *w / 3<99\2DL 3)-););)=>A1>  ,..2.?.?.E.E.GHdaQTH + 0 [M&++-335;;DA!FYZJ [8BCueEKKM*C Cc*eC O.DEF):_] :: n MM       - ! KKe f%EERDH DM !  3 3>I [ [Cs<#HI I I11I. I(7I?II I%c,t|jSr;)r^rrAs r" vocab_sizezCLIPTokenizer.vocab_sizeLs4<<  r,cBt|jfi|jSr;)rradded_tokens_encoderrs r" get_vocabzCLIPTokenizer.get_vocabPsDLL>D$=$=>>r, token_ids_0 token_ids_1returncl|jg}|jg}|||z|zS||z|z|z|z|zS)a Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and adding special tokens. A CLIP sequence has the following format: - single sequence: `<|startoftext|> X <|endoftext|>` Pairs of sequences are not the expected use case, but they will be handled without a separator. Args: token_ids_0 (`list[int]`): List of IDs to which the special tokens will be added. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens. ) bos_token_id eos_token_idrArrr}r~s r" build_inputs_with_special_tokensz.CLIPTokenizer.build_inputs_with_special_tokensSsU(&&' &&'  {*Y6 6;&2Y>LyXXr,already_has_special_tokensc|rt|||dS|dgdgt|zzdgzSdgdgt|zzdgzdgzdgt|zzdgzS)a Retrieve sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` method. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: `list[int]`: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. T)rrrrr)rget_special_tokens_maskr^)rArrrrs r"rz%CLIPTokenizer.get_special_tokens_maskns& &72'[]a3   31#K 001QC7 7sqcC ,,-3qc9aS3{CS=STXYWZZZr,c|jg}|jg}|t||z|zdgzSt||z|z|z|z|zdgzS)a Create a mask from the two sequences passed. CLIP does not make use of token type ids, therefore a list of zeros is returned. Args: token_ids_0 (`list[int]`): List of IDs. token_ids_1 (`list[int]`, *optional*): Optional second list of IDs for sequence pairs. Returns: `list[int]`: List of zeros. r)rrr^rs r"$create_token_type_ids_from_sequencesz2CLIPTokenizer.create_token_type_ids_from_sequencessp &&' &&'  y;.:;qcA A9{*Y6B[PS\\]ab`cccr,c|jvrj|St|dd|ddzfz}t|}|s|dzS t|fd}|jvrn|\}}g}d}|t |kr |j ||} |j||| | }|||k(r6|t |dz kr%||dz|k(r|j||z|dz }n|j|||dz }|t |krt|}|}t |dk(rn t|}dj|}|j|<|S#t$r|j||dYiwxYw) Nr]cNjj|tdS)Ninf)rgetfloat)pairrAs r"z#CLIPTokenizer.bpe..s1C1CD%PU,1Wr,keyrrr.) rrr+minrr^indexrL ValueErrorrrN) rArRr'r(bigramfirstsecondnew_wordr`js ` r"bpezCLIPTokenizer.bpes DJJ ::e$ $U3BZ E"I$6#88$6> !$WXFT^^+"ME6HAc$i- 5!,A OOD1I.A7e#CIM(9d1q5kV>SOOEFN3FAOODG,FAc$i- XHD4yA~!$9:xx~  5 +"OODH-sEE98E9cg}j+djjj|}n(t j|j }t jj|D]a}djfd|jdD}|jdj|jdDc|S)zTokenize a string.r.rXc3<K|]}j|ywr;)r).0r!rAs r" z*CLIPTokenizer._tokenize..s!)*!!!$sruc3 K|]}|ywr;rh)r bpe_tokens r"rz*CLIPTokenizer._tokenize..sTIiTs ) rrNrrTr3rJr/findallrencoderLrr5)rAr2 bpe_tokensrRs` r" _tokenizezCLIPTokenizer._tokenizes == 88DHH--d34D#DMM$$78>>@DZZ$/ UEGG.3ll7.CE   T%9N9Ns9ST T  U r,c|jj||jj|jS)z0Converts a token (str) in an id using the vocab.)rrr|)rArRs r"_convert_token_to_idz"CLIPTokenizer._convert_token_to_ids,||t||'7'7'GHHr,c8|jj|S)z=Converts an index (integer) in a token (str) using the vocab.)rr)rArs r"_convert_id_to_tokenz"CLIPTokenizer._convert_id_to_tokens||&&r,cdj|}t|Dcgc]}|j|c}}|jd|jj ddj }|Scc}w)z:Converts a sequence of tokens (string) in a single string.rXru)r{rr.)rN bytearrayrdecoder{replacer1)rAr6r2c byte_arrays r"convert_tokens_to_stringz&CLIPTokenizer.convert_tokens_to_stringsiwwvdC 1 1! 4CD    =EEfcRXXZ  DsA5save_directoryfilename_prefixc Ptjj|stj d|dytjj ||r|dzndt dz}tjj ||r|dzndt dz}t|dd 5}|jtj|jd d d dzdddd}t|dd 5}|jdt|jjdD]M\}} || k7rtjd|d| }|jdj |dz|dz }O ddd||fS#1swYxYw#1swY||fSxYw)NzVocabulary path (z) should be a directory-rXr r wrurvrTF)indent sort_keys ensure_asciirxrz#version: 0.2 c |dS)Nrrh)kvs r"rz/CLIPTokenizer.save_vocabulary..s Y[\]Y^r,rzSaving vocabulary to zZ: BPE merge indices are not consecutive. Please check that the tokenizer is not corrupted!r.r)ospathisdirrerrorrNVOCAB_FILES_NAMESrwriterdumpsrsortedrrwarning) rArrr merge_filefrwriterr token_indexs r"save_vocabularyzCLIPTokenizer.save_vocabularysww}}^, LL,^,<rs% oo   H %   0 ^^BF&'F&R  r,