rhdZddlZddlmZddlmZddlmZeje Z ddiZ d Z Gd d eZ d gZy) zTokenization classes for ESM.N)Optional)PreTrainedTokenizer)logging vocab_file vocab.txtct|d5}|jj}|Dcgc]}|jc}cdddScc}w#1swYyxYw)Nr)openread splitlinesstrip)rflinesls {/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/esm/tokenization_esm.pyload_vocab_filersS j# *!##%#()a )**)**s#AAAAA!c eZdZdZeZddgZ dfd Zdede fdZ de defd Z d Z d Z de defd Zdede fd Z ddeedeeedeefdZ ddedeededeefdZdZedefdZxZS) EsmTokenizerz& Constructs an ESM tokenizer. input_idsattention_maskc Vt||_tt|j|_t|jD cic]\}} | | c} }|_t |d|||||d||j|_|j|jycc} }w)N) unk_token cls_token pad_token mask_token eos_token) r all_tokensdict enumerate _id_to_token _token_to_idsuper__init__unique_no_split_tokens _update_trie) selfrrrrrrkwargsindtok __class__s rr%zEsmTokenizer.__init__+s**5 4??!;<6?6PQ(#sS#XQ  !    '+oo# $556Rs B%indexreturncN|jj||jSNr"getrr(r-s r_convert_id_to_tokenz!EsmTokenizer._convert_id_to_tokenG  $$UDNN;;tokenc|jj||jj|jSr0r#r2rr(r7s r_convert_token_to_idz!EsmTokenizer._convert_token_to_idJ0  $$UD,=,=,A,A$..,QRRr6c "|jSr0)split)r(textr)s r _tokenizezEsmTokenizer._tokenizeMszz|r6cp|jj}|j|j|Sr0)r#copyupdateadded_tokens_encoder)r( base_vocabs r get_vocabzEsmTokenizer.get_vocabPs0&&++- $334r6c|jj||jj|jSr0r9r:s r token_to_idzEsmTokenizer.token_to_idUr<r6cN|jj||jSr0r1r3s r id_to_tokenzEsmTokenizer.id_to_tokenXr5r6 token_ids_0 token_ids_1c|jg}|jg}||j||zS||z|zS|j td||z|z|z|zS)Nz=Cannot tokenize multiple sequences when EOS token is not set!) cls_token_id eos_token_id ValueError)r(rKrLclsseps r build_inputs_with_special_tokensz-EsmTokenizer.build_inputs_with_special_tokens[s  !  !    ([(([(3..    &\] ][ 3&4s::r6already_has_special_tokensc|r-| td|Dcgc]}||jvrdndc}Sdgdgt|zzdgz}||dgt|zdgzz }|Scc}w)a Retrieves sequence ids from a token list that has no special tokens added. This method is called when adding special tokens using the tokenizer `prepare_for_model` or `encode_plus` methods. Args: token_ids_0 (`list[int]`): List of ids of the first sequence. token_ids_1 (`list[int]`, *optional*): List of ids of the second sequence. already_has_special_tokens (`bool`, *optional*, defaults to `False`): Whether or not the token list is already formatted with special tokens for the model. Returns: A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token. zYou should not supply a second sequence if the provided sequence of ids is already formatted with special tokens for the model.r)rPall_special_idslen)r(rKrLrTr7masks rget_special_tokens_maskz$EsmTokenizer.get_special_tokens_maskis$ && R LWW%$"6"66AA=W WsqcC ,,-3  " QC#k**aS0 0D XsA!ctjj||r|dznddz}t|d5}|j dj|j ddd|fS#1swY|fSxYw)N-rw )ospathjoinr writer)r(save_directoryfilename_prefixrrs rsave_vocabularyzEsmTokenizer.save_vocabularyskWW\\.O?S3Hacgr2rs *c " 0a GGDIIdoo. / 0} 0}s +A--A8c,t|jSr0)rXr)r(s r vocab_sizezEsmTokenizer.vocab_sizes4??##r6)zzzzzr0)NF)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr%intstrr4r;r@rFrHrJlistrrSboolrZrfpropertyrh __classcell__)r,s@rrr#s*$&67 78<#<#<S#S#S SSS<<<JN ;9 ;3;DI3F ; c ;in.6tnae c> $C$$r6r)rlr`typingrtokenization_utilsrutilsr get_loggerriloggerrmrr__all__rr6rr|sT$ 5   H %!;/* m$&m$`  r6