rhZdZddlmZddlmZddlmZddlmZddl m Z eje Z d d d d Zd diZGddeZdgZy)zTokenization classes for Qwen2.)Optional) AddedToken)PreTrainedTokenizerFast)logging)Qwen2Tokenizerz vocab.jsonz merges.txtztokenizer.json) vocab_file merges_filetokenizer_filezqwen/qwen-tokenizericdeZdZdZeZddgZeZ d fd Z d de de e de e fdZ xZS) Qwen2TokenizerFastac Construct a "fast" Qwen2 tokenizer (backed by HuggingFace's *tokenizers* library). Based on byte-level Byte-Pair-Encoding. Same with GPT2Tokenizer, this tokenizer has been trained to treat spaces like parts of the tokens so a word will be encoded differently whether it is at the beginning of the sentence (without space) or not: ```python >>> from transformers import Qwen2TokenizerFast >>> tokenizer = Qwen2TokenizerFast.from_pretrained("Qwen/Qwen-tokenizer") >>> tokenizer("Hello world")["input_ids"] [9707, 1879] >>> tokenizer(" Hello world")["input_ids"] [21927, 1879] ``` This is expected. This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should refer to this superclass for more information regarding those methods. Args: vocab_file (`str`, *optional*): Path to the vocabulary file. merges_file (`str`, *optional*): Path to the merges file. tokenizer_file (`str`, *optional*): Path to [tokenizers](https://github.com/huggingface/tokenizers) file (generally has a .json extension) that contains everything needed to load the tokenizer. unk_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this token instead. Not applicable to this tokenizer. bos_token (`str`, *optional*): The beginning of sequence token. Not applicable for this tokenizer. eos_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The end of sequence token. pad_token (`str`, *optional*, defaults to `"<|endoftext|>"`): The token used for padding, for example when batching sequences of different lengths. input_idsattention_maskc Dt|trt|ddddn|}t|trt|ddddn|}t|trt|ddddn|}t|trt|ddddn|}t |d|||||||d|y)NFT)lstriprstripspecial normalized)r r r unk_token bos_token eos_token pad_token) isinstancestrrsuper__init__) selfr r r rrrrkwargs __class__s /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/qwen2/tokenization_qwen2_fast.pyrzQwen2TokenizerFast.__init__Ss$)S) yudW\ ] )S) yudW\ ] )S) yudW\ ] )S) yudW\ ]   !#)  save_directoryfilename_prefixreturncf|jjj||}t|S)N)name) _tokenizermodelsavetuple)rr$r%filess r"save_vocabularyz"Qwen2TokenizerFast.save_vocabularys+%%**>*PU|r#)NNN <|endoftext|>Nr/r/)N)__name__ __module__ __qualname____doc__VOCAB_FILES_NAMESvocab_files_namesmodel_input_namesr slow_tokenizer_classrrrr,r. __classcell__)r!s@r"rr%se'R*$&67)!!!. bcHSM]bcf]gr#rN)r3typingrtokenization_utilsrtokenization_utils_fastrutilsrtokenization_qwen2r get_loggerr0loggerr4MAX_MODEL_INPUT_SIZESr__all__rr#r"rBsg&,>.   H %&/6a0aH r#