rh. d Z ddlZddlmZ ddlmZ ddlmZ ddlm Z m
Z
e rdd l m Z ndZ e
j e Zd
d d Z G d
de ZdgZy)z Tokenization classes for XGLM. N)copyfile)Optional )PreTrainedTokenizerFast)is_sentencepiece_availablelogging )
XGLMTokenizerzsentencepiece.bpe.modelztokenizer.json)
vocab_filetokenizer_filec e Zd ZdZeZddgZeZ d
fd Z dde
e de e
e de
e fdZ
dde
e de e
e de
e fd Zdd
ed e e dee fd Z xZS )XGLMTokenizerFasta{
Construct a "fast" XGLM tokenizer (backed by HuggingFace's *tokenizers* library). Adapted from [`RobertaTokenizer`]
and [`XLNetTokenizer`]. Based on
[BPE](https://huggingface.co/docs/tokenizers/python/latest/components.html?highlight=BPE#models).
This tokenizer inherits from [`PreTrainedTokenizerFast`] which contains most of the main methods. Users should
refer to this superclass for more information regarding those methods.
Args:
vocab_file (`str`):
Path to the vocabulary file.
bos_token (`str`, *optional*, defaults to `""`):
The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.
When building a sequence using special tokens, this is not the token that is used for the beginning of
sequence. The token used is the `cls_token`.
eos_token (`str`, *optional*, defaults to `""`):
The end of sequence token.
When building a sequence using special tokens, this is not the token that is used for the end of sequence.
The token used is the `sep_token`.
sep_token (`str`, *optional*, defaults to `""`):
The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences for
sequence classification or for a text and a question for question answering. It is also used as the last
token of a sequence built with special tokens.
cls_token (`str`, *optional*, defaults to `""`):
The classifier token which is used when doing sequence classification (classification of the whole sequence
instead of per-token classification). It is the first token of the sequence when built with special tokens.
unk_token (`str`, *optional*, defaults to `""`):
The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be this
token instead.
pad_token (`str`, *optional*, defaults to `""`):
The token used for padding, for example when batching sequences of different lengths.
additional_special_tokens (`list[str]`, *optional*, defaults to `["NOTUSED", "NOTUSED"]`):
Additional special tokens used by the tokenizer.
input_idsattention_maskc * d| _ t | j D
cg c] }
d|
d
} }
| j dg xs g | d< | dxx | D cg c] } | | d vs | c} z
cc< t
| |f|||||||d| || _ y c c}
w c c} w )N z additional_special_tokens)r bos_token eos_token sep_token cls_token unk_token pad_token)num_madeup_wordsrangegetsuper__init__r )selfr r r r r r r r kwargsi madeup_wordsword __class__s /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/xglm/tokenization_xglm_fast.pyr zXGLMTokenizerFast.__init__X s !" 49$:O:O4PQq+aS*Q Q.4jj9TVX.Y._]_*+*+ )0
T@[9\-\D0
+
)
%' R0
s
B B
B
token_ids_0 token_ids_1returnc \ || j g|z S | j g}||z |z |z |z S )a
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating and
adding special tokens. An XLM-RoBERTa sequence has the following format:
- single sequence: ` X `
- pair of sequences: ` A B `
Args:
token_ids_0 (`list[int]`):
List of IDs to which the special tokens will be added.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of [input IDs](../glossary#input-ids) with the appropriate special tokens.
) sep_token_idr r' r( seps r& build_inputs_with_special_tokensz2XGLMTokenizerFast.build_inputs_with_special_tokens{ sF ( %%&4 4 ![ 3&,{:: c z | j g}|t ||z dgz S t ||z |z |z |z dgz S )a
Create a mask from the two sequences passed to be used in a sequence-pair classification task. XLM-RoBERTa does
not make use of token type ids, therefore a list of zeros is returned.
Args:
token_ids_0 (`list[int]`):
List of IDs.
token_ids_1 (`list[int]`, *optional*):
Optional second list of IDs for sequence pairs.
Returns:
`list[int]`: List of zeros.
r )r+ lenr, s r& $create_token_type_ids_from_sequencesz6XGLMTokenizerFast.create_token_type_ids_from_sequences sU $ ! s[()QC/ /3$s*S0;>?1#EEr/ save_directoryfilename_prefixc | j s t d t j j | st
j
d| d y t j j ||r|dz ndt d z }t j j | j t j j | k7 rt | j | |fS )NzhYour fast tokenizer does not have the necessary information to save the vocabulary for a slow tokenizer.zVocabulary path (z) should be a directory.- r ) can_save_slow_tokenizer
ValueErrorospathisdirloggererrorjoinVOCAB_FILES_NAMESabspathr r )r r3 r4 out_vocab_files r& save_vocabularyz!XGLMTokenizerFast.save_vocabulary s ++
ww}}^, LL,^,<rE rD zz)N)__name__
__module__ __qualname____doc__r@ vocab_files_namesmodel_input_namesr
slow_tokenizer_classr listintr r. r2 strtuplerC
__classcell__)r% s @r& r r $ s -^ *$&67( !%H JN; 9;3;DI3F;
c;4 JNF 9F3;DI3FF
cF0!c !HSM !]bcf]g !r/ r )rI r: shutilr typingr tokenization_utils_fastr utilsr r tokenization_xglmr
get_loggerrF r= r@ r __all__ r/ r& rZ sc % > 8 0M
H %#