rh,hdZddlZddlmZmZddlmZddlmZddl m Z gd Z Gd d eZ d gZ y) z Processor class for EVOLLA. N)OptionalUnion) BatchFeature)ProcessorMixin) AutoTokenizer)aa_seqfoldseekmsac eZdZdZddgZdgZdZdZdZdfd Z ddZ dde fd Z dd e eeeefd e eeeeeefd e e de e fd ZdZdZdZdZfdZefdZxZS)EvollaProcessoran Constructs a EVOLLA processor which wraps a LLama tokenizer and SaProt tokenizer (EsmTokenizer) into a single processor. [`EvollaProcessor`] offers all the functionalities of [`EsmTokenizer`] and [`LlamaTokenizerFast`]. See the docstring of [`~EvollaProcessor.__call__`] and [`~EvollaProcessor.decode`] for more information. Args: protein_tokenizer (`EsmTokenizer`): An instance of [`EsmTokenizer`]. The protein tokenizer is a required input. tokenizer (`LlamaTokenizerFast`, *optional*): An instance of [`LlamaTokenizerFast`]. The tokenizer is a required input. protein_max_length (`int`, *optional*, defaults to 1024): The maximum length of the sequence to be generated. text_max_length (`int`, *optional*, defaults to 512): The maximum length of the text to be generated. protein_tokenizer tokenizersequence_max_lengthr c | td| tdt| ||d|j_||_||_y)Nz+You need to specify an `protein_tokenizer`.z"You need to specify a `tokenizer`.z<|reserved_special_token_0|>) ValueErrorsuper__init__r pad_tokenprotein_max_lengthtext_max_length)selfrrrrkwargs __class__s /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/models/evolla/processing_evolla.pyrzEvollaProcessor.__init__;sS  $JK K  AB B *I6#A "4.c dg}|D]}|jd}|jd}djt||Dcgc]&\}}|j|j z(c}}} |j | |j j|dd|d} | Scc}}w)Nr r ptT)return_tensors truncation max_lengthpadding)getjoinzipupperlowerappendrbatch_encode_plus) rproteinsr sa_sequencesproteinr r sf sa_sequence sa_tokenss rprocess_proteinsz EvollaProcessor.process_proteinsGs  -G[[*F{{:.H''SQYEZ"[TQ1779qwwy#8"[\K    ,  - **<< $K]gk=  #\s+B, rcg}|D]1}|jj|dd}|j|3|j|dddd|}|S)NFT)tokenizeadd_generation_promptr longest)add_special_tokensr!r$r"r#)rapply_chat_templater*)rtextsrpromptsmessagesprompt prompt_inputss r process_textzEvollaProcessor.process_textTsw  #H^^77&*8F NN6 "  # $& ' rr, messages_listrc || td||n |j}||n |j}t|tr|g}t|t t frt|dt t fs|g}t|t t frtd|Ds tdt|t t fr6td|Ds$tddjtd|t|t t fr|D]}t|t t fstd t|d td |Ds td td |Dstd|Dsstd|ntdt|d |j||}|j||}t|d|d|d|ddS)avThis method takes batched or non-batched proteins and messages_list and converts them into format that can be used by the model. Args: proteins (`Union[List[dict], dict]`): A list of dictionaries or a single dictionary containing the following keys: - `"aa_seq"` (`str`) -- The amino acid sequence of the protein. - `"foldseek"` (`str`) -- The foldseek string of the protein. messages_list (`Union[List[List[dict]], List[dict]]`): A list of lists of dictionaries or a list of dictionaries containing the following keys: - `"role"` (`str`) -- The role of the message. - `"content"` (`str`) -- The content of the message. protein_max_length (`int`, *optional*, defaults to 1024): The maximum length of the sequence to be generated. text_max_length (`int`, *optional*, defaults to 512): The maximum length of the text. Return: a dict with following keys: - `protein_input_ids` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The input IDs for the protein sequence. - `protein_attention_mask` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The attention mask for the protein sequence. - `text_input_ids` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The input IDs for the text sequence. - `text_attention_mask` (`torch.Tensor` of shape `(batch_size, sequence_length)`) -- The attention mask for the text sequence. z3You need to specify `messages_list` and `proteins`.rc3<K|]}t|tywN isinstancedict.0ps r z+EvollaProcessor.__call__..s:aST:a;N:azUThe proteins should be a list of dictionaries, but not all elements are dictionaries.c3\K|]$}td|jD&yw)c3,K|] }|tvywrC)PROTEIN_VALID_KEYS)rHks rrJz5EvollaProcessor.__call__...s:A'':sN)allkeysrGs rrJz+EvollaProcessor.__call__..s&; ?@C:: :; s*,z2There should be a list of dictionaries with keys: z, z for each protein.But got: z;Each messages in messages_list should be a list instead of .c3<K|]}t|tywrCrDrHms rrJz+EvollaProcessor.__call__..sA1:a.ArKzfEach message in messages_list should be a list of dictionaries, but not all elements are dictionaries.c3TK|] }t|jdk7"yw)rN)lenrQrTs rrJz+EvollaProcessor.__call__..s.s+D=>CMfi%88Ds(*zlEach message in messages_list should be a list of dictionaries with two keys: 'role' and 'content'.But got: zFThe messages_list should be a list of lists of dictionaries, but it's input_idsattention_mask)protein_input_idsprotein_attention_maskr\r])data)rrrrErFlisttuplerPr&rNtypeanyr3r?r) rr,r@rrrr<r2 text_tokenss r__call__zEvollaProcessor.__call__ls,B  }4RS S3E3Q/W[WnWn-<-H/dNbNb h % zH mdE] 3J}UVGWZ^`eYf"?   rc:|jj|i|SrC)r batch_decoderargsrs rrhzEvollaProcessor.batch_decodes*t~~**D;F;;rc:|jj|i|SrC)rdecoderis rrlzEvollaProcessor.decodes$t~~$$d5f55rc:|jj|i|SrC)rrhris rprotein_batch_decodez$EvollaProcessor.protein_batch_decodes 2t%%22DCFCCrc:|jj|i|SrC)rrlris rprotein_decodezEvollaProcessor.protein_decodes ,t%%,,d=f==rc |jjtjj ||j d|j v}|r|j jdnd}|r||j jdt||fi|}|r||j j|d|S)Nr) rsave_pretrainedospathr&protein_tokenizer_dir_name attributesindexremoverinsert)rsave_directoryrprotein_tokenizer_presentprotein_tokenizer_indexoutputsrs rrrzEvollaProcessor.save_pretraineds ..rww||NDLkLk/lm%84??$J!Pi$//"7"78K"Los $)@)L OO " "#6 7').CFC $)@)L OO " "#: (  rr)rrstypingrrfeature_extraction_utilsrprocessing_utilsrautor rNr__all__rrrs@ "4!3TnTn  r