rh "ddlmZddlZddlZddlZddlmZ ddlmZddl Z ddl m Z ddl mZerddl mZej eZGdd eZy#e$r ddl mZYCwxYw) ) annotationsN) TYPE_CHECKING)Self) AutoTokenizer) InputModule)PreTrainedTokenizerceZdZUdZdgZded< d dfd ZddZdddd Ze d dd Z e d dd Z dd Z dd Z d ddZxZS)SparseStaticEmbeddinga SparseStaticEmbedding module for efficient sparse representations. This lightweight module computes sparse representations by mapping input tokens to static weights, such as IDF (Inverse Document Frequency) weights. It is designed to encode queries or documents into fixed-size embeddings based on the presence of tokens in the input. A common scenario is to use this module for encoding queries, and using a heavier module like SPLADE (MLMTransformer + SpladePooling) for document encoding. Args: tokenizer (PreTrainedTokenizer): PreTrainedTokenizer to tokenize input texts into input IDs. weight (torch.Tensor | None): Static weights for vocabulary tokens (e.g., IDF weights), shape should be (vocab_size,). If None, initializes weights to a vector of ones. Default is None. frozen (bool): Whether the weights should be frozen (not trainable). Default is False. frozenz list[str] config_keysct|||_|(tjj || |_n[tjj tjt|jj| |_||_ |j jd|_ |jj|_y)N) requires_gradr)super__init__ tokenizertorchnn Parameterweightoneslen get_vocabr sizenum_dimensionsmodel_max_lengthmax_seq_length)selfrrr __class__s /var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/sentence_transformers/sparse_encoder/models/SparseStaticEmbedding.pyrzSparseStaticEmbedding.__init__-s "  ((,,Vv:,NDK((,,UZZDNNE__Q' Z1D1DYM]M]ejepepq 9n=$dkk1   )#&88J)3%&r Tsafe_serializationcn|j||j|||j|y)Nr2)save_tokenizersave_torch_weights save_config)r output_pathr3argskwargss rsavezSparseStaticEmbedding.saveUs2 K(  @RS %r c jtjj|s ddlm}||d||||}t |5} tj| } dddt j\} } |jt| } tj| tj} t!| d z}tj"|tj}t| | D] \}}|||< |d ||d |S#t $rt d|dwxYw#1swYxYw) a Create an SparseStaticEmbedding module from a JSON file containing token to IDF weight mappings. Args: json_path (str): Path to the JSON file containing token to IDF weight mappings. tokenizer (PreTrainedTokenizer): Tokenizer to use for converting tokens to IDs. token (bool | str | None): Token for Hugging Face authentication cache_folder (str | None): Cache folder for Hugging Face revision (str | None): Model revision local_files_only (bool): Whether to only load local files **config: Additional configuration options for the IDF model. Returns: SparseStaticEmbedding: An initialized SparseStaticEmbedding model. r)hf_hub_downloadzidf.json)repo_idfilenametoken cache_dirrevisionlocal_files_onlyzIDF JSON file not found at z. Please provide a valid path.N)r&r'rr)ospathexistshuggingface_hubr= ValueErroropenjsonloadzipitemsconvert_tokens_to_idslistrtensorfloat32maxr*)cls json_pathrr@ cache_folderrBrCconfigr=fInidftokensweights token_ids max_token_idrtoken_idws r from_jsonzSparseStaticEmbedding.from_jsonZs*4ww~~i( j;+%'*%%5  )_ !))C.C !syy{+33DLA ,,wemm<9~) \?y'2 !KHa F8  !@&I@@@ j #>ykIg!hii j ! !sD D) D&)D2c ,|j||||||}tj||||||} |jdd} | *| j dr|j | | f||||d|S|d d| d|} |j ||||||| } | S) a Load the SparseStaticEmbedding module with its tokenizer. Args: model_name_or_path (str): Path to the directory containing the saved model. subfolder (str): Subfolder within the model directory token (bool | str | None): Token for Hugging Face authentication cache_folder (str | None): Cache folder for Hugging Face revision (str | None): Model revision local_files_only (bool): Whether to only load local files **kwargs: Additional keyword arguments Returns: SparseStaticEmbedding: The loaded SparseStaticEmbedding module. )model_name_or_path subfolderr@rWrBrC)rdr@rArBrCrGNz.json)r@rArBrCrD)rcrdr@rWrBrCmodelrE) load_configrfrom_pretrainedpopendswithraload_torch_weights) rUrcrdr@rWrBrCr:rXrrGres rrMzSparseStaticEmbedding.loads41%- ! "11 "-  zz&$'   g 6 3==&!!1  ?49??&&1%-'  r cd|jjj}d|jd|j|dS)Nz , tokenizer=zSparseStaticEmbedding(z, dim=))rr__name__get_config_dictr)rtokenizer_infos r__repr__zSparseStaticEmbedding.__repr__sL'(@(@(I(I'JK'(<(<(>'?vdFYFYEZ[iZjjkllr c|jS)N)r)rs r get_sentence_embedding_dimensionz6SparseStaticEmbedding.get_sentence_embedding_dimensions"""r c @t|j||dddS)NTptF)padding truncationreturn_tensorsadd_special_tokens)dictr)rtextsrus rtokenizezSparseStaticEmbedding.tokenizes) NN5'dSWlqN r  r )NF)rrrztorch.Tensor | Noner bool)r-dict[str, torch.Tensor]returnr})r8strr3r|r~None)NNNF) rVrrrr@bool | str | NonerW str | NonerBrrCr|)NNNF)rcrrdrr@rrWrrBrrCr|r~r)r~r)r~int)T)rzz.list[str] | list[dict] | list[tuple[str, str]]ruz str | boolr~r})rm __module__ __qualname____doc__r __annotations__rr1r; classmethodrarMrprrr{ __classcell__)rs@rr r sW$'ZK' '+ >&>$> >$,HL&  $(#'#!&4A4A'4A! 4A ! 4A  4A4A4Al#'#'#!&DDD! D ! D  DD DDLm#\` C NX  r r ) __future__rrLloggingrFtypingrr ImportErrortyping_extensionsr transformersr(sentence_transformers.models.InputModulerr getLoggerrmloggerr rEr rrsa"  ' &@0   8 $L KL '&'sA A$#A$