rh#/4ddlmZddlZddlmZmZddlmZddlm Z m Z eje Z erddlmZmZmZ ddlmZ ddlmZ d d d Z d dd Zy#e$rYEwxYw#e$rYIwxYw)) annotationsN) TYPE_CHECKINGLiteral)save_or_push_to_hub_model)disable_datasets_cachingis_datasets_available CrossEncoderSentenceTransformer SparseEncoder)OVQuantizationConfig)QuantizationConfigc ddlm}m}m} ddlm} m} ddlm} t||xr7t|xr*t|ddxrt|dj| } t||xr7t|xr*t|ddxrt|dj| } t||xrt|j| }| s|s | s td| s| r|dj}n |j}| j!|tt"rTdvr td d d }t%| d xs)j&j(j+d |'j&j(j+dt-fdd|||d| y #t$r tdwxYw)a Export a quantized ONNX model from a SentenceTransformer, SparseEncoder, or CrossEncoder model. This function applies dynamic quantization, i.e. without a calibration dataset. Each of the default quantization configurations quantize the model to int8, allowing for faster inference on CPUs, but are likely slower on GPUs. See the following pages for more information & benchmarks: - `Sentence Transformer > Usage > Speeding up Inference `_ - `Cross Encoder > Usage > Speeding up Inference `_ Args: model (SentenceTransformer | SparseEncoder | CrossEncoder): The SentenceTransformer, SparseEncoder, or CrossEncoder model to be quantized. Must be loaded with `backend="onnx"`. quantization_config (QuantizationConfig): The quantization configuration. model_name_or_path (str): The path or Hugging Face Hub repository name where the quantized model will be saved. push_to_hub (bool, optional): Whether to push the quantized model to the Hugging Face Hub. Defaults to False. create_pr (bool, optional): Whether to create a pull request when pushing to the Hugging Face Hub. Defaults to False. file_suffix (str | None, optional): The suffix to add to the quantized model file name. Defaults to None. Raises: ImportError: If the required packages `optimum` and `onnxruntime` are not installed. ValueError: If the provided model is not a valid SentenceTransformer, SparseEncoder, or CrossEncoder model loaded with `backend="onnx"`. ValueError: If the provided quantization_config is not valid. Returns: None rr )ORTModel ORTQuantizer)AutoQuantizationConfigzPlease install Optimum and ONNX Runtime to use this function. You can install them with pip: `pip install optimum[onnxruntime]` or `pip install optimum[onnxruntime-gpu]` auto_modelz}The model must be a Transformer-based SentenceTransformer, SparseEncoder, or CrossEncoder model loaded with `backend="onnx"`.)arm64avx2avx512 avx512_vnnizqquantization_config must be an QuantizationConfig instance or one of 'arm64', 'avx2', 'avx512', or 'avx512_vnni'.NF) is_static_ _quantizedc,j|S)N) file_suffixquantize)save_dirrquantization_config quantizers y/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/sentence_transformers/backend/quantize.pyz5export_dynamic_quantized_onnx_model..ps););L 5-. 6 J 6 E!Hl + 6 uQx**H 5  5-( 6 J 6 E!Hl + 6 uQx**H 5  ! 5[*U[[RZ:[O / L  /#Ah11 #kk ,,Y7I%s+ &P PD $7q#9 Rg&<>QR]bc!s(;(I(I(N(N(T(T(V'WWXYqXr%s ,::??EEGH S sB"- Y   8   s F44G c  ddlm} m} m}  ddlm} m}m}ddlm }ts tdt| xr7txr*tddxrtdj|}t| xr7txr*tddxrtdj|}t| xrtj |}|s|s |s t#d|s|rdj}n j }||}| | |j%|t'd |||fDr!t)d |||fDs t#d fd||nd}||nd}||nd}ndt+5j-||fd| |j.nd|d d d t1fdd||||| d y #t$r tdwxYw#1swY;xYw)aw Export a quantized OpenVINO model from a SentenceTransformer, SparseEncoder, or CrossEncoder model. This function applies Post-Training Static Quantization (PTQ) using a calibration dataset, which calibrates quantization constants without requiring model retraining. Each default quantization configuration converts the model to int8 precision, enabling faster inference while maintaining accuracy. See the following pages for more information & benchmarks: - `Sentence Transformer > Usage > Speeding up Inference `_ - `Cross Encoder > Usage > Speeding up Inference `_ Args: model (SentenceTransformer | SparseEncoder | CrossEncoder): The SentenceTransformer, SparseEncoder, or CrossEncoder model to be quantized. Must be loaded with `backend="openvino"`. quantization_config (OVQuantizationConfig | dict | None): The quantization configuration. If None, default values are used. model_name_or_path (str): The path or Hugging Face Hub repository name where the quantized model will be saved. dataset_name(str, optional): The name of the dataset to load for calibration. If not specified, the `sst2` subset of the `glue` dataset will be used by default. dataset_config_name (str, optional): The specific configuration of the dataset to load. dataset_split (str, optional): The split of the dataset to load (e.g., 'train', 'test'). Defaults to None. column_name (str, optional): The column name in the dataset to use for calibration. Defaults to None. push_to_hub (bool, optional): Whether to push the quantized model to the Hugging Face Hub. Defaults to False. create_pr (bool, optional): Whether to create a pull request when pushing to the Hugging Face Hub. Defaults to False. file_suffix (str, optional): The suffix to add to the quantized model file name. Defaults to `qint8_quantized`. Raises: ImportError: If the required packages `optimum` and `openvino` are not installed. ValueError: If the provided model is not a valid SentenceTransformer, SparseEncoder, or CrossEncoder model loaded with `backend="openvino"`. ValueError: If the provided quantization_config is not valid. Returns: None rr )OVConfigr OVQuantizer)OVModelzPlease install datasets, optimum-intel and openvino to use this function. You can install them with pip: `pip install datasets optimum[openvino]`zaPlease install datasets to use this function. You can install it with pip: `pip install datasets`rzThe model must be a Transformer-based SentenceTransformer, SparseEncoder, or CrossEncoder model loaded with `backend="openvino"`.N)r c3$K|]}|du ywN.0params r" z9export_static_quantized_openvino_model..s j5  jc3$K|]}|du ywrHrIrJs r"rMz9export_static_quantized_openvino_model..sw#TwrNzEither specify all of `dataset_name`, `dataset_config_name`, `dataset_split`, and `column_name`, or leave them all unspecified.c.j|dddS)N max_lengthiT)paddingrQ truncation) tokenizer)examplesr/s r"preprocess_functionzCexport_static_quantized_openvino_model..preprocess_functionsx#Z^__r$gluesst2trainsentencec|SrHrI)rU column_namerVs r"r#z8export_static_quantized_openvino_model..s1DXkEZ1[r$i,) dataset_namedataset_config_namerV num_samples dataset_splitc,j|S)N)save_directory ov_configr)rcalibration_datasetrcr!s r"r#z8export_static_quantized_openvino_model..s);); I*<* r$&export_static_quantized_openvino_modelopenvinor')r0r r r optimum.intel.openvinorDr rEoptimum.intel.openvino.modelingrFr3rr4r5r6rr/r7r8anyallrget_calibration_datasetr_r)r/r r+r]r^r`r\r,r-rr r r rDr rErFr>r?r@ov_modelrdrcrVr!s` ` @@@@r"rere|s~^WV    < ! " o  5-. 5 J 5 E!Hl + 5 uQx**G 4  5-( 5 J 5 E!Hl + 5 uQx**G 4  ! 5Z*U[[RY:ZO / P  /!!H//!KK"24-@AI++H5I j<9Lm]h*i jjsvw(46I=Ze'fwt N  `$0#;<L1D1P-V\%2%>MGM!,!8+jK ! # '??% 3 [;N;Z+77`c' @   F"- }   V   j  sG")G:"G7:H)FFN)r/2SentenceTransformer | SparseEncoder | CrossEncoderr zFQuantizationConfig | Literal['arm64', 'avx2', 'avx512', 'avx512_vnni']r+r9r,boolr-rnr str | NonereturnNone)NNNNFFqint8_quantized)r/rmr z"OVQuantizationConfig | dict | Noner+r9r]ror^ror`ror\ror,rnr-rnrr9rprq) __future__rloggingtypingrr#sentence_transformers.backend.utilsrsentence_transformers.utilrr getLogger__name__loggerr0r r r optimum.intelr r3r2rr%rerIr$r"r|sT")IV   8 $VV 6 H" a =a_aa a  a  a aP $&* $"(B =B;BB B $ B  BBBBB BY        s$BBB  B BB