rht= :ddlZddlZddlZddlmZddlmZmZmZm Z ddl Z GddeZ ddl m Z ddlmZmZmZdZdd Z ddd dd d eeeeee eeegeffffd Zej2eZeGddZy#e$rd ZYXwxYw)N)Enum)AnyCallableOptionalUnionc,eZdZdZdZdZdZdZdZdZ dZ y ) RequestStatusz5Status of a generation request through its lifecycle.pending prefillingprefilling_splitsplit_pending_remainderdecodingfinishedfailedN) __name__ __module__ __qualname____doc__PENDING PREFILLINGPREFILLING_SPLITSPLIT_PENDING_REMAINDERDECODINGFINISHEDFAILEDm/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/utils/metrics.pyr r s*?GJ)7HH Frr )metrics)Status StatusCode get_tracerTFc$tsdSfd}|S)a Decorator that attaches a tracer to a class. This decorator should be applied to classes that need OpenTelemetry tracing. It adds a tracer attribute to the class instance that can be used by the traced decorator. Args: tracer_name_template: Optional template string for the tracer name. If provided, it should contain {module} which will be replaced with the class's full module path and {class_name} for the class name. If None, a default naming scheme will be used where: - If the module already starts with "transformers.", it will use that directly - Otherwise, it will prepend "transformers." to the module name Returns: Class decorator function c|SNr)clss rzattach_tracer..2s3rcpjtjfd}|_S)Nc|g|i|j}j}"|jdr|d|}nd|d|}nj||}t ||_y)N transformers..)module class_name)rr startswithformatr"tracer) selfargskwargs module_namer- tracer_namer& original_inittracer_name_templates rinit_with_tracerz:attach_tracer..decorator..init_with_tracer7s $ 0 0 0..K))J#+))/:%0M:,"?K$1+a |"LK299Yc9d $[1DKr)__init__ functoolswraps)r&r8r6r7s` @r decoratorz attach_tracer..decorator4s5  ' 2 ( 2 (  r)_has_opentelemetry)r7r<s` r attach_tracerr>s$ . r) span_name standaloneadditional_attributesrAc.fd}||S||S)a Decorator to trace function calls with OpenTelemetry. Can be used as @traced or @traced(span_name="custom_name") Args: func: The function to trace span_name: Optional custom name for the span (defaults to function name) standalone: If True, creates a parentless span additional_attributes: Optional list of additional attributes to set on the span. Each item is a tuple of (instance_attribute_name, span_attribute_key, value_or_transform_function) where: - instance_attribute_name: Name of the attribute to get from the class instance - span_attribute_key: Key to use when setting the attribute on the span - value_or_transform_function: Either a raw value to use directly, or a function to transform the attribute value before setting it on the span Returns: Decorated function with tracing c\tsSddl}|jfd}|S)Nrc @|rtdrj|dnd}|du}|rt|dr |j}n%tdjdj }xs j }r |j n |j}||5}|jdj |jdj|jd||r{t|D]m\}} t| ttttfs| |jd |t| G|jd |tt| o|r|j!D]m\} } t| ttttfs| |jd | t| G|jd | tt| orP|rND]I} | \} }}t|| st#|| }t%|r ||}n|}|j||K |i|}|cdddS#t&$r?}|j)t+t,j.|j1|d}~wwxYw#1swYyxYw) N__self__rr0r*r+z function.namezfunction.modulezfunction.is_methodzargs.zkwargs.)hasattrrEr0r"rr start_spanstart_as_current_span set_attribute enumerate isinstancestrintfloatbooltypeitemsgetattrcallable Exception set_statusr r!ERRORrecord_exception)r2r3instance is_methodr0namespan_fnspaniargkeyvalue attr_configinstance_attribute_namespan_attribute_keyvalue_or_transform_functionattribute_valuetransformed_valueresulterAfuncr?r@s rwrapperz*traced..decorator..wrapperps|"&GD*,E$--JctAwjnH ,IWXx8!#mDOO3DAdmm_$UV- D+5f''6;W;WG# $""?DMMB""#4dooF""#7C"+D/L3%cCeT+BCs{ ..qc{CHE ..qc{CS NK L &,llnR U%ec3t-DE ..E K ..DK@PQ R )Y'<V cn`/1CE`"8-DE.5h@W.XO'(CD4OP_4` 14O 1 ../ACTUV!4262F!?# # @!OOF:+;+;$<=))!,A# # s1E%J?6J6 I  J:J  JJJ)r=r:r;)rir:rjrAr?r@s` rr<ztraced..decoratorjs3!K  .  . `rr)rir?r@rAr<s ``` rtracedrkNs 87r | T?rceZdZdZdefdZdZedede ddfd Z ed e ddfd Z edd Z ed ededdfdZedede ddfdZy)ContinuousBatchProcessorMetricsz0Metrics collection for ContinuousBatchProcessor.max_batch_tokensc2||_|jy)zInitialize metrics for continuous batch processor. Args: max_batch_tokens: Maximum number of tokens in a batch N)rn_setup_metrics)r1rns rr9z(ContinuousBatchProcessorMetrics.__init__s !1 rcJtstjdytjd|_gd}|j j ddd||_|j jd d d |_ |j jd dd |_ gd}|j j ddd||_ |j jddd |_ |j jddd |_|j jddd |_gd}|j j ddd||_|j jddd |_|j jd!d"d |_y)#zIInitialize OpenTelemetry metrics and tracing if the library is available.zIOpenTelemetry is not installed. Metrics and tracing will not be recorded.Nz2transformers.generation.continuous_batch_processor) 2Kdi,i'ttft_millisecondsz#Time to first token in millisecondsms)rZ descriptionunit#explicit_bucket_boundaries_advisoryactive_requests_countz3Number of active requests currently being processedrequests)rZrrwaiting_requests_countz*Number of requests waiting to be processed) rtrvryrzr{r|r}i Ni0ui`request_latency_millisecondsz9End-to-end latency for completed requests in millisecondsdecode_prefill_ratioz3Ratio of decode tokens to prefill tokens in a batchratioprefill_tokens_processedz"Number of prefill tokens processedtokensdecode_tokens_processedz!Number of decode tokens processed) rr(rt<FPZ_brvbatch_fill_percentagez5Percentage of max_batch_tokens utilized in each batchpercentkv_cache_free_memory_bytesz/Free memory of the PagedAttentionCache in bytesbyteskv_cache_memory_bytesz0Memory usage of the PagedAttentionCache in bytes)r=loggerinfor get_metermetercreate_histogramttft_histogram create_gaugeactive_requests_gaugewaiting_requests_gaugerequest_latency_histogramdecode_prefill_ratio_gaugecreate_counterprefill_tokens_counterdecode_tokens_counterbatch_fill_percentage_histogramkv_cache_free_memory_gaugekv_cache_memory_gauge)r1 ttft_bucketslatency_bucketsbatch_fill_bucketss rrpz.ContinuousBatchProcessorMetrics._setup_metricss" KKc d &&'[\ _ "jj99$=0< : &*ZZ%<%<(M&=& " '+jj&=&=)D'>' #\)-)D)D/S0? *E* &+/***A*A'M+B+ ' '+jj&?&?+<'@' # &*ZZ%>%>*;&?& "R/3zz/J/J(O0B 0K0 ,+/***A*A-I+B+ ' &*ZZ%<%<(J&=& "r created_time request_idreturnNctsytj|z dz} |jj|tj d|d|ddy#t $r"}tjd|Yd}~yd}~wwxYw)zRecord Time to First Token (TTFT). Args: created_time: The time the request was created request_id: The ID of the request N@@zRecorded TTFT for request : .2frzFailed to record TTFT metric: )r=timerrecordrdebugrTwarning)r1rrttft_msrhs rrecord_ttft_metricz2ContinuousBatchProcessorMetrics.record_ttft_metric s" 99;-7 A    & &w / LL5j\GC=PRS T A NN;A3? @ @ A8A B%BBrequests_in_batchc tr|syd}d}|D]j}|jtjk(r|dz }&|jtjtj fvsS|t |jz }l||z} |dkDr|jj||dkDr|jj||dkDr ||z }|jj|||jz dz}|jj|t j#d|d|d|dd |d |jd y#t$$r"}t j'd |Yd}~yd}~wwxYw) zRecord metrics about the batch composition including decode/prefill ratio and batch fill percentage. Args: requests_in_batch: List of request states in the current batch NrgY@zBatch metrics: z decode tokens, z prefill tokens, batch fill: rz% (/)z Failed to record batch metrics: )r=statusr rrrlen prompt_idsraddrrsetrnrrrrrTr) r1r decode_tokensprefill_tokensstatetotal_batch_tokensrfill_percentagerhs rrecord_batch_metricsz4ContinuousBatchProcessorMetrics.record_batch_metricss"):  & 8E||}555" -":":MContinuousBatchProcessorMetrics.record_kv_cache_memory_metricsFs" ' M#..U5G5G1HHOU__-J(-{{u}}enn6U'U![\  !"""#++,.. !   & & e(()*""#++,.. !   & &   & & * *< 8  + + / /0A B LL#LK$@#EF /0%2B2B1CD#e&6&66rtuplerLrk getLoggerrrrmrrrrs 11  D %BB ,` W_c W $DsCsHcUTWZDX?X9Y/Y)Z$[\ Wt   8 $tOtOtOcsBBB