rh[ddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl m Z mZddlmZmZddlmZddl mZddlmZmZmZmZddlmZddlmZdd lmZddlZdd l m!Z!m"Z"dd l#m$Z$m%Z%m&Z&m'Z'm(Z(d d l)m*Z*m+Z+m,Z,m-Z-m.Z.d dl/m0Z0m1Z1d dl2m3Z3m4Z4ddl)m5Z5e3rddl6Z6ddlm7Z7m8Z8m9Z9m:Z:e%rddl;Z;e'xre$xre(xre&Zm?Z?m@Z@ddlAmBZBddlCmDZDmEZEddlFmGZGddlHmIZIddlJmKZKddlLmMZMmNZNmOZOmPZPmQZQddlRmSZSddlTmUZUmVZVmWZWmXZXmYZYmZZZm[Z[m\Z\m]Z]m^Z^m_Z_m`Z`maZambZbmcZcddldmeZeddlfmgZgmhZhmiZiGddeed !ZjGd"d#eSd !ZkGd$d%eId !ZlehejZmehekZnehelZohd&Zphd'Zqhd(Zre4jetZud)d*d+d,iZvewevjZyGd-d.ejZ{d/efd0Z|d1e}d2d3d4d3fd5Z~Gd6d7ZGd8d9ZeGd:d;ZGd<d=e5Zetd>k(reZej yy)?N)ArgumentParser Namespace) dataclassfield)BytesIO)Thread) GeneratorIterableOptionalUnion) model_info)HF_HUB_OFFLINE)Image)!MODEL_FOR_CAUSAL_LM_MAPPING_NAMES*MODEL_FOR_IMAGE_TEXT_TO_TEXT_MAPPING_NAMES)is_fastapi_availableis_librosa_availableis_openai_availableis_pydantic_availableis_uvicorn_available) AutoConfigLogitsProcessorListPreTrainedTokenizerFastProcessorMixinTextIteratorStreamer)ContinuousBatchingManager RequestStatus)is_torch_availablelogging)BaseTransformersCLICommand) AutoProcessorBitsAndBytesConfigGenerationConfigPreTrainedModel)FastAPI HTTPException)CORSMiddleware) JSONResponseStreamingResponse) Transcription)TranscriptionCreateParamsBase)ChatCompletionMessageParam)ChatCompletionChunkChoice ChoiceDeltaChoiceDeltaToolCallChoiceDeltaToolCallFunction)CompletionCreateParamsStreaming)ResponseResponseCompletedEventResponseContentPartAddedEventResponseContentPartDoneEventResponseCreatedEvent ResponseErrorResponseErrorEventResponseFailedEventResponseInProgressEventResponseOutputItemAddedEventResponseOutputItemDoneEventResponseOutputMessageResponseOutputTextResponseTextDeltaEventResponseTextDoneEvent)ResponseCreateParamsStreaming) BaseModel TypeAdapterValidationErrorceZdZUdZeed<y))TransformersResponseCreateParamsStreamingz OpenAI's ResponseCreateParamsStreaming with an additional field for the generation config (as a json string). generation_configN__name__ __module__ __qualname____doc__str__annotations__p/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/transformers/commands/serving.pyrIrIr rSrIF)totalceZdZUdZeed<y)+TransformersCompletionCreateParamsStreamingz OpenAI's CompletionCreateParamsStreaming with an additional field for the generation config (as a json string). rJNrKrRrSrTrXrXyrUrSrXc:eZdZUdZeed<eed<dZee ed<y)%TransformersTranscriptionCreateParamsz OpenAI's TranscriptionCreateParamsBase with an additional field for the generation config (as a json string). filerJFstreamN) rLrMrNrObytesrQrPr\r boolrRrSrTrZrZs"  !&&rSrZ> textuserstorepromptinclude reasoning background truncation tool_choice service_tier top_logprobsmax_tool_callsprevious_response_id>nstopr`audioralogprobsmetadata functions modalities predictionrgrhri function_callstream_optionsresponse_formatpresence_penaltyreasoning_effortweb_search_optionsparallel_tool_callsmax_completion_tokens>rbrclanguagervchunking_strategytimestamp_granularitiesqwenz z )startendceZdZdZdZdZdZy)ModalityLLMVLMSTTTTSN)rLrMrNrrrrrRrSrTrrs C C C CrSrargsct|S)z~ Factory function used to instantiate serving server from provided command line arguments. Returns: ServeCommand ) ServeCommand)rs rTserve_command_factoryrs  rSreqmodel_generation_configr%returnc `|jd"td itj|d}nt j |}|j d i|}|jD]\}}| t||||jdt|d|_ |jdt|d|_ |jdt|d|_ |jd |d|_ |jd |d|_|jd+t|d|_t|ddk(rd |_|jd t|d |_|jd t%j&|d |S) a Creates a generation config from the parameters of the request. If a generation config is passed in the request, it will be used as a baseline for parameterization. Otherwise, we will use the model's default generation config. Other parameters in the request will be applied on top of the baseline. Args: req (`dict`): The request which may optionally contain generation parameters. model_generation_config (`GenerationConfig`): The model's default generation config. kwargs (`dict`): Additional parameters to set in the generation config. Returns: The prepared `GenerationConfig` object. rJmax_output_tokens max_tokensfrequency_penalty logit_biasrm temperaturegFtop_pseedrR)getr%jsonloadscopydeepcopyupdateitemssetattrintmax_new_tokensfloatrepetition_penalty sequence_bias stop_stringsr do_samplertorch manual_seed)rrkwargsrJnon_standard_kwargskvs rT!create_generation_config_from_reqrs. ww"#/,Ttzz#>Q:R/ST MM*AB2+22Reset the tool call state (assumes we're outside a tool call).FrN)inside_tool_callhas_tool_name_definedarg_nesting_levelbufferrs rTrzToolState.resets! %%*"!" rSN)rLrMrNrOrrrRrSrTrrsArSrc FeZdZdZ d dddedeedfdZd Zd Z d Z y) TimedModelz A class that holds a PreTrainedModel instance and its associated processor. Automatically deletes the instances after a specified timeout. Nmodelr&timeout_seconds processor)rrc||_t|j|_||_||_t j|j |j|_ |jjyr) rrP name_or_path _name_or_pathrr threadingTimer _delete_model_timerr)rrrrs rTrzTimedModel.__init__-s[   !3!34".ood&:&:DDelete the wrapped model and processor and clean up resources.rNz was removed from memory after z seconds of inactivity) hasattrrrgccollectrcuda is_available empty_cacheloggerinforrrs rTrzTimedModel._delete_model@s 4 !djj&< DJ!DN JJLzz&&( &&( KK%%&&EdFZFZE[[qr '= !rSc<t|d xs|jduS)z)Check if the instances have been deleted.rN)rrrs rT is_deletedzTimedModel.is_deletedQs 4))?TZZ4-??rSr) rLrMrNrOrr r rrrrrRrSrTrr'sJSW    E"MNO  "@rSrcVeZdZUdZedddiZeed<eddgdd Ze eed <ed dd iZ e ed <edddiZ e eed<ed ddiZ e ed<ed ddiZe ed<eddddgd Zeed<ed ddiZe ed<edddiZeed<edddiZeed <ed!dd"iZeed#<ed$dd%iZeed&<eddd'iZe eed(<ed dd)iZe ed*<ed dd+iZe ed,<eddd-iZe eed.<y)/ServeArgumentsz Arguments for the serve CLI. See the metadata arg for each argument's description -- the metadata will be printed with `transformers serve --help` autohelpzfDevice to use for inference; will default to `auto` andplace the model on an accelerator if available.)defaultrpdevicezOverride the default `torch.dtype` and load the model under this dtype. If `'auto'` is passed, the dtype will be automatically derived from the model's weights.)rbfloat16float16float32)rchoices torch_dtypeFz2Whether to trust remote code when loading a model.trust_remote_codeNzWhich attention implementation to use; you can run --attn_implementation=flash_attention_2, in which case you must install this manually by running `pip install flash-attn --no-build-isolation`.attn_implementationzIWhether to use 8 bit precision for the base model - works only with LoRA. load_in_8bitzIWhether to use 4 bit precision for the base model - works only with LoRA. load_in_4bitnf4zQuantization type.fp4bnb_4bit_quant_typez#Whether to use nested quantization.use_bnb_nested_quant localhostz$Interface the server will listen to.hosti@zPort the server will listen to.porti,z@Time in seconds after which a model will be removed from memory. model_timeoutrz8Logging level as a string. Example: 'info' or 'warning'. log_levelz1The default seed for torch, should be an integer. default_seedztWhether to enable CORS. Some apps that make requests from external domains (e.g. Cursor) require CORS to be enabled. enable_corsz+Whether to turn on strict input validation.input_validationzName of the model to be forced on all requests. This is useful for testing Apps that don't allow changing models in the request. force_model)rLrMrNrOrrrPrQrr rr^rrrrrrrrrrrrrrrRrSrTrrVs  > FC"'PA "K#$)] ^t*/ r *#efL$efL$ %UFZhmotgu=vww!&uHm?n!o$okV=c4deD#edf6W-XYD#Y\]M3 &*d!eIs#(([\#L(3- & K# B d "' 2 "K#rSrcDeZdZedefdZdefdZdedddd d e fd Z defd Z defd Z defdZ d0deedeedeedeedeedeeddefdZdddefdZdZej*ddeeeeffdZdedeeddffd Zededefd!Zed"efd#Zdedeeddffd$Zdedeeddffd%Zdedeeddffd&Z dede!fd'Z"ededed(fd)Z#d*edefd+Z$d,efd-Z%d,ede&ee'ffd.Z(d,ede&ee)ffd/Z*y)1rparsercdtf}|jd|}|jty)z Register this command to argparse so it's available for the transformer-cli Args: parser: Root parser to register command-specific arguments serve)dataclass_types)funcN)r add_parser set_defaultsr)rr serve_parsers rTregister_subcommandz ServeCommand.register_subcommands3*+((/(R !!'<!=rSrcts td||_|jjdk(|_|jj |_|jj )tj|jj tjd}|jtj|jjjtjd}|jtj|jjji|_d|_d|_d|_d|_y)NzaMissing dependencies for the serving CLI. Please install with `pip install transformers[serving]` sdpa_paged transformersz+transformers.generation.continuous_batching)serve_dependencies_available ImportErrorrruse_continuous_batchingrrrrr get_loggersetLevel log_levelsrlower loaded_models#running_continuous_batching_manager last_messages last_kv_cache last_model)rrtransformers_logger cb_loggers rTrzServeCommand.__init__s+s   '+yy'D'D 'T$9900 99 ! ! -   dii44 5&00@$$W%7%7 8K8K8Q8Q8S%TU&&'TU 7--dii.A.A.G.G.IJK57X\0"!rSrequestschema_TypedDictMeta validatorrF unused_fieldsc0tjd|t|j}|j}||z }|r(tj d|t dd||jjrB |j|||z} | r(tj d| t dd| yy#t$rF}tj d|jt d|jd}~wwxYw)a Validates the request against the schema, and checks for unexpected keys. Args: request (`dict`): The request to validate. schema (`_TypedDictMeta`): The schema of the request to validate. It is a `TypedDict` definition. validator (`TypeAdapter`): The validator to use to validate the request. Built from `schema`. unused_fields (`set`): Fields accepted by `schema`, but not used in `transformers serve`. Raises: HTTPException: If the request is invalid or contains unexpected or unused fields. zValidating request: z Unexpected keys in the request: i) status_codedetailzValidation error: NzUnused fields in the request: ) rdebugsetkeys__mutable_keys__errorr(rrvalidate_pythonrGerrors) rr r rr input_keys possible_keysunexpected_keyseunused_fields_in_requests rT_validate_requestzServeCommand._validate_requests.  +G956( // $}4  LL;O;LM NC:Z[jZk8lm m 99 % % H))'2 (2M'A $' =>V=WXY# #.LMeLf,g( &# H 1!((*>?#AHHJGG HsC DADDcF|j|tttyN)r r rr)r rIresponse_validatorUNUSED_RESPONSE_FIELDSrr s rTvalidate_response_requestz&ServeCommand.validate_response_requests! <(0  rScF|j|tttyr")r rXcompletion_validatorUNUSED_CHAT_COMPLETION_FIELDSr%s rT validate_chat_completion_requestz-ServeCommand.validate_chat_completion_requests! >*7  rScF|j|tttyr")r rZtranscription_validatorUNUSED_TRANSCRIPTION_FIELDSr%s rTvalidate_transcription_requestz+ServeCommand.validate_transcription_requests! 8-5  rSN request_idcontentrrole finish_reason tool_callsr2rc t|ttj|tt |||d|gdd}d|j d d S) a Builds a chunk of a streaming OpenAI Chat Completion response. IMPORTANT: The serialized chunk won't contain empty fields (fields with `None`). Some downstream apps, like Cursor, assume that when the field exists, it has data. Args: request_id (`str`): The request ID. content (`str`, *optional*): Content of the response from the model. model (`str`, *optional*): The model that generated the content. role (`str`, *optional*): The role of the next content, until a new role is defined. finish_reason (`str`, *optional*): The reason the generation by the model has finished. tool_calls (`list[ChoiceDeltaToolCall]`, *optional*): Data about the tool calls, when they are triggered. Returns: `str`: The built chunk, a string containing a JSON string with the payload. )r0r1r3r)deltaindexr2rzchat.completion.chunk)idcreatedrrsystem_fingerprintobjectdata: T exclude_none )r/rtimer0r1model_dump_json)rr/r0rr1r2r3chunks rTbuild_chat_completion_chunkz(ServeCommand.build_chat_completion_chunk'sq@$ $% '!#- "/  "*! $--4-@AFFrSresponserEc.d|jddS)a Builds a event of a streaming OpenAI Response response. IMPORTANT: The serialized chunk won't contain empty fields (fields with `None`). Some downstream apps, like Cursor, assume that when the field exists, it has data. Args: response (`BaseModel`): The response to build an event from. One of the multiple OpenAI Response output types Returns: `str`: The built chunk, a string containing a JSON string with the payload. r;Tr<r>)r@)rrCs rTbuild_response_eventz!ServeCommand.build_response_event[s"00d0CDDIIrSc4t}jr|jtdgddgdg|j ddt ffd }|j ddt ffd }d d lm}|j d d|ffd }|jd |jd fd}tj|jjjjjjy)N*T) allow_originsallow_credentials allow_methods allow_headersz/v1/chat/completionsr cj|jrj|}nj|}t |dSNr text/event-stream media_type)r*r#continuous_batching_chat_completiongenerate_chat_completionr+r outputrs rTchat_completionz)ServeCommand.run..chat_completionysK  1 1' 1 B++AA'J66w?$V8KL LrSz /v1/responsescdj|j|}t|dSrM)r&generate_responser+rTs rT responsesz#ServeCommand.run..responsess2  * *7 * ;++G4F$V8KL LrSr)Requestz/v1/audio/transcriptionsc K|j4d{}t|djd{|d}tj d|dj d|dj d|djdz dd dddd{j j|}t|d S7778#1d{7swYHxYww) Nr[r)r[rzReceived file: z ; MIME type: z; size: z.2fz KiBrNrOrP) formrZreadrrfilename content_typesizer.generate_transcriptionr+)r r]parsed_requestrUrs rTaudio_transcriptionsz.ServeCommand.run..audio_transcriptionss||~  !F#F|0022w-"  %d6l&;&;%.get_all_modelss 64;N;N;P QR RrS)rrr)r'radd_middlewarer)postdictfastapirZoptionsruvicornrunrrrr)rapprVrYrZrdris` rTrpzServeCommand.runksi      "e"&"e"e   ( ) MT M * M / " Mt M # M $ , - M M . M" \ "   S  # S  Cdiinn499>>TYYM`M`arS)maxsizec gd}trQ|Dcgc]E}|dtjjj|j dddGc}S|Dcgc] }t |}}|Dcgc]5}|j d|jj|jd7c}Scc}wcc}wcc}w)a. This is by no means a limit to which models may be instantiated with `transformers serve`: any chat-based model working with generate can work. This is a limited list of models to ensure we have a discoverable /v1/models endpoint for third-party integrations. ) zMenlo/Jan-nanozMenlo/Jan-nano-128kzQwen/Qwen2.5-0.5B-InstructzQwen/Qwen2.5-3B-InstructzQwen/Qwen2.5-7B-InstructzQwen/Qwen2.5-14B-Instructz meta-llama/Llama-3.1-8B-Instructz meta-llama/Llama-3.2-1B-Instructz!meta-llama/Llama-3.3-70B-InstructzHuggingFaceTB/SmolVLM-Instructz!ibm-granite/granite-vision-3.2-2bzQwen/Qwen2.5-VL-7B-Instructr/r)r7r:r8owned_by) rdatetimenow timestampsplitr r7 created_atauthor)rmodelsr model_infoss rTrhzServeCommand.get_gen_modelss   $  %'00446@@B % C 0 3  ;AA:e,AKA)  ((%$//99; %  BsA B5 B:8:B?rc  jd jk7} _|r0j$jjddd_j \}}t |dr |j n|}t|j|j|jddd dd d jK|jd _tj_ jj|jdddj!|j"} fd}||dS)a' Generates an OpenAI Chat Completion using continuous batching. Args: req (`dict`): The request to generate an OpenAI Chat Completion for. Returns: `Generator[str, None, None]`: A generator that yields the OpenAI Chat Completion chunks. rNTr)blocktimeout tokenizerFr!r\ fifo) r eos_token_id pad_token_id use_cache num_blocks block_sizermax_batch_tokens scheduler)rJ streamingmessagespt)return_tensorsadd_generation_promptc3K jj|jdj}d} j |d jD]}|j |k7rjd"|s |j tjk(rDd}|j tjk(rdnd}|j tjk(r j ||y j ||j y#t$r9}tjt|d t|d Yd}~yd}~wwxYww) Nr/)r/rF assistantr1rTrmr2r)r/r0rdata: {"error": ""})r add_requestrrrBr/statusrFINISHED next_token ExceptionrrrP) _inputsr/queue_is_flushedresultr2rrJmodel_id_and_revisionrrs rTstream_chat_completionzPServeCommand.continuous_batching_chat_completion..stream_chat_completions_! 7!EEQQ (=N_NnNnR $) 66z [p6qq"FFF((J6 ww|,8AQ!==M,B,BB$/3,.4mm}?U?U.UF[_M}} (>(>>">>&mK`?">>'16;L;LTi?!( 7 SV$*3q6(#66 7s;E$C6D;E$<"DE$ E!(/EE$E!!E$r)process_model_namer rrmload_model_and_processorrrrrJrrinit_continuous_batchingrlogit_processorrapply_chat_templatetor) rrmust_discard_cacherrrinputsrrJrs `` @@rTrRz0ServeCommand.continuous_batching_chat_completionsc!% 7 7G E2dooE/ 77C88==DRS=T;?8889NOy+29k+JI''PY = $)$;$;"//"//    3 3 ;7<7U7U"3t8V8D 4 H[G\D 4 4 D  4 4 : : <..s:tko.pss LL " 7H&fQi00rSc|jj}|tjvrtj }|S|t jvrtj}|Std|)NzUnknown modality: ) __class__rLrvaluesrrrr ValueError)rmodel_classnamemodalitys rTget_model_modalityzServeCommand.get_model_modality,sm//22 HOOQ Q||H   A H H J J||H1/1BCD DrSrc g}|D]~}|dgd}|tjk(r't|dtr|dn|dd}||d<n(|tjk(rt|dtr|dj d|ddn|dD]}|ddk(r|dj | |ddk(s)d|dd vrt jd d |dd }tjttj|}tjd d }|j} |j!|jn|dd } |dj d| d|j ||S)Nr1r1r0r0r_)typer_r image_urlbase64urlz^data:image/.+;base64,rz.pngF)suffixdeleteimage)rr)rr isinstancerPrappendresubropenrr b64decodetempfileNamedTemporaryFilenamesave) rrprocessor_inputsmessageparsed_messager0 image_datarr[rs rT*get_processor_inputs_from_inbound_messagesz7ServeCommand.get_processor_inputs_from_inbound_messages8s 4G&-fo"EN8<<'0:79;Ms0S'),Y`ajYklrYs,3y)X\\)gi0#6"9-44fgV_N`5ab#*9#5\"6?f4*95< rScjjjj|d<|d}|dddk(ryj|djk7}_j \}j }j ||}dtD]/}|jjdjvs-|n|j|d|jd dd dd }|jj}|jd d d} djjdjvrd} t|| d} t!|j"} d} j%|r|s j&} i|| | d| dfd} | | S)a Generates an OpenAI Chat Completion using `generate`. Args: req (`dict`): The request to generate an OpenAI Chat Completion for. Returns: `Generator[str, None, None]`: A generator that yields the OpenAI Chat Completion chunks. Nrrr1rrTtoolsr)rrr return_dicttokenizer/req_0gptossFskip_special_tokens skip_promptr)streamerrJreturn_dict_in_generatepast_key_valuesc3Kd}d}d jjdjvrd}d} fd}t| }d} |j t }j d  |D]2}d jjdjvr |jd r|dtd  }||z }|r||vrd}[\|jtd k(rd|_ |jtd k(r(|jj |dd|jr@|xj|z c_ |jsYtj d|j} | | j#d} d|_t%t'| dd|dz} n|dk(rWd|jvrg|xj(|j+dz c_|xj(|j+dzc_|j(dkr&dj-|j/ddddz}t%t'|dd} j |d| g|dk7sj ||5j |d|j-|j-y#t0$r9} t2j5t7| d t7| d!Yd} ~ Nd} ~ wwxYw#|j-wxYww)"NFrrT<|channel|>final<|message|>cLjdi|}|j_yNrRgeneraterrrgenerate_outputrrs rTgenerate_with_cachezbServeCommand.generate_chat_completion..stream_chat_completion..generate_with_cache$"0%..":6":%4%D%D"rStargetrrrr <|return|>rrr3)r/r1r2rz\"name\": \"(.*?)\"r!)rfunction _tool_call)rr6rr7z"arguments": {{}) arguments)rr6r)r/r1r3r)r0rrmrrr)config architecturesrrrrrBendswithlenstrip_TOOL_CALL_TOKENSrrrrrsearchgroupr2r3rcountjoinryrrrrP)r _request_id filter_cot cot_trace_endrthreadresults tool_stater tool_nametoolrgeneration_kwargsrrr/rtool_model_familys rTrzEServeCommand.generate_chat_completion..stream_chat_completions_J M5<<55a8>>@@! =  E#6?PQFGh  &[ 66z [p6qq&WF5<<#=#=a#@#F#F#HH!??<8%+,@s->?P-QRY-ZZ:>J7$"<<>->?P-QRW-XX&,,."&"B"B+6%).:&; #C#%%66&--7-$.#C#C,.II6LjN_N_,` #,#4$,090BICG @':-Hi-X*+)3'2\'A ("$*R<$,$4:;L;L#L$,!+ < < S@Q Q < * < < S@Q Q <#-#?#?!#C-/WWV\\#5Fs5K-Ls-RF':-HSY-Z*+)3(" #'"B"B+6Ttf\q#C#%|">>'?T?kWp66{RX`u6vv   7 SV$*3q6(#66 7  sCAM I K*?K*M* L,3/L'"L/'L,,L//MM)rrrr rrr_MODELS_WITH_TOOL_SUPPORTrrrrrrrrrrJis_continuationr)rrrrrrrsupported_model_familiesrrgeneration_streamerrJrrrrrr/rs` @@@@@rTrSz%ServeCommand.generate_chat_completion]s 99 ,9900CL9c[`[r[rs    $-? ..M  +!2'+,  y y v&&9:FFrSc jd jk7} _j \ }tdtr'dvr dddgng}|j dddntdt r8dvr.ddddk7rdddgd}nYd}d|dd <nHd}nBtdtr$dvr dddgng}|j dn td |j|d d }|j j}jdd d }d jjdjvrd}t!||d }t# j$}d}j'r|s j(}|t+j,|||d |d  fd} | | S)a  Generates an OpenAI Response using `generate`. Args: req (`dict`): The request to generate an OpenAI Response for. Returns: `Generator[str, None, None]`: A generator that yields the OpenAI Response events. rinput instructionssystemrr`rr1r0z%inputs should be a list, dict, or strTr)rrrkrrFrrN)rattention_maskrrJrrc3L Kd}d}djjdjvrd}d}fd}t|}d}d}d} |j t j } t d|td | d jd d d diidggjdddjd } |dz }j| td|td | djd d d diidggjdddjd } |dz }j| td||tddddg} |dz }j| tdd|||tdd g!"} |dz }j| d }|D]}djjdjvr |jd#r|dt!d# }||z }|r ||vrd}d }\]t#d$d||||d d%d&g'}|dz }j|t%d(d||d|d d%d&g)}|dz }j|t'd*d|||td|j(g!"}|dz }|dz }j|t+d+||tddd,d|j,gg-}|dz }|dz }j|t/d.|td | d,jd d d dii|j0gdgjdddjd/ }|dz }j||j3|j3y#t4$r}t6j9d0t;|t=d1|t;|2}|dz }j|t?d3|td  d4jd d d diigdgddjdtAd5t;|67 }|dz }j|Yd}~d}~wwxYw#|j3wxYww)8NFrrTrcLjdi|}|j_yrrrs rTrzTServeCommand.generate_response..stream_response..generate_with_cachenrrSrzresponse.createdresp_queuedrformatrr_rCrzrrp) r7rzrrrr_r:rrUrzrgrp)rsequence_numberrCr!zresponse.in_progress in_progresszresponse.output_item.addedmsg_rr)r7rrr1r0)rr  output_indexitemzresponse.content_part.added output_textr)rr_ annotations)ritem_idr r  content_indexpartrzresponse.output_text.deltagX@)tokenlogprob)rrr r rr5rozresponse.output_text.done)rrr r rr_rozresponse.content_part.donezresponse.output_item.done completed)r7rrr1r0rzresponse.completed) r7rzrrrr_rUr:rrzrgrpz"Exception in response generation: r)rr rzresponse.failedfailed server_error)coder) r7rzrrrr_rUr:rrzrgrpr)!rrrrrr?r9r5rrEr=r>r@r7rArrrBrCr8r_r?rr6rrrrrrPr;r<r:)rrrrrrr r rrzresponse_createdresponse_in_progressresponse_output_item_addedresponse_content_part_addedrrresponse_output_text_deltaresponse_output_text_doneresponse_content_part_doneresponse_output_item_doneresponse_completedr error_eventresponse_failedrrrrr/rs rTstream_responsez7ServeCommand.generate_response..stream_responsedsJ M5<<55a8>>@@! =  E#6?PQFOLMN  !YY[ $8+$3%":,/#-'3%(WW^%<&(89) !,/GG4I5,Q$*!$!4 $ $ 1$//0@AA'>/$3%":,/#-,3%(WW^%<&(89) !,/GG4I5,Q$*!$!4 ($$ 1$//0DEE.J5$3!-.!*.Y}[fpr .* 1$//0JKK/L6":,/$3!-"/+RUWX /+ 1$//0KLL&PF5<<#=#=a#@#F#F#HH!??<8%+,@s >?) ,/GG4I5,Q$*!$!4 &"$ 1$//0BCC J I! A A#a&JK0 $3F  1$// <<"5*$3%":,/#-'3%(WW^%<&(89!) ,1$*!$!4+!/$'F#, 1$//@@C! AH s>A R$MN4#R$4 R =CRRR  RR!!R$)rr rrrPrrfrlrrrrrrrrrrrJrrr ones_like)rrrrrrrrJrr%rrrr/s`` @@@@rTrXzServeCommand.generate_response%s1!% 7 7G E2dooE/889NOy c'lC (M[_bMbxC4GHIhjF MM6c'lC D G d +$w<?6*h6'/CF1Ii(W G d +M[_bMbxC4GHIhjF MM#g, 'DE E..vTbf.g5<<(WW3W= # u||11!4::< <"' 2  3  >c[`[r[rs    $-? ..M#oof5+!2'+,  a a F2J??rSc( ts td|j|d}|j|\ t j dd}t | j} jj}tj|d}tj||d\}} ||d j j d j j  d <||dd  fd } | S) a Generates an OpenAI Transcription using the audio file. Args: req (`dict`): The request containing the audio file and model information. Returns: `Generator[str, None, None]`: A generator that yields the transcription result. z]Missing librosa dependency for audio transcription. Please install with `pip install librosa`rTrrr[)srmonor) sampling_raterinput_features)rrJrc3Kjdi}j|jdd}t|}|j dyw)NT)rr)r_r<rR)r batch_decode sequencesr,r@) generated_idstranscription_text transcription audio_inputs audio_modelaudio_processorrs rT_generate_transcriptionzDServeCommand.generate_transcription.._generate_transcriptionqsg0K00U<UCTUM!0!=!=m>U>Uko!=!pqr!s )/ABM"222EF GsAA)rrrload_audio_model_and_processorrrrrJfeature_extractorr*iorlibrosaloadrrdtype)rrrrrJmodel_sampling_rate audio_bytes audio_array_r5r2r3r4rs @@@@rTrbz#ServeCommand.generate_transcriptionIs"$%o !% 7 7G E'+'J'JK`'a$ _2  % %4T > )F)F  .??MMjjV-  k6IPTU Q&{BUfjknn    *66F)G)J)J;K\K\)] %&,!2'+   H '((rSc>|jdxs|jd}d}|jd}n`t|jt|k\rd}n p5CZCZCiCilpCp ! "%=59E # # 2 m$9#:;<n$$rSc||jvs|j|jrG|j|\}}t||jj ||j|<||fS|j|j |j|j}|j|j}||fS)a\ Loads the text model and processor from the given model ID and revision into the ServeCommand instance. Args: model_id_and_revision (`str`): The model ID and revision to load. Returns: `tuple[PreTrainedModel, PreTrainedTokenizerFast]`: The loaded text model and processor. rr rrr[rrrrrr)rrrrs rTrz%ServeCommand.load_model_and_processors !(:(: :d>P>PQf>g>r>r>t#BBCXY E98B $ 7 7#9D  4 5i   4 5 A A C&&'<=CCE**+@AKKIirSc||jvs|j|jrG|j|\}}t||jj ||j|<||fS|j|j |j|j}|j|j}||fS)aU Loads the audio model and processor from the given model ID and revision into the ServeCommand instance. Args: model_id_and_revision (`str`): The model ID and revision to load. Returns: `tuple[PreTrainedModel, ProcessorMixin]`: The loaded audio model and processor. r]r^)rrr3r4s rTr6z+ServeCommand.load_audio_model_and_processors !(:(: :d>P>PQf>g>r>r>t+/+N+NOd+e (K8B $ 7 7)9D  4 5O++   4 5 A A C,,-BCIIK"001FGQQOO++rS)rNNNNN)+rLrMrN staticmethodrrrrrlrr r&r*r.r rPrfrBrErp functools lru_cacheanyrhr rRr&rrrrSrXrbr^rrIrr[tuplerrrr6rRrSrTrrs >N > >^@//!/! /  /b     d %'!%#"'+<@2GSM2G#2G} 2G sm 2G  } 2GT"7892G 2GhJ[JSJ 8btY&+T#s(^ 4+'+ZW1tW1 #tUY/@ZW1r / h  " x" " HFGDFGYsD$5OFGPb@Tb@iT4.Hb@H .)$.)9S$_3M.)`+4+D+<#n#BV9W##8"3"3"";%C;%z c eO]tLtFu 2,C,ERacqRqLr,rSr__main__)rrrvenumrarr8rrrrr?argparserr dataclassesrrrrtypingr r r r huggingface_hubr huggingface_hub.constantsrPILrr&transformers.models.auto.modeling_autorrtransformers.utils.import_utilsrrrrrrrrrrrgeneration.continuous_batchingrrutilsrr r"rr#r$r%r&r9rrormr'r(fastapi.middleware.corsr)fastapi.responsesr*r+ openai.types.audio.transcriptionr,.openai.types.audio.transcription_create_paramsr-openai.types.chatr.'openai.types.chat.chat_completion_chunkr/r0r1r2r3*openai.types.chat.completion_create_paramsr4openai.types.responsesr5r6r7r8r9r:r;r<r=r>r?r@rArBrC-openai.types.responses.response_create_paramsrDpydanticrErFrGrIrXrZr#r(r,r$r)r-rrLrrrfrrEnumrrrlrrrrrrrprRrSrTr|sU    .(77&4V/(k 4 6k;O;QkViVk .6A>\<["\@@4QY^6U]b'0MUZ'%%NO&'RS)*OP %!.#   H %    !!2!7!7!9:tyy 8 8/8 8v  ,@,@^ WW WtA,-A,H$ z NE IIKrS