rh\0ddlZddlZddlZddlZej eZejddfdZ ejZ d dZ ejddfdZ d dZ d dZd d Z dd Zdd Zy)NFctjd|j|fztj}|j\}}t j |} t j ||| } t j||} |dk(rt j}|rKtjd|zt j} || _ t j| | |} d} |D]}|jd}| j|| j||\}}|| z }| j||| j| |z } tjd| tj|z fz| j!tjdtj|z | fz| j"| j$fS) zComputes the exact KNN search results for a dataset that possibly does not fit in RAM but for which we have an iterator that returns it block by block. z%knn_ground_truth queries size %s k=%dkeep_maxrrunning on %d GPUscongpur%d db elements, %.3f szGT time: %.3f s (%d vectors))LOGinfoshapetimefaissis_similarity_metric ResultHeap IndexFlat get_num_gpusGpuMultipleClonerOptionsshardindex_cpu_to_all_gpusaddsearch add_resultresetfinalizeDI)xq db_iteratork metric_typerr t0nqdrrhindexr i0xbinirrs r/var/www/html/ai-insurance-compliance-backend/venv/lib/python3.12/site-packages/faiss/contrib/exhaustive_search.pyknn_ground_truthr,s HH 4!} DE B HHEB))+6H   "a( 3B OOA{ +E rz!!#  %,-  + + -++EbtD BD YYq\ #||B"1 R a  b )Rr1A,BBCDKKMHH +tyy{R/?.D DE 44:c P|j\}}t|tj}tj|j }|r t |n t|}t|j|} tjd|d| d|d|tj} |j|| \} } tj| z } |r|dzdksJ| jd} d}d }||s| d d | d z f|k}n| d d | d z f|kD}|jdkDrtjd |jztj} t|t j"rN|}|rtj$|dz}n tj&||j }|j)||j+|||\}}}|r|jd}tj| z }tjd tj} |rtj,ntj.}||| ||} tj0}|| |_|| |_|C||_||_||j;d |_||_t!j@|d zd }|jC|||d}t!j@|| jD}t!j@|d }|jG||||tj| z }!tjd| dd|dd|!dd|||fS)aAGPU does not support range search, so we emulate it with knn search + fallback to CPU index. The index_cpu can either be: - a CPU index that supports range search - a numpy table, that will be used to construct a Flat index if needed. - None. In that case, at most gpu_k results will be returned z GPU search z queries with k=z is_binary_index=z keep_max=iint16rNzCPU search remain %dcombineint64dtyperztimes z.3fzs s)*r isinstancer IndexBinaryrr"intfloatminntotalr debugrrastypesumnpndarrayIndexBinaryFlatrr range_searchCombinerRangeKNNint16CombinerRangeKNNfloatswig_ptrrrmaskD_remainview lim_remainI_remainempty compute_sizesr5 write_resultrange METRIC_L2appendcumsumlenhstack)"rr2 index_gpu index_cpugpu_kr$r%is_binary_indexrr!r#rrt1t2rJrGxbrHrKCombinerRangeKNNcombinerspL_resnresD_resI_resnrinvl0l1dit3s" r+range_search_gpurk:s/ HHEB E,=,=>O)))*?*?@H#RrB I  e$AII bT*!u,>oa-@ (1NP B   B "DAq r B1uu}} HHW  BJQAX;#DQAX;#D 88:> II,txxz9 :B)RZZ0" % 5 5a!e NFB KKM br 'AAYq1u BBw! Br"I&! Br"I&  ' b )Rr1A,BBC#D&hhq(Ghhq *G9:;A!r'"))A,w .;A;9:;A!r'"))A,w .;A; SV E  u:   88BF( +DyyDH 1ryy| ++= ! . <; s L L-#L #L?Lc|r||kD}n||k}tj|}d}t|D].\}} t| } |||| zj ||<|| z }0|||||fS)z select a set of results r)r@ zeros_like enumerater9r?) radisidsthreshrrGnew_nresorerds r+threshold_radius_nresrsV|V|}}T"H A42 W1QVn((*  R SYD ))r-c|r||kD}n||k}tj|}t|dz }t|D].}||||dz} } |||| | j z||dz<0|||||fS)z= restrict range-search results to those below a given radius r1)r@r}rSrOr?) rzrrrrrGnew_limsnrergrhs r+threshold_radiusrsV|V|}}T"H D A A 1X:a$q1u+B"1+R (99Q: SYD ))r-c <tj|Dcgc]\}}}| c}}}t||kDsJ|r)|jt||z dz |d|z }n|j|||}|jdk(r t |}n t |}tjd|zd}t|D]3\}\} }} t| || ||\} }} |t|z }| || f||<5tjd|z||fScc}}w)zfind radius that reduces number of results to target_nres, and applies it in-place to the result batches used in range_search_max_resultsr1rrmz setting radius to %srrz. updated previous results, new nb results %d) r@rTrS partitionr5r:r9r r=r~r) res_batches target_nresr_ralldisradiustotresrerars r+ apply_maxresrs+YY[9 39 :F v; $$ $V{2Q67[()% $ ||y vVII'&01 F(5( D#s. #sFX7c3#c(sC A ( II>GH 6>+:s D c t|tj}||Jtd|z}||Jt|dz}|dk(rtj}|rLt j d|ztj} || _tj|| |} nd} tj} dx} } dx}x}}g}|D]u}tj}t jdt|d | rt||| |\}}}n|j||\}}}|d d|ddz }|t|z }|t|z }tj}|r|jd }|t|z }|j!|||f|K||kDrFt j d ||fzt#|||j$tj&k( \}}tj}| ||z z } | ||z z } t jdtj| z ||fzxt j d| | ||fz|r1||kDr,t#|||j$tj&k( \}}t)j*|Dcgc]\}}}| c}}}}t)j*|Dcgc]\}}}| c}}}}t)j*|Dcgc]\}}}| c}}}}t)j,t|d zd}t)j.||d d||||fScc}}}wcc}}}wcc}}}w)aPerforms a range search with many queries (given by an iterator) and adjusts the threshold on-the-fly so that the total results table does not grow larger than max_results. If ngpu != 0, the function moves the index to this many GPUs to speed up search. Ng?g?rrrrz searching z vectorsr1r0z-too many results %d > %d, scaling back radiusrz' [%.3f s] %d queries done, %d resultszBsearch done in %.3f s + %.3f s, total %d results, end threshold %grnr4)r7rr8r9rr r rrrrr=rSrkrCr>rQrr"METRIC_INNER_PRODUCTr@rTrprR) r'query_iteratorr max_results min_resultsrr clip_to_minrYr rVt_startt_searcht_post_processqtotr raw_totresrxqir#rsrtrunres_irZr[dis_iids_irarrrzs r+range_search_max_resultsrs\!(9(9:O&&&# +, &&&++,  rz!!#  %,-  + + -//"4H  iikG !!H~!""D"6JK2 YY[ Js3xj12 -c69eLNFB"//VAA)i)F)NNFrF) i N)rrnumpyr@logging getLogger__name__r rPr,knnrkr{rrrrrr-r+rs  g!5:__EXZ$N ii [|@E"3,l * *:<@>CY"x r-