o j)@sUdZddlZddlZddlZddlZddlmZmZm Z m Z ddl m Z ddl mZeeeefeeeeeeeeffffZiddddd d d d d ddddddddddddddddddd d!d"d#d$d%d&d'd(d)d*d+d,d-d.d/d0d1d2d3d4 Zeeefed5<d6d7eDZeeefed8<d9ed:efd;d<Zejd:efd=d>Ziddddd dd dddddddddddddd dd"dd$dd&dd'dd)dd(dd d d d d d dddddddd? Zeeefed@<d9edAedBedCeedDeeeeeffd:edBf dEdFZdGedHeeeeedBe effdIed:e!fdJdKZ"d9edAedBedCeed:edBf dLdMZ#dNedAedBed:eeefdBfdOdPZ$dQZ%d9edRee d:ee fdSdTZ&dS)UaConfusion group resolution for similar single-byte encodings. At runtime, loads pre-computed distinguishing byte maps from confusion.bin and uses them to resolve statistical scoring ties between similar encodings. Build-time computation (``compute_confusion_groups``, ``compute_distinguishing_maps``, ``serialize_confusion_data``) lives in ``scripts/confusion_training.py``. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N) BigramProfile get_enc_indexget_idf_weightsscore_with_profile)DetectionResult)lookup_encodingLuLlLtLmLoMnMcMeNd Nl No Pc Pd PsPePiPfPoSmScSkSoZsZlZpCcCfCsCoCn) _INT_TO_CATEGORYcCsi|]\}}||qSrD).0kvrDrD`/var/www/html/fyndo/pharma/fyndo/venv/lib/python3.10/site-packages/chardet/pipeline/confusion.py GsrI_CATEGORY_TO_INTdatareturncCs"i}d}td||\}|d7}t|D]z}td||\}|d7}||||d}||7}td||\}|d7}||||d}||7}td||\} |d7}g} i} t| D]#}td||\} } }|d7}| | t| d t|d f| | <q`t| | f|||f<q|S) zLoad confusion group data from raw bytes. :param data: The raw binary content of a confusion.bin file. :returns: A :data:`DistinguishingMaps` dictionary keyed by encoding pairs. rz!Hr z!Br zutf-8z!BBBr r5)struct unpack_fromrangedecodeappendrCget frozenset)rKresultoffset num_pairs_ name_a_lenname_a name_b_lenname_b num_diffsdiff_bytes_list categoriesbv cat_a_int cat_b_intrDrDrH%deserialize_confusion_data_from_bytesJs4      rbc Cstjdd}|}|stjdtddiSzt|}Wnt j t fy8}z d|}t ||d}~wwi}| D]\\}}}t|pJ|} t|pP|} ||| | f<q?|S)zLoad confusion group data from the bundled confusion.bin file. :returns: A :data:`DistinguishingMaps` dictionary keyed by encoding pairs. zchardet.modelsz confusion.binuZchardet confusion.bin is empty — confusion resolution disabled; reinstall chardet to fixr ) stacklevelzcorrupt confusion.bin: N) importlib resourcesfilesjoinpath read_byteswarningswarnRuntimeWarningrbrMerrorUnicodeDecodeError ValueErroritemsr) refrawraw_mapsemsg normalizedabvaluenorm_anorm_brDrDrHload_confusion_datars,     r{) r,r-r.r/r0r2r1r4r3r5rrr_CATEGORY_PREFERENCEenc_aenc_b diff_bytesr^c Csd}d}t||@}|sdS|D])}||\} } t| d} t| d} | | kr/|| | 7}q| | kr9|| | 7}q||kr@|S||krF|SdS)aResolve between two encodings using Unicode category voting. For each distinguishing byte present in the data, compare the Unicode general category under each encoding. The encoding whose interpretation has the higher category preference score gets a vote. The encoding with more votes wins. :param data: The raw byte data to examine. :param enc_a: First encoding name. :param enc_b: Second encoding name. :param diff_bytes: Byte values where the two encodings differ. :param categories: Mapping of byte value to ``(cat_a, cat_b)`` Unicode general category pairs. :returns: The winning encoding name, or ``None`` if tied. rN)rSr|rR) rKr}r~rr^votes_avotes_brelevantr_cat_acat_bpref_apref_brDrDrHresolve_by_category_votings&     rprofileindexenccs(||}|s dStfdd|DS)zDReturn the best bigram score across all language variants for *enc*.gc3s"|] \}}}t||VqdS)N)r)rErWmodel model_keyrrDrH s  z&_best_variant_score..)rRmax)rrrvariantsrDrrH_best_variant_scores  rcCst|dkrdSt}i}tt|dD]'}||}||d}||vr*||vr*q|d>|B} || d|| || <q|sAdSt|} t} t| | |} t| | |} | | kr[|S| | kra|SdS)aResolve between two encodings by re-scoring only distinguishing bigrams. Builds a focused bigram profile containing only bigrams where at least one byte is a distinguishing byte, then scores both encodings against their best language model. :param data: The raw byte data to examine. :param enc_a: First encoding name. :param enc_b: Second encoding name. :param diff_bytes: Byte values where the two encodings differ. :returns: The winning encoding name, or ``None`` if tied. r Nr rr)lenrrOrRrfrom_weighted_freqrr)rKr}r~ridffreqib1b2idxrrbest_abest_brDrDrHresolve_by_bigram_rescores,      rmapscCs,||f|vr ||fS||f|vr||fSdS)zEFind the canonical key for a pair of encodings in the confusion maps.NrD)rr}r~rDrDrH_find_pair_keys  rg{Gzt?resultscs"t|dkr|S|d}|jdur|St}|j}tdt|D]n|}|jdur,q dkr:||jtkr:|St||j|j}|durGq ||\}}|\} } t|| | ||} t|| | |} | durf| n| } | dur| |jkrt |j|j|j |j }fddt |D}|g|Sq |S)a Resolve confusion between similar encodings in the top results. Checks the top result against each candidate within a confidence band. Always checks position 1 (preserving original top-2 behavior); for positions 2+ only checks within the band. Uses bigram re-scoring with category voting as fallback. :param data: The raw byte data to examine. :param results: Detection results sorted by confidence descending. :returns: A reordered list of :class:`DetectionResult` with the winner first. r rNr csg|] \}}|kr|qSrDrD)rEjrrrDrH fsz,resolve_confusion_groups..) rencodingr{ confidencerO_CONFUSION_BANDrrrrlanguage mime_type enumerate)rKrtoprtop_conf candidatepair_keyrr^r}r~ cat_winner bigram_winnerwinnerpromotedrestrDrrHresolve_confusion_groups.sF     r)'__doc__ functoolsimportlib.resourcesrdrMrichardet.modelsrrrrchardet.pipelinerchardet.registryrdicttuplestrrSintDistinguishingMapsrC__annotations__rorJbytesrbcacher{r|rlist memoryviewfloatrrrrrrDrDrDrHsJ         ""(     " *  /