o j. @s$UdZddlZddlZddlZddlZddlZddlZddlm Z m Z e dj Z e dj ZdZiZeeefed<e D]ZeejdkrPejdeej<q?d ed eeeefeeefffd d Zejd eeeefeeefffd dZd eeeffddZ deeefd eee!eedBeefffddZ"ejd eee!eedBeefffddZ#ded edBfddZ$ded e%fddZ&d eeeffddZ'ejd e(fddZ)Gdd d Z* !d)d"e*d#e(eBd$ed efd%d&Z+ d*d eded"e*dBd eeedBffd'd(Z,dS)+zModel loading and bigram scoring utilities. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N)REGISTRYlookup_encodingz>Iz>dsCMD2_SINGLE_LANG_MAPdatareturnc Csz|ddtkrd}t|d}t||\}|d7}|dkr)d|d}t|g}i}t|D]<}t||\}|d7}|dkrKd|d }t|||||d }||7}t||\} |d 7}||| ||<q1t||d} |d } t | | krd t | d| }t|t | } i} t |D]\}}|d }| ||d | |<qW| |fStj y}z d|}t||d}~wt j tfy}z d|}t||d}~ww)zParse the v2 dense zlib-compressed models.bin format. :param data: Raw bytes of models.bin (must be non-empty). :returns: A ``(models, norms)`` tuple. :raises ValueError: If the data is corrupt or truncated. Nz&corrupt models.bin: missing CMD2 magici'zcorrupt models.bin: num_models=z exceeds limitzcorrupt models.bin: name_len=z exceeds 256zutf-8z&corrupt models.bin: decompressed size z != expected zcorrupt models.bin: ) _V2_MAGIC ValueError_unpack_uint32rangedecode_unpack_float64appendzlib decompresslen memoryview enumerateerrorstructUnicodeDecodeError)rmsgoffset num_modelsnamesnorms_name_lennamenormblob expected_sizemvmodelsistarter+]/var/www/html/fyndo/pharma/fyndo/venv/lib/python3.10/site-packages/chardet/models/__init__.py_parse_models_bins`                r-cCs>tjdd}|}|stjdtddiifSt|S)zkLoad and parse models.bin, returning (models, norms). Cached: only reads from disk on first call. chardet.modelsz models.binuXchardet models.bin is empty — statistical detection disabled; reinstall chardet to fix stacklevel) importlib resourcesfilesjoinpath read_byteswarningswarnRuntimeWarningr-refrr+r+r,_load_models_dataasr<cC tdS)zLoad all bigram models from the bundled models.bin file. Each model is a memoryview of length 65536 (256*256). Index: (b1 << 8) | b2 -> weight (0-255). :returns: A dict mapping model key strings to 65536-byte lookup tables. rr<r+r+r+r, load_modelsvs r?r'cCsti}|D]\}}|dd\}}||g|||fqt|D]}t|}|dur7||vr7||||<q#|S)zBuild a grouped index from a models dict. :param models: Mapping of ``"lang/encoding"`` keys to 65536-byte tables. :returns: Mapping of encoding name to ``[(lang, model, model_key), ...]``. /rN)itemssplit setdefaultrlistr)r'indexkeymodellangencenc_name canonicalr+r+r,_build_enc_indexs  rLcCs ttS)zTReturn a pre-grouped index mapping encoding name -> [(lang, model, model_key), ...].)rLr?r+r+r+r, get_enc_indexs rMencodingcCs t|S)zReturn the language for a single-language encoding, or None. :param encoding: The canonical encoding name. :returns: An ISO 639-1 language code, or ``None`` if the encoding is multi-language. )rgetrNr+r+r,infer_languages rQcCs |tvS)zReturn True if the encoding has language variants in the model index. :param encoding: The canonical encoding name. :returns: ``True`` if bigram models exist for this encoding. )rMrPr+r+r,has_model_variantss rRcCr=)zAReturn cached L2 norms for all models, keyed by model key string.rr>r+r+r+r,_get_model_normss rScCsVtjdd}|}t|dkr'tjdt|dtddt ddSt |S) uReturn a 65536-byte IDF weight table for bigram profile construction. Loads a precomputed table from ``idf.bin`` (generated at training time). For each bigram index, the weight reflects how discriminative that bigram is across all models: - Bigrams in every model (common ASCII) → weight 1 (minimal signal) - Bigrams in one model → weight 255 (maximum signal) - Bigrams not in any model → weight 1 (unknown, treat as neutral) r.zidf.binr z chardet idf.bin has wrong size (z"), falling back to uniform weightsr/r0) r2r3r4r5r6rr7r8r9 bytearrayr:r+r+r,get_idf_weightss   rVc@sDeZdZdZdZdeddfddZedee e fddfd d Z dS) BigramProfileuPre-computed bigram frequency distribution for a data sample. Computing this once and reusing it across all models reduces per-model scoring from O(n) to O(distinct_bigrams). Stores a dense ``freq`` list of length 65536 indexed by bigram index, plus a ``nonzero`` list of indices with non-zero frequency for fast iteration. Each bigram is weighted by its IDF (inverse document frequency) across all models — bigrams unique to few models get high weight, bigrams common to all models get weight 1. )freq input_normnonzero weight_sumrrNc Cst|d}|dkrg|_g|_d|_d|_dSt}dgd}g}d}t|D])}||d>||dB}||} ||dkrE||||| 7<|| 7}q(||_||_||_d} |D] }||} | | | 7} q_t | |_dS)a?Compute the bigram frequency distribution for *data*. Each bigram is weighted by its IDF (inverse document frequency) across all loaded models. Bigrams unique to few models get high weight; bigrams common to all models get weight 1. :param data: The raw byte data to profile. rrNr r ) rrXrZr[rYrVrrmathsqrt) selfr total_bigramsidfrXrZw_sumr(idxwnorm_sqvr+r+r,__init__s4      zBigramProfile.__init__ weighted_freqcCsz|d}dgd}g}|D]\}}|||<|r||q||_||_t||_ttdd|D|_ |S)aLCreate a BigramProfile from pre-computed weighted frequencies. Computes ``weight_sum`` and ``input_norm`` from *weighted_freq* to ensure consistency between the stored fields. :param weighted_freq: Mapping of bigram index to weighted count. :returns: A new :class:`BigramProfile` instance. rr css|]}||VqdSNr+).0rfr+r+r, sz3BigramProfile.from_weighted_freq..) rArrXrZsumvaluesr[r]r^rY)clsrhprofilerXrZrccountr+r+r,from_weighted_freqs  z BigramProfile.from_weighted_freq) __name__ __module__ __qualname____doc__ __slots__bytesrg classmethoddictintrrr+r+r+r,rWs  ( rWrprG model_keyc Cs|jdkrdSt}|r||nd}|dur1d}tdD]}||}|r+|||7}qt|}|dkr7dSd}|j} |jD] } ||| | | 7}q?|||jS)zSScore a pre-computed bigram profile against a single model using cosine similarity.r\Nrr )rYrSrOrr]r^rXrZ) rprGr}r model_normsq_sumr(rfdotrXrcr+r+r,score_with_profiles&     rc Csx|s|durdSt}||}|durdS|durt|}d}d}|D]\}}} t||| } | |kr7| }|}q$||fS)aScore data against all language variants of an encoding. Returns (best_score, best_language). Uses a pre-grouped index for O(L) lookup where L is the number of language variants for the encoding. If *profile* is provided, it is reused instead of recomputing the bigram frequency distribution from *data*. :param data: The raw byte data to score. :param encoding: The canonical encoding name to match against. :param profile: Optional pre-computed :class:`BigramProfile` to reuse. :returns: A ``(score, language)`` tuple with the best cosine-similarity score and the corresponding language code (or ``None``). N)r\Nr\)rMrOrWr) rrNrprEvariants best_score best_langrHrGr}sr+r+r,score_best_language6s"   r)r|rj)-rv functoolsimportlib.resourcesr2r]rr7rchardet.registryrrStruct unpack_fromrrr rrzstr__annotations__rn_encr languagesr"rxtuplerfloatr-cacher<r?rDrLrMrQboolrRrSrUrVrWrrr+r+r+r,sr    D(  *  P