o jv@sUdZddlZddlmZddlmZddlmZmZm Z m Z ddl m Z m Z mZmZmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlm Z ddl!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.ede dddZ/dZ0dZ1e2hdZ3e2e4e5d<e2hdZ6e2e7e5d<e2hdZ8e2e7e5d<e2hdZ9e2e7e5d<e2hd Z:e2e7e5d!<e6e8e9e:d"Z;eed?ZKd*e?d@eAeJe4eLfd;eJe-d iso8859-1 iso8859-15cp1252_COMMON_LATIN_ENCODINGS>._ISO_8859_10_DISTINGUISHING>r'r(r*r+r,r-r/r0r1r3r4r5r6r7r8r:r;r<r=r>r?r@rArIrR_ISO_8859_14_DISTINGUISHING>rXrYrZr[_WINDOWS_1254_DISTINGUISHING>rBrCrDrErFrGrJr]rY_HP_ROMAN8_DISTINGUISHING)z iso8859-10z iso8859-14cp1254z hp-roman8_DEMOTION_CANDIDATES> r'r(r+r8_KOI8_T_DISTINGUISHINGcp932cp949)shift_jis_2004euc_kr_MARKUP_SUPERSET_PROMOTIONSdata markup_resultallowedreturnc Cs|jdur|St|j}|dus||vr|St|}z |j|ddWn ttfy1|YSwt}t|t|j|}t|||}||krRt ||j |j |j S|S)aEPromote a markup-declared encoding to its superset when structural evidence supports it. If the declared encoding has a known superset, the superset validates the data, and the superset's structural score is materially better, return a new result using the superset encoding. Otherwise return the original. Nstricterrors) rr~getrdecodeUnicodeDecodeError LookupErrorr rr r r!r")rrr superset_name superset_infoctx base_scoresuperset_scorerc/var/www/html/fyndo/pharma/fyndo/venv/lib/python3.10/site-packages/chardet/pipeline/orchestrator.py_try_promote_markup_supersets,   rr param_namecCs:||vrtj|d|dtddtgSt|dddgS)zReturn a low-confidence result for *encoding*, or ``encoding=None`` if filtered out. ``stacklevel=5`` targets the public caller: detect() -> run_pipeline() -> _run_pipeline_core() -> _make_fallback_or_none().  zL is excluded by include_encodings/exclude_encodings; returning encoding=None) stacklevelg?N)rr r!)warningswarn UserWarningrr )rrrrrr_make_fallback_or_nones rcs.t|dur dStfdd|D S)auReturn True if encoding is a demotion candidate with no distinguishing bytes. Checks whether any non-ASCII byte in *data* falls in the set of byte values that decode differently under the given encoding vs iso-8859-1. If none do, the data is equally valid under both encodings and there is no byte-level evidence for preferring the candidate encoding. NFc3s |] }|dkr|vVqdSNr.0bdistinguishingrr 7z!_should_demote..)rprany)rrrrr_should_demote,s rg?gffffff?valid_candidates.rcCsg}|D]W}|jrVt|||}||j|j<|tkrq|jdur,t|t|dt|_|jt kr2qt ||||jd}||j |j<|t krFq|jt krVt|||}|tkrVq||qt|S)aEliminate CJK multi-byte candidates that lack genuine multi-byte structure. Four checks are applied in order to each multi-byte candidate: 1. **Structural pair ratio** (valid_pairs / lead_bytes) must be >= ``_CJK_MIN_MB_RATIO``. Catches files with many orphan lead bytes. 2. **Minimum non-ASCII byte count**: the data must contain at least ``_CJK_MIN_NON_ASCII`` bytes > 0x7F. Tiny files with 1-5 high bytes can accidentally form perfect pairs and score 1.0 structurally. 3. **Byte coverage** (non-ASCII bytes in valid multi-byte sequences / total non-ASCII bytes) must be >= ``_CJK_MIN_BYTE_COVERAGE``. Latin text has many high bytes that are NOT consumed by multi-byte pairs; genuine CJK text has nearly all high bytes accounted for. 4. **Lead byte diversity**: the number of distinct lead byte values in valid pairs must be >= ``_CJK_MIN_LEAD_DIVERSITY``. Genuine CJK text draws from a wide repertoire of lead bytes; European false positives cluster in a narrow band (e.g. 0xC0-0xDF for accented Latin). Returns the filtered candidate list. Structural scores are cached in ``ctx.mb_scores`` for reuse in Stage 2b. N)non_ascii_count) is_multibyter mb_scoresname_CJK_MIN_MB_RATIOrlen translater _CJK_MIN_NON_ASCIIr mb_coverage_CJK_MIN_BYTE_COVERAGE_CJK_DIVERSITY_MIN_NON_ASCIIr_CJK_MIN_LEAD_DIVERSITYappendtuple)rrrgatedencmb_score byte_coveragelead_diversityrrr_gate_cjk_candidatesXs.         rstructural_scoresc sdd|Dtfdd|D}tdd|D}tt|dtg||R}g}|D]+}|jr<|j|jdnd} | dkrU|t|j|j d | |j |j q/||q/|j d d d d |S)aScore structurally-valid CJK candidates using statistical bigrams. When multiple CJK encodings score equally high structurally, statistical scoring differentiates them (e.g. euc-jp vs big5 for Japanese data). Single-byte candidates are also scored and included so that the caller can compare CJK vs single-byte confidence. Multi-byte candidates with high byte coverage (>= 0.95) receive a confidence boost proportional to coverage. When nearly all non-ASCII bytes form valid multi-byte pairs, the structural evidence is strong and should increase the candidate's ranking relative to single-byte alternatives whose bigram models may score higher on small samples. Note: boosted confidence values may exceed 1.0 and are used only for relative ranking among candidates. ``run_pipeline`` clamps all confidence values to [0.0, 1.0] before returning to callers. cSsi|] }|jr|j|qSr)rrrerrr s z0_score_structural_candidates..c3s$|] \}}|vr|VqdSNr)rr_sc enc_lookuprrrsz/_score_structural_candidates..css|]}|js|VqdSr)rrrrrrsNgffffff?cSs|jSr)r xrrrsz._score_structural_candidates..Tkeyreverse) rlistr_STAT_SCORE_MAX_BYTESrrrrr r r!r"sort) rrrrvalid_mb single_byteresultsboostedrcoveragerrr_score_structural_candidatess,  rrcst|dkrS|djdurSt|dj|rS|dj|dj}|ddD]-jtvrRtj|jj}fdd|D}fdd|D}|g||Sq%|S)aDemote niche Latin encodings when no distinguishing bytes are present. Some bigram models (e.g. iso-8859-10, iso-8859-14, windows-1254) can win on data that contains only bytes shared with common Western Latin encodings. When there is no byte-level evidence for the winning encoding, promote the first common Western Latin candidate to the top and push the demoted encoding to last. rrNcs"g|] }|jkr|ur|qSrrrrdemoted_encodingrrr sz'_demote_niche_latin..csg|] }|jkr|qSrrr)rrrrs)rrrr r&r r!r")rrtop_confpromotedothersdemoted_entriesrrr_demote_niche_latins$      rcs|r |djdkr |Stddt|Dddur|Stdd|DrI|}|dj}t|j||j|j}fddt|D}|g|S|S) aPromote KOI8-T over KOI8-R when Tajik-specific bytes are present. KOI8-T and KOI8-R share the entire 0xC0-0xFF Cyrillic letter block, making statistical discrimination difficult. However, KOI8-T maps 12 bytes in 0x80-0xBF to Tajik-specific Cyrillic letters where KOI8-R has box-drawing characters. If any of these bytes appear, KOI8-T is the better match. rzkoi8-rcss"|] \}}|jdkr|VqdS)zkoi8-tNrrirrrrrs z!_promote_koi8t..Ncss |] }|dkr|tvVqdSr)ryrrrrrrcsg|] \}}|kr|qSrrr koi8t_idxrrrsz"_promote_koi8t..)rnext enumeraterr r r!r")rr koi8t_resultrrrrrr_promote_koi8ts"   ric CsB|dkr|Sz |j|ddjdddWStttfy YdSw)aPDecode data from encoding and re-encode as UTF-8 for language scoring. Returns None if the encoding is unknown. For UTF-8, returns data as-is. Uses ``errors="ignore"`` because the data already passed byte-validity filtering for the detected encoding; any residual invalid bytes are irrelevant for language scoring. utf-8ignorer surrogatepassN)rencoder TypeError ValueError)rrrrr_to_utf8src Csg}d}d}|D]}|j}|dur`|jdur`t|j}|dur8|r8t|jr8|dur.t|}t||j|d\}}|dur`|r`tdr`t||j}|r`|dusS|jdkrWt|}t|d|d\}}|j} | durp|jdurndnd} ||jksz| |jkr|t |j|j || q||q|S)aFill in language and mime_type for results missing them. **Language** (only for text results where ``encoding is not None``): Tier 1: single-language encodings via hardcoded map (instant). Tier 2: multi-language encodings via statistical bigram scoring (lazy). Tier 3: decode to UTF-8, score against UTF-8 language models (universal fallback). **MIME type**: text results default to ``"text/plain"``, binary results (``encoding is None``) default to ``"application/octet-stream"``. N)profilerz text/plainr) r!rrrrrrr"rr r ) rrfilledr utf8_profileresultlang_ utf8_datamimerrr_fill_metadatas>     rcCst||}t||}t||S)zGApply confusion resolution, niche Latin demotion, and KOI8-T promotion.)rrr)rrrrr_postprocess_resultsOs   rr%rinclude_encodingsexclude_encodingsno_match_encodingempty_input_encoding encoding_era max_bytesrrrrcCsXt}|d|}t|||}tdd|D} |s t|| dSt|} | dur0| j| vr0| gSt|} | dur@| j| vr@| gSt|} | durU| jdurU| j| vrU| gSt|} | dur`| gSt |}t |}|dury|duryt ||dryt gSt |}|dur|j| vrt||| }|gS|dur|j| vr|gS|dur|j| vr|gSt||}|st|| dSt|||}|st|| dSg}|D]"}|jr|j|j}|durt|||}|dkr||j|fq|r|jdd d d |d \}}|tkrt||||}|rt||S|dt}tt|t|}|s't|| dSt||S) zBCore pipeline logic. Returns list of results sorted by confidence.Ncss|]}|jVqdSr)r)rrrrrrksz%_run_pipeline_core..r)rrrcSs|dS)Nrrrrrrrsz$_run_pipeline_core..Trr)r r frozensetrrrrrrrr r_BINARY_RESULTrrrrrrrrrrr _STRUCTURAL_CONFIDENCE_THRESHOLDrrrrrr)rrrrrrrr candidatesr bom_resultutf1632_result escape_result magic_result utf8_precheckascii_precheckrrrrscorer best_scorer stat_datarrr_run_pipeline_coreYs                 rc CsFt|||||||d}t|dt|}|sd}t|dd|DS)aURun the full detection pipeline. :param data: The raw byte data to analyze. :param encoding_era: Filter candidates to a specific era of encodings. :param max_bytes: Maximum number of bytes to process. :param include_encodings: If not ``None``, only return these encodings. :param exclude_encodings: If not ``None``, never return these encodings. :param no_match_encoding: Encoding returned when no candidate survives. :param empty_input_encoding: Encoding returned for empty input. :returns: A list of :class:`DetectionResult` sorted by confidence descending. rNz/pipeline must always return at least one resultcSs6g|]}|jdkrt|jt|jd|j|jn|qS)g?)r r rminr!r")rrrrrr s  z run_pipeline..)rr_LANG_SCORE_MAX_BYTES RuntimeError) rrrrrrrrmsgrrr run_pipelines  r)V__doc__rchardet._utilsr chardet.enumsrchardet.modelsrrrrchardet.pipelinerr r r r chardet.pipeline.asciir chardet.pipeline.binaryrchardet.pipeline.bomrchardet.pipeline.confusionrchardet.pipeline.escaperchardet.pipeline.magicrchardet.pipeline.markuprchardet.pipeline.statisticalrchardet.pipeline.structuralrrrchardet.pipeline.utf8rchardet.pipeline.utf1632rchardet.pipeline.validityrchardet.registryrrrrrrrr&str__annotations__rUintr\r_rnrpdictryr~bytesrrrboolrrrrrrrrfloatrrrrrrrrrrrrrs\              7*   $    6  2   $ 6