fj&yPUdZddlZddlmZddlmZddlmZmZm Z m Z ddl m Z m Z mZmZmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlm Z ddl!m"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,m-Z-m.Z.ede ddZ/dZ0dZ1e2hdZ3e2e4e5d<e2hdZ6e2e7e5d<e2hdZ8e2e7e5d<e2hdZ9e2e7e5d<e2hd Z:e2e7e5d!<e6e8e9e:d"Z;eeeAed-eAefd?ZNd*e?d>eAed-eAefd@ZOdAZPd*e?d/e4d-e?dzfdBZQd*e?d>eAed-eAefdCZRd*e?d>eAed-eAefdDZSefdddEdFdGd*e?dHedIe7dJe2e4dzdKe2e4dzdLe4dMe4d-eAefdNZTefdddEdFdGd*e?dHedIe7dJe2e4dzdKe2e4dzdLe4dMe4d-eAefdOZUdS)PuPipeline orchestrator — runs all detection stages in sequence. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N)DEFAULT_MAX_BYTES) EncodingEra) BigramProfilehas_model_variantsinfer_languagescore_best_language) _NONE_RESULTDETERMINISTIC_CONFIDENCE HIGH_BYTESDetectionResultPipelineContext) detect_ascii) is_binary) detect_bom)resolve_confusion_groups)detect_escape_encoding) detect_magic)detect_markup_charset)score_candidates)compute_lead_byte_diversitycompute_multibyte_byte_coveragecompute_structural_score) detect_utf8)detect_utf1632_patterns)filter_by_validity)REGISTRY EncodingInfoget_candidatesapplication/octet-stream)encoding confidencelanguage mime_typeg333333?i@> iso8859-1 iso8859-15cp1252_COMMON_LATIN_ENCODINGS>._ISO_8859_10_DISTINGUISHING>r(r)r+r,r-r.r0r1r2r4r5r6r7r8r9r;r<r=r>r?r@rArBrJrS_ISO_8859_14_DISTINGUISHING>rYrZr[r\_WINDOWS_1254_DISTINGUISHING>rCrDrErFrGrHrKr^rZ_HP_ROMAN8_DISTINGUISHING)z iso8859-10z iso8859-14cp1254z hp-roman8_DEMOTION_CANDIDATES> r(r)r,r9_KOI8_T_DISTINGUISHINGcp932cp949)shift_jis_2004euc_kr_MARKUP_SUPERSET_PROMOTIONSdata markup_resultallowedreturnc|j|St|j}|||vr|St|} ||dn#t t f$r|cYSwxYwt}t|t|j|}t|||}||kr!t||j |j |j S|S)aEPromote a markup-declared encoding to its superset when structural evidence supports it. If the declared encoding has a known superset, the superset validates the data, and the superset's structural score is materially better, return a new result using the superset encoding. Otherwise return the original. Nstricterrors) r rgetrdecodeUnicodeDecodeError LookupErrorr rr r!r"r#)rrr superset_name superset_infoctx base_scoresuperset_scores qC:\Users\Terasoftware\OneDrive\Desktop\faahhh\fyndo\fyndo\venv\Lib\site-packages\chardet/pipeline/orchestrator.py_try_promote_markup_supersetrs /33M4JKKM W <]+M M( 3333  ,   C)$9O0PRUVVJ-dM3GGN "    $  "  #    sAA-,A-r param_namec||vr*tj|d|dtdtgSt |ddgS)zReturn a low-confidence result for *encoding*, or ``encoding=None`` if filtered out. ``stacklevel=5`` targets the public caller: detect() -> run_pipeline() -> _run_pipeline_core() -> _make_fallback_or_none().  zL is excluded by include_encodings/exclude_encodings; returning encoding=None) stacklevelg?N)r r!r")warningswarn UserWarningr r )r rrs r_make_fallback_or_nonersuw  L LH L L L      ~ X$ N N N OOcxt|dStfd|D S)auReturn True if encoding is a demotion candidate with no distinguishing bytes. Checks whether any non-ASCII byte in *data* falls in the set of byte values that decode differently under the given encoding vs iso-8859-1. If none do, the data is equally valid under both encodings and there is no byte-level evidence for preferring the candidate encoding. NFc3,K|]}|dk|vVdSN).0bdistinguishings r z!_should_demote..7s1AA1DA1&AAAAAAr)rqrany)r rrs @r_should_demoter,sK*--h77NuAAAAAAAAA AArg?gffffff?valid_candidates.rc,g}|D]}|jrt|||}||j|j<|tkr6|j=t |t |dtz |_|jtkrt||||j}||j |j<|tkr|jtkrt|||}|tkr||t#|S)aEliminate CJK multi-byte candidates that lack genuine multi-byte structure. Four checks are applied in order to each multi-byte candidate: 1. **Structural pair ratio** (valid_pairs / lead_bytes) must be >= ``_CJK_MIN_MB_RATIO``. Catches files with many orphan lead bytes. 2. **Minimum non-ASCII byte count**: the data must contain at least ``_CJK_MIN_NON_ASCII`` bytes > 0x7F. Tiny files with 1-5 high bytes can accidentally form perfect pairs and score 1.0 structurally. 3. **Byte coverage** (non-ASCII bytes in valid multi-byte sequences / total non-ASCII bytes) must be >= ``_CJK_MIN_BYTE_COVERAGE``. Latin text has many high bytes that are NOT consumed by multi-byte pairs; genuine CJK text has nearly all high bytes accounted for. 4. **Lead byte diversity**: the number of distinct lead byte values in valid pairs must be >= ``_CJK_MIN_LEAD_DIVERSITY``. Genuine CJK text draws from a wide repertoire of lead bytes; European false positives cluster in a narrow band (e.g. 0xC0-0xDF for accented Latin). Returns the filtered candidate list. Structural scores are cached in ``ctx.mb_scores`` for reuse in Stage 2b. N)non_ascii_count) is_multibyter mb_scoresname_CJK_MIN_MB_RATIOrlen translater _CJK_MIN_NON_ASCIIr mb_coverage_CJK_MIN_BYTE_COVERAGE_CJK_DIVERSITY_MIN_NON_ASCIIr_CJK_MIN_LEAD_DIVERSITYappendtuple)rrrgatedencmb_score byte_coveragelead_diversitys r_gate_cjk_candidatesrXs1:!#E   /c3??H&.CM#( #++ " X&)$ii#dnnT:6V6V2W2W&W#"%77 ;c30CM)6COCH %55 "&BB != 0.95) receive a confidence boost proportional to coverage. When nearly all non-ASCII bytes form valid multi-byte pairs, the structural evidence is strong and should increase the candidate's ranking relative to single-byte alternatives whose bigram models may score higher on small samples. Note: boosted confidence values may exceed 1.0 and are used only for relative ranking among candidates. ``run_pipeline`` clamps all confidence values to [0.0, 1.0] before returning to callers. c,i|]}|j |j|Sr)rrres r z0_score_structural_candidates..s3+++q~+ +++rc36K|]\}}|v |VdSNr)rr_sc enc_lookups rrz/_score_structural_candidates..sJ&T3tz?Q4rc3(K|] }|j |VdSr)rrs rrz/_score_structural_candidates..s)JJa1>JJJJJJJrNgffffff?c|jSr)r!xs rz._score_structural_candidates..sq|rTkeyreverse) rlistr_STAT_SCORE_MAX_BYTESr rrrr r!r"r#sort) rrrrvalid_mb single_byteresultsboostedrcoveragers @r_score_structural_candidatesrsm.++++++J*;HJJ#3JJJJJK44457P7PK7P7PQQG &(G   ;<:N3?&&qz37773 t   NNJ H =qz1;     NN1     LL++TL::: Nrrcxt|dkr|djt|dj|r||dj|dj}|ddD]WjtvrGt j|jj}fd|D}fd|D}|g||cSX|S)aDemote niche Latin encodings when no distinguishing bytes are present. Some bigram models (e.g. iso-8859-10, iso-8859-14, windows-1254) can win on data that contains only bytes shared with common Western Latin encodings. When there is no byte-level evidence for the winning encoding, promote the first common Western Latin candidate to the top and push the demoted encoding to last. rrNc2g|]}|jk |u|Srr )rrdemoted_encodingrs r z'_demote_niche_latin..sC!*8H*HMNVWZrc*g|]}|jk |Srr)rrrs rrz'_demote_niche_latin..s&"X"X"XGW9W"X1"X"X"Xr)rr rr!r'r r"r#)rrtop_confpromotedothersdemoted_entriesrrs @@r_demote_niche_latinrs G q= AJ = 71:. 5 5= #1:.1:( = =Az44 =*J!*ak&#Y"X"X"Xg"X"X"X <6.s2QQDAq!*:PQaQQQQQQrNc34K|]}|dk|tvVdSr)rz)rrs rrz!_promote_koi8t..s1 A A1D A1& & A A A A A Arc&g|] \}}|k |Srr)rrr koi8t_idxs rrz"_promote_koi8t..s&EEE1a9nE!EEEr)r next enumeraterr!r r"r#)rr koi8t_resultrrrrs @r_promote_koi8trs gaj)X5QQIg$6$6QQQSWXXI A A A A AAA #y) 1:("  !   !  "    FEEE ' 2 2EEE"6"" Nric|dkr|S ||dddS#tttf$rYdSwxYw)aPDecode data from encoding and re-encode as UTF-8 for language scoring. Returns None if the encoding is unknown. For UTF-8, returns data as-is. Uses ``errors="ignore"`` because the data already passed byte-validity filtering for the detected encoding; any residual invalid bytes are irrelevant for language scoring. utf-8ignorer surrogatepassN)rencoder TypeError ValueError)rr s r_to_utf8rsx7 {{8H{55<< O=    J /tts+6AAc g}d}d}|D]5}|j}||jt|j}|A|r?t|jr+|t |}t ||j|\}}|Y|rWtdrHt ||j}|r1| |jdkrt |}t |d|\}}|j} | |jdnd} ||jks | |jkr1|t|j|j ||  ||7|S)aFill in language and mime_type for results missing them. **Language** (only for text results where ``encoding is not None``): Tier 1: single-language encodings via hardcoded map (instant). Tier 2: multi-language encodings via statistical bigram scoring (lazy). Tier 3: decode to UTF-8, score against UTF-8 language models (universal fallback). **MIME type**: text results default to ``"text/plain"``, binary results (``encoding is None``) default to ``"application/octet-stream"``. N)profilerz text/plainr) r"r rrrrrr#rr r!) rrfilledr utf8_profileresultlang_ utf8_datamimes r_fill_metadatars%'F$(G)-L!"!"  FO !&/22D V V);FO)L)L V2+D11G-dFOWUUU4  );G)D)D $T6?;; #@v'/I@'4Y'?'? 1!7LGAt  ?0 /  6? " "df.>&> " MM1BD$OO     MM& ! ! ! ! Mrcbt||}t||}t||S)zGApply confusion resolution, niche Latin demotion, and KOI8-T promotion.)rrr)rrs r_postprocess_resultsrOs2 'tW55G!$00G $ ( ((rr&rinclude_encodingsexclude_encodingsno_match_encodingempty_input_encoding encoding_era max_bytesr r r r cLt}|d|}t|||}td|D} |st|| dSt |} | | j| vr| gSt |} | | j| vr| gSt|} | | j | j| vr| gSt|} | | gSt|}t|}||t||rtgSt|}||j| vrt||| }|gS| |j| vr|gS| |j| vr|gSt||}|st|| dSt!|||}|st|| dSg}|D]]}|jrT|j|j}|t+|||}|dkr||j|f^|rR|dd |d \}}|t0kr$t3||||}|rt5||S|dt6}t9t;|t=|}|st|| dSt5||S) zBCore pipeline logic. Returns list of results sorted by confidence.Nc3$K|] }|jV dSr)r)rrs rrz%_run_pipeline_core..ks$'G'GS'G'G'G'G'G'Grr )rr rc|dS)Nrrrs rrz$_run_pipeline_core..s QqTrTrr)r r frozensetrrr rrrrrr_BINARY_RESULTrrrrrrrrrrr _STRUCTURAL_CONFIDENCE_THRESHOLDrrrrrr)rr rr r r r r candidatesr bom_resultutf1632_result escape_result magic_result utf8_precheckascii_precheckrrrrscorer best_scorer stat_datas r_run_pipeline_corerYs   C   D  .?ARSSJ''G'GJ'G'G'GGGG  % '+A   D!!J*"5"@| -T22N n&=&H  +400M  "  "g -  %%L~ %%M"$''N    di 0 0 0  *$//M]%;w%F4T='RR  n&=&H ]%;w%F*$ ;; W%&7BUVVV,D2BCHH W%&7BUVVV24<<   <M%%ch//E A0sC@@s{ <!((#(E):;;;;>>4@@@)!, : 9 9 ;2')93G ;+D'::: +++,I#Iu5E/F/FGGHHG W%&7BUVVV g . ..rc t|||||||}t|dt|}|sd}t|d|DS)aURun the full detection pipeline. :param data: The raw byte data to analyze. :param encoding_era: Filter candidates to a specific era of encodings. :param max_bytes: Maximum number of bytes to process. :param include_encodings: If not ``None``, only return these encodings. :param exclude_encodings: If not ``None``, never return these encodings. :param no_match_encoding: Encoding returned when no candidate survives. :param empty_input_encoding: Encoding returned for empty input. :returns: A list of :class:`DetectionResult` sorted by confidence descending. rNz/pipeline must always return at least one resultc g|]C}|jdkr4t|jt|jd|j|jn|DS)g?)r!r r minr"r#)rrs rrz run_pipeline.. s^     <#   C c$:$:AJ TTT    r)rr_LANG_SCORE_MAX_BYTES RuntimeError) rr rr r r r rmsgs r run_pipeliner&s*! +++1GT"8#8"897CCG  ?3      r)V__doc__rchardet._utilsr chardet.enumsrchardet.modelsrrrrchardet.pipeliner r r r r chardet.pipeline.asciirchardet.pipeline.binaryrchardet.pipeline.bomrchardet.pipeline.confusionrchardet.pipeline.escaperchardet.pipeline.magicrchardet.pipeline.markuprchardet.pipeline.statisticalrchardet.pipeline.structuralrrrchardet.pipeline.utf8rchardet.pipeline.utf1632rchardet.pipeline.validityrchardet.registryrrrrrrrr'str__annotations__rVintr]r`rorqdictrzrbytesrrrboolrrrrrrrrfloatrrrr#rrrrr&rrrr@s,,,,,,%%%%%% 0/////------++++++??????:::::://////999999999999 .-----<<<<<<888888CCCCCCCCCC  ' ( $( +4)++3/8i///1/1/Ys^111n/8i   "/"/Ys^"""T09y(((00in-6I--9S>:.-** 33d3 #./*3LLL** #//T#s(^ ! !"!s^! !!!!HPP s^PP / PPPP* BS B B$ B B B B$   "3 3L#-.3 3 <  3333l/ /E#u*-./L#-./  /  / ////d  / " /@  / " /H5CEDL$3 3/3 /3333l) ) / ") /))))'J/ 04/3% 'J/J/J/ J/J/J/ !~, J/ !~, J/J/J/ /J/J/J/J/`', 04/3% ',,, ,,, !~, , !~, ,,, /,,,,,,r