fjA @UdZddlmZddlmZmZddlmZdede e e e ffdZ dede e e e ffdZ dede e e e ffd Zdede e e e ffd Zdede e e e ffd Zdede e e e ffd Zdede e e e ffd Zdede e e e ffdZdede e e e ffdZe e eeeeeedZeeeege e e e fffed<dededede e e e fdzfdZdededede fdZ ddededede dzde f dZdededede fdZdS)aStage 2b: Multi-byte structural probing. Computes how well byte patterns in the data match the expected multi-byte structure for a given encoding. Used after byte-validity filtering (Stage 2a) to further rank multi-byte encoding candidates. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. )Callable) HIGH_BYTESPipelineContext) EncodingInfodatareturncd}d}d}t}d}t|}||kr||}d|cxkrdksnd|cxkrdkrqnnn|dz }|dz|krZ||dz}d|cxkrdksnd |cxkrd kr3nn0|dz }|||dz }|d kr|dz }|d z }|dz }n|dz }||k|dkr||z nd } | |t|fS)zSingle-pass Shift_JIS structural analysis. Lead bytes: 0x81-0x9F, 0xE0-0xEF Trail bytes: 0x40-0x7E, 0x80-0xFC Returns (pair_ratio, mb_bytes, lead_diversity). r@~setlenadd r lead_count valid_countmbleadsilengthbtrailratios oC:\Users\Terasoftware\OneDrive\Desktop\faahhh\fyndo\fyndo\venv\Lib\site-packages\chardet/pipeline/structural.py_analyze_shift_jisr%sJK BeeE A YYF f* G A        41          !OJ1uv~ QU ETtu1$KIIaLLL!GBt| aFA FAA FA# f*$)3Q ?K* $ $CE "c%jj  cd}d}d}t}d}t|}||kr||}d|cxkrdksnd|cxkrdkrqnnn|dz }|dz|krZ||dz}d|cxkrdksnd |cxkrdkr3nn0|dz }|||dz }|d kr|dz }|d z }|dz }n|dz }||k|dkr||z nd } | |t|fS) aBSingle-pass CP932 structural analysis. Lead bytes: 0x81-0x9F, 0xE0-0xFC Trail bytes: 0x40-0x7E, 0x80-0xFC Extends Shift_JIS by raising the lead byte ceiling from 0xEF to 0xFC, covering IBM vendor-defined characters (NEC-selected, IBM extensions). Returns (pair_ratio, mb_bytes, lead_diversity). rr r r rrrrrrrrrrs r$_analyze_cp932r(EsJK BeeE A YYF f* G A        41          !OJ1uv~ QU ETtu1$KIIaLLL!GBt| aFA FAA FA# f*$)3Q ?K* $ $CE "c%jj  r&cd}d}d}t}d}t|}||kr@||}|dkrR|dz }|dz|kr>d||dzcxkrdkr(nn%|dz }|||dz }|dz }a|dz }n|dkrk|dz }|dz|krWd||dzcxkrdkrAnn>d||dzcxkrdkr(nn%|dz }|||d z }|d z }|dz }nhd|cxkrdkrVnnS|dz }|dz|kr?d||dzcxkrdkr)nn&|dz }|||dz }|dz }5|dz }n|dz }||k@|dkr||z nd }||t|fS) zSingle-pass EUC-JP structural analysis. Two-byte: Lead 0xA1-0xFE, Trail 0xA1-0xFE SS2 (half-width katakana): 0x8E + 0xA1-0xDF SS3 (JIS X 0212): 0x8F + 0xA1-0xFE + 0xA1-0xFE Returns (pair_ratio, mb_bytes, lead_diversity). rrrrr rrrrrrr r!r#s r$_analyze_euc_jpr1nsJK BeeE A YYF f*$ G 9"  !OJ1uv~ $$q1u+         q  ! aQ FAA $Y  !OJA DQK    +/     DQK    ,0      q  ! aQ FAA Q    $      !OJ1uv~ $$q1u+         q  ! aQ FAA FAI f*$J)3Q ?K* $ $CE "c%jj  r&cd}d}d}t}d}t|}||kru||}d|cxkrdkrUnnR|dz }|dz|kr>d||dzcxkrdkr(nn%|dz }|||dz }|dz }j|dz }n|dz }||ku|dkr||z nd}||t|fS)zSingle-pass EUC-KR structural analysis. Lead 0xA1-0xFE; Trail 0xA1-0xFE Returns (pair_ratio, mb_bytes, lead_diversity). rr+r.rrrrr0s r$_analyze_euc_krr3sFJK BeeE A YYF f*  G 1          !OJ1uv~ $$q1u+         q  ! aQ FAA FA f* )3Q ?K* $ $CE "c%jj  r&cd}d}d}t}d}t|}||kr||}d|cxkrdksnd|cxkrdkrnn}|dz }|dz|kri||dz}d|cxkrdks!nd |cxkrd ksnd|cxkrd kr3nn0|dz }|||dz }|d kr|dz }|d z }|dz }n|dz }||k|dkr||z nd} | |t|fS)atSingle-pass CP949 (Unified Hangul Code) structural analysis. Lead bytes: 0x81-0xC8, 0xCA-0xFD Trail bytes: 0x41-0x5A, 0x61-0x7A, 0x81-0xFE Extends EUC-KR by lowering the lead byte floor from 0xA1 to 0x81 and adding ASCII letter trail ranges plus 0x81-0xA0. 0xC9 is not a valid UHC lead byte. Returns (pair_ratio, mb_bytes, lead_diversity). rr rAZazr.rrrrrs r$_analyze_cp949r<sJK BeeE A YYF f* G A        41          !OJ1uv~ QU U    &*        )-        *.       1$KIIaLLL!GBt| aFA FAA FA+ f*,)3Q ?K* $ $CE "c%jj  r&cd}d}d}t}d}t|}||kr||}d|cxkrdkrnn|dz }|dz|krpd||dzcxkrdkrZnnWd||dzcxkrdkrAnn>d||dzcxkrdkr(nn%|dz }|||dz }|d z }d |cxkrd krJnnG|dz|kr>d ||dzcxkrdkr(nn%|dz }|||dz }|dz }|dz }n|dz }||k|dkr||z nd }||t|fS) aSingle-pass GB18030 / GB2312 structural analysis. Only counts strict GB2312 2-byte pairs (lead 0xA1-0xF7, trail 0xA1-0xFE) and GB18030 4-byte sequences. The broader GBK extension range (lead 0x81-0xFE, trail 0x40-0x7E / 0x80-0xFE) is intentionally excluded because it is so permissive that unrelated single-byte data (EBCDIC, DOS codepages, etc.) can score 1.0, leading to false positives. Returns (pair_ratio, mb_bytes, lead_diversity). rr r.rr/09rr+rrr0s r$_analyze_gb18030rBszJK BeeE A YYF f* G 1          !OJA DQK    +/     DQK    ,0     DQK    ,0      q  ! aQq    D     QUV^ QU     t     q  ! aQ FAA FA3 f*4)3Q ?K* $ $CE "c%jj  r&cd}d}d}t}d}t|}||kr||}d|cxkrdkrqnnn|dz }|dz|krZ||dz}d|cxkrdksnd|cxkrdkr3nn0|dz }|||dz }|dkr|dz }|d z }|dz }n|dz }||k|dkr||z nd } | |t|fS) zSingle-pass Big5 structural analysis. Lead 0xA1-0xF9; Trail 0x40-0x7E, 0xA1-0xFE Returns (pair_ratio, mb_bytes, lead_diversity). rr+rrrr.rrrrrs r$ _analyze_big5rE'sJK BeeE A YYF f* G 1          !OJ1uv~ QU ETtu1$KIIaLLL!GBt| aFA FAA FA# f*$)3Q ?K* $ $CE "c%jj  r&cd}d}d}t}d}t|}||kr||}d|cxkrdkrqnnn|dz }|dz|krZ||dz}d|cxkrdksnd|cxkrdkr3nn0|dz }|||dz }|dkr|dz }|d z }|dz }n|dz }||k|dkr||z nd } | |t|fS) aWSingle-pass Big5-HKSCS structural analysis. Lead bytes: 0x87-0xFE Trail bytes: 0x40-0x7E, 0xA1-0xFE Extends Big5 by lowering the lead byte floor from 0xA1 to 0x87 and raising the ceiling from 0xF9 to 0xFE. 0x7F and 0x80-0xA0 are not valid Big5/HKSCS trail bytes. Returns (pair_ratio, mb_bytes, lead_diversity). rr.rrrr+rrrrrs r$_analyze_big5hkscsrHLsJK BeeE A YYF f* G 1          !OJ1uv~ QU ETtu1$KIIaLLL!GBt| aFA FAA FA# f*$)3Q ?K* $ $CE "c%jj  r&cd}d}d}t}d}t|}||kr||}d|cxkrdks!nd|cxkrdksnd|cxkrdkrwnnt|dz }|dz|kr`||dz}d |cxkrd ksnd |cxkrd kr9nn6|dz }|||d kr|dz }|d kr|dz }|dz }|dz }n|dz }||k|dkr||z nd} | |t|fS)zSingle-pass Johab structural analysis. Lead: 0x84-0xD3, 0xD8-0xDE, 0xE0-0xF9 Trail: 0x31-0x7E, 0x91-0xFE Returns (pair_ratio, mb_bytes, lead_diversity). rr rDr1rr.rrrrrs r$_analyze_johabrPvsJK BeeE A YYF f* G A        41        $!    t      !OJ1uv~ QU ETtu1$KIIaLLL4x at| aFA FAA FA# f*$)3Q ?K* $ $CE "c%jj  r&)shift_jis_2004cp932 euc_jis_2004euc_krcp949gb18030 big5hkscsjohab _ANALYZERSnamectxNc|j|}||St|}|dS||}||j|<|S)z/Return cached analysis or compute and cache it.N)analysis_cachegetrY)rrZr[cachedanalyzerresults r$ _get_analysisrbsd  # #D ) )F  ~~d##Ht Xd^^F%Ct Mr& encoding_infoc\|r|jsdSt||j|}|dS|dS)aReturn 0.0--1.0 indicating how well *data* matches the encoding's structure. For single-byte encodings, always returns 0.0. For empty data, always returns 0.0. :param data: The raw byte data to analyze. :param encoding_info: Metadata for the encoding to probe. :param ctx: Pipeline context for caching analysis results. :returns: A structural fit score between 0.0 and 1.0. rNr is_multibyterbrZrrcr[ras r$compute_structural_scorerhsF }1s 4!3S 9 9F s !9r&non_ascii_countc|r|jsdSt||j|}|dS|d}||n7t|t|dt z }|dkrdS||z S)avRatio of non-ASCII bytes that participate in valid multi-byte sequences. Genuine CJK text has nearly all non-ASCII bytes paired into valid multi-byte sequences (coverage close to 1.0), while Latin text with scattered high bytes has many orphan bytes (coverage well below 1.0). :param data: The raw byte data to analyze. :param encoding_info: Metadata for the encoding to probe. :param ctx: Pipeline context for caching analysis results. :param non_ascii_count: Pre-computed count of non-ASCII bytes, or ``None`` to compute from *data*. :returns: A coverage ratio between 0.0 and 1.0. rNrr)rfrbrZr translater)rrcr[riramb_bytes non_asciis r$compute_multibyte_byte_coveragerns& }1s 4!3S 9 9F sayH  ? YYT^^D*==>> > A~s i r&c\|r|jsdSt||j|}|dS|dS)aCount distinct lead byte values in valid multi-byte pairs. Genuine CJK text uses lead bytes from across the encoding's full repertoire. European text falsely matching a CJK structural scorer clusters lead bytes in a narrow band. :param data: The raw byte data to analyze. :param encoding_info: Metadata for the encoding to probe. :param ctx: Pipeline context for caching analysis results. :returns: The number of distinct lead byte values found. rNrrergs r$compute_lead_byte_diversityrqsF }1q 4!3S 9 9F s !9r&)N)__doc__collections.abcrchardet.pipelinerrchardet.registryrbytestuplefloatintr%r(r1r3r<rBrErHrPrYdictstr__annotations__rbrhrnrqr&r$r~sc   %$$$$$88888888)))))) #! #! 5#s?#!#!#!#!L&! &! 5#s?&!&!&!&!R7! 7! 5#s?7!7!7!7!t! ! 5#s?!!!!@+! +! 5#s?+!+!+!+!\.! .! 5#s?.!.!.!.!b"! "! 5#s?"!"!"!"!J'! '! 5#s?'!'!'!'!T#! #! 5#s?#!#!#!#!V) # #  D D DhweS#o(>>?? @       !0  5#s?d"    (  ,3B 6#' $ $ $ $  $ 4Z $  $ $ $ $ N  ,3Br&