o žªj÷>ã @sêUdZddlmZddlmZmZddlmZdede e e e ffdd„Z dede e e e ffd d „Z dede e e e ffd d „Zdede e e e ffd d„Zdede e e e ffdd„Zdede e e e ffdd„Zdede e e e ffdd„Zdede e e e ffdd„Zdede e e e ffdd„Ze e eeeeeedœZeeeege e e e fffed<dededede e e e fdBfdd„Zded edede fd!d"„Z d(ded eded#e dBde f d$d%„Zded edede fd&d'„ZdS))a™Stage 2b: Multi-byte structural probing. Computes how well byte patterns in the data match the expected multi-byte structure for a given encoding. Used after byte-validity filtering (Stage 2a) to further rank multi-byte encoding candidates. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. é)ÚCallable)Ú HIGH_BYTESÚPipelineContext)Ú EncodingInfoÚdataÚreturnc Csd}d}d}tƒ}d}t|ƒ}||kr|||}d|kr!dks.nd|kr,dkrtnnF|d7}|d|kro||d}d|krHdksUnd |krSd kronn|d7}| |¡|d7}|d krj|d7}|d 7}q|d7}n|d7}||ks|dkr„||nd } | |t|ƒfS)z°Single-pass Shift_JIS structural analysis. Lead bytes: 0x81-0x9F, 0xE0-0xEF Trail bytes: 0x40-0x7E, 0x80-0xFC Returns (pair_ratio, mb_bytes, lead_diversity). rééŸéàéïéé@é~é€éüééç©ÚsetÚlenÚadd© rÚ lead_countÚ valid_countÚmbÚleadsÚiÚlengthÚbÚtrailÚratio©r"úa/var/www/html/fyndo/pharma/fyndo/venv/lib/python3.10/site-packages/chardet/pipeline/structural.pyÚ_analyze_shift_jiss2 .  .  ïr$c Csd}d}d}tƒ}d}t|ƒ}||kr|||}d|kr!dks.nd|kr,dkrtnnF|d7}|d|kro||d}d|krHdksUnd |krSdkronn|d7}| |¡|d7}|d krj|d7}|d 7}q|d7}n|d7}||ks|dkr„||nd } | |t|ƒfS) aBSingle-pass CP932 structural analysis. Lead bytes: 0x81-0x9F, 0xE0-0xFC Trail bytes: 0x40-0x7E, 0x80-0xFC Extends Shift_JIS by raising the lead byte ceiling from 0xEF to 0xFC, covering IBM vendor-defined characters (NEC-selected, IBM extensions). Returns (pair_ratio, mb_bytes, lead_diversity). rrr r rr r rrrrrrrr"r"r#Ú_analyze_cp932Es2 .  .  ïr%c CsÎd}d}d}tƒ}d}t|ƒ}||krÖ||}|dkrL|d7}|d|krGd||dkr3dkrGnn|d7}| |¡|d7}|d7}q|d7}n†|dkr‘|d7}|d|krŒd||dkrhdkrŒnn"d||dkrxdkrŒnn|d7}| |¡|d 7}|d 7}q|d7}nAd|kr›dkrÎnn1|d7}|d|krÉd||dkrµdkrÉnn|d7}| |¡|d7}|d7}q|d7}n|d7}||ks|dkrÞ||nd }||t|ƒfS) zóSingle-pass EUC-JP structural analysis. Two-byte: Lead 0xA1-0xFE, Trail 0xA1-0xFE SS2 (half-width katakana): 0x8E + 0xA1-0xDF SS3 (JIS X 0212): 0x8F + 0xA1-0xFE + 0xA1-0xFE Returns (pair_ratio, mb_bytes, lead_diversity). réŽr é¡éßrééþérr© rrrrrrrrr!r"r"r#Ú_analyze_euc_jpnsR ,       ,  Ü%r-c CsÚd}d}d}tƒ}d}t|ƒ}||kr\||}d|kr!dkrTnn1|d7}|d|krOd||dkr;dkrOnn|d7}| |¡|d7}|d7}q|d7}n|d7}||ks|dkrd||nd}||t|ƒfS)z†Single-pass EUC-KR structural analysis. Lead 0xA1-0xFE; Trail 0xA1-0xFE Returns (pair_ratio, mb_bytes, lead_diversity). rr'r*r rrrr,r"r"r#Ú_analyze_euc_kr¨s* ,  ô r.c Cs0d}d}d}tƒ}d}t|ƒ}||kr‡||}d|kr!dks.nd|kr,dkrnnQ|d7}|d|krz||d}d|krHdks`nd |krSd ks`nd|kr^d krznn|d7}| |¡|d7}|d kru|d7}|d 7}q|d7}n|d7}||ks|dkr||nd} | |t|ƒfS)atSingle-pass CP949 (Unified Hangul Code) structural analysis. Lead bytes: 0x81-0xC8, 0xCA-0xFD Trail bytes: 0x41-0x5A, 0x61-0x7A, 0x81-0xFE Extends EUC-KR by lowering the lead byte floor from 0xA1 to 0x81 and adding ASCII letter trail ranges plus 0x81-0xA0. 0xC9 is not a valid UHC lead byte. Returns (pair_ratio, mb_bytes, lead_diversity). rréÈéÊéýr éAéZéaézr*rrrrrr"r"r#Ú_analyze_cp949Ès6.    ër6c Cs‚d}d}d}tƒ}d}t|ƒ}||kr°||}d|kr!dkr¨nn…|d7}|d|krod||dkr;dkronn2d||dkrKdkronn"d||dkr[dkronn|d7}| |¡|d7}|d 7}qd |kryd kr£nn(|d|kr£d ||dkrdkr£nn|d7}| |¡|d7}|d7}q|d7}n|d7}||ks|dkr¸||nd }||t|ƒfS) aÕSingle-pass GB18030 / GB2312 structural analysis. Only counts strict GB2312 2-byte pairs (lead 0xA1-0xF7, trail 0xA1-0xFE) and GB18030 4-byte sequences. The broader GBK extension range (lead 0x81-0xFE, trail 0x40-0x7E / 0x80-0xFE) is intentionally excluded because it is so permissive that unrelated single-byte data (EBCDIC, DOS codepages, etc.) can score 1.0, leading to false positives. Returns (pair_ratio, mb_bytes, lead_diversity). rrr*r r+é0é9rér'é÷rrr,r"r"r#Ú_analyze_gb18030ös<      D  çr;c Csd}d}d}tƒ}d}t|ƒ}||krq||}d|kr!dkrinnF|d7}|d|krd||d}d|kr=dksJnd|krHdkrdnn|d7}| |¡|d7}|dkr_|d7}|d 7}q|d7}n|d7}||ks|dkry||nd } | |t|ƒfS) zSingle-pass Big5 structural analysis. Lead 0xA1-0xF9; Trail 0x40-0x7E, 0xA1-0xFE Returns (pair_ratio, mb_bytes, lead_diversity). rr'éùr r rr*rrrrrr"r"r#Ú _analyze_big5's2   .  ïr=c Csd}d}d}tƒ}d}t|ƒ}||krq||}d|kr!dkrinnF|d7}|d|krd||d}d|kr=dksJnd|krHdkrdnn|d7}| |¡|d7}|dkr_|d7}|d 7}q|d7}n|d7}||ks|dkry||nd } | |t|ƒfS) aWSingle-pass Big5-HKSCS structural analysis. Lead bytes: 0x87-0xFE Trail bytes: 0x40-0x7E, 0xA1-0xFE Extends Big5 by lowering the lead byte floor from 0xA1 to 0x87 and raising the ceiling from 0xF9 to 0xFE. 0x7F and 0x80-0xA0 are not valid Big5/HKSCS trail bytes. Returns (pair_ratio, mb_bytes, lead_diversity). ré‡r*r r rr'rrrrrr"r"r#Ú_analyze_big5hkscsLs2  .  ïr?c Cs8d}d}d}tƒ}d}t|ƒ}||kr‹||}d|kr!dks9nd|kr,dks9nd|kr7dkrƒnnJ|d7}|d|kr~||d}d |krSd ks`nd |kr^d kr~nn|d7}| |¡|d krq|d7}|d kry|d7}|d7}q|d7}n|d7}||ks|dkr“||nd} | |t|ƒfS)z«Single-pass Johab structural analysis. Lead: 0x84-0xD3, 0xD8-0xDE, 0xE0-0xF9 Trail: 0x31-0x7E, 0x91-0xFE Returns (pair_ratio, mb_bytes, lead_diversity). ré„éÓéØéÞr r<r é1ré‘r*rrrrrr"r"r#Ú_analyze_johabvs4 D  .  ïrF)Úshift_jis_2004Úcp932Ú euc_jis_2004Úeuc_krÚcp949Úgb18030Ú big5hkscsÚjohabÚ _ANALYZERSÚnameÚctxNcCsD|j |¡}|dur |St |¡}|durdS||ƒ}||j|<|S)z/Return cached analysis or compute and cache it.N)Úanalysis_cacheÚgetrO)rrPrQÚcachedÚanalyzerÚresultr"r"r#Ú _get_analysis¬s   rWÚ encoding_infocCs0|r|jsdSt||j|ƒ}|durdS|dS)a•Return 0.0--1.0 indicating how well *data* matches the encoding's structure. For single-byte encodings, always returns 0.0. For empty data, always returns 0.0. :param data: The raw byte data to analyze. :param encoding_info: Metadata for the encoding to probe. :param ctx: Pipeline context for caching analysis results. :returns: A structural fit score between 0.0 and 1.0. rNr©Ú is_multibyterWrP©rrXrQrVr"r"r#Úcompute_structural_scoreÀs r\Únon_ascii_countcCsh|r|jsdSt||j|ƒ}|durdS|d}|dur|n t|ƒt| dt¡ƒ}|dkr0dS||S)avRatio of non-ASCII bytes that participate in valid multi-byte sequences. Genuine CJK text has nearly all non-ASCII bytes paired into valid multi-byte sequences (coverage close to 1.0), while Latin text with scattered high bytes has many orphan bytes (coverage well below 1.0). :param data: The raw byte data to analyze. :param encoding_info: Metadata for the encoding to probe. :param ctx: Pipeline context for caching analysis results. :param non_ascii_count: Pre-computed count of non-ASCII bytes, or ``None`` to compute from *data*. :returns: A coverage ratio between 0.0 and 1.0. rNr r)rZrWrPrÚ translater)rrXrQr]rVÚmb_bytesÚ non_asciir"r"r#Úcompute_multibyte_byte_coverage×s ÿýracCs0|r|jsdSt||j|ƒ}|durdS|dS)aáCount distinct lead byte values in valid multi-byte pairs. Genuine CJK text uses lead bytes from across the encoding's full repertoire. European text falsely matching a CJK structural scorer clusters lead bytes in a narrow band. :param data: The raw byte data to analyze. :param encoding_info: Metadata for the encoding to probe. :param ctx: Pipeline context for caching analysis results. :returns: The number of distinct lead byte values found. rNérrYr[r"r"r#Úcompute_lead_byte_diversityþs rc)N)Ú__doc__Úcollections.abcrÚchardet.pipelinerrÚchardet.registryrÚbytesÚtupleÚfloatÚintr$r%r-r.r6r;r=r?rFrOÚdictÚstrÚ__annotations__rWr\rarcr"r"r"r#Ús  ÿ  þ&ÿ  þ)ÿ  þ:ÿ  þ ÿ  þ.ÿ  þ1ÿ  þ%ÿ  þ*ÿ  þ+*ø ÿÿÿ þÿÿÿ þüÿþýü û'ÿÿÿþ