o j5,@sUdZddlZddlmZmZmZdZdZdZdZ dZ d Z d Z d eZ eed <d ededefddZd ededBfddZd ededBfddZd ededBfddZdedefddZddededefddZdS) aStage 1a+: UTF-16/UTF-32 detection for data without BOM. This stage runs after BOM detection but before binary detection. UTF-16 and UTF-32 encoded text contains characteristic null-byte patterns that would otherwise cause binary detection to reject the data. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N)ASCII_TEXT_BYTESDETERMINISTIC_CONFIDENCEDetectionResulti gQ??gffffff?g333333?_NULL_SEPARATOR_ALLOWEDdata null_fracreturncCs|tkrdS|dt S)uReturn True if the data looks like ASCII with null byte separators. :param data: The raw byte sample to examine. :param null_frac: The positional null fraction for this UTF-16 candidate (i.e. fraction of null bytes in even positions for BE, or odd positions for LE) — not the total null fraction across all bytes. Checks two conditions: 1. The positional null fraction is below ``_NULL_SEPARATOR_MAX_FRACTION`` 2. Every non-null byte is printable ASCII or common whitespace When both conditions are met, the nulls are likely field separators (e.g. ``find -print0``), not UTF-16 encoding artifacts. FN)_NULL_SEPARATOR_MAX_FRACTION translater )r r r^/var/www/html/fyndo/pharma/fyndo/venv/lib/python3.10/site-packages/chardet/pipeline/utf1632.py_is_null_separator_pattern6srcCs8|dt}t|tkrdSt|}|dur|St|S)aDetect UTF-32 or UTF-16 encoding from null-byte patterns. UTF-32 is checked before UTF-16 since UTF-32 patterns are more specific. :param data: The raw byte data to examine. :returns: A :class:`DetectionResult` if a strong pattern is found, or ``None``. N) _SAMPLE_SIZElen_MIN_BYTES_UTF16 _check_utf32 _check_utf16)r sampleresultrrrdetect_utf1632_patternsJs  rcs`ttd}|tkrdSd||d}tfddtdtdD}tfddtdtdD}||krc||dkrczd}t|rXtdtdd WSWn tybYnwtfd dtd tdD}tfd dtd tdD}||kr||dkrzd}t|rtdtdd WSWdStyYdSwdS)aCheck for UTF-32 encoding based on 4-byte unit structure. For valid Unicode (U+0000 to U+10FFFF = 0x0010FFFF): - UTF-32-BE: the first byte of each 4-byte unit is always 0x00 - UTF-32-LE: the last byte of each 4-byte unit is always 0x00 For BMP characters (U+0000 to U+FFFF), additionally: - UTF-32-BE: the second byte is also 0x00 - UTF-32-LE: the third byte is also 0x00 Nc3 |] }|dkrdVqdSrNr.0ir rr tz_check_utf32..rc3s$|] }|ddkrdVqdS)rrNrrr!rrr"v"rz utf-32-beencoding confidencelanguagec3rrrrr!rrr"r#c3rrrrr!rrr"r#z utf-32-le) r_MIN_BYTES_UTF32sumrangedecode_looks_like_textrrUnicodeDecodeError)r trimmed_len num_units be_first_nullbe_second_nulltext le_last_null le_third_nullrr!rr`sL  ""  ""  rc sttt}||d8}|tkrdS|d}tfddtd|dD}tfddtd|dD}||}||}g}|tkrStd||sS|d|f|tkrgtd||sg|d |f|skdSt|dkr|dd}zd| |} t | rt |t dd WSWdSt yYdSwd} d } |D]%\}} z d| |} Wn t yYqwt| } | | kr| } |} q| dur| tkrt | t dd SdS) aCheck for UTF-16 via null-byte patterns in alternating positions. UTF-16 encodes each BMP character as two bytes. For characters whose code-point high byte is 0x00 (Latin, digits, basic punctuation, many control structures), one of the two bytes in each unit will be a null. Even for non-Latin scripts (Arabic, CJK, Cyrillic, etc.) a significant fraction of code units still contain at least one null byte. Non-UTF-16 single-byte encodings never contain null bytes, so even a small null-byte fraction in alternating positions is a strong signal. When both endiannesses show null-byte patterns (e.g., Latin text where every other byte is null), we disambiguate by decoding both ways and comparing text-quality scores. r*Nc3rrrrr!rrr"r#z_check_utf16..rc3rrrrr!rrr"r#rz utf-16-lez utf-16-ber%)minrrrr,r-_UTF16_MIN_NULL_FRACTIONrappendr.r/rrr0 _text_quality_MIN_TEXT_QUALITY)r sample_lenr2 be_null_count le_null_countbe_fracle_frac candidatesr&r5 best_encoding best_quality_qualityrr!rrsp          rr5cCs6|sdS|dd}tdd|D}|t|tkS)z9Quick check: is decoded text mostly printable characters.FNcss$|] }|s |dvrdVqdS) rN) isprintable)rcrrrr"r$z#_looks_like_text..)r,r_MIN_PRINTABLE_FRACTION)r5r printablerrrr/s  r/rHlimitc Cs |d|}t|}|dkrdSd}d}d}d}d}|D]>} t| } | ddkr8|d7}t| dkr7|d7}q| ddkrC|d7}q| dksK| d vrP|d7}q| dd krZ|d7}q||d krcdS||d krkdS||} | ||d 7} |dkr|dkr| d 7} | S)uScore how much *text* looks like real human-readable content. Returns a score in the range [-1.0, ~1.6). Higher values indicate more natural text. The practical maximum is 1.5 for all-ASCII-letter input (1.6 approaches as sample size grows with all ASCII letters plus whitespace). A score of -1.0 means the content is almost certainly not valid text (too many control characters or combining marks). Scoring factors: * Base score: ratio of Unicode letters (category ``L*``) to sample length. * ASCII bonus: additional 0.5x weight for ASCII letters. This is the primary signal for disambiguating endianness — correct decoding of Latin-heavy text produces ASCII letters, wrong decoding produces CJK. * Space bonus: +0.1 when the sample contains at least one whitespace character and is longer than 20 characters. * Rejection: returns -1.0 if >10% control characters or >20% combining marks (category ``M*``). Nrr8LrMZsrICg?g?r)r unicodedatacategoryord) r5rNrnlettersmarksspacescontrols ascii_lettersrKcatscorerrrr<s@          r<)rH)__doc__rUchardet.pipelinerrrrr+rr:r=rLr r bytes__annotations__floatboolrrrrstrr/intr<rrrrs" 8T