|jY4UdZddlZddlmZmZmZdZdZdZdZ dZ d Z d Z d Z d ezZeed <dededefdZdededzfdZdededzfdZdededzfdZdedefdZddededefdZdS)aStage 1a+: UTF-16/UTF-32 detection for data without BOM. This stage runs after BOM detection but before binary detection. UTF-16 and UTF-32 encoded text contains characteristic null-byte patterns that would otherwise cause binary detection to reject the data. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N)ASCII_TEXT_BYTESDETERMINISTIC_CONFIDENCEDetectionResulti gQ??g?gffffff?g?_NULL_SEPARATOR_ALLOWEDdata null_fracreturncT|tkrdS|dt S)uReturn True if the data looks like ASCII with null byte separators. :param data: The raw byte sample to examine. :param null_frac: The positional null fraction for this UTF-16 candidate (i.e. fraction of null bytes in even positions for BE, or odd positions for LE) — not the total null fraction across all bytes. Checks two conditions: 1. The positional null fraction is below ``_NULL_SEPARATOR_MAX_FRACTION`` 2. Every non-null byte is printable ASCII or common whitespace When both conditions are met, the nulls are likely field separators (e.g. ``find -print0``), not UTF-16 encoding artifacts. FN)_NULL_SEPARATOR_MAX_FRACTION translater )r r s /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/chardet/pipeline/utf1632.py_is_null_separator_patternrFs-000u~~d$;<< <<c|dt}t|tkrdSt|}||St |S)aDetect UTF-32 or UTF-16 encoding from null-byte patterns. UTF-32 is checked before UTF-16 since UTF-32 patterns are more specific. :param data: The raw byte data to examine. :returns: A :class:`DetectionResult` if a strong pattern is found, or ``None``. N) _SAMPLE_SIZElen_MIN_BYTES_UTF16 _check_utf32 _check_utf16)r sampleresults rdetect_utf1632_patternsrZsS-<- F 6{{%%%t& ! !F     rc ttdzz }|tkrdSd||dz}tfdtdtdD}tfdtdtdD}||krV||z dkrM d}t |rt dtdSn#t$rYnwxYwtfd td tdD}tfd td tdD}||krV||z dkrM d }t |rt d tdSn#t$rYnwxYwdS)aCheck for UTF-32 encoding based on 4-byte unit structure. For valid Unicode (U+0000 to U+10FFFF = 0x0010FFFF): - UTF-32-BE: the first byte of each 4-byte unit is always 0x00 - UTF-32-LE: the last byte of each 4-byte unit is always 0x00 For BMP characters (U+0000 to U+FFFF), additionally: - UTF-32-BE: the second byte is also 0x00 - UTF-32-LE: the third byte is also 0x00 Nc34K|]}|dkdVdSrN.0ir s r z_check_utf32..+JJaT!W\\\\\\JJrrc3:K|]}|dzdkdVdS)r!rNr"r#s rr&z_check_utf32..s5OOqd1q5kQ>N>N>N>N>N>NOOrrz utf-32-beencoding confidencelanguagec34K|]}|dkdVdSr r"r#s rr&z_check_utf32..s+IIQDGqLLqLLLLIIrc34K|]}|dkdVdSr r"r#s rr&z_check_utf32..r'rz utf-32-le) r_MIN_BYTES_UTF32sumrangedecode_looks_like_textrrUnicodeDecodeError)r trimmed_len num_units be_first_nullbe_second_nulltext le_last_null le_third_nulls` rrrps1d))s4yy1}-K%%%t   Dq IJJJJ5CIIq#9#9JJJJJMOOOOE!SYY$:$:OOOOON !!ny&@3&F&F ;;{++D%% &(7!  "    D IIII%3t99a"8"8IIIIILJJJJ5CIIq#9#9JJJJJMy  ]Y%>%D%D ;;{++D%% &(7!  "    D  4s$>:C:: DD:G GGcttt}||dzz}|tkrdS|dz}t fdt d|dD}t fdt d|dD}||z }||z }|t kotd|| }|t kotd|| }|s|sdSd||fd||fg} ||kr| d} d } d } d} d }| D]y\}}} d| |}n#t$rY1wxYw|rd }t|sI| dz } t|}|| | dkrtnd zkr|} |} |} z| dS| r| dkp | tk}n |o | tk}|rt| t d SdS)aCheck for UTF-16 via null-byte patterns in alternating positions. UTF-16 encodes each BMP character as two bytes. For characters whose code-point high byte is 0x00 (Latin, digits, basic punctuation, many control structures), one of the two bytes in each unit will be a null. Even for non-Latin scripts (Arabic, CJK, Cyrillic, etc.) a significant fraction of code units still contain at least one null byte. Non-UTF-16 single-byte encodings never contain null bytes, so even a small null-byte fraction in alternating positions is a strong signal. The null pattern only establishes that the data is UTF-16-like. Byte order is always chosen by decoding both ways and comparing text-quality scores, with the null signal breaking near-ties (see below). r0Nc34K|]}|dkdVdSr r"r#s rr&z_check_utf16..+KKad1gllllllKKrrc34K|]}|dkdVdSr r"r#s rr&z_check_utf16..r@rr!z utf-16-lez utf-16-begFTgr))minrrrr2r3_UTF16_MIN_NULL_FRACTIONrreverser4r6r5 _text_quality_QUALITY_TIE_MARGIN_MIN_TEXT_QUALITYrr)r sample_lenr8 be_null_count le_null_countbe_fracle_frac le_qualified be_qualifiedsides best_encoding best_qualitybest_qualifiedviablequalified_side_decodedr*_frac qualifiedr;qualityaccepteds` rrrs SYY --J*q. J$$$taIKKKK5J#:#:KKKKKMKKKK5J#:#:KKKKKMi'Gi'G ++ G*4  +O.O "/    4s!D?? E  E r;c|sdS|dd}td|D}|t|z tkS)z9Quick check: is decoded text mostly printable characters.FNc3JK|]}|s|dvdVdS) r!N) isprintable)r$cs rr&z#_looks_like_text..s3JJ!JAMMAMMMMJJr)r2r_MIN_PRINTABLE_FRACTION)r;r printables rr5r5sO u $3$ZFJJvJJJJJI s6{{ "%< <10% control characters or >20% combining marks (category ``M*``). NrgLr!MZsr\Cg?g?r)r unicodedatacategoryord) r;rarnlettersmarksspacescontrols ascii_lettersr^catscores rrErEsA(&5&\F F AAvvtG E FHM   "1%% q6S== qLG1vv||" Vs]] QJEE D[[AMM aKFF Vs]] MH!|ct qy3t aKE ma3 &&E2vv&1**   Lr)rZ)__doc__richardet.pipelinerrrrr1rrCrGrFr_rr bytes__annotations__floatboolrrrrstrr5intrEr"rrr|s   XXXXXXXXXX  " $")+;!;;;;=U=u=====( % Od,B    ,5u54!75555pdud4!7ddddN=3=4====999C9%999999r