|j$dZddlmZmZmZmZmZmZddlm Z dZ de de de dzfd Z de d ee dee fd ZdS) aThree-tier language detection for filling DetectionResult languages. Tier 1: hardcoded mapping for single-language encodings (e.g. Big5 -> Chinese). Tier 2: statistical bigram scoring against the encoding's language-model variants. Tier 3: decode to UTF-8 and score against the UTF-8 byte-level language models. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. )_THIN_RARE_MAX_BYTESRARE_LANGUAGES BigramProfilehas_model_variantsinfer_languagescore_best_language)DetectionResultidataencodingreturnNc|dkr|S ||dddS#tttf$rYdSwxYw)aPDecode data from encoding and re-encode as UTF-8 for language scoring. Returns None if the encoding is unknown. For UTF-8, returns data as-is. Uses ``errors="ignore"`` because the data already passed byte-validity filtering for the detected encoding; any residual invalid bytes are irrelevant for language scoring. utf-8ignore)errors surrogatepassN)decodeencode LookupError TypeError ValueError)r r s /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/chardet/pipeline/language.py_to_utf8rsv7 {{8H{55<< O=    J /tts+6AAresultsc |dt}dt|cxko tknc}g}d}d}d}|D]}|o|jduo|jtvo|jdu}|j |j|s||K|j} |rdnt| } | >|r1 ,t+  ? "v'B7'B MM& ! ! ! ?(> MM& ! ! ! ! \ MM& ! ! ! ! MM&*;T6CSTT     M)__doc__chardet.modelsrrrrrrchardet.pipeliner rbytesstrrlistr0r1rr9s  -,,,,, 5CEDL$b b/b /bbbbbbr1