|jsUdZddlZddlZddlZddlZddlZddlZddlZddl Z ddl m Z ddl m Z mZddlmZmZmZmZmZddlmZmZejdejdfZe jdZejjd kZed e d DZ!iZ"e#e$e$fe%d <ej&D]+Z'e(e'j)d kre'j)de"e'j<,ej*de+e#e$efe#e$e,fffdZ-de#e$effdZ.de#e$efde#e$e/e+e$dzee$fffdZ0ej*de#e$e/e+e$dzee$fffdZ1de$de$dzfdZ2de$de3fdZ4de#e$e,ffdZ5ej*de#e$effdZ6ej*de#e$e,ffdZ7dZ8ej*defdZ9ej*defdZ:GddZ; d1d d!d"e;d#d$d%e$d&e3de,f d'Z<e=hd(Z>e=e$e%d)<d*Z?d+Z@d,ZA d2d d d-d.ede$d"e;dzd/e3d&e3de+e,e$dzff d0ZBdS)3zModel loading and bigram scoring utilities. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N)_kernel) dot_packed pack_profile)langnorm_from_tableparse_langnorm_binparse_models_binparse_rowmax_binrowmax_from_table)REGISTRYlookup_encodingi)z.soz.pydcpythonc#&K|] }|dvrdndV dS)) rN).0bs /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/chardet/models/__init__.py rCsB  DE8 8 8AAa      _SINGLE_LANG_MAPrreturnctjdd}|}|s t jdtdiifSt|S)zkLoad and parse models.bin, returning (models, norms). Cached: only reads from disk on first call. chardet.models models.binuXchardet models.bin is empty — statistical detection disabled; reinstall chardet to fix stacklevel) importlib resourcesfilesjoinpath read_byteswarningswarnRuntimeWarningrrefdatas r_load_models_datar2Os}   # #$4 5 5 > >| L LC >>  D   '      2v D ! !!rc*tdS)zLoad all bigram models from the bundled models.bin file. Each model is a bytes object of length 65536 (256*256). Index: (b1 << 8) | b2 -> weight (0-255). :returns: A dict mapping model key strings to 65536-byte lookup tables. rr2rrr load_modelsr5ds   q !!rmodelsc,i}|D]J\}}|dd\}}||g|||fKt |D]"}t |}|||vr ||||<#|S)zBuild a grouped index from a models dict. :param models: Mapping of ``"lang/encoding"`` keys to 65536-byte tables. :returns: Mapping of encoding name to ``[(lang, model, model_key), ...]``. /r)itemssplit setdefaultappendlistr )r6indexkeymodellangencenc_name canonicals r_build_enc_indexrEos=?Ellnn== UIIc1%% c b!!(($s);<<<<KK//#H--  Ye%;%;$XE)  Lrc8ttS)zTReturn a pre-grouped index mapping encoding name -> [(lang, model, model_key), ...].)rEr5rrr get_enc_indexrGs KMM * **rencodingc6t|S)zReturn the language for a single-language encoding, or None. :param encoding: The canonical encoding name. :returns: An ISO 639-1 language code, or ``None`` if the encoding is multi-language. )rgetrHs rinfer_languagerLs    ) ))rc"|tvS)zReturn True if the encoding has language variants in the model index. :param encoding: The canonical encoding name. :returns: ``True`` if bigram models exist for this encoding. )rGrKs rhas_model_variantsrNs } &&rc*tdS)zAReturn cached L2 norms for all models, keyed by model key string.rr4rrr_get_model_normsrPs   q !!rc6t}tjd} |d}t j|d}n#ttf$rd}d}YnwxYwt||t|}||S|rtjdtdd |DS) uReturn per-model row-maximum tables for upper-bound prescreening. For each model, entry ``b1`` of its 256-byte table holds the maximum weight in the model's row for lead byte ``b1``. Because every bigram weight is bounded by its row maximum, a dot product against the row maxima (256 terms) upper-bounds the dot product against the full table (65536 terms) — statistical scoring uses this to rule out candidate models without scoring them fully. Loads the precomputed ``rowmax.bin`` (written by ``chardet.models._format.write_model_artifacts`` in the same model order as ``models.bin``). The file starts with a ``CRM1`` magic and the SHA-256 of the ``models.bin`` it was derived from: a stale or mismatched file would silently under-estimate row maxima and break the upper bound that pruning depends on, so anything that does not match the *current* ``models.bin`` byte-for-byte is rejected and the tables are derived from the models directly (slower, but always correct). r"z rowmax.binr#rNzpchardet rowmax.bin is missing or does not match models.bin; deriving row maxima from the models (slower startup)r$r%c4i|]\}}|t|Sr)r )rr?tables r zget_rowmax..s' K K Kjc5C"5)) K K Kr)r5r'r(r)r*r+hashlibsha256digestFileNotFoundErrorOSErrorr r=r,r-r.r9)r6r)r1 models_digesttabless r get_rowmaxr\s (]]F   % %&6 7 7E~~l++6688 NN< ( ( 3 3 5 5  &((   w ' dM4<< @ @F      C      L KFLLNN K K KKsA2B""B:9B:cXt}tjd} |d}t j|d}n#ttf$rd}d}YnwxYwt||t|}||S|rtjdtdt!fd |DS) aReturn per-model norms for language scoring, keyed by model key. The language tiers score in an IDF-weighted space (see :meth:`BigramProfile.for_language`), where a model's length is its norm after each weight is scaled by the square root of its bigram's IDF. Loads the precomputed ``langnorm.bin`` (written by ``chardet.models._format.write_model_artifacts`` in the same model order as ``models.bin``) and, like :func:`get_rowmax`, rejects a file whose embedded digest does not match the current ``models.bin``: a stale norm table would tilt every language comparison. The fallback derives the norms from the tables directly, which is correct but costs a pass over every model. r"z langnorm.binr#rNzvchardet langnorm.bin is missing or does not match models.bin; deriving language norms from the models (slower startup)r$r%c8i|]\}}|t|Sr)r)rr?rSidfs rrTz&get_language_norms..s* R R RZS%C$UC00 R R Rr)r5r'r(r)r*r+rUrVrWrXrYrr=r,r-r.get_idf_weightsr9)r6r)r1rZnormsr_s @rget_language_normsrbs1]]F   % %&6 7 7E~~n--88:: NN< ( ( 3 3 5 5  &((   w '  t]DLL A AE      G        C R R R R6<<>> R R RRsA2B##B;:B;ct}tdtdDtfd|DS)a{Return the 65536-byte root-IDF table the language profile multiplies by. Entry ``i`` is ``round(16 * sqrt(idf[i]))``. The language tiers compare an IDF-weighted profile against models weighted by the square root of IDF; folding the model-side factor into the profile, as the product of this table and the IDF table per occurrence, leaves the model tables untouched, so the same scoring loop serves both metrics and only the norm differs (see :func:`get_language_norms`). A byte table keeps the profile walk on native indexing: a list of the products costs about as much again as the walk itself. c3hK|]-}tttj|zV.dSN)round_LANGUAGE_SCALE_FACTORmathsqrt)rvs rrz%get_language_scale.. s8SS1%.1=>>SSSSSSrrc3(K|] }|V dSrfr)rrkrootss rrz%get_language_scale.. s'''aq''''''r)r`bytesrange)r_rms @rget_language_scalerpsU   C SSc SSS S SE ''''3''' ' ''rctjdd}|}t |dkr2t jdt |dtdddzS|S) uReturn a 65536-byte IDF weight table for bigram profile construction. Loads a precomputed table from ``idf.bin`` (generated at training time). For each bigram index, the weight reflects how discriminative that bigram is across all models: - Bigrams in every model (common ASCII) → weight 1 (minimal signal) - Bigrams in one model → weight 255 (maximum signal) - Bigrams not in any model → weight 1 (unknown, treat as neutral) r"zidf.binz chardet idf.bin has wrong size (z"), falling back to uniform weightsr$r%) r'r(r)r*r+lenr,r-r.r/s rr`r`s   # #$4 5 5 > >y I IC >>  D 4yyE  .s4yy . . .       Krc eZdZdZdZdeddfdZededdfdZde e d e e d e e d e ddf d Z ed e e e fddfdZ dS) BigramProfileupPre-computed bigram frequency distribution for a data sample. Computing this once and reusing it across all models reduces per-model scoring from O(n) to O(distinct_bigrams). Each bigram is weighted by its IDF (inverse document frequency) across all models — bigrams unique to few models get high weight, bigrams common to all models get weight 1. ``nonzero`` lists the indices carrying weight, in first-encounter order. :meth:`for_language` builds the same profile under the language tiers' heavier weighting instead. The weights are reachable two ways, and which one a profile fills depends on how it was built: * ``idx_arr``/``val_arr`` — parallel ``array('i')`` buffers, what :func:`score_with_profile` reads for a streaming profile. Filled by the streaming constructor only. * ``values`` — a plain list parallel to ``nonzero``, filled by :meth:`from_weighted_freq` for the small focused profiles confusion resolution builds, which are scored inline instead. * ``gather``/``dense_vals`` — an :func:`operator.itemgetter` over ``nonzero`` and the weights it picks out of the dense table, filled for a streaming profile only when the kernel is not compiled and the interpreter is CPython. ``row_freq`` aggregates the weights by lead byte (256 entries) and ``nonzero_rows`` lists the lead bytes with non-zero total; together with per-model row maxima (:func:`get_rowmax`) they let statistical scoring compute a cheap upper bound on a model's score. **Input limit.** Weights are packed as int32, which holds any value an input of at most 16 MB can produce (``255`` per occurrence against a 2.1-billion ceiling). Detection truncates to ``max_bytes`` long before that; construct a profile directly from a larger buffer and packing raises :exc:`OverflowError`. ) dense_valsfreqgatheridx_arr input_normnonzero nonzero_rowsrow_freqval_arrvalues weight_sumr1r Nc"t|dz }|dkrGg|_g|_g|_t\|_|_g|_g|_d|_ d|_ dSt}dgdz}g}d}t|D]o}||}||dz} || krt|r)|dz| z} || } || dkr|| || xx| z cc<|| z }p|||g|dS)a?Compute the bigram frequency distribution for *data*. Each bigram is weighted by its IDF (inverse document frequency) across all loaded models. Bigrams unique to few models get high weight; bigrams common to all models get weight 1. :param data: The raw byte data to profile. rrNrr)rtrxr|r _EMPTY_PACKEDrzrr~r}rr{r`ro_ASCII_WHITESPACE_TABLEr<_finish) selfr1 total_bigramsr_rxr|w_sumr b1b2idxws r__init__zBigramProfile.__init__[s=D A A  #%DI&(DL%'DK)6 &DL$,')DM+-D #$DO%(DO F#+}%%  AaBa!eB Rxx3B7x7b.CCACyA~~s### IIINIII QJEE T7B.....rcV|d}t|dz }|dkr|St}dgdz}g}t|D]h}||}||dz} || krt|r)|dz| z} || dkr|| || xx|| z cc<it } d} d} |D]'} || }| ||zz } || | z}||| <| |z } (|||g| ttj | z|_ |S)uBuild the profile the language tiers score with. Same bigram walk as the streaming constructor, but each occurrence carries IDF times :func:`get_language_scale` (a byte-quantized root of IDF) rather than IDF alone. Scored against :func:`get_language_norms`, this is the cosine between the IDF-weighted input and models weighted by the square root of IDF. The plain cosine lets one high-IDF bigram that two languages share, such as the UTF-8 pair for ``é``, outweigh every bigram that tells them apart: it rates the model by the share that bigram takes of its *unweighted* length, so an accent-heavy language wins French text on the strength of ``é`` alone. Weighting the model side too charges each model for its rare bigrams, and the common bigrams decide again. :param data: The raw byte data to profile. :returns: A profile for use with ``language_weighting=True``. rrrrrr) rtr`rorr<rprrhrirjr{)clsr1profilerr_rxr|r rrrscalernorm_sqrkrs r for_languagezBigramProfile.for_languagesh(#c((D A A  N#+}%% " "AaBa!eBRxx3B7x7b.CCyA~~s### IIIS !IIII#$$  CS A q1u GE#JADI QJEEgr51113di6H6HHrrxr|rrc ||_||_||_d}dgdz |rItt |D]+}||}|||zz } ||dz xx|z cc<,n(|D]%}||}|||zz } |dz xx|z cc<&t j||_ |_ fdtdD|_ |stst\|_ |_ nt||\|_ |_ |strgn||_d|_d|_|jrAt |dkr0t$r+t'j|} | |_| ||_dSdSdSdS)aQStore the frequency data and derive the norm and row aggregates. Single finalization path shared by both constructors so pruning fields cannot silently diverge between them. Exactly one of *freq* and *values* carries the weights, never both. The streaming constructor passes the dense *freq* table it had to build and leaves *values* empty; the sparse constructor fills *values* parallel to *nonzero* and passes no table, which is what lets it skip a 65536-entry allocation per call. Neither is retained. *freq* is read here to derive the norm, the row aggregates and the packed buffers, and is then dropped -- scoring reads ``idx_arr``/``val_arr`` or ``values``, never the dense table, so keeping it alive would pin 512 KB per profile for nothing. rrrc$g|] }| | Srr)rrr~s r z)BigramProfile._finish..s"EEEB EREEErNrr)r|rrrortrirjr{r~r}_KERNEL_COMPILEDrrzrrrxryrw _GATHER_DOToperator itemgetter) rrxr|rrrr rkrryr~s @rrzBigramProfile._finishs.  $ cCi  (3w<<(( / /1I1q5 q)))Q.)))) /  ( (I1q5 """a'"""")G,,  EEEE%**EEE  E) E)6 &DL$,,)5gt)D)D &DL$,!@%5@BBD DH +- 9 +W))k)('2F DK$fTllDOOO + +))))r weighted_freqc|d}g}g}d}|D]6\}}|r/||||||z }7|g||||S)uCreate a BigramProfile from pre-computed weighted frequencies. Computes ``weight_sum`` and ``input_norm`` from *weighted_freq* to ensure consistency between the stored fields. Deliberately does not build the dense 65536-entry ``freq`` table the streaming constructor uses. Callers here pass a handful of bigrams — confusion resolution's focused profiles hold a median of eight — and allocating a 65536-element list per call cost about 24us, which measured as roughly 40% of the whole bigram-rescore stage. Scoring reads ``nonzero``/``values``, so the table is never needed. :param weighted_freq: Mapping of bigram index to weighted count. :returns: A new :class:`BigramProfile` instance. rr)r9r<r)rrrr|rrrcounts rfrom_weighted_freqz BigramProfile.from_weighted_freqs$#c(('--//  JC s### e$$$GVU333r)__name__ __module__ __qualname____doc__ __slots__rnr classmethodrr=intrdictrrrrrvrv(s""H I,/U,/t,/,/,/,/\44/444[4l9+3i9+c9+S 9+  9+  9+9+9+9+vtCH~/[rrvFlanguage_weightingrr@zbytes | bytearray | memoryview model_keyrcTt|tst|}|jdkrdS|rtn t }|r||nd}|Y|rt |t}n:d}tdD]}||}|r|||zz }tj |}|dkrdS|j } |j } | rbrcygagdRARE_LANGUAGESzxxgQ?)demote_thin_rarerr1rc|s|dSt}||}|dS|+|rt|nt|}d}d}d} d} |D]C\} } } t || | |}||kr|}| }|r| t vr| t kr || kr|} | } D|r||t vr| || z tkr| }||fS)aScore data against all language variants of an encoding. Returns (best_score, best_language). Uses a pre-grouped index for O(L) lookup where L is the number of language variants for the encoding. If *profile* is provided, it is reused instead of recomputing the bigram frequency distribution from *data*. :param data: The raw byte data to score. :param encoding: The canonical encoding name to match against. :param profile: Optional pre-computed :class:`BigramProfile` to reuse. :param demote_thin_rare: Pass true only when the caller has judged the *original* input thin (under :data:`_THIN_RARE_MAX_BYTES` before any transcoding). A :data:`RARE_LANGUAGES` winner that leads the best prevalent-language variant by less than the measured noise band is then reported under the prevalent language instead. The length judgment deliberately lives with the caller: this function may receive transcoded bytes or a profile without its source data, so ``len(data)`` here is not a reliable proxy for input size. Language-fill callers pass this; encoding-ranking callers must not, so that candidate ordering stays byte-identical. :param language_weighting: Score in the language tiers' IDF-weighted space (see :meth:`BigramProfile.for_language`). A *profile* passed alongside must have been built by that constructor. The encoding-ranking caller leaves this off for the same reason as *demote_thin_rare*: its scores feed the confidence and ordering of encodings, and the weighted cosine lives on a different scale. :returns: A ``(score, language)`` tuple. The score is always the best cosine similarity across variants; the language matches it except when ``demote_thin_rare`` fires, in which case the label is the best prevalent-language variant's while the score remains the rare winner's. N)rNrr)rGrJrvrrr ART_LANGUAGE_THIN_RARE_MARGIN)r1rHrrrr>variants best_score best_langbest_prevalentbest_prevalent_langrAr@rss rscore_best_languagerusDR GOy OOEyy""Hy" %M & &t , , ,t$$  J IN&*"*''eY  UI:L    z>>JI  'N** $$N""N"&  (  !  ' '  +  '*; ; ;' y  r)rrf)Crarray functoolsrUimportlib.resourcesr'rirsysr,chardetrchardet._kernelrrchardet.models._formatrrrr r chardet.registryr r r__file__endswithrimplementationnamerrnrorrrstr__annotations__r_encrt languagescachetuplefloatr2r5r=rErGrLboolrNrPr\rbrhrpr`rvr frozensetrr_THIN_RARE_MAX_BYTESrrrrrrsg   4444444476666666S!!;5;s#3#34 #,,_==  %2  %  INs    $&$sCx.%%% HO  88D s4>a&*nQ&7# "5c5j!14U 3C!CD""""("T#u*%"""" e  #tE#*eS012 23. +tCeC$Js,B&C!DDE++++ *S*S4Z****''''''"$sEz*""""  (LDe$(L(L(L(LV #SDe,#S#S#S#SR (E((((" 0ooooooooj;3 % ;3;3;3 ;3 +;3;3  ;3  ;3;3;3;3D"++C+C+C!D!D #DDD   %)U! #$ U!U!U! U!U!T !U!  U!  U! 5#* U!U!U!U!U!U!r