|j"dZddlmZmZmZmZmZmZddlm Z ddl m Z m Z ddl mZdZdedeed fd ed eeeeed zffd Zdeed fd ed eeeeed zeeefeeeeeed zeefffdZdeed fd ed eeeeed zffdZdddedeed fded ee fdZd S)zStage 3: Statistical bigram scoring. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. ) BigramProfile_get_model_norms get_enc_index get_rowmaxscore_best_languagescore_with_profileDetectionResult)forced_encodings scoring_floor) EncodingInfo@data candidates.profilereturnNcg}|D]?}t||j|\}}|dkr||j||f@|S)zFScore every candidate fully (no pruning). Returns (enc, score, lang).)r)rnameappend)rrrscoresencslangs /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/chardet/pipeline/statistical.py _score_allrs\ 35F//%dCHgFFF4 s77 MM38Q- . . . Mc t}t}t}|j}|j}|j}g}g} |D]} || j} | | jr9t| D](\} \} }}| | j| ||| f)_t| D]\} \} }}||}d}|D]}|||||zz }||}|td}n|dkr |||zz }nd}| || | j| ||f| dd|| fS)u~Flatten candidate model variants for pruned scoring. Returns ``(mb_entries, sb_entries)`` where multi-byte entries are ``(enc, lang, table, key, variant_index)`` and single-byte entries are ``(upper_bound, variant_index, enc, lang, table, key)`` sorted by descending bound. The upper bound multiplies each lead byte's total profile weight by the model's maximum weight for that lead byte — at most 256 terms versus one term per distinct bigram for a full score. ``variant_index`` is the variant's position in the encoding index, so exact score ties resolve to the same variant the full path keeps. Nrinfrc|dS)Nr)es rz!_split_variants..Zs !A$rTkeyreverse) rrrrow_freq nonzero_rows input_normgetris_multibyte_cjk enumeraterfloatsort)rrindexnormsrowmaxr'r(r) mb_entries sb_entriesrvariantsvirtabler%rmub_dotb1 model_normubs r_split_variantsr<'s$ OOE   E \\FH'L#J@BJGIJDD99SX&&      *3H*=*= D D&&T5#!!38T5#r"BCCCC &/&9&9 D D "B"ucBF" 0 0"R&8B<//3J!5\\c!!zJ67   r2sxucB C C C C D OOO555 z !!rc t}t||\}}iiiddddtdtdtdzdtddf fd }|D]%\}}}} } ||t ||| || &|D]>\} } }}}} t } | | krn||t ||| || ?t fd D} t| }|rc|D]`}|j |vr t| |j gD]+\} \}}} ||j t ||| || ,afd |DS) ucScore candidates, skipping single-byte variants that provably cannot matter. Multi-byte variants are always scored fully — the orchestrator may later boost their confidence based on structural coverage, so no raw-score bound can rule them out. Single-byte variants are scored in descending upper-bound order (see :func:`_split_variants`) and skipped once their bound falls below the pruning contract's ``scoring_floor``: such variants can affect neither the winner, nor position 1, nor any candidate a rank correction can examine. Encodings that ``postprocess_results`` inspects regardless of rank (the common Western Latin trio for niche-Latin demotion, KOI8-T for the KOI8-R promotion) are force-scored when their trigger could fire. Because confusion resolution can promote position 1 or any candidate within the band into position 0 before those triggers are evaluated, the trigger check covers every encoding near the top, not just the statistical winner. Returns (enc, score, lang) tuples for encodings scoring above zero, in candidate order. renc_namerrNr5rc|}|||ks||kr||krdS||<||<||<| kr|dS|kr || dS| kr| dSdS)N)r*) r?rrr5prev best_lang best_scorebest_vitop1top1_enctop2s rrecordz_score_pruned..records~~h''  Ta4iiB'(BSz!_score_pruned..s'GGGdaA4F4F4F4F4Frcg|]A}|jddk!|j|j|jfBS)r)r*r)rJrrBrCs rrLz!_score_pruned..sU     >>#(C ( (3 . . :ch'38)<= . . .r) rr<strr-intrr itemsr rr,r*)rrr/r2r3rHr?rr6r%r5r; thresholdnear_topforcedrrBrCrDrErFrGrKs @@@@@@@r _score_prunedrT^sR2 OOE,ZAAJ #%J')I GH D DcDjcd&+5LL&$sBx+GUC@@$KKKK.8 L L*B$s"$-- >> Ex+GUC@@$KKKK"$--MGGGGj..00GGGH h ' 'F T T TCxv%%+4EIIch4K4K*L*L T T&&T5#sx!3GUC!H!H$PRSSSS T        rF) full_rankingrUc|r|sgSt|}|jdkrgS|st|jtkrt |||}nt ||}|ddd|DS)aScore all candidates and return results sorted by confidence descending. :param data: The raw byte data to score. :param candidates: Encoding candidates to evaluate. :param full_ranking: When ``True``, score every candidate fully so the returned list is complete (needed by ``detect_all``). When ``False`` (the default), single-byte candidates that provably cannot affect the top of the ranking may be skipped; the winner, position 1, and all candidates within the confusion band of the top score are identical to the full ranking. :returns: A list of :class:`DetectionResult` sorted by confidence. rc|dS)Nr!)xs rr#z"score_candidates..s adrTr$c:g|]\}}}t|||S))encoding confidencelanguager )rJrrrs rrLz$score_candidates..s=    D!T !dCCC   r)rr)lennonzero_MIN_NONZERO_FOR_PRESCREENrrTr.)rrrUrrs rscore_candidatesras$ z D!!GS   4s7?++.HHHD*g66z733 KKNNDK111  #   r)__doc__chardet.modelsrrrrrrchardet.pipeliner chardet.pipeline.postprocessr r chardet.registryr r`bytestuplelistrNr-rrOr<rTboolrar!rrrks.-,,,,,HHHHHHHH))))))    lC'(   %UC$J& '(    4"lC'(4" 4" sC$JsC/ 01uc3d E36 78:4"4"4"4"n[lC'([ [ %UC$J& '([[[[D """ "lC'(" "  / """"""r