|j* UdZddlmZmZmZddlmZmZmZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZddlmZehdZeeed <ehd Zeeed <ehd Zeeed <ehdZeeed<ehdZ eeed<eeee dZ!e"eeefed<ehdZ#eeed<de!$DZ%e"ee&fed<e&e#Z'e&ed<de&de de de(fdZ)de*e de fdZ+de&d e*e de*e fd!Z,de&d e*e de*e fd"Z-d#Z.d$d$d%d&Z/e"eefed'<eZ0d(Z1d)Z2d*Z3d+edefd,Z4de&d+ed-d.de(fd/Z5de&d e*e de*e fd0Z6d e*e d1ede*e fd2Z7de&d e*e de*e fd3Z8d4Z9d5Z:d e*e de*e fd6Z;de&d e*e de*e fd7Ze9d;ezzZ?dZAd?e*ede*efd@ZBdAdBde&d e*e d9e(de*e fdCZCdDS)Eu\Stage 13: post-processing rank corrections. After statistical scoring produces a ranked list of candidates, a chain of rank corrections fixes up the ranking when bigrams alone are insufficient — see :func:`postprocess_results` for the order. The steps: dead-heat priors (superset preference, era prevalence), rare-language arbitration (ADR-0005), confusion-group resolution (delegated to :mod:`chardet.pipeline.confusion`), niche Latin demotion, KOI8-T promotion, classic-Mac line-ending promotion, and last of all the decode-safety flip, which hands a winner whose only multi-byte evidence is an undecodable trailing sequence to the best rival that can decode the caller's complete input. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. )dangling_tail_with_ascii_prefixdecodes_completelydecodes_without_error) ART_LANGUAGERARE_LANGUAGES get_enc_index) _COMPAT_NAMES)DetectionResult)CONFUSION_BANDCONFUSION_FLOOR_RATIOSTRICT_TIER_MAX_CONF_comparable_languagesarbitrate_distinguishing_bytesconfusion_pair_winnerdiffering_high_bytesresolve_confusion_groups)REGISTRY> iso8859-1 iso8859-15cp1252_COMMON_LATIN_ENCODINGS>._ISO_8859_10_DISTINGUISHING>rrrrrrr r!r"r$r%r&r'r(r)r+r,r-r.r/r0r1r2r:rC_ISO_8859_14_DISTINGUISHING>rIrJrKrL_WINDOWS_1254_DISTINGUISHING>r3r4r5r6r7r8r;rNrJ_HP_ROMAN8_DISTINGUISHING)z iso8859-10z iso8859-14cp1254z hp-roman8_DEMOTION_CANDIDATES> rrrr)_KOI8_T_DISTINGUISHINGc4i|]\}}|t|S)bytes).0encbyte_sets /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/chardet/pipeline/postprocess.py rrs1&&&*S(Cx&&&_DEMOTION_DELETE_KOI8_T_DELETEdatatoptargetreturnc |jpd}t|dt|t|krdS|j|jz t krdSt |||jpdt||jdnt|jf|jdnt|jf}||kS)u}Return True if *top*, a demotion candidate, has no byte evidence over *target*. Callers guarantee ``top.encoding`` is in :data:`_DEMOTION_CANDIDATES`. Two questions, cheapest first. Does *data* contain any byte the candidate decodes differently from ISO-8859-1? If not, the data is equally valid under both encodings, nothing at the byte level favors the candidate, and it is demoted. If such bytes are present, do they favor the candidate? Presence alone is symmetric evidence: a Windows-1252 file whose only non-ASCII letter is an ``Ö`` carries 0xD6, which HP-Roman8 reads as ``ø``, so both candidates "contain" the byte and the question is which reading holds up. That is a confusion-style arbitration between the candidate and its swap target on the distinguishing bytes alone (see :func:`~chardet.pipeline.confusion.arbitrate_distinguishing_bytes`), each side scored under the variant that actually won its slot. The models decide when they can: one Welsh ``ŵ`` keeps ISO-8859-14 because the Welsh model knows that bigram and the Windows-1252 reading is a ``ð`` no Welsh model has seen. When the models are silent, word shape decides: a Kven ``đ`` keeps a Finnish file on ISO-8859-10 because a letter between letters beats the superscript ``¹`` Windows-1252 reads there. A lone ``Ö`` mid-word decides nothing either way, and the more prevalent encoding takes the evidence-free tie. The arbitration is only asked when the candidate's lead over the swap target is within :data:`~chardet.pipeline.confusion.CONFUSION_BAND`. A win by more than the band was decided on the full statistics, and re-litigating it on a handful of bytes is neither sound nor free: the scan is Python level, and mainstream Turkish text tops as windows-1254 with hundreds of distinguishing bytes and a lead of 0.2. NTF languages_a languages_b) encodinglen translatert confidencer rralanguage frozenset)rvrwrxrwinners rq_should_demoters@|!rH 4>>$ 0 : ; ;<<D IIt ~))N::u + 2X&L0DDi6P6P"O3DDFOCU9V9V F X rs candidatescvtd|Dfd|D}t|dS)aSPick the common Latin candidate that replaces a demoted top. Among the candidates within :data:`_DEAD_HEAT_EPSILON` of the highest-scoring one, era prevalence chooses (windows-1252 over iso-8859-1): inside that band the confidence order is noise, the very premise of the demotion. A candidate trailing the best common Latin by more than the epsilon lost to it on real evidence and stays put. Equal era ranks (iso-8859-1 against iso-8859-15, both legacy ISO) keep confidence order, since ``min`` returns the first of equals and the candidates arrive ranked. c3$K|] }|jV dSN)rrnrs rq z_swap_target..s$55QAL555555rsc:g|]}|jz tk|Srl)r_DEAD_HEAT_EPSILON)rnr lead_confs rq z _swap_target..s+WWWQ AL(@DV(V(Vq(V(V(Vrsc.t|jpdS)Nr{) _era_rankr)rs rqz_swap_target..si 0@b&A&Ars)key)maxmin)rin_bandrs @rq _swap_targetr sQ55*55555IWWWW*WWWG wAA B B BBrsresultsc t|dks|djtvr|Sd|ddD}|s|St| t ||d s|S|dj|dj}t j| j j} fd|D}|r |djn|fd|D}|g||S) aDemote a niche Latin top that its distinguishing bytes do not support. Some bigram models (iso-8859-10, iso-8859-14, windows-1254, hp-roman8) can win on data that contains only bytes shared with the common Western Latin encodings, or on a lone shared byte the models cannot arbitrate. When :func:`_should_demote` finds no byte-level evidence for the winning encoding, promote the swap target :func:`_swap_target` picks among the common Latin candidates and push the demoted encoding to last. The demoted entries take the confidence of the candidate they now sit behind. Rank position alone does not survive the trip out to callers: ``detect_all`` re-sorts by confidence, and a stable sort hands an entry that kept the top score its old place back. :param data: The raw byte data the results were produced from. :param results: A list of :class:`DetectionResult` ranked by confidence. :returns: A new list (or the same list) with the demotion applied. rc.g|]}|jtv|Srl)rrrs rqrz'_demote_niche_latin..7s%RRRAJ:Q,Q,Q!,Q,Q,QrsNc2g|]}|jk |u|Srlr)rnxdemoted_encodingrxs rqrz'_demote_niche_latin..Bs, W W WAAJ2B$B$BqPVarsc g|]A}|jk t|jt|j|j|jBSrl)rr rrr mime_type)rnrr tail_confs rqrz'_demote_niche_latin..Ds]     :) ) )  J  i ( ( J K    * ) )rs) rrrarrrr rr) rvrrtop_confpromotedothersdemoted_entriesrrrxs @@@rq_demote_niche_latinrs2. 7||a71:.6JJJRRWQRR[RRRJ  * % %F $ F 3 3qz*qz$H6?F4DHX W W W W W W WF)/=r %%XI        O  0v 0 00rsc|r|djdkr|Stdt|Dd}||St|dt t|krt ||S|S)aPromote KOI8-T over KOI8-R when Tajik-specific bytes are present. KOI8-T and KOI8-R share the entire 0xC0-0xFF Cyrillic letter block, making statistical discrimination difficult. However, KOI8-T maps 12 bytes in 0x80-0xBF to Tajik-specific Cyrillic letters where KOI8-R has box-drawing characters. If any of these bytes appear, KOI8-T is the better match. rkoi8-rc36K|]\}}|jdk|VdS)koi8-tNr)rnirs rqrz!_promote_koi8t..`s2QQDAq!*:P:Pa:P:P:P:PQQrsN)rnext enumeraterrru_promote_to_top)rvr koi8t_idxs rq_promote_koi8trQs gaj)X55QQIg$6$6QQQSWXXI 4>>$ / /00CII==w 222 Nrsg-C6?cp932cp949) shift_jisshift_jis_2004euc_kr_DEAD_HEAT_SUPERSETSi@rcftj|}|dSt|j}|| zS)zHReturn the lowest era bit for *encoding* (lower = more prevalent today).Ni@)rgetintera)rinfors rqrrs4 < ! !D |w dh--C #:rsrz str | Nonect|}|sdS|dt}t}|d}t dt |D]3}||}|dks|dkr||dz|z|}4|sdS|D] \} } } || |kr|D]} | | rdS!dS)u0Return True if *encoding*'s winning model weights a high-byte bigram present in *data*. A candidate whose model assigns zero weight to every non-ASCII bigram in the data earned its statistical score purely from ASCII bigrams — noise that cannot distinguish encodings. Only the variant that actually won (*language*) counts: another language's variant having weight for those bytes says nothing about why *this* result is on top. Only called on dead heats, so the Python-level scan of the (capped) data is off the hot path. FNrrrbT)rr_EVIDENCE_SCAN_MAX_BYTESsetrangeradd) rvrrvariantswindowseenprevrblangtable_keyidxs rq_has_high_byte_evidencers""8,,H u +++ ,FUUD !9D 1c&kk " " 1I 4<<199 HHdai1_ % % % u%eT  DH$4$4   CSz ttt   5rsc |r|dnd}||jt|dkr|Sd}t|j}tdt|D]J}||}|j|j|jz t krnt|j}||kr|}|}K|dkr|St ||j|jst||S||jpd}t|j|td|j||jfD} t||j|t|j|| | } | |krt||S|S)u]Break statistical dead heats in favor of the more prevalent era. When several encodings score within :data:`_DEAD_HEAT_EPSILON` of the top result, the ranking among them is mostly an artifact of ASCII-bigram noise. Promote the candidate from the most prevalent era (modern web > legacy ISO > Mac > regional > DOS > mainframe) so evidence-free dead heats resolve to the likeliest real-world answer. A top result whose models carry no weight for any high-byte bigram in the data has no evidence at all and yields outright. One whose models do weight an observed bigram is not thereby safe: an English file with one capital ``É`` ranks MacRoman first because the MacRoman model reads 0xC9 as the ellipsis English text is full of, a lead worth 2e-5. Such a top is arbitrated against the prevalent candidate on the bytes the two read differently, under the languages the two can be compared in (see :func:`~chardet.pipeline.confusion.arbitrate_distinguishing_bytes` and confusion's ``_comparable_languages``): the Windows-1252 French model knows ``École`` even when the English one does not, while its Icelandic model may not read a Welsh ``dŵr`` as ``dðr`` against an encoding that models no Icelandic. The prevalent candidate is promoted only when it wins outright; a tie keeps the top, so an ISO-8859-1 result tied with Windows-1252 on data without C1 bytes stays where the statistics put it. Genuine MacRoman text never reaches the arbitration, since its hundreds of distinguishing bytes put Windows-1252 far outside the band. rNrrr{c3K|]}||V dSrrlrnrs rqrz1_prefer_prevalent_on_dead_heat..s4     rsr|) rrrrrrrrrrrrr) rvrrwbest_idx best_rankrrrankrival comparablers rq_prefer_prevalent_on_dead_heatrs< )'!**TC {cl*c'llQ.>.>H#,''I 1c'll # #   AJ :   >AL (+= = = E$$ )  IH1}} "4s| D D2w111 H  & ,"E&    wx'8'AB     J,   S\511 Fw111 Nrsrc|}t|j|dj|j|j}fdt |D}|g|S)zDMove ``results[i]`` to the top, carrying the current top confidence.rc&g|] \}}|k |Srlrl)rnjrrs rqrz#_promote_to_top..s" 7 7 7$!QQArs)r rrrrr)rrrrrests ` rqrrsb A GAJ)1:q{H 8 7 7 7)G,, 7 7 7D  t rsc~|r|dnd}||jt|dkr|St|j}||St dt|D]Q}||}|j|jz t krn.|j|kr"t||rt||cSR|S)zAPromote a Windows superset over its base encoding on a dead heat.rNrr) rrrrrrrrr)rvrrwsupersetrrs rq_promote_superset_on_dead_heatr s  )'!**TC {cl*c'llQ.>.>#'' 55H 1c'll # #// AJ >AL (+= = = E : ! !&;D(&K&K !"7A.. . . . Nrsg{Gz?g333333?cx|r|dnd}|8|j1|jtvs#|jtkst |dkr|St dt |D]S}||}|j|jz tkrn0|j|j3|jtvrt||cST|S)aDemote a rare-language winner that leads a prevalent rival by a coin flip. Fires only when the winner's language is in :data:`~chardet.models.RARE_LANGUAGES`, its absolute confidence is inside the evidence-free zone, and a prevalent-language candidate sits within :data:`_RARE_ARBITRATION_MARGIN`. Genuine rare-language text fails both gates: even short files score confidently, and their entire neighborhood is same-language variants. rNrr) rrrr _RARE_ARBITRATION_MAX_CONFIDENCErr_RARE_ARBITRATION_MARGINr)rrwrrs rq_arbitrate_rare_languager7s )'!**TC <  <~ - - >= = = w<AL (+C C C E : !3  :^ + +"7A.. . . . , Nrsc|r|dnd}||jt|dkr|St|jtkr|S|jt kr|S|ddks|dtkr|Stdt|D]}||}|j |j z tkrn|j{t|jtkr^td|j|jfD}t||j|j||jkrnt||cS|S)uNPromote a classic-Mac candidate when line endings are bare ``\r``. Classic Mac OS is the only platform that terminated lines with a lone carriage return, so data with several ``\r`` bytes and no ``\n`` is near-certainly Mac-era text. When a LEGACY_MAC candidate scores within :data:`_CR_MAC_BAND` of a non-Mac top result, promote it — unless the pair has a distinguishing-byte map and the byte-level evidence says the current top wins: a platform prior must not overturn direct evidence that confusion resolution may have just used to establish the top. rNr  rc3K|]}||V dSrrlrs rqrz2_promote_mac_on_cr_line_endings..ys0t?O?O?O?O?Ors)rrr_LEGACY_MAC_ERArrfindcount_CR_MAC_MIN_LINESrr _CR_MAC_BANDrrr)rvrrwrrlangss rq_promote_mac_on_cr_line_endingsrWst )'!**TC {cl*c'llQ.>.>/11  ||## yy1 5 1 14E E E 1c'll # #// AJ >AL (< 7 7 E : !i &;&;&N&N"%, !;E&dCL!*eLL<  "7A.. . . . Nrscvt||sdStj|}|dupt||S)a3Check that *data* decodes completely under *encoding* and its output name. The flip's promise is that the caller's ``data.decode(result)`` works, and the caller sees the *public* name: ``compat_names=True`` (the default) can remap to a strictly narrower codec (``euc_jis_2004`` is reported as ``EUC-JP``), so a rival must decode under both names to be promoted. ``prefer_superset=True`` can also narrow (cp125x leaves codepoints undefined that iso-8859-x maps), but that output transform checks the data itself before remapping. FN)rr r)rvrdisplays rq_decodes_under_public_namesrsE dH - -u))G d? ?0w???rsinput_truncatedcv|r|dnd}|sR|P|jIt|dks6td|ddDrt||js|St dt|D]8}||}|jt ||js't ||cS|S)aPromote a strictly decoding rival over a winner with no real evidence. Byte-validity filtering runs incremental decoders with ``final=False``, tolerating an incomplete multi-byte sequence at the end because detection input is often a prefix of a larger whole. When chardet examined the caller's *entire* input, that tolerance can hand back an encoding the caller's very next ``data.decode()`` will reject --- a four-byte ``iso-8859-1`` word ending in ``0xE1`` detected as utf-8 (issue #380). Fires only when *data* is the whole of what the caller handed over --- *input_truncated* is False --- the tail can actually hold a dangling sequence (a high byte in the final four, multi-byte winner), and the winner's tolerant decode is **non-empty pure ASCII** --- its only multi-byte evidence is the dangling tail itself. An empty tolerant decode (the whole input is one clipped sequence) is zero evidence, not ASCII evidence, and disqualifies the flip. The best-ranked rival that decodes the input completely under both its internal and public names then takes the top slot, regardless of the confidence gap: an all-ASCII-evidence winner detected nothing the rival did not also detect, and any statistical lead it holds comes from ASCII bigrams the rival matched equally well. The scan sees the ranking as given, which under ``full_ranking=False`` is pruned; if no listed rival decodes, the winner stands (measured across a 648-case accent-final sweep, the pruned ranking always carried a decodable rival). The pure-ASCII condition is what makes the unconditional flip safe. A short mid-character CJK cut has a correct answer that cannot decode the input --- flipping it to whichever single-byte codec happens to decode the bytes trades a right answer for a wrong one, and a 5-40 byte sweep measured exactly that under a gap-based rule (34 correct CJK answers lost, Big5 becoming cp1125). Such a winner has decoded real multi-byte characters and keeps its ranking; with the pure-ASCII condition in place, the same sweep measures zero lost answers at any gap. rNrc3"K|] }|dkV dS)rbNrl)rnrs rqrz+_prefer_decodable_on_tie..s&0019000000rsr)rranyrrrr)rvrrrwrrs rq_prefer_decodable_on_tiersR )'!**TC ; <  w<. s( 7 71$ $ 7 7 7 7 7 7rsrr)rextendrappend)rforceds rqforced_encodingsrs]F 7 7h 7 7 777/ -...8 h MrsFrct||}t||}t|}t||}t ||}t ||}t ||}t|||S)aApply rank corrections to the statistically scored results. Steps run in sequence, weakest evidence first: dead-heat priors (superset preference, era prevalence), then confusion-group resolution, niche Latin demotion, and KOI8-T promotion (byte-level evidence), and finally the classic-Mac line-ending promotion (platform evidence that should override the priors). The decode-safety tiebreak runs last of all: whatever the ranking settled on, a winner that cannot decode the caller's complete input, and whose own evidence is nothing but the undecodable tail, yields to the best-ranked rival that can decode it. :param data: The raw byte data the results were produced from. :param results: A list of :class:`DetectionResult` ranked by confidence. :param input_truncated: True when *data* is a chardet-made slice rather than the caller's whole input --- the ``max_bytes`` slice, the evidence-cap slice, or ``UniversalDetector``'s buffer cap. Any of them means the bytes here are not the whole story, so the decode-safety tiebreak, which reasons about what the caller will decode, stands down. :returns: A new list (or the same list) with rank corrections applied. r )rrrrrrrr)rvrrs rqpostprocess_resultsr s6-T7;;G,T7;;G&w//G&tW55G!$00GT7++G-dG<rs$ GFFFFFFFFF......,,,,,,                    &%%%%%+4)++3/8i///1/1/Ys^111n/8i   "/"/Ys^"""T09y(((00in-6I--9S>B.-** 33d3 #./*3LLL** #&&.B.H.H.J.J&&&$sEz"455555--_-o-RV----`CT/2CCCCC"01 01 / "01 /01010101f  / " /8((d38n  !%3,SWBE E / "E /EEEEPT/2stO?T  / " /J  $(  / " /@/ / / "/ /////d@e@s@t@@@@"? ? / "? ?  / ????T-q>/AAUu4tCyT#Y." "T"T"T "T / ""T "T  / "T"T"T"T"T"Trs