|j2UdZddlmZddlZddlmZddlmZideddhd ed d hd ed hd edhdedhdeddhdeddhdedhdedhdedhdedhdedhdehdd ed!d"hd#ed$hd%edhd&ed'hed(hed)hed*hed+hed,hed-hed.d/hed0hed0hed1d2hed3hed3hd4 Z d5e d6<d7Z d8e d9<d:Z d8e d;<d<fd]dAZ e e Zd5e dB<d^dGZiZd5e dH<e D]K\ZZeepeZedIeDZeeeezee<Le e dJKZd5e dL<d_dNZd_dOZd`dQZedRdShZdTe dU<dadXZdbd[Zdbd\ZdS)caAccuracy-evaluation tables and predicates. Tools for asking *"is this detection acceptable, given what was expected?"* Used by the test suite and by benchmarking/diagnostic scripts (e.g., ``scripts/compare_detectors.py``, ``scripts/diagnose_accuracy.py``). This module owns three kinds of tables: 1. **Directional supersets** (:data:`SUPERSETS`) -- detecting a superset encoding when the expected encoding is a subset is correct (e.g., detecting UTF-8 when expected is ASCII), but not the reverse. 2. **Bidirectional encoding groups** (:data:`BIDIRECTIONAL_GROUPS`) -- groups of encodings where any member is interchangeable with any other. 3. **Bidirectional language groups** (:data:`LANGUAGE_EQUIVALENCES`) -- ISO 639-1 codes for languages that are nearly indistinguishable by statistical detection (e.g., Slovak/Czech, Malay/Indonesian). The corresponding predicates are :func:`is_correct`, :func:`is_equivalent_detection`, and :func:`is_language_equivalent`. ) annotationsN)Callablelookup_encodingASCIIzutf-8cp1252zTIS-620z iso8859-11cp874z ISO-8859-11GB2312gb18030GBKBig5 big5hkscscp950 Shift_JIScp932shift_jis_2004zShift-JISX0213zEUC-JP euc_jis_2004z EUC-JISX0213zEUC-KRcp949CP037cp1140z ISO-2022-JP> iso2022_jp_2iso2022_jp_extiso2022_jp_2004z ISO2022-JP-1rrz ISO2022-JP-3rz ISO-8859-1z ISO-8859-2cp1250cp1251cp1256cp1253cp1255cp1254cp1257z utf-16-lez utf-16-bezutf-16z utf-32-lez utf-32-bezutf-32) z ISO-8859-5z ISO-8859-6z ISO-8859-7z ISO-8859-8z ISO-8859-9z ISO-8859-13zUTF-16z UTF-16-LEz UTF-16-BEzUTF-32z UTF-32-LEz UTF-32-BEdict[str, frozenset[str]] SUPERSETS))rrrtuple[tuple[str, ...], ...]BIDIRECTIONAL_GROUPS))skcs)ukrubgbe)msid)nodasvLANGUAGE_EQUIVALENCESc|SN)xs /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/chardet/evaluation.pyr6wsgroups normalizeCallable[[str], str]returncri}|D]0}tfd|D}|D]}|||<1|S)zJBuild a lookup: key -> frozenset of all equivalent keys in the same group.c3.K|]}|VdSr2r3).0nr9s r5 z%_build_group_index..|s+77A99Q<<777777r7) frozenset)r8r9resultgroupnormednames ` r5_build_group_indexrFusi )+F--777777777 - -D&,F99T?? # # - Mr7_LANGUAGE_EQUIVexpectedstrdetectedboolcV||krdSt|}|duo||vS)aCheck whether *detected* is an acceptable language for *expected*. Returns ``True`` when *expected* and *detected* are the same ISO 639-1 code, or belong to the same equivalence group in :data:`LANGUAGE_EQUIVALENCES`. :param expected: Expected ISO 639-1 language code. :param detected: Detected ISO 639-1 language code. :returns: ``True`` if the languages are equivalent. TN)rGget)rHrJrCs r5is_language_equivalentrNs<8t    ) )E   2U!22r7_NORMALIZED_SUPERSETSc#8K|]}t|p|VdSr2r)r>ss r5r@r@s/DDA**/aDDDDDDr7c$t|p|Sr2r)r?s r5r6r6soa.@.@.EAr7)r9_NORMALIZED_BIDIR str | Nonec||duS|dSt|p|}t|p|}||kS)aCheck whether *detected* names the very same encoding as *expected*. Only alias and case differences are normalized away, so ``windows-1252`` matches ``cp1252``. Unlike :func:`is_correct`, this credits neither supersets nor byte-order variants: detecting UTF-8 for an ASCII file is ``False`` here. This is the stricter convention some other detectors score themselves against. Reporting it alongside :func:`is_correct` makes the two conventions directly comparable instead of leaving the difference implicit. :param expected: The expected encoding name, or ``None`` for binary files. :param detected: The detected encoding name, or ``None``. :returns: ``True`` if both names resolve to the same encoding. NF)rlowerrHrJnorm_expnorm_dets r5is_exact_matchrZsc"4ux((cs r5r@z#_strip_combining..s2CC+*?*B*BC1CCCCCCr7)rbr9join)r]nfkds r5_strip_combiningrgs5   . .D 77CCdCCC C CCr7)¤€)rirhzfrozenset[tuple[str, str]]_EQUIVALENT_SYMBOL_PAIRSabcl||krdS||ftvrdSt|t|kS)uReturn True if characters *a* and *b* are functionally equivalent. Equivalent means: - Same character, OR - Same base letter after stripping combining marks, OR - An explicitly listed symbol equivalence (e.g. ¤ ↔ €) T)rjrg)rkrls r5_chars_equivalentrnsC Avvt 1v)))t A  "21"5"5 55r7databytesc||duS|dSt|p|}t|p|}||krdS ||}||}n#ttf$rYdSwxYw||krdSt |t |krdSt dt||dDS)uCheck whether *detected* produces functionally identical text to *expected*. Returns ``True`` when: 1. *detected* is not ``None`` and both encoding names normalize to the same codec, OR 2. Decoding *data* with both encodings yields identical strings, OR 3. Every differing character pair is functionally equivalent: same base letter after stripping combining marks, or an explicitly listed symbol equivalence (e.g. ¤ ↔ €). Returns ``False`` if *detected* is ``None``, either encoding is unknown, or either encoding cannot decode *data*. :param data: The raw byte data that was detected. :param expected: The expected encoding name, or ``None`` for binary files. :param detected: The detected encoding name, or ``None``. :returns: ``True`` if decoding with *detected* yields functionally identical text to decoding with *expected*. NFTc3<K|]\}}t||VdSr2)rn)r>rkrls r5r@z*is_equivalent_detection..4s1XX41a A&&XXXXXXr7)strict)rrVdecodeUnicodeDecodeError LookupErrorlenallzip)rorHrJrXrYtext_exptext_dets r5is_equivalent_detectionr|s.4ux((rs.#"""""$$$$$$,,,,,,+( YY* + ++( yy,011+(99gY''+( ii $$ +(  99i[ ! ! +(  II{G, - - +(G%5677+(ii!1 233+( ii())+(II~.//+( ii ""+( YYz " "+(.99RRRSS/+(0II~/?@AA1+(2II01223+(6))XJ''7+(8))XJ''9+(:)XJ'')XJ'')XJ'')XJ'')XJ''9hZ((ik233H:&&H:&&ik233H:&&H:&&U+(+(+( ++++h56&'2k     .@-?@U-V-VVVVV3333*465555$??,,YYGZ ?7 # # .wDiDDDDDDDG"7";";D))++"N"NQX"X$0B/A$E$E000     4####LDDDD8Ay88 6 6 6 6 .Y.Y.Y.Ybr7