|jERUdZddlZddlmZddlmZddlmZddlm Z m Z m Z ddl m Z ddlmZmZd d d Zeeeeeffed <d ZehdZejdejZejdejZejdejejzZejdejZejdZ ejdZ!ejdejZ"ejdejZ#e$e%ddZ&dZ'de$de dzfdZ(dededzfdZ)de$de dzfdZ*d e$de$fd!Z+dededzfd"Z,d#e$dee$e$ffd$Z-d e$dee$fd%Z.de$de dzfd&Z/de$d'e d(eede fd)Z0de$d*ede1fd+Z2de$d*edzdedzfd,Z3dS)-zStage 1b: charset declaration extraction (HTML/XML/PEP 263) and promotion. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N)Iteratordecodes_without_error) EncodingEra)DETERMINISTIC_CONFIDENCEDetectionResultPipelineContext)compute_structural_score)REGISTRYlookup_encoding) shift_jiscp932)euc_krcp949)shift_jis_2004r_MARKUP_SUPERSET_PROMOTIONSi>utf-16utf-32 utf-16-be utf-16-le utf-32-be utf-32-les-<\?xml\s[^<>]*encoding\s*=\s*['"]([^'"]+)['"]s]*)>s4([^\s=/>]+)(?:\s*=\s*(?:(["'])(.*?)\2|([^\s>"']+)))?s@charset[\t\n\f\r ]*=[\t\n\f\r ]*(?:(["'])([^"']+)\1|([^\s;"']+))s&^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)s^[ \t\f]*(?:[#\r]|$)z<(?:meta|\?xml)[^<>]*zM(?:charset|encoding)\s*=\s*(?:[^\sA-Za-z0-9._-]\s*)?([A-Za-z][A-Za-z0-9._-]+)g?datareturnc|dt}t|t|dtz }|t|tzkrdS|dd}t |D]}t| dD]}t| d }|nt|j tjzrOt!||r?| dddk}t#|t$d|rdnd ccSdS) uLook for a charset declaration in EBCDIC-encoded markup. The ASCII regexes cannot see declarations in EBCDIC bytes, so when the head looks like EBCDIC text (dominated by high bytes), decode it through cp037 — the EBCDIC page whose letter and digit positions are shared by all variants — and scan the decoded text. Only declarations naming a MAINFRAME-era encoding are honoured, and the declared encoding must decode the whole window. The MIME type follows the tag the declaration sits in. Ncp037replace)errorsr?text/xml text/htmlencoding confidencelanguage mime_type) _SCAN_LIMITlen translate_MARKUP_HIGH_BYTES_EBCDIC_SCAN_MIN_HIGH_FRACTIONdecode_EBCDIC_TAG_REfinditer_EBCDIC_DECL_REgroupr stripr erar MAINFRAME_validate_bytesrr)rhead high_countdecodedtagmatchr&is_xmls /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/chardet/pipeline/markup.py_detect_ebcdic_declarationr?fsj   DTS6H!I!IJJJJCII >>>>tkk')k44G &&w//$--ciill;;  E&u{{1~~';';'='=>>H$X&*[-BB%#D(33%1aC/&%7!,2Cjj   4raw_namec|dddd}|dks|drdS|dvs|drd St|S) aResolve a coding cookie the way :func:`tokenize.detect_encoding` does. Python folds the ``utf-8-*`` spellings to UTF-8 and ``latin-1``, ``iso-8859-1``, and ``iso-latin-1`` (with or without a suffix) to Latin-1 before it consults the codec registry, so ``iso-latin-1`` is a valid cookie even though no codec answers to it, and ``utf-8-sig`` names plain UTF-8 rather than a file that starts with a mark. N _-utf-8zutf-8-)zlatin-1z iso-8859-1z iso-latin-1)zlatin-1-z iso-8859-1-z iso-latin-1-z iso8859-1)lowerr startswithr )rAnames r>_pep263_encodingrJs CRC=   ( (c 2 2D w$//(33w 7774??3<<7{ 8 $ $$r@c`|dt}d|vrdS|dd}t|d}|St |dkr@t |dr t|d}|r |dd}n#ttf$rYdSwxYwt|t|}|t|tdd SdS) aCheck the first two lines of *data* for a PEP 263 encoding declaration. PEP 263 declarations (e.g. ``# -*- coding: utf-8 -*-``) are only valid on line 1 or line 2 of a Python source file. Both lines have to start inside the first :data:`_SCAN_LIMIT` bytes, the same bound the markup scan uses. :param data: The raw byte data to scan. :returns: A :class:`DetectionResult` with confidence 0.95, or ``None``. N# rr!asciiz text/x-pythonr%)r*split _PEP263_REr<r+_PEP263_BLANK_REr3r/r4UnicodeDecodeError ValueError_declared_encodingrJrr)rr8linesr<rAr&s r>_detect_pep263rWs4   D 4t JJua E   U1X & &E }Ua,<,B,B58,L,L  q**  {{1~~,,W55;;==HH"J/   44 %d,`` comments. A declaration inside a comment is not a declaration, and the WHATWG prescan skips comments for that reason. As in the prescan, the dashes that open a comment may also close it (```` is a whole comment), and a comment that never closes runs to the end of the head. srNr@N)findappendjoin)r8startpartsposends r>_strip_commentsras IIg  E qyy E C 1** T#e)_%%%ii ** 7788E?? "Ag '3'' 1** LLcdd 88E??r@cn|dkrdSt|}|tvrdS|S)z9Resolve a charset label found in ASCII-compatible markup.zx-user-definedcp1252rF)rGr _SELF_CONTRADICTING_LABELS)rAr&s r>_markup_encodingres@~~+++xx((H---w Or@bodyci}t|D]a}|d}||vr.|d}||d}||||<b|S)z:Return the attributes of one meta tag, lowercased by name.r!rYN)_ATTR_REr1r3rG)rfattrsr<rIvalues r>_meta_attributesrls "E""4((  {{1~~##%% 5==  A =KKNNE  E$K Lr@c#Kt|D]!}t|d}|d}||r|VU|d}||d}|d}|||dkr]t|}|rA|dr|dn|d V#dS) aYield charset labels from real meta declarations, in document order. ``charset=`` inside some other attribute (a refresh URL, a description) is not a declaration. A declaration is a ``charset`` attribute, or the ``charset`` parameter of an ``http-equiv="content-type"`` tag. ``name="content-type"`` is wrong by the spec but common enough in the wild that it counts too. r!scharsetNs http-equivsnamescontents content-typerNrY) _META_TAG_REr1rlr3getr4rG_CONTENT_CHARSET_REsearch)r8r;rjcharsetequivcontentfounds r>_iter_html_charsetsrvs-$$T**KK 1..))J''  7==?? MMM  -(( =IIg&&E))J''  # ##%%88'..w77E K(- AJekk!nnnEKKNNJJJ#KKr@c|sdSt|dt}t|}|r |dd}n#ttf$rd}YnwxYwt||rt|nd}|t|tddSt|D]t} |d}n#t$rY7wxYwt|t|}|t|tddcSut|}||St|S)aScan the first bytes of *data* for a charset declaration. Checks for: 1. ```` 2. ```` 3. ```` (also with ``name`` in place of ``http-equiv``) 4. PEP 263 ``# -*- coding: ... -*-`` (first two lines only) Markup inside ```` comments is not scanned. A UTF-16 or UTF-32 label is read as UTF-8 and ``x-user-defined`` as windows-1252, as in the WHATWG prescan, since a declaration found in ASCII bytes cannot be describing UTF-16 text. A declaration counts only if the declared encoding decodes the whole of *data*, or, failing that, if its Windows superset does (see :func:`_declared_encoding`). A declaration that fails is skipped and the next one is tried, so a stale prolog does not hide a correct meta. :param data: The raw byte data to scan. :returns: A :class:`DetectionResult` with confidence 0.95, or ``None``. Nr!rOr#r%r$)rar*_XML_ENCODING_RErqr3r/r4rSrTrUrerrrvr?rW)rr8r<rAr&raw ebcdic_results r>detect_markup_charsetr|s0 t 4  - . .D  # #D ) )E  {{1~~,,W55;;==HH"J/   HHH % B"8,,,d    "!3$  #4((   zz'**0022HH!    H %d,promote_markup_supersetrTs.%+// 0FGGI$-!NMG##  H   D   C)$9O0PRUVVJ-dH]4KSQQN "" } 5 5 O T>22T=11 r@r&c"t||S)a}Check that the whole of *data* decodes under *encoding*. A declaration is an answer only if the caller can decode what chardet examined with it. The decoder stops at the first bad sequence, so a declaration that lies about the header fails as fast as a head-only check would, and a cut multi-byte sequence at the very end is tolerated as everywhere else. r)rr&s r>r7r7s !x 0 00r@c|dSt||r|St|}|t||dr|dSdS)aReturn the name to report for a declaration of *encoding*, or ``None``. When the declared encoding cannot decode the window but its Windows superset can (a declared EUC-KR page whose first CP949-only hangul sits past the header, a declared Shift_JIS page using NEC extensions), the superset is the answer the caller can use, so it is reported directly. :func:`promote_markup_superset` would reach the same name one window decode later. Nr!)r7rro)rr&rs r>rUrUsattX&&+//99Iy|!D!D| 4r@)4__doc__recollections.abcrchardet._utilsr chardet.enumsrchardet.pipelinerrr chardet.pipeline.structuralr chardet.registryr r rdictstrtuple__annotations__r* frozensetrdcompile IGNORECASEryrnDOTALLrirprQrRr0r2bytesranger-r.r?rJrWrarerlrvr|rboolr7rUr@r>rs $$$$$$000000%%%%%%WWWWWWWWWW@@@@@@66666666-!;;T#uS#X"67  'YLLL2:92=rz. >> 2:@MBI  !bjLMRZB C C 2:677*4bmDD"*TMU55u--.."&#U#/E####L%s%sTz%%%%&!!?T#9!!!!H%E0ssTz 5 T%,%7    KeKKKKK:@ @ /D*@@ @ @ @ F6 6"6s^6 6666r 1% 13 14 1 1 1 1UcDjS4Zr@