jQ1ZdZddlZddlZddlmZmZddlmZdedede fdZ eGd d Z eGd d Z ed GddZ eGddZeGddZeGddZeGddZeGddZejdZeGddZdefdZded edefd!Zdefd"Zded ed#edefd$Zd(d&edeefd'ZdS))aIData models for the chunking pipeline. ID formats are a public contract (stable across versions, snapshot-tested): - element: ``p{page}.b{box}`` (1-based original page number, 0-based box index on that page). Element ids are scoped to (document bytes, package version, parse options): parse modes that re-split boxes (e.g. HTML table rendering) produce different box indices, so ids must not be cached across parse-option changes. - chunk: ``c{n}`` - table: ``t{n}`` - figure: ``f{n}`` (same numbering as ``[Figure f{n}: WxH]`` placeholders) - section: ``s{n}`` N) dataclassfield)Optionalpageboxreturncd|d|S)z$Public element id: 'p{page}.b{box}'.pz.b)rrs /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/pymupdf4llm/helpers/chunking/models.py element_idr s t  s  ceZdZUdZeed<eed<eed<eed<eed<eed<eed<d Zeed <d Z e ed <d Z eed <dZ e e ed<dZe e ed<dZeed<dZe eed<dZeed<dZeed<dZeed<dZeed<dZeed<dZeed<dZe eed<dZe eed<dZe eed<dZe eed<eddZe eed<eed Z eed!<dS)" SentenceUnitzA sentence-level unit extracted from a LayoutBox. For table/figure boxes, this represents the entire box content as one unit. sent_idtext norm_textpage_no box_indexboxclassbboxr token_countfont_size_dominantfont_flags_dominantNline_gap_beforeline_gap_afterFis_heading_hintheading_level_hint is_list_itemis_table_contentis_figure_related is_footnoteis_header_footer is_captioncaption_target_typetable_markdown table_html figure_numberdefaultrepr image_datadefault_factoryr,_source_box_indices)!__name__ __module__ __qualname____doc__int__annotations__strtuplerrfloatrrrrrboolrr r!r"r#r$r%r&r'r(r)rr-byteslistr0r rr rrsLLL IIINNNLLLNNNMMM KKKK!$###    (,OXe_+++&*NHUO***"OT!!!(, ,,,L$"d"""#t###K"d"""J)-#--- %)NHSM((( $J $$$$(M8C='''#(%5"A"A"AJAAA!&d G G GGGGGGrrc*eZdZUdZeed<eeZeed<dZ e ed<dZ eed<d Z e ed <dZeed <dZeed <eeZeed <eeZeed<dZee ed<eeZeed<eedZeed<dS) ProtoChunkzAn intermediate chunk before split/merge refinement. Holds a contiguous run of units; ``text`` is only materialized at serialization time (``unit_range`` + precomputed token sums keep assembly free of repeated joining/tokenizing). chunk_idr/sent_idsrrrrr unit_range page_startpage_end box_indicesbboxesN primary_typetypesFr. _sentences)r1r2r3r4r5r6rr<rArr7rrDr8rErFrGrHrIrrJrKr rr r>r>Qs! MMMU4000Hd000D#NNNKJJHcd333K3335...FD...#'L(3-&&&%---E4---uT>>>J>>>>>rr>T)frozenc^eZdZUdZeed<eed<eed<eed<eed<eed<dZe ed <d S) ElementuOne layout box, chunk-addressable evidence with canonical text. The registry keeps every box — including header/footer boxes excluded from chunk text — so nothing the parser saw is unreachable (D8). idrrrrrFr$N) r1r2r3r4r7r6r5r8r$r:r rr rNrNmse GGG III HHHMMM KKK III"d"""""rrNc eZdZUdZeed<eeed<eed<eed<eed<dZ eeed<dZ eeed <e e Z e ed <dZeeed <dZeeed <dZeeed<dZeed<edefdZdS) TableChunkaMA table exposed in the tables view; links back to its chunk. ``markdown`` and ``html`` are mutually exclusive per parse mode: in HTML table mode the parser carries markdown=None and html is the canonical rendering; otherwise markdown is canonical (possibly ``""`` for a degenerate table) and html is None (D15). rOr?r rrNmarkdownhtmlr@headerscaptioncaption_element_id section_idrrrc0|j|jS|jpdS)z'Canonical rendering for the parse mode.NrB)rSrRselfs r rzTableChunk.texts! 9 9 }""r)r1r2r3r4r7r6rr5r8rRrSrr<rTrUrVrWrpropertyrr rr rQrQ}s  GGGsmOOO III KKK"Hhsm"""D(3-E$///GT///!GXc]!!!(, ,,, $J $$$K #c###X###rrQc"eZdZUdZeed<eeed<eed<eed<eed<dZ eed<d Z eeed <d Z eed <d Z eeed <d Z eeed<d Zeeed<ed dZeeed<edefdZd S) FigureChunkzFA figure/formula exposed in the figures view; links back to its chunk.rOr?r rrpicturerNocr_textrB placeholderrUrVrWFr*imagerc\t|jo|jSN)r:r_striprYs r has_textzFigureChunk.has_texts%DM;dm&9&9&;&;<<Section body text, assembled lazily from the element registry.z c3@K|]}|j |j|jVdSrc)rr$).0es r z$SectionChunk.text..sK  v 0 F      r)joinrprYs r rzSectionChunk.texts8{{   N      r)r1r2r3r4r7r6r5rjrrr<rkrlr8rmrrpr[rr rr rgrgs@@ GGG JJJ JJJOOOMMM(, ,,,t,,,D$,,, L%   !E$777OT777KuReUCCCIuCCC  c   X   rrgcleZdZUdZdZeed<dZeed<dZ e eed<e e Z e ed <dS) SectionNodezCA node in ChunkedDocument.hierarchy (root has level 0, no section).rBrhrriNrWr@children)r1r2r3r4rhr7r6rir5rWrrr<ryr rr rxrxscMME3OOOE3NNN $J $$$U4000Hd00000rrxceZdZUdZdZeed<dZeed<ee Z e ed<ee Z e ed<dZ e eed <ee Ze ed <dZeed <ee Ze ed <ee Ze ed <ee Ze ed<ee Ze ed<dZeed<dZe eed<dZe eed<dS) ChunkMetadatazsMetadata attached to a final chunk (payload-ready: every field is json-safe and lands in ``to_dicts()`` as-is).rrErFr@rHrJNrW section_pathr element_ids table_ids figure_idslistsFocr file_path page_count)r1r2r3r4rEr5r6rFrr<rHrJrWrr7r|rr}r~rrrr:rrr rr r{r{sk55JHc5...FD...%---E4--- $J $$$t444L$444Kd333K333eD111It111uT222J222%---E4---C $Ix}### $J $$$$$rr{z\s+ceZdZUdZeed<eed<dZeed<eeZ eed<edd d Z e eed <e d efd Z dS)ChunkaA finalized, retrieval-ready chunk. Treat a chunk as read-only once ``to_chunks()`` has returned it. ``content_hash`` is computed once and cached, and ``ChunkedDocument.text`` / ``ChunkedDocument.get()`` cache over the chunk list, so assigning to ``text`` (or mutating ``metadata``) leaves those caches describing the previous content. To change chunk boundaries call ``reassemble_chunks()``; to carry your own fields, keep them beside the chunk (its id is the stable key) rather than on it. rOrrBtagged_contentr@metadataNFrn _content_hashrc|jptd|j}t j|d|_|jS)zLazy sha256 of whitespace-normalized text (D17). Stable across runs for identical content; whitespace-normalized so that byte-identical content is not reported as changed by rendering-neutral whitespace drift. N zutf-8) r_WS_REsubrrdhashlibsha256encode hexdigest)rZnorms r content_hashzChunk.content_hashsa   %::c49--3355D!( G0D0D!E!E!O!O!Q!QD !!r)r1r2r3r4r7r6rrr{rrrr[rr rr rrs   GGG IIINC#eMBBBHmBBB#(5E5#Q#Q#QM8C=QQQ  "c " " "X " " "rrc|jsdS|js|js |jsdS|jdS|jdkr |jrdS|jdkr |jrdSdS)zCheck if a caption SentenceUnit matches an element SentenceUnit (one direction). Returns True if caption targets the element type, or if target_type is None and element is table/figure/list. FNTtablefigure)r%r!r"r r&)rUelements r caption_matches_elementrs  u  $(AWEYu"*t"g--'2J-t"h..73L.t 5rbbox_abbox_bc|\}}}}|\}}}}t||z ||z }|dkrdStdt||t||z }||z S)zCompute horizontal overlap ratio between two bboxes. Returns overlap / min_width, or 1.0 if either bbox has zero width. rg?r)minmax) rrax0_ax1bx0bx1 min_widthoverlaps r horizontal_overlap_ratior&sr NCCNCCC#IsSy))IA~~s#s3}}s3}}455G Y rctdx}}tdx}}|D]G\}}}}t||}t||}t||}t||}H|tdkrdS||||fS)zKCompute the union bounding box from an iterable of (x0, y0, x1, y1) tuples.infz-inf)rrrr)r9rr) rHx0y0x1y1rby0rby1s r union_bboxr4sEllBFmmB$S#s S\\ S\\ S\\ S\\ U5\\| B rtolc|\}}}}|\}}} } t||z |kot|| z |k} t||z |kot|| z |k} | p| S)uCheck if two bboxes share a matching edge *pair*. Returns True when (x0 AND x1) or (y0 AND y1) both match within *tol*. A single edge match (e.g. only x1) is not enough — it would cause chain-reaction grouping across spatially unrelated boxes. )abs) rrrray0ray1rrrrx_pairy_pairs r _has_shared_edgerBs{ Cc3Cc3 s^^s " z group_bboxes..is& 0 0 0AJtAww   0 0 0r)r<lenrrrappend)rHrboxesgroupsr last_unions r group_bboxesrPs LLE   5zzQE{{"'( FQRRy!!VBZ 0 011 JY 7 7 ! 2J  c " " " " MM3% 0 0 0 0 00r)r)r4rre dataclassesrrtypingrr5r7r rr>rNrQr]rgrxr{compilerrr:rr8r9rrrr<rr rr rs9   ((((((((Sss  2H2H2H2H2H2H2H 2Hj ??????? ?6 $ # # # # # # # # ####### #: ======= =(          0 1111111 1 %%%%%%% %6 F   """"""" "B& U E e     %     U E  $    11E1T%[111111r