jYdZddlmZmZmZmZmZmZehdZ GddZ de e fdZ de e fdZd e e de e fd Zd e e de e fd Zd S) aSteps C+D: Assemble initial chunks and refine via split/merge. Token budgeting works on per-unit token counts precomputed once by the pipeline and summed here (D12 error contract: the sum of unit counts may differ from tokenizing the joined text by up to the number of joins). Assembly never re-tokenizes or re-joins chunk text; ``ProtoChunk.text`` stays empty until serialization. ) SentenceUnit ProtoChunk union_bbox group_bboxeshorizontal_overlap_ratiocaption_matches_element>listtablefigurec eZdZdZ d1deded ed ed ed ef d Zde e de ede e fdZ de e de e fdZ de de defdZede de defdZede de defdZde de e fdZde e de e fdZde de defdZede d edefd!Zde e de e fd"Zde de defd#Zed2d%ed&ed'edefd(Zed2d)e d*e d'edefd+Zd,ede e de fd-Zede e deee effd.Zede e de e fd/Zd0S)3ChunkAssemblerzAssembles SentenceUnits into ProtoChunks based on boundary scores, then refines by splitting oversized and merging undersized chunks. x?preserveT max_tokens min_tokens threshold table_modemerge_small_chunksrespect_section_startscZ||_||_||_||_||_||_dSN)rrrr merge_smallr)selfrrrrrrs /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/pymupdf4llm/helpers/chunking/chunk_assembler.py__init__zChunkAssembler.__init__s6%$"$-&<###sentsscoresreturnc|sgSg}|dg}|dj}t|D]\}}||dz}||jk} | s||d|} | s||jz|jk} | r.|j|djkr|j|djkrd} | rA||t|||g}|j}||||jz }|r6||t|||S)zForm initial chunks from sentences + boundary scores. A new chunk starts when: - break_score > threshold, OR - structural break (heading, type transition), OR - adding the next sentence would exceed max_tokens rF) token_count enumerater_is_structural_breakr box_indexpage_noappend_make_proto_chunklen) rrr chunks current_sentscurrent_tokensiscore next_sentdo_breaks rassemblezChunkAssembler.assemble&sy Iq q-!&)) 8 8HAua!e It~-H S44]25F RR T)I,AADOS !Y0M"4E4OOO!*!2mB6G6O!O!O  8 d44S[[-PPQQQ!* !*!6$$Y///)"77  N MM$00VmLL M M M rr-cbg}|D]P}|j|jkr)|||;||Q|js||S||}||}||S)uNRefine chunks via split and multi-depth merge. 1. Split — break oversized chunks at sentence boundaries 2. Semantic merge — heading+content, caption+element 3. Budget merge — combine undersized neighbours within max_tokens (a pair is merged only while one side is below min_tokens) ) r%rextend _split_chunkr*r _renumber_merge_semantic _merge_budget)rr-splitchunksemanticmergeds rrefinezChunkAssembler.refineTs#% $ $E 4?22 T..u556666 U#### )>>%(( (''..##H--~~f%%%rprevr2c|||rdS|||rdS|j|jkrdS|j|jkrdS|j|jkrdS|jrdSdS)z9Check if there's a forced structural break between units.FT)_is_caption_element_pair_is_heading_element_pairis_header_footeris_table_contentis_figure_relatedis_heading_hint)rr@r2s rr'z#ChunkAssembler._is_structural_breakps  ( (y 9 9 5  ( (y 9 9 5  %)> > >4  %)> > >4  &$*@ @ @4  $ 4urc|jsdS|j|jkrdS|js |jsdSt|j|jS)z>Check if prev is a heading and next is its associated element.F)rGr)rErFr _has_horizontal_overlapbbox)r@r2s rrCz'ChunkAssembler._is_heading_element_pairs]# 5 <9, , ,5* i.I 555diPPPrabc|j|jkrdSt|j|jsdSt ||pt ||S)z>Check if a and b form a caption-element pair on the same page.F)r)r rIrJr)rKrLs rrBz'ChunkAssembler._is_caption_element_pairsY 9 ! !555afafEE 5&q!,,M0G10M0MMrr<c|j}|s|gS|jdkr|jdkr|gSg}g}d}|D]}|j|jkrZ|r-||d|g}d}||d|gm||jz|jkrU|rRt|}|j|djkr|j |dj kr|j}|j } t|dz } | dkrO|| j|kr>|| j | kr-| dz} | dkr"|| j|kr|| j | k-| dkrz.ChunkAssembler._split_chunk..s$*R*RQ1=*R*R*R*R*R*RrNc3$K|] }|jV dSrrPrQs rrTz.ChunkAssembler._split_chunk..s$$D$DqQ]$D$D$D$D$D$Dr) _sentences primary_typerr%rr*r+r,r(r)sum)rr<rresultcurrentr/sent split_pointbipgj carry_tokenskeepcarrys rr7zChunkAssembler._split_chunks  7N   ( (T_ -J-J7N$ 3$ 3D$/11'MM$"8"8G"D"DEEE G%&N d44Q??@@@ 004?BBwB "'ll Ngbk&;;; LGBK,???BBG q(Aa%%GAJ$8B$>$>71:CUY[C[C[Qa%%GAJ$8B$>$>71:CUY[C[C[1uu'**R*R'!a%&&/*R*R*R'R'R '$*::doMM*+a%K| |, -CMM$"8"8D"A"ABBB4&.!$$D$DG$D$D$D!D!Dt$$$$"22  > MM$00G<< = = = rct|dkr|Sg}d}|t|kr||}|dzt|krC||||dzr#t|j}|dz}|t|kr||d|||rh|||j|dz }|t|kr0||d|||h|d|}t|jdkrt|j|_|||}n|||dz }|t|k|S)zMerge semantic pairs in a single forward pass. Pairs: heading + following content, caption + adjacent element. Semantic pairs always produce a single union bbox (not a list). rr#) r,_is_semantic_pairr rVr+r6bboxes_union_per_pager*)rr-rYr0r<combinedr_pcs rr9zChunkAssembler._merge_semantics v;;!  M#% #f++oo1IE1us6{{""t'='=eVAPQE]'S'S" 011E#f++oo$*@*@**1h77++oOOF1I$8999FA #f++oo$*@*@**1h77++o ++Ax88ry>>A%% / : :BI b!!! e$$$Q)#f++oo, rc |jdks |jdkrdS|jr|jr|jdjrdS|j|jz|jkrdSt |j|jz dkrdS| |j |j sdS|jdkr |jdkrdS|jdkr |jdkrdSt|jotd|jD}|r)|jtvr|||jSt|jotd |jD}|jtvr|r|||jSdS) z?Check if (a, b) form a semantic pair that should stay together. header_footerFr#rheadingTr c3$K|] }|jV dSr is_captionrQs rrTz3ChunkAssembler._is_semantic_pair..$1U1U1!,1U1U1U1U1U1Urc3$K|] }|jV dSrrmrQs rrTz3ChunkAssembler._is_semantic_pair.. ror)rWrrVrGr%rabspage_end page_start_any_bbox_overlapreboolall_ELEMENT_TYPES_caption_target_matches)rrKrL a_is_caption b_is_captions rrdz ChunkAssembler._is_semantic_pairs >_ , ,/0Q0Q5  ' AL LO3 5 =1= (4? : :5 qzAL( ) )A - -5%%ah99 5 >Y & &1>Y+F+F4 >V # #&(@(@4AL))Uc1U1U 1U1U1U.U.U  CANn<<//1>BB BAL))Uc1U1U 1U1U1U.U.U >^ + + +//1>BB Bur caption_chunk element_typec4d|jD}|sdS||vS)z8Check if caption's target type matches the element type.c*h|]}|j |jS)caption_target_typerQs r z9ChunkAssembler._caption_target_matches..)s4   &'$ !   rT)rV)r{r| target_typess rrxz&ChunkAssembler._caption_target_matches&s=  +8+C    4|++rct|dkr|S|dg}tdt|D]}|d}||}|||rL|j|jz}|d|}t |j|jz|_||d<t|||S)aMerge adjacent undersized chunks that fit within max_tokens. Preserves bbox structure from earlier phases: instead of rebuilding bboxes from scratch, concatenates the bbox lists of both chunks. This keeps semantic-merge union bboxes intact. rr#r$)r,range_can_budget_mergerVr+_regroup_page_bboxesrer*)rr-r>r0r@currcombined_sentsrhs rr:zChunkAssembler._merge_budget3s v;;!  M)q#f++&& $ $A":D!9D%%dD11 $!%4?!B++A~>>1t{1JKK r  d#### rch|jdks |jdkrdS|jdkrdS|jr|jr|jdjrdS|jdkr|jdks |jdkrdS|j|jz|jkrdS|jdkr"|j|jkr|j|jkrdSdS)u-Check if two adjacent chunks can be merged within budget. Spatial proximity (bbox overlap, page distance) is intentionally NOT checked here — budget merge is purely about filling token capacity with sequentially adjacent chunks. min_tokens is the floor this step works towards: a pair is merged only while at least one side is still below it, so two chunks that already stand on their own are left alone (merge_small_chunks semantics). min_tokens=0 restores unconditional greedy packing. rjFrkr#isolater T)rWrrVrGrr%rr)rrKrLs rrz ChunkAssembler._can_budget_mergeOs >_ , ,/0Q0Q5 >Y & &5  ' AL LO3 5 ?i ' '~((ANg,E,Eu =1= (4? : :5 Oa  MT_44MT_445tr333333?bbox_abbox_b min_ratioc*t|||kS)z>Check if two single bboxes have sufficient horizontal overlap.)r)rrrs rrIz&ChunkAssembler._has_horizontal_overlapvs(779DDrbboxes_abboxes_bc|D]]}t|dkr |ddn|}|D]9}t|dkr |ddn|}t|||krdS:^dS)zCheck if any bbox from list a overlaps horizontally with any from list b. Handles both 4-tuple (x0,y0,x1,y1) and 5-tuple (page,x0,y0,x1,y1). rNTF)r,r)rrrbaba4bbbb4s rrtz ChunkAssembler._any_bbox_overlap{s   BGGqLL"QRR&&bC   #B1 bff"+C55BB444C urchunk_idc||\}}t|}ttd|D}t |d|Dt d|D|dj|djdzf|dj|dj||||t| S)zCreate a ProtoChunk from a contiguous run of SentenceUnits. Text is NOT materialized here (serialization does that once); token_count is the sum of precomputed unit counts (D12). c32K|]}|j|jfVdSr)r)r(rQs rrTz3ChunkAssembler._make_proto_chunk..s<) ) )*QY $) ) ) ) ) ) rcg|] }|j Sr)sent_idrQs r z4ChunkAssembler._make_proto_chunk..s///Aai///rc3$K|] }|jV dSrrPrQs rrTz3ChunkAssembler._make_proto_chunk..s$99aAM999999rr#r$r) rsent_idsr% unit_rangersrr box_indicesrerWtypesrV) _infer_chunk_types_group_bboxes_by_pager dictfromkeysrrXrr))rrrrW all_typesrers rr+z ChunkAssembler._make_proto_chunks #'"9"9%"@"@ i&u--4==) ) .3) ) )     /////99599999a(%)*;a*?@Qx'2Y&#%E{{    rct}g}|D]z}d}|jrd}n>|jrd}n4|jrd}n*|jrd}n |jrd}n|jrd}n |jrd}nd }||vr*||| |{d }|D] }||vr||fcS d|vrd|fSd|vrt|d krd|fSd|vrd|fSd |fS) zCollect all content types and determine the primary type. Returns (primary_type, all_types) where all_types preserves reading order and primary_type is the dominant element type. Nrjr r r rkcaptionfootnote paragraph)r r r r) setrDrErF is_list_itemrGrn is_footnoteaddr*r,)rseenrrSt_ELEMENT_PRIORITYets rrz!ChunkAssembler._infer_chunk_typesscuu  $ $AA! ## $  "   }}   ###8# % %BTzz9}$$$ d " ""I- -   Uqi' '   y( (I%%rc>t|D] \}}||_ |S)z Renumber chunk IDs sequentially.)r&r)r-r0r<s rr8zChunkAssembler._renumbers-"&))  HAuENN rN)rrrrTT)r)__name__ __module__ __qualname____doc__intfloatstrrurr rrr4r?r' staticmethodrCrBr7r9rdrxr:rtuplerIrtr+rr8rrrr r s $#''+==== =  = ! =!%====$*d<0*$u+*$zJZ****\&T*-&$z2B&&&&8,SW$Q|Q QQUQQQ\QNLN\NdNNN\NA*Aj1AAAAAJ!d:&6!4 ;K!!!!F$:$*$$$$$L,z,,QU,,,\,D$4j9I8#:#*#####NEEEuEEY]EEE\E  D D U UY   \  # d<6H Z    8*&$|"4*&sDI~9N*&*&*&\*&X$z*tJ/?\rr r!cfddlm}g}g}|D]f}|jr|||r$|t |g}||j|jfg|r"|t ||t}|D] \}}|||!g}t| D]B}tt||} | D]}||g|RC|S)aJGroup sentence bboxes by page, then by edge-pair within each page. Consecutive list-items are pre-merged into a single union bbox before edge-pair grouping, since individual list-item x1 varies by text length while they share the same x0 (indentation level). Returns list of (page, x0, y0, x1, y1) 5-tuples. r# defaultdict) collectionsrrr*r6_flush_list_runr)rJr sortedkeysriter) rrmerged_entrieslist_runrS page_bboxesr)rJrYgroupeds rrrsx('''''NH 77 > 7 OOA     %%oh&?&?@@@  ! !19af"5 6 6 6 69oh77888+d##K'** G##D)))) F+**,,--,,tK$899:: , ,D MM7*T** + + + + , Mrcddlm}|t}|D]'}||j|j(g}t |D]}||}td|D}td|D}td|D} td|D} ||||| | ff|S)z=Union consecutive list-item sentences into one bbox per page.r#rc3&K|] }|dV dS)r#NrrRrLs rrTz"_flush_list_run.. &&&!1&&&&&&rc3&K|] }|dV dS)rNrrs rrTz"_flush_list_run.. rrc3&K|] }|dV dS)Nrrs rrTz"_flush_list_run.. rrc3&K|] }|dV dS)Nrrs rrTz"_flush_list_run.. rr) rrr r)r*rJrrminmax) itemsrby_pagerSrYr)rex0y0x1y1s rrrs''''''k$G ** !!!&)))) F',,..))33! &&v&&& & & &&v&&& & & &&v&&& & & &&v&&& & & wRR 012222 MrrcLddlm}|t}|D])}|\}}}}}||||||f*g} t |D]=} t t|| } | | g| R>| S)z:Union all bboxes on each page into a single bbox per page.r#r)rrr r*rrrr) rrrpbpagerrrrrYr)us rrfrfs''''''k$G//!b"b" b"b"-.... F',,..))%% tGG,-- . . wmmm$$$$ MrcVddlm}|t}|D])}|\}}}}}||||||f*g} t |D]B} t t|| } | D]} | | g| RC| S)zRe-group 5-tuple bboxes (page, x0, y0, x1, y1) by page. Preserves page info while merging compatible bboxes within each page. r#r)rrr r*rrrr) rrrrrrrrrrYr)rrJs rrr s ('''''k$G//!b"b" b"b"-.... F',,..)),,tGG$45566 , ,D MM7*T** + + + + , MrN)rmodelsrrrrrr frozensetrwr r rrrrfrrrrrs&zyyyyyyyyyyyyyyy66677HHHHHHHHV#DK####Ld5k" e e    d5kd5kr