j=dZddlZddlmZmZmZmZmZmZm Z ddl m Z m Z ej dZej dZGdd Zd efd Zd efd Zd Zd eefdZd efdZdS)uSStep E: Serialize ProtoChunks into Chunks, plus table/figure/section views. The serializer materializes chunk text once (assembly never joins text), builds the tables/figures/sections views with bidirectional links (chunk.metadata.table_ids/figure_ids/section_id <-> view.chunk_id(s)), and attaches citation metadata (section_path, pages, element ids). section_path derives from the layout-detected heading structure — it is the innermost owning section's path (D18). It never consults the PDF TOC, so it stays populated and consistent with the sections view on documents without bookmarks. N)Chunk ChunkMetadata FigureChunk ProtoChunk SectionChunk TableChunk element_id)extract_table_headers table_contentz ^[#>*_`\s]+z [*_`\s]+$creZdZdZdZdeefdZdedefdZ dZ defd Z ded ee de fd Z d S) ChunkSerializerz?Converts ProtoChunks into Chunks and builds the document views.c@||_d|jD|_dS)Nc>h|]}t|dd|jS) full_ocredF)getattr page_number).0ps /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/pymupdf4llm/helpers/chunking/serializer.py z+ChunkSerializer.__init__..%s;   <0O0O M   )docpages _ocr_pages)selfrs r__init__zChunkSerializer.__init__#s/  #&9   r proto_chunksc|sggggfSfd|D}||\}}}t||D]*\}}||jj|_+||||fS)zConvert ProtoChunks to Chunks. Returns (chunks, tables, figures, sections) where tables/figures/ sections are the document views, bidirectionally linked to chunks. c:g|]}|S) _make_chunk)rpcrs r z-ChunkSerializer.serialize..2s'>>>2$""2&&>>>r) _build_viewszip_build_tagged_contentmetadata section_pathtagged_content)rrchunkstablesfiguressectionsr#chunks` r serializezChunkSerializer.serialize)s  "r2r> !>>>>>>>$($5$5lF$K$K!\622 1 1IB#'#=#=EN/$1$1E vw00rr#returnc dd|jD|_t|j|jd|jD|jp |jpdg|j |j t|jtfd|jDj jj j }t!d|j|jd| S) z@Create a Chunk from a ProtoChunk (text materialized here, once). c3$K|] }|jV dSN)textrss r z.ChunkSerializer._make_chunk..?s$::qAF::::::rc4g|]\}}t||Sr!)r )rrbs rr$z/ChunkSerializer._make_chunk..Ds&EEEdaAq))EEEr paragraphc34K|]}|jjvVdSr5)page_nor)rr8rs rr9z.ChunkSerializer._make_chunk..Ls,HHQAI0HHHHHHr) page_startpage_end element_idstypes token_countbboxeslistsocr file_path page_countc)idr6r*r()join _sentencesr6rr?r@ box_indicesrB primary_typerCrD_group_list_itemsanyrfilenamerHrchunk_id)rr#r(s` rr"zChunkSerializer._make_chunk=s))::BM::::: }[EEbnEEE(>r=+>9#BM22HHHH"-HHHHHh'x*     2;       rcB g}g}g}i}g}i}d} |jjD]#} | || j<| t| jz } $d} t ||D]\} } t | jD]\}}|jrt| j|d}tdt|| j t|j |j|j |j|j|jt%|j|r|jnd|rt|j |jnd|j }||| jj|j |jr|j||j}||jdr|jnd}t| j|d }t9d |j| j t|j |j|j |j|j|rdn|j||r|jnd|rt|j |jnd|j }|||j<||| jj|j |jds7|j r$|j d z|jz!n|j|_ |j | jjvr$| jj|j |j"rB|j#pd }|r:|dd|kr(|$|r|dd|k(tJ&dtN&d|j!}tQdt||||j |j t|j |jd|D|gz||j d|jz| f}||| || |||f|D]0\}}| || tS|j*| j*|_*1|r=|dd }|j | j_+tY|j-| j_.t |D]J\}}| }||d zdD]!}|j/|j/kr|j0d}n"|j0d|f|_0K|1|}|rte|||}d|D}||fD]4}|D]/} || j3}!|!|!jj+| _+05|||fS)z1Build t/f/s views with bidirectional chunk links.rc|j|jvr;|j|j|xj|jjz c_dSdSr5)rKchild_chunk_idsappendrCr()sectionr/s r_linkz+ChunkSerializer._build_views.._linkisRxw666'..ux888##u~'AA####76rtabletN) rKrSr pagebboxmarkdownhtmlheaderscaptioncaption_element_idrCz[FigurerJfiguref) rKrSr r\r]boxclassocr_text placeholderrarbimager3rr8c"g|] \}}|j Sr!)title)r_lsecs rr$z0ChunkSerializer._build_views..sEEEGBciEEEr)rKrklevelr?r@heading_element_idpath element_spanci|] }|j| Sr!rKrrIs r z0ChunkSerializer._build_views..s///1qtQ///r)4rrrlenboxesr& enumeraterMis_table_content_nearest_captionr rKr r> box_indexr]table_markdown table_htmlr r6rCrWr( table_idsis_figure_related figure_numberget startswithrre image_data figure_idsrfstripis_heading_hintheading_level_hintpop_MD_DECOR_TRAIL_REsub _MD_DECOR_RErmaxr@ section_idlistrpr)rnrq_unchunked_tables_merge_unchunked_tablesrS)"rrr+r,r-r. seen_figures open_sections box_offset total_boxesr\rYr#r/idxr8caprZfigrgrnrkrX_level innermostirmendlater unchunked chunk_by_idviewitemowners" rr%zChunkSerializer._build_viewsZs.    HN + +D+6Jt' ( 3tz?? *KK B B B \622V CV CIB#BM22L ;L ;Q%K;*2=#wGGC&,s6{{,,!&#-ai#E#EYV!"!1\ 5al C C,/ 9TFISJs{CMBBBt$%M   EMM%(((N,33EH====(8;Q_-H&**1?;;C{010A0A)0L0L&TaffRT .r}c8LL)41?44%*X'1!)Q['I'I!"!"%&Z-8%DTTaf(303$=CHHJM W 3; F F FSW"#,   9< Q_5s+++188@@@@ !v00;;I/2|-HS\D-@16-I,P,P,R,R,RAB L 6)BBB!N5<.B.B+ )) : :FAsC!!a%&&)  ;#)++,Q/CE,!$ 0 3S9C   **<88  H,VYGGF0//// W% @ @D @ @# 66$&+n&?DO @ w((rcd|D}g}|jjD]}t|jD]\}}|jdvr|j|f|vrt |\}}||d}|tddt|j||j|j |j |j |j f||t||S)u)Table boxes that produced no unit, as chunk-less TableChunks. A table box whose rendering is empty yields no SentenceUnit, so no chunk carries it. It stays in the element registry, but without this it would also disappear from ``tables`` and therefore from ``diagnostics["degenerate_tables"]`` — exactly the case the diagnostic exists to surface. Such a table is exposed with ``chunk_id=None``: it is addressable and reported, and its empty rendering is visible in ``TableChunk.text``. cLh|]!}|jD]}|j |j|jf"Sr!)rMryr>r{)rr#r8s rrz4ChunkSerializer._unchunked_tables..sO***2=**./(*AIq{+****r)rZztable-fallbackNrJ)rKrSr r\r]r^r_r`)rrrxrwrerr rWr r x0y0x1y1r ) rrchunkedmissingr\box_idxboxr^r_s rrz!ChunkSerializer._unchunked_tabless**)***HN  D )$* 5 5   <'BBB$g.'99!.s!3!3$# !Hz!)$*:GDD)&#&#&#&9%1$77        $r hierarchycg}|r+|dd||j|jkr|d|jn%|d|jd|jd|jpgD}|r+|dd||d |jd |S) z*Build context-enriched text for embedding.z [Section] z > z[Page] z[Pages] -cg|] }|dk| S)r<r!)rr[s rr$z9ChunkSerializer._build_tagged_content..s"DDD!1 3C3CA3C3C3Crz[Type] z, z [Markdown] r3)rWrLr?r@rBr6)rr#rpartsnon_paras rr'z%ChunkSerializer._build_tagged_contents  ? LL=ejj&;&;== > > > =BK ' ' LL22=22 3 3 3 3 LLABMAABKAA B B BDDBDDD  : LL8499X#6#688 9 9 9 -BG--...yyrN)__name__ __module__ __qualname____doc__rrrr0rr"r%rstrr'r!rrrr sII   1d:&61111( j U    :E)E)E)N"""""L  tCy S      rrr1cl|jt|jdddfS)z=(page, box) reading-order key from a TableChunk's element id.z.brri)r\intr rsplit)rZs r_table_addressrs/ JE,33D!<..s % % %QT1 % % %rN) sortedrrrxrKr(r~clearrSrrW) r,rr+mergedrrZby_idr/rs rrr#sDLL4 ??2 G G GFf%%5q77 % %f % % %E))  &&((((66-2^E %.)))   N $ + +EH 5 5 5 Mrcd}d}t|D]7\}}|js |jd|fvrt||z }|||kr||}}8|S)zHNearest caption unit to sents[idx] targeting *target_type* (or untyped).N)rx is_captioncaption_target_typeabs)sentsr target_typebest best_distjr8dists rrzrz8sx DI%  &&1|   {(; ; ; 1s7||  y 0 0)D Krcg}g}|D]E}|jr|||r$|t|g}F|r"|t||S)uGroup consecutive list-item sentences into logical list groups. Each group gets one union bbox *per page* and an ordered items list. Non-list sentences break the current group, so two separate runs of list-items produce two distinct list groups. Returns:: [{"items": [{"text": str, "page": int, "bbox": (x0, y0, x1, y1)}], "bboxes": [(page, x0, y0, x1, y1), ...]}] A list continuing across a page break yields one entry in ``bboxes`` per page, in page order — unioning across pages would place the whole group in a rectangle that exists on neither page. Item bboxes stay 4-tuples and carry their page next to them, matching ``Element.bbox``. ) is_list_itemrW_finalize_list_group)rgroups current_itemsr8s rrPrPGs"FM ## > #   # # # # # 2=AABBB " ; *=99::: Mrc d|D}i}|D]5}||jg|j6g}t |D]}||}||t d|Dt d|Dt d|Dt d|Df||dS)z=Build a list group dict from consecutive list-item sentences.c8g|]}|j|j|jdS))r6r\r])r6r>r]r7s rr$z(_finalize_list_group..ks8 16BBrc3&K|] }|dV dS)rNr!rr;s rr9z'_finalize_list_group..w&))!))))))rc3&K|] }|dV dS)rNr!rs rr9z'_finalize_list_group..xrrc3&K|] }|dV dS)Nr!rs rr9z'_finalize_list_group..yrrc3&K|] }|dV dS)Nr!rs rr9z'_finalize_list_group..zrr)itemsrD) setdefaultr>rWr]rminr)rentriesby_pager8rDr> page_boxess rrris GG 9919b))008888 F'??  W%   ))j))) ) ) ))j))) ) ) ))j))) ) ) ))j))) ) )       / //r)rremodelsrrrrrr r text_sourcer r compilerrrtuplerrrrzdictrPrr!rrrss   >=======rz.)) RZ --y y y y y y y y |CUCCCC $*   T D04000000r