j-dZddlZddlmZddlmZddlmZddlm Z e hdZ e Z Gd d eZd ed efd Zd efdZd efdZded efdZdZdS)aChunkedDocument: the return type of to_chunks. A Sequence[Chunk] carrying the element registry, the table/figure/section views (bidirectionally linked to chunks), a section hierarchy tree, ingestion diagnostics, and cheap re-chunking from retained units. N)Sequence)fields)MappingProxyType) SectionNode> max_tokens min_tokens table_modemerge_small_chunksbreakpoint_thresholdrespect_section_startsc eZdZdZddddddddddZdZdZd Zed e fd Z ed e fd Z ed e fd Zefde fdZddded efdZddded e fdZdZed efdZddde dzfdZddde dzfdZdS)ChunkedDocumenta!Retrieval-ready chunks plus document-level structure. - Sequence protocol: ``len(cd)``, ``cd[i]``, iteration, slicing. - ``elements``: every layout box the parser saw (header/footer included), addressable as ``p{page}.b{box}``. - ``tables`` / ``figures`` / ``sections``: views linked to chunks both ways (view.chunk_id(s) <-> chunk.metadata.table_ids/figure_ids/ section_id). - ``hierarchy``: sections as a tree of SectionNode (root level 0). - ``diagnostics``: structured input for an ingestion gate. Nr)elementstablesfiguressectionsparamsdoc all_unitsheader_footer_excludedct||_t||_t||_t||_t||_tt|pi|_ ||_ t||_ | |_ d|_ d|_d|_|jD]-} | j\} } t|j| | | _.dSN)tuple_chunkslistrrrrrdictr_doc _all_units_header_footer_excluded_text_by_id _hierarchy element_span _elements) selfchunksrrrrrrrrslohis /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/pymupdf4llm/helpers/chunking/chunked_document.py__init__zChunkedDocument.__init__,sV}} X 6ll G}} X &tFLb'9'9::  y//'=$   6 6A^FB be 455AKK 6 6c*t|jSr)lenrr's r,__len__zChunkedDocument.__len__Ds4<   r.czt|trt|j|S|j|Sr) isinstanceslicerr)r'indexs r, __getitem__zChunkedDocument.__getitem__Gs6 eU # # - U+,, ,|E""r.c dt|jdt|jdt|jdt|jd S)NzChunkedDocument(chunks=z , tables=z , figures=z , sections=))r0rrrrr1s r,__repr__zChunkedDocument.__repr__Lsl2#dl*;*;22dk**2269$,6G6G22 ..222 3r.returnc|jS)z=The chunks as a tuple (== tuple(self)); explicit-access form.rr1s r,r(zChunkedDocument.chunksQs |r.cp|j)dd|jD|_|jS)z6All chunk text joined in reading order (lazy, cached).Nz c3$K|] }|jV dSr)text).0cs r, z'ChunkedDocument.text..\s$$B$BQV$B$B$B$B$B$Br.)r"joinrr1s r,r@zChunkedDocument.textXs8 : $B$BT\$B$B$BBBDJzr.c|jt}|g}|jD]}t|j|j|j}t |dkrS|dj|jkr=|t |dkr|dj|jk=|dj || |||_|jS)z5Sections nested as a tree; the root node has level 0.N)titlelevel section_idr) r$rrrFrGidr0popchildrenappend)r'rootstackr)nodes r, hierarchyzChunkedDocument.hierarchyas ? "==DFE] # #"./d444%jj1nnrAG)C)CIIKKK%jj1nnrAG)C)Cb "))$/// T"""""DOr.rJcZ|jWi}|jD] }|||j< |j|j|jfD]}|D] }|||j< |jD] }|||j< ||_|r |j|tnt}|tur|turt||S|S)zLook up any public id: c{n}, t{n}, f{n}, s{n}, p{page}.b{box}. Raises KeyError for an unknown id unless *default* is given. ) r#rrJrrrrget_MISSINGKeyError) r'rJdefaultr6rBgroupitemelfounds r,rSzChunkedDocument.getss ; E\  ad +t|T]C * *!**D%)E$'NN*m " "!be DK13A H--- H  (""rll"N r.Tinclude_taggedr\c*fd|jDS)u>Chunks as flat, json-safe, payload-ready dicts (schema §4.5).c0g|]}t|Sr)_chunk_to_dict)rArBr\s r, z,ChunkedDocument.to_dicts..s#HHHaq.11HHHr.r=)r'r\s `r,to_dictszChunkedDocument.to_dictssHHHH4<HHHHr.c |dttj||fi|S)NrVr[) setdefault _json_defaultjsondumpsra)r'r\ json_kwargss r,to_jsonzChunkedDocument.to_jsonsJy-888z MMM 8 8     r.c t|tz }|r5tdttdt|dddlm}i|j|}||j|j|S)uHRe-run assembly from retained units with new budget params. Only assembly-tier parameters are accepted (REASSEMBLY_PARAMS); substrate parameters (sentence_splitter, header_footer_mode, tokenizer, weights) and parse options raise ValueError — call to_chunks() on a new parse for those. zreassemble_chunks only accepts z; z require a new to_chunks() callr) _chunk_units) setREASSEMBLY_PARAMS ValueErrorsortedrjrrr )r'runknownrjmergeds r,reassemble_chunksz!ChunkedDocument.reassemble_chunkssf++ 11  D&9J2K2KDD'??DDD  #"""""*DK*6*|DIt???r.c |jrd|jjDng}t}|jD]$}|t |%g}|jsf|js|dnItd|jDr|dn|dt|jt|j t|j t|j t|j t|tt||z |d|j Dd|j D|jd S) a Structured ingestion-gate input, derived from the registry. Consumers decide PROCEED/HOLD/REJECT; this only reports facts: counts, zero-chunk causes, text-less figures (OCR candidates), degenerate tables, and header/footer exclusions. cg|] }|j Sr) page_number)rAps r,r`z/ChunkedDocument.diagnostics..s8881888r.no_text_extractedc3$K|] }|jV dSr)is_header_footer)rAus r,rCz.ChunkedDocument.diagnostics..s%AAAQ'AAAAAAr.all_units_header_footerall_units_filteredc*g|]}|j |jSr)has_textrJ)rAfs r,r`z/ChunkedDocument.diagnostics..s0%8%8%8a,-J%8QT%8%8%8r.cNg|]"}|j|j#Sr)r@striprJ)rAts r,r`z/ChunkedDocument.diagnostics..s8"9"9"91)*"9!$"9"9"9r.) chunk_count element_count table_count figure_count section_count page_countpages_without_chunkszero_chunk_causesfigures_without_textdegenerate_tablesr)rpagesrkrupdate _chunk_pagesr rMallr0rrrrrnr!)r'rpages_with_chunksrBrs r, diagnosticszChunkedDocument.diagnosticss=AIM88 88882EE 6 6A  $ $\!__ 5 5 5 5| ?? ?!(()<====AAAAAAA ?!(()BCCCC!(()=>>>t|,, //t{++ -- //e**$*3u::8I+I$J$J!2%8%84<%8%8%8"9"9 "9"9"9&*&B   r.)doc_idrc:ddlmfd|jDS)zChunks as langchain Documents (requires langchain-core). See :meth:`to_llama_nodes` for what *doc_id* is for. r)Documentc vg|]5}t|j|jt|6S))rJ page_contentmetadata _export_idrJr@_export_metadata)rArBrrs r,r`z:ChunkedDocument.to_langchain_documents..sZ    H 1400"#&.q11 3 3 3   r.)langchain_core.documentsrr)r'rrs `@r,to_langchain_documentsz&ChunkedDocument.to_langchain_documentssL 655555     \     r.c:ddlmfd|jDS)uChunks as llama-index TextNodes (requires llama-index-core). Chunk ids are document-local ("c0" starts every document), so two documents in one store collide on the exported id and the later write replaces the earlier record. Pass *doc_id* — any stable per-document key — to export ``"{doc_id}:{chunk.id}"`` instead. The chunk's own id is always in ``metadata["chunk_id"]``, so a retrieved record still addresses ``ChunkedDocument.get()``. r)TextNodec vg|]5}t|j|jt|6S))id_r@rr)rArBrrs r,r`z2ChunkedDocument.to_llama_nodes..sY    HAD&11.q11 3 3 3   r.)llama_index.core.schemarr)r'rrs `@r,to_llama_nodeszChunkedDocument.to_llama_nodessL 544444     \    r.)__name__ __module__ __qualname____doc__r-r2r7r:propertyrr(strr@rrQrTrSboolrrarhrrrrrrrr.r,rrs  ,.b"Ttr()666660!!!### 333 X cX ;   X "$,c226III$I$IIII15        @@@*# T# # # X# N>B    d     6:   d       r.rchunk_idr;c|r|d|n|S)z8Store-facing id: document-scoped when a doc_id is given.:r)rrs r,rrs %+ 9f ! !x ! ! !9r.cFt|dd}|j|d<|S)z?Framework metadata: the ChunkMetadata fields plus the chunk id.Frr)r_rJ)chunkrs r,rrs'eU++J7H 8HZ Or.chd|jjD}|jjD]X}|dd} |t |ddI#t $rYUwxYw|s4tt|jj |jj dz}|S)aHPages that actually put content into *chunk*. ``page_start..page_end`` is a span, not a coverage claim: a chunk bridging pages 1 and 3 says nothing about page 2, so counting the span would hide an empty page from ``pages_without_chunks``. The pages come from the chunk's own bbox/element addresses instead. c"h|] }||d S)rr)rAbs r, z_chunk_pages.. s! 6 6 6aA 6QqT 6 6 6r.z.brrN) rbboxes element_ids partitionaddintrmrkrange page_startpage_end)rrelementpages r,rrs 7 65>0 6 6 6E>-  &&q)  IIc$qrr(mm $ $ $ $    H  8E%.3.1A57788 Ls*A,, A98A9r\cjjjfdtjDd}|r j|d<|S)NcPi|]"}|jtj|j#Sr)namegetattrr)rArrs r, z"_chunk_to_dict..s;    FGENAF33   r.)rJr@ content_hashrtagged_content)rJr@rrrr)rr\ds` r,r_r_slh *    EN++     A3#2  Hr.ct|trdSt|trt|St dt |)Nznot JSON serializable: )r4bytesrr TypeErrortype)os r,rdrd(sS!Ut!UAww 9d1gg99 : ::r.)rrecollections.abcr dataclassesrtypesrmodelsr frozensetrlobjectrTrrrrrrkrrr_rdrr.r,rsX $$$$$$"""""" I 688V V V V V hV V V r:::::: t3*  $  4     ;;;;;r.