j H,dZddlZddlmZmZmZmZejddddddddddZ e d krT ddl Z ej d Z e jd Zee jd kree jd ZndZee jd kree jd ZndZejeZeD]Ze eeeZeZeeD]C\ZZeeeejdze ee De!e e"e#e$dddSdS)a This is an advanced PyMuPDF utility for detecting multi-column pages. It can be used in a shell script, or its main function can be imported and invoked as descript below. Features --------- - Identify text belonging to (a variable number of) columns on the page. - Text with different background color is handled separately, allowing for easier treatment of side remarks, comment boxes, etc. - Uses text block detection capability to identify text blocks and uses the block bboxes as primary structuring principle. - Supports ignoring footers via a footer margin parameter. - Returns re-created text boundary boxes (integer coordinates), sorted ascending by the top, then by the left coordinates. Restrictions ------------- - Only supporting horizontal, left-to-right text - Returns a list of text boundary boxes - not the text itself. The caller is expected to extract text from within the returned boxes. - Text written above images is ignored altogether (option). - This utility works as expected in most cases. The following situation cannot be handled correctly: * overlapping (non-disjoint) text blocks * image captions are not recognized and are handled like normal text Usage ------ - As a CLI shell command use python multi_column.py input.pdf footer_margin header_margin Where margins are the height of the bottom / top stripes to ignore on each page. This code is intended to be modified according to your need. - Use in a Python script as follows: ---------------------------------------------------------------------------------- from multi_column import column_boxes # for each page execute bboxes = column_boxes(page, footer_margin=50, no_image_text=True) bboxes is a list of pymupdf.IRect objects, that are sorted ascending by their y0, then x0 coordinates. Their text content can be extracted by all PyMuPDF get_text() variants, like for instance the following: for rect in bboxes: print(page.get_text(clip=rect, sort=True)) ---------------------------------------------------------------------------------- Dependencies ------------- PyMuPDF v1.24.2 or later Copyright and License ---------------------- Copyright 2024 Artifex Software, Inc. License GNU Affero GPL 3.0 N) WHITE_CHARS are_disjointintersect_rects bbox_is_emptyT2F) footer_margin header_margin no_image_texttextpagepathsavoid ignore_imagescf&'()*d}d'd(d))fd} d} d} d} (fd } |j &&xj|zc_&xj|z c_| &fd |D}|!|&t j }g}g}|||g}g*|D]}|d }|dx}dn|dz}|jdkr |xj |zc_ |xj |z c_ |j dkr |xj|zc_|xj|z c_* |* d|durE|D]0}|||d1|d}|D]}t j|d}|r '||r, |dd}n#t$$rYHwxYwt'd|ddz dkr| |t j}|dD]Q}t j|d}dd|dD}||s||z}R| }|js| || '*fd|gkrgS|dg}|dd }i}t/|D]\} }!d}"t1t3|D]c}#||#}$|! |$j |!j ks|!j |$j kr-(|$*|(|!*|krJ|!|$z}%| |%|$||}"|"durnd|"s/| |!t3|dz }#||#}%| |%|!||}"|"dur| |!n|%||#<d || <| |}t3|dkr|S| |}| |*|}|S)aDetermine bboxes which wrap a column on the page. Args: footer_margin: ignore text if distance from bottom is less header_margin: ignore text if distance from top is less no_image_text: ignore text inside image bboxes textpage: use this textpage instead of creating one paths: use these drawings instead of extracting here avoid: ignore text in any of these areas c*tj|S)zCheck for relevant text.)r issupersettexts /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/pymupdf4llm/helpers/multi_column.pyis_whitezcolumn_boxes..is_white`s%d+++cDt|dD] \}}||vr|cSdS);Return 1-based number if a bbox contains bb, else return 0.startr) enumerate)bbbboxesibboxs rin_bboxzcolumn_boxes..in_bboxds> q111  GAtTzzqrc t|dt|}||}||Sd}t|dD] \}}||vr|}n|||<|S)r_Nrrr)idgetr)rrcache cache_keycachedindexrr s rin_bbox_using_cachez)column_boxes..in_bbox_using_cacheks:"vv,,6 ,, 9%%  M q111  GAtTzz!i rc:tfd|DS)z4Return True if a bbox touches bb, else return False.c3>K|]}t|d VdS)T)strictN)r).0r rs r z:column_boxes..intersects_bboxes..~s5NNt|BT::::NNNNNNr)any)rrs` rintersects_bboxesz'column_boxes..intersects_bboxes|s&NNNNvNNNNNNrc~|D]8}||s(|%||kstt||dr6dSdS)aGDetermines whether rectangle 'temp' can be extended by 'bb' without intersecting any of the rectangles contained in 'bboxlist' or 'vert_bboxes'. Items of bboxlist may be None if they have been removed. Returns: True if 'temp' has no intersections with items of 'bboxlist'. NT) bbox_onlyF)rr)temprbboxlist vert_bboxesbr1s r can_extendz column_boxes..can_extendsg  A$$T;77  77 qD!I!I!IJJ55trc2t|}|dkr|S|dz }t|ddD]}||}||dz }||kr||=t|dkr|S|dj}d}d}tdt|D][}||} t| j|z dkr4||kr%t |||dzd|||dz<| j}|}|}\||kr%t |||dzd|||dz<|S) zDo some elementary cleaning.rrc|jSNx0r7s rz5column_boxes..clean_nblocks..sADrkeyc|jSr>r?rAs rrBz5column_boxes..clean_nblocks..sadr)lenrangey1abssorted) nblocksblenrrbb1bb0rHi0i1b1s r clean_nblocksz#column_boxes..clean_nblocksse7|| !88Nqub"%%  A!*C!a%.CczzAJ w<<1  N QZ]  q#g,,''  AB252:""77+1R!V ,..,,,GBaK(UBB 77#)'"rAv+*>NN#S#S#SGBaK rcd}|dd}g}|rq|d}d}|rKd}tt|dz ddD]%}||z||zjr|||z}||=d}&|K|||d=|q|S)zPostprocess identified text blocks, phase 1. Joins any rectangles that "touch" each other. This means that their intersection is valid (but may be empty). To prefer vertical joins, we will ignore small gaps. )rrr NrTFrr;)rGrFis_validappend)rdeltaprects new_rectsprect0repeatrs rjoin_rects_phase1z'column_boxes..join_rects_phase1s  AYFF &s6{{Q266&&A%6!94>&&)+"1I!%  &   V $ $ $q  rcf|dd}tt|D]S}||tfd|D}tfd|D}|_|_||<T|d|dg}|ddD]}|d}t|j|jz d krKt|j|jz d kr+t|j|j z d kr ||z}||d<u| ||S) zPostprocess identified text blocks, phase 2. Increase the width of each text block so that small left or right border differences are removed. Then try to join even more text rectangles. Nc^g|])}t|jjz dk"|j*Sr<)rIr@r.rr7s r z;column_boxes..join_rects_phase2..5GGGBEADL0A0AQ0F0Fbe0F0F0Frc^g|])}t|jjz dk"|j*Sr_)rIx1r`s rraz;column_boxes..join_rects_phase2..rbrc|j|jfSr>)r@y0rAs rrBz9column_boxes..join_rects_phase2..14,rrCrrr;r<rT) rGrFminmaxr@rdsortrIrHrfrV) rrXrr@rdrYrr0r7s @rjoin_rects_phase2z'column_boxes..join_rects_phase2sas6{{##  Aq AGGGGfGGGHHBGGGGfGGGHHBADADF1II  .. ///AYK  A2BAD25L!!Q&&ru %%** %%++a " "    Q    rc&|dd}g}|r|d}d}|rd}tt|dz ddD]}||}|j|jks|j|jkr+||||||krH||zt fd||zD} | t t |t |fkr||z}||d<||=d}||||d=| g} |D]\tfd|Dd } | r| djjf} njjf} | | f]| d d | D}g} || zS) NrTFrr;cXg|]&}|t|'S) intersectstuple)r.r7r4s rraz;column_boxes..join_rects_phase3..s0TTTadASASTqTTTrcg|]Q}|jjkr?j|jcxkr jks!nj|jcxkr jkLnn|RSrp)rdr@rfrH)r.rkboxs rraz;column_boxes..join_rects_phase3..6stcf}}141111361111SVqt5M5M5M5Msv5M5M5M5M5M6N5M5Mrc|jSr>)rd)rks rrBz9column_boxes..join_rects_phase3..=sadrrCc|dS)Nrrp)srs rrBz9column_boxes..join_rects_phase3..Ds r!urcg|] }|d S)rrp)r.rws rraz;column_boxes..join_rects_phase3..Es000rRU000r) rGrFr@rdsetrrrVrJrfrj)r path_rectsr&rXrYrZr[rprect1test sort_rects left_rectsrD shadow_rectsrtr4r*s @@rjoin_rects_phase3z'column_boxes..join_rects_phase3sT  AYFF &s6{{Q266&&A#AYFy69,, FI0E0E +* E,,VZGGHH!!F?DTTTT6I+=TTTDsE&MM5==#ABBBB&($*q "1I!%+ &,   V $ $ $q 5 8 *  * *C &#N   J '!"~(#&1vsv&   sCj ) ) ) ),,---00Z000  <''rNclg|]0}|djjk|djjk.|1S)rect)widthheight)r.pclips rraz column_boxes..VsI   y++& 0@4;0N0N 0N0N0Nr)rflagsrrg?rc|j|jfSr>rfr@rAs rrBzcolumn_boxes..yrgrrCFblocksr linesrdirgMbP?cg|] }|d Srrp)r.ss rraz column_boxes..s===!AfI===rspansc6||j|jfSr>r)kr!rzs rrBzcolumn_boxes..swwq*55qtQTBrT)rrHrf get_drawings get_textpagepymupdfTEXT_ACCURATE_BBOXESextendrr@rdrrVrj get_imagesget_image_rects extractDICTRect IndexErrorrI EMPTY_RECTjoinis_emptyrrGrF)+pagerr r r r r rrr8rRr\rmrr img_bboxesr6rprectr#lwidthitemrr7r line0srectlinelbboxrrKr&rrcheckjnbbr4rr!r*r1rzs+ @@@@@r column_boxesrJs,,,,"OOO*&&&P0$$$LS(S(S(S(S(l I:DGG}GGGG}GG }    &&((    $$$g6R$SS FJ %   KJ  ! !& j1q3w ;!   HH HH HH HH <1   HH HH HH HH%    OO..O///OO%% = =D   d2247;; < < < < ! ! # #H -F  |AfI&&  WWT:66   gJqMEE    H  q5<?" # #d * *   t $ $ $ "$$gJ  DLf..E77==tG}===>>D8D>> v} MM$    KKBBBBBKCCC|| aykG ABBZF E6""!!2s7||$$  A!*CzSVbe^^rusv~~#"3 E::>Q>QJ??8DJtS';??E}}  NN2   G q A1:D 4V[99 E>> NN2    GAJq mG$$G 7||q  ((GU;;G Ns#G22 G?>G?__main__redrr:r<)rr ))colorz.pdfz -blocks.pdf)%__doc__rpymupdf4llm.helpers.utilsrrrrTOOLSunset_quad_correctionsr__name__syspdfcolorREDargvfilenamerFintrr opendocrr new_shapeshaperrr draw_rect insert_texttlstrfinishcommitez_savereplacerprrrsL<<| $$T***    SSSSSl  z JJJ  5 !Cx{H s38}}qCHQK((   s38}}qCHQK((   ',x C  ]      !y(( D DGAt OOD ! ! !   dg/Qs  C C C C  3  KK  7788888er