j`dZddlmZddlmZddlZddlmZmZm Z m Z m Z m Z dZ dZdZd Zd Zd Zd Zed dZdZddddZdZdZdZdZdZdZdZdZdZdZ dZ!dZ"dZ#dZ$d Z%dddd!Z&dS)"a Copyright (C) 2023 Artifex Software, Inc. This file is part of PyMuPDF. PyMuPDF is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. PyMuPDF is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details. You should have received a copy of the GNU Affero General Public License along with MuPDF. If not, see Alternative licensing terms are available from the licensor. For commercial licensing, see or contact Artifex Software, Inc., 39 Mesa Street, Suite 108A, San Francisco, CA 94129, USA, for further information. --------------------------------------------------------------------- PyMuPDF table union stage, behind find_tables(union=True): fuse the layout analyzer's table grids with the line-based finder's candidates. Table, TableFinder and _iou come from pymupdf.table; find_tables is imported lazily. ) bisect_left) namedtupleN)CHARSEDGESTable TableFinder_cells_to_rows_iou lines_strictg?Tg@g333333?g?g? _UnionContentz1supported dense_rows dense_columns concrete slotsc $ g}|jpgD]}t|ts|ddkr3|d}|d}|r|b|d|d|d|d f\ }}gfd |jDz|gz} g fd |jDz|gz}g} t t|dz D]r} g} t t|dz D]9} | || || || dz|| dzf:| | s|tj |dd | f|S) atPrimary table grids from the raw layout analyzer result. Reads page.layout_information in its raw (return_raw=True) form and yields a ``(bbox, grid)`` pair per "table" group -- ``grid`` is the full row-major cell grid built from the group box plus its interior GridPrediction lines, in layout (reading) order. Boxes without a usable grid are skipped. class_nametable group_bbox table_gridNrcg|]}|zSr).0hy0s /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/pymupdf/_table_union.py z'_layout_table_grids..X<<.Yrr) layout_information isinstancedictgeth_linesv_linesrangelenappendpymupdfRect)pagegridsgroupr grid_predx1y1r&r'gridirowjr rs @@r_layout_table_gridsr7Cs E)/R;;%&&   99\ " "g - - YY|,, IIl++  Y. #A 1 z!}jQRmSBB$<<<<)*;<<<<tC$<<<<)*;<<<<tCs7||a'((  AC3w<22D9:::: Lr add_lines add_boxesc~ddlm}gfd}||td|||}||fS)anLine-based table candidates for the union stage as ``(bbox, grid)`` pairs. Runs a nested find_tables (strategy=_UNION_STRATEGY, use_layout=False) and keeps each detected table's bbox and row-major cell grid (None for a gap), deduped by rounded bbox. Caller-supplied virtual lines / boxes are forwarded to that nested finder so they take part in the same union decisions as PDF-native vector rules. Admission (see ``admit``) runs on the finder's cell groups, before a Table is built for any of them. Returns ``(candidates, finder)``; the finder is reused as the returned TableFinder shell. r) find_tablesc g}t}d}|D][}tjtd|Dtd|Dt d|Dt d|D}|jr~dt |D}|std|D}||vrt||rR|t}t||} | j r)| j dkr| j dkr| jdz| jks|||| ||f]|S) aKeep the cell groups that may become tables, in detection order. A candidate is admitted as before: the line grid is evidence in its own right and the layout model can miss a table entirely. The one exception is a candidate sitting inside a region the layout model called a picture, which has to show table-shaped text instead. That cheap geometric test therefore gates the character scan, and the scan itself is built once for the whole call. Nc3&K|] }|dV dS)rNrrcells r z8_union_line_candidates..admit..&..DG......rc3&K|] }|dV dS)rNrr?s rrAz8_union_line_candidates..admit..rBrc3&K|] }|dV dSrNrr?s rrAz8_union_line_candidates..admit..rBrc3&K|] }|dV dS)rNrr?s rrAz8_union_line_candidates..admit..rBrcg|] }|j Sr)cellsrr5s rrz9_union_line_candidates..admit..s???#CI???rc34K|]}t|VdSN)round)rvalues rrAz8_union_line_candidates..admit..s(77e 777777rr)setr+r,minmaxis_emptyr tuple_union_candidate_inside_picture_union_char_midpoints_union_grid_content_support supported dense_rows dense_columnsconcreteslotsaddr*) live_pagegroupskeptseenpointsrHbboxr3keycontent candidatess radmitz%_union_line_candidates..admittsuu" ," ,E<............................ D } ??)>)>???D 77$77777Cd{{.y$?? >244F5dFCC %*a//-22(1, == HHSMMM KK      tTl + + + + rF)strategy use_layoutr9r:_cell_group_filter) pymupdf.tabler<_UNION_STRATEGYclear)r-r9r:r<refinderrds @r_union_line_candidatesrmds*)))))J00000d[   F~ v rctdt|j|jz tdt|j|jz zSN)rPfloatr1r r2r)rects r_union_rect_arearssC sE$'DG+,, - -Ctw?P9Q9Q0R0R RRrctt|jt|j}tt|jt|j}t t|jt|j}t t|jt|j}||ks||krdS||z ||z zSro)rPrqr rrOr1r2)leftrightr rr1r2s r_union_intersection_arearws U47^^U58__ - -B U47^^U58__ - -B U47^^U58__ - -B U47^^U58__ - -B Rxx288s GR  rc tdtt|jt|jtt|jt|jz Sro)rPrOrqr1r )rurvs r_union_x_overlaprysM sCdgeh883uTW~~uUZU];_;__ ` ``rc6t|}|dkrdSd}d}d}t|D]i\}}t|}t||} | dks|dkr1| |z } | |z } | tkr | |kr|}| }Q| tks | tkrd}j||fS)a2The primary a split candidate belongs inside, plus an ambiguity flag. Returns ``(owner_index, ambiguous)``: owner is the best-contained primary (candidate>=_UNION_OWNER_CONTAINMENT inside it), else None; ambiguous is True when the candidate overlaps some primary enough to be unsafe to append.r)NTNrpFT)rs enumeraterw_UNION_OWNER_CONTAINMENT_UNION_OWNER_AMBIGUOUS_OVERLAP) candidate_bboxexisting_bboxescandidate_area best_ownerbest_containment ambiguousindex existing_bbox existing_area inter_areacandidate_containmentexisting_coverages r_union_find_ownerrs &n55NzJI )/ : :  }(77 -nmLL ??mq00  *^ ;&6 $< < z._union_grid_content_support.. s4YYYYYHXeDGnnHXHXHXHXrc34K|]}t|VdSrKr)rIs rrAz._union_grid_content_support.. s(**cS******rr)defaultcDg|]\}}||tj|fSrK)r+r,)rrr@s rrz/_union_grid_content_support..s1cccRVRb%d++,RbRbRbrc*g|]\}}|j ||fSr)rQ)rrrrs rrz/_union_grid_content_support..s'MMM;5$t}M%MMMrrcg|];\}}tj|cxkrtjk6nn|."sY!TW~~;;;;U47^^;;;;;;;;rc3<K|]}t|dkVdSrEr)rcovereds rrAz._union_grid_content_support..*s-AA'CLLA%AAAAAArrc3XK|]#tfdDdkV$dS)c3 K|]}|vV dSrKr)rrrs rrAz8_union_grid_content_support...+s( 9 9'Fg  9 9 9 9 9 9rrN)sum)rr row_columnss @rrAz._union_grid_content_support..*sbOO  9 9 9 9[ 9 9 999Q>OOOOOOrc3:K|]\}}|dko|dz|kVdSrrNrrfilledtotals rrAz._union_grid_content_support..0s8YY]VUuqy8VaZ5%8YYYYYYrc3:K|]\}}|dko|dz|kVdSrrrs rrAz._union_grid_content_support..1sL   AI -&1*-      r)rVrWrXrYrZ) sortedrPr)rNr{_union_row_points_union_rect_has_pointupdater*rr(r zip)r3r`columnswidthcolumn_populatedcolumn_concrete row_countsrYrZr5r row_populatedrectsbandrspannedrVrrrs @@rrUrUs*YYYYYZZG **T***A 6 6 6EsU{cEkOKJH E77 S%% cc #cccMM%MMMCJJ  4$UF33D$ 4 4 t&&&!+&&&,T488"  '''1,'''%.w%7%7 w25(33337###=#e**56666AA[AAAAAQF 3OOOOCLL))OOOLL L I YYjYYYYY  !$%5!G!G         rctd|D}td|D}|t||ft||fS)z?The y-sorted midpoints falling in one grid row's vertical band.c3DK|]\}}t|jVdSrK)rqrr_indexrrs rrAz$_union_row_points..<. 6 6 U47^^ 6 6 6 6 6 6rc3DK|]\}}t|jVdSrK)rqr2rs rrAz$_union_row_points..=rr)rOrPr)rr`rr2s rrr:sb 6 6 6 6 6 6 6B 6 6 6 6 6 6 6B +fre,,{6B5/I/II JJrct|jt|jt|jt|jf\}}}}|D](\}}||cxkr|krn||cxkr|krn%dS)dS)zJWhether any midpoint of the row band lies in rect (half-open, as extract).TF)rqr rr1r2)rrrr rr1r2rrs rrrAs47^^U47^^U47^^U47^^SNBB u          e 0 0 0 0b 0 0 0 0 044 5rcLg}|jpgD]}t|ts|d}|s0 t j|dd}n#t tf$rYawxYw|jrm| |d|f|S)zKThe page's layout groups as ``(class_name, rect)``, skipping unusable ones.rNr!r) r"r#r$r%r+r,rrrQr*)r-r]r/rrrs r_union_layout_groupsrJs F)/R 7 7%&&  YY|,,    < 2A2//DD:&    H  =   uyy..56666 MsAA.-A.ct|}|dkrdSt|z}t|D]7\}}|dkr t|||kr!|t|kr5dSdS)aWhether a candidate sits inside a layout picture group without covering it. The layout model classified that region as a picture, so a line grid found almost entirely inside one is as likely to be a chart's gridlines as a table, and overriding the model's verdict needs table-shaped text. The shape, not the amount, is what tells them apart: a chart's text sits along its two axes, so exactly one row and one column are well filled and its partial gridlines leave most of the grid without a cell at all, while a table has a header row and a label column that are each at least half full over mostly complete cell coverage -- which a sparse matrix of scattered marks still satisfies. See the admission rule in _union_line_candidates. A candidate larger than the picture contains it instead -- a bordered table with an image in one of its cells -- and keeps the ordinary rule. rFpictureT)rs_UNION_PICTURE_CONTAINMENTrrw)r-r~r thresholdrrrs rrSrS]s &n55Nu*^;I066 D  " "  #ND 9 9I E E  ,T22 2 2 tt 5rc@t|dd}||Sg}|ddgpgD]}|dgpgD]}|dgpgD]}t|dpds9|d }|sQt j|}|js|| t|d|n#t$rYnwxYw|S) zNon-empty page text-span rects, cached on the page. Drives the grid-ref span-multiplicity gate: every non-blank span as a bare rect._union_text_spans_cacheNr$blockslinesspansrrra) getattrget_textr%rrr+r,rQr*setattr Exception)r-cachedrblocklinespanrarrs r_union_text_span_rectsr|s\ T4d ; ;F  Ev&&**8R88>B ' 'IIgr**0b ' 'D"--3 ' '488F++1r2288::xx''|D))}'LL&&& ' ' /7777      Ls<D DDcng}|D]X}t|jt|jzdz }|t|jdz ks|t|jdzkrft||dkr{t t|jt|j}t t|jt|j}||krt|D]D\}\}} t||z dkr&| ||f||zdz | f||<nE| |||fgfZd} |D]Q\} } d} d} t| D](\}}| || z dkr| dz } |} t | |} )t | | } R| S)zNMax horizontally-separated text-span groups on any single text line in a cell.rg?g@rNr) rqrr2ryrPr rOr1r{absr*r)r@ text_spans line_bandsrcenter_yr r1rband_y intervalsbest_r]last_x1s r_union_cell_span_group_countrsJ66$'NNU47^^3s: eDGnns* * *htw#9M.M.M  D$ ' '3 . .  twtw 0 0 twtw 0 0 88 *3J*?*? 6 6 &E&FI8f$%%,,  "b***&,x&73%> $J 5!-   x2r(4 5 5 5 D" ! ! 9Y'' + +FB"w,"4"4! gr**4   Krc|dSt|}d}|D]P}|D]K}|tj|}|jr!t ||}|dkr||nt ||}LQ|t |ndS)zEMax cell span-group count over a grid (high => under-segmented grid).Nr)rr+r,rQrrPrq)r-r3rrr5r@rrcounts r_union_span_multiplicityrs |t'--J D C C C CD|<%%D} 0zBBEqyy $ uu#dE2B2B C*5;;;4rci}i}t|D]\}\} } t|D]\} \} } t| | }||krp||g| t |f|| g|t |fi}t }|D]\}}t|dkr|d\} }t|| gdkrL|r%t||| d}|||krs|| ||<| | ||fS)a]Primaries a candidate can grid-ref (replace grid with), 1:1 by IoU. ``existing``/``candidates`` are ``(bbox, grid)`` lists. Returns ``(refs, consumed)``: ``refs`` maps a primary index to the candidate supplying its grid (mutual 1:1 IoU>=threshold matches passing the span-multiplicity gate), ``consumed`` the candidate indexes used.rr) r{r setdefaultr*rqrNitemsr)r%rr[)existingrd iou_thresholdr-span_mult_gatespan_mult_thresholdexisting_matchescandidate_matchesexisting_indexr_gridcandidate_indexr~_cgridiourefsconsumedmatchesr span_mults r_union_one_to_one_grid_refsrs2;H2E2Egg..9B:9N9N g g 5O5nf}n55Cm## ++NB??FFY^_bYcYcGdeee!,,_bAAHH.Z_`cZdZdIefff  g DuuH#3#9#9#;#; & & w<<1   $QZ  $$_b99 : :a ? ?   0z/7RST7UVVI$6I)I)I)/:^ _%%%% >rcd|D}|rt||||||\}} nit} }i} g} t|D]g\} } | | vr t| d|\}}|*| |g| O|rR| | hd| D}g}t|D]h\}}||vr|||%||vr*||d||dfS||i|| |S)aFuse primary and candidate ``(bbox, grid)`` entries. Applies grid-ref replacement, split replacement (>=2 candidates owned by one primary replace it, ordered by y0/x0) and append of unowned candidates, returning the fused entry list in the contractual order.cg|] }|d S)rr)rentrys rrz)_union_replace_append..s666EuQx666r)rr-rrrNc`i|]+\}}t|dk|t|d,S)rcjt|djt|djfSr)rqrr )rs rz2_union_replace_append... s(eAhk0B0BE%PQ(+DVDV/Wr)rb)r)r)rrrs r z)_union_replace_append.. sF E5 u::?? ve!W!WXXX ??rr)rrNr{rrr*rextend)rrdr-grid_ref grid_ref_iourrr grid_refsr replacementsappend_candidatesr candidate owner_indexrfinal_replacementsentriesrrs r_union_replace_appendrs 76X666O (9  &) 3     88!#%%8 L&/ &;&; 0 0" h & & !29Q<!Q!Q Y  "  # #K 4 4 ; ;I F F F F  0   $ $Y / / / /(..00 G!(++"" u & & & NN-e4 5 5 5 5 i   NNE!Hi&6q&9: ; ; ; ; NN5 ! ! ! ! NN$%%% Nrc |j|dt|}t|||\}}t |||dt t t}|5tj tj t|}g}|D]9\}} d| D} | s| t|| |:||_|S)aDetect a page's tables by fusing layout grids with line-based candidates. Ensures the raw layout (computed only when page.layout_information is None, like the official use_layout path), reads primary grids, detects candidates (forwarding the caller's virtual lines / boxes to the nested finder), applies grid-ref / split / append, and returns a TableFinder whose .tables carry the fused grids in contractual order (grid-ref tables keep their explicit layout bbox).NT) return_rawr8)r-rrrrc g|] }|D]}|| SrKrrs rrz&_find_tables_union..>s*IIIII8H8H8H8H8Hr)ra)r" get_layoutr7rmr_UNION_GRID_REF_IOU_UNION_GRID_REF_SPAN_MULT_GATE#_UNION_GRID_REF_SPAN_MULT_THRESHOLDrrkrrr*rtables) r-r9r: primariesrdrlr rrar3flats r_find_tables_unionrs  & 4(((#D))I/ YJ$ (5?G~   T"" F44 dIIIII   eD$T2223333FM Mr)'__doc__bisectr collectionsrr+rirrrrr r rjrrrr|r}rr r7rmrsrwryrrTrUrrrrSrrrrrrrrrrs<""""""PPPPPPPPPPPPPPPP"!!%&)#!%  H B/3dMMMMM`SSS!!!aaa!!!6*AAAHKKK&>6D555&@---`+/$'''''''r