j+|dZddlZddlZddlmZmZmZmZmZ ddZ dddddd d Z e d krddl Z ej d ZejeZd ZeD]$Zee eddzedzdzz Z%e jejdedSdS)z This script accepts a PDF document filename and converts it to a text file. Dependencies ------------- PyMuPDF v1.24.2 or later Copyright and License ---------------------- Copyright 2024 Artifex Software, Inc. License GNU Affero GPL 3.0 N)is_whitealmost_in_bbox are_disjoint bbox_is_emptyTYPE3_FONT_NAMETc|}d}t|tjs|td| ||j}|$d|dD}g}t |D]Z\} } t| d|rt | dD]&\} } t| d|r| d} |rtd | d z d krDt | d D]\}}t|d r|d ts|dd kr|rJtj |d}| dd |_ | dd|_||d<t|d|ds| |d<| |d<| |d<||͐(\|sgS|dg}|d g} |d d}|d dD]}|d}| dd}t|j|jz |ks t|j |j z |kr| |||z}s|| } ||| g|g} |}|| } ||| g|S)aExtract the text spans from a TextPage in natural reading sequence. All spans roughly on the same line are joined to generate an improved line. This copes with MuPDF's algorithm that generates new lines also for spans whose horizontal distance is larger than some threshold. Result is a sorted list of line objects that consist of the recomputed line boundary box and the sorted list of spans in that line. This result can then easily be converted e.g. to plain text and other formats like Markdown or JSON. Args: textpage: TextPage object. Can be None if blocks are given. blocks: (list) if given, use these blocks instead of extracting them from the TextPage. This allows to re-use blocks extracted by the caller. clip: (Rect) specifies a sub-rectangle of the textpage rect (which in turn may be based on a sub-rectangle of the full page). tolerance: (float) put spans on the same line if their top or bottom coordinate differ by no more than this value. ignore_invisible: (bool) if True, invisible text is ignored. This may have been set to False for pages with OCR text. Returns: A sorted list of items (rect, [spans]), each representing one line. The spans are sorted left to right. Span dictionaries have been changed: - "bbox" has been converted to a Rect object - "line" (new) the line number in TextPage.extractDICT - "block" (new) the block number in TextPage.extractDICT This allows to detect where MuPDF has generated line breaks to indicate large inter-span distances. c|dtt|dz ddD]}||dz }||}|ddz}|dj|z|djks(|d |d d zf|d |d d zfkrh|d |d ks|d|dkr|d xx|d z cc<|dxx|dzcc<||=|||dz <|S) aSort and join the spans in a re-synthesized line. The PDF may contain "broken" text with words cut into pieces. This funtion joins spans representing the particles and sorts them left to right. Arg: A list of spans - as derived from TextPage.extractDICT() Returns: A list of sorted, and potentially cleaned-up spans c|djS)Nbboxx0ss /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/pymupdf4llm/helpers/get_text_lines.pyz7get_raw_lines..sanitize_spans..Ss & keyrsizeg?r flags char_flagstext)sortrangelenx1r)lineis0s1deltas rsanitize_spansz%get_raw_lines..sanitize_spansFs> ,, ---s4yy1}a,,  Aa!eBaBvJ$E&z}u$r&z}447 < 2%9 7 < 2% 99&zRZ''2f:F+C+C6 bj( vJJJ"V* $JJJQDQKK rNz+Either textpage or blocks must be provided.cTg|]%}|ddkt|d#|&S)typerr )r).0bs r z!get_raw_lines..ws=   yA~~mAfI&>&>~ ~~rblocksr linesdirrrgMbP?spansrfontalphargRQ?)portionr!blockc:|dd |djfS)Nr.rr )y1rs rrzget_raw_lines..sqx{lAfIL9rrr) isinstancepymupdfTextPage ValueErrorrect extractDICT enumeraterabsr startswithrRecty0r5rappendr)textpager,clip toleranceignore_invisibleonly_horizontaly_deltar&r/bnor*lnor!line_dirsnorsbboxnlineslrectsbbox0s r get_raw_linesrPsRRG(((T h 0 1 1HfnFGGG |,} ~  ))++H5    EF##" " Q & 4 ( (  "1W:..  ICDL$// E{H $'HQK$8$84$?$?#DM22  QAfI&&& ,,_==' a(( QvY// <?<?!& %aitDDD&  ' #% Q/  B   JJ99J::: F !H:D !HV E 122Y& b&! ux&)# $ $ / /3ux&)7K3L3LPW3W3W KKNNN UNE ~d##  udm$$$s >$  D MM5$-   Mr F)rBrCseprDocrctj}||s|jntj|}|dkr|nd}|4|dur|||}n|dd}n|}t||| } |sd}| sdSd} |sjd } | D]^\} } | d d }|| kr| d z } |} | d d}| D](}|d}|d}||kr| |z } n| ||zz } |})| d z } _| d z } | S g}g}d }| D]\} } t| dkr|s| | d ddzz } /|s0d| D| d dj gz}t| }dg|z}| D]u\}}tt|dz D]P}||||dz}}t|j |z dkst|j |z dkr|||<Qv| ||rtdd|d zdz}| |z } | dt|d zdzz } |ddD] }| dd|zdzz } !| d z } | S)aExtract text by line keeping natural reading sequence. Notes: Internally uses "dict" to select lines and their spans. Returns plain text. If originally separate MuPDF lines in fact have (approximatly) the same baseline, they are joined into one line using the 'sep' character(s). This method can be used to extract text in reading sequence - even in cases of text replaced by way of redaction annotations. Args: page: (pymupdf.Page) textpage: (TextPage) if None a temporary one is created. clip: (rect-like) only consider spans inside this area sep: (str) use this string when joining multiple MuPDF lines. Returns: String of plain text in reading sequence. |NF)rCri,T)dpifull)rCrDrrr3 r!rrz c(g|]}|djS)r r )r)rs rr+z"get_text_lines..s222qy|222rr rz| z|---)r7TEXT_MEDIABOX_CLIPremove_rotationr:r? get_textpageget_textpage_ocrrPrr rr=rrAjoin)pagerBrCrRrDrS textflagsprecttpr-alltextprev_bnorNr!rHline_norrIstextrowsxvalues col_countrowrlr"rr s rget_text_linesrnsJ&*I! 9DIIw|D'9'9E##C %<<""Y"??BB&&3T&::BB  "5I > > >E   rG    KE4q''"Ch4H1gfoG  i& c>>u$GGsU{*G tOGG4 DGI t t99>>'> tAwv1 1G  "22T222d2hv6F6I5JJGD IdY  DAq3w<>Q&&#adRi..A*=*=CF  C CHHT!W%%%-36CQLL(5008 3 3C sSXXc]]*U2 2GG4 Nr__main__rrV )rRrY z.txt)NNNrTT)__doc__sysr7pymupdf4llm.helpers.utilsrrrrrrPrn__name__pathlibargvfilenameopendocrr`chrPathname write_bytesencoderrrs~      ddddN&*$!QV`````F zNNNx{H ',x C DFF t---4ss2ww>EEGLCH"""##// >>>>>r