j 2 d Z ddlZddlZddlZddlmZ ddlmZ ddlm Z ddl
Z
ddl
m Z ddl m
Z
ddlmZ dd lmZmZmZmZmZmZmZmZmZmZ dd
lmZ n# e $ r dd l m Z Y nw xY we
j d d
Z! G d d Z" G d d Z#e G d d Z$d=dZ%d Z&d Z'd Z(d Z)d Z*d Z+ddddddd d d d dd dddd d dd!d"dd#dddddd$d%e,fd&Z-d' Z.d( Z/d) Z0e1d*k rddl2Z2ddl3Z3ddl4Z4 e3j5 d+ Z6nA# e7$ r9 e8d,ej9 : e; d- e3j< Y nw xY w e4j= Z> e
j? e6 Z@e3j5 d.d ZA eBe@jC ZD eEeA d.k reAd d/k rg ZDeAd+ F d0e@jC ZGeGH d1 D ]lZId2eIv rE eJeKeIH d2 \ ZLZMeDN eBeLd+z
eM KeDO eKeI d+z
m ePd3 eDD dd4 ZQeQ eP k r e3j< d5eQ d6e@ d7 e-e@eD8 ZRe@jS ZTeTd9z ZU e2jV eU W eRX e4j= ZY e8d:eTd; eZeYe>z
d. d< dS dS )>a
This script accepts a PDF document filename and converts it to a text file
in Markdown format, compatible with the GitHub standard.
It must be invoked with the filename like this:
python pymupdf_rag.py input.pdf [-pages PAGES]
The "PAGES" parameter is a string (containing no spaces) of comma-separated
page numbers to consider. Each item is either a single page number or a
number range "m-n". Use "N" to address the document's last page number.
Example: "-pages 2-15,40,43-N"
It will produce a markdown text file called "input.md".
Text will be sorted in Western reading order. Any table will be included in
the text in markdwn format as well.
Dependencies
-------------
PyMuPDF v1.25.5 or later
Copyright and License
----------------------
Copyright (C) 2024-2025 Artifex Software, Inc.
PyMuPDF4LLM is free software: you can redistribute it and/or modify it under the
terms of the GNU Affero General Public License as published by the Free
Software Foundation, either version 3 of the License, or (at your option)
any later version.
Alternative licensing terms are available from the licensor.
For commercial licensing, see or contact
Artifex Software, Inc., 39 Mesa Street, Suite 108A, San Francisco,
CA 94129, USA, for further information.
N)
b2a_base64) defaultdict) dataclass)mupdf)
get_raw_lines) column_boxes)
BULLETSREPLACEMENT_CHARACTERalmost_in_bbox are_disjoint bbox_in_bbox
bbox_is_emptyintersect_rects is_ocr_textis_whitestartswith_bullet)tqdm) ProgressBarTz
c D e Zd ZdZ d
dedededefd Zdd
e d efd Z
dS )IdentifyHeadersa Compute data for identifying header text.
All non-white text from all selected pages is extracted and its font size
noted as a rounded value.
The most frequent font size (and all smaller ones) is taken as body text
font size.
Larger font sizes are mapped to strings of multiples of '#', the header
tag in Markdown, which in turn is Markdown's representation of HTML's
header tags to .
Larger font sizes than body text but smaller than the font size are
represented as .
N docpages
body_limit
max_levelsc t |t r|t dd vrt d t |t j r|}nt j | }|j rtt j | }t j
t j | t j d }|
t j d |t |j }t t }|D ]} | | }
|
dt j d } d
| D D ]L} t' | d }
||
xx t) | d z
cc< M||k r| i _ t1 d
| D d }|r"t5 ||d d _ n| _ t1 fd| D d d| }t; |d D ]\ }}d|z dz j |< j r0t= j dz
_ dS dS )zRead all text and make a dictionary of fontsizes.
Args:
doc: PDF document or filename
pages: consider these page numbers only
body_limit: treat text with larger font size as a header
z-max_levels must be an integer between 1 and 6RootStructTreeRootNdictflagsblocksc h g | ]/}|d D ]$}|d D ]}t |d |%0S )linesspanstext)r ).0blss /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/pymupdf4llm/helpers/pymupdf_rag.py
z,IdentifyHeaders.__init__..z sp 7 7 & ** sizer) c g | ] \ }}||f
S r3 )r* kvs r. r/ z,IdentifyHeaders.__init__.. s 2 2 21aV 2 2 2r0 c " | d | d fS )Nr r r3 is r. z*IdentifyHeaders.__init__.. s 1Q41, r0 keyr c * g | ]}|j k
|S r3 )r )r* fselfs r. r/ z,IdentifyHeaders.__init__.. s% @ @ @1A,?,?Q,?,?,?r0 T)reverse)start# )
isinstanceintrange
ValueErrorpymupdfDocumentopenis_pdf_as_pdf_documentr pdf_dict_get pdf_trailerPDF_NAME pdf_dict_del
page_countr load_pageget_textTEXTFLAGS_TEXTroundlenstripclose header_idsorteditemsmaxr keys enumeratemin)r? r r r r mydocmypdfroot fontsizespnopager% spanfontsztempsizesr8 r1 s ` r. __init__zIdentifyHeaders.__init__V s *c** Nja .K.K LMM M c7+ , , &EEL%%E < B,U33E%!%(('*:6*B*B D
g ./? @ @ A A A =%*++E $$ ? ?C??3''D]]61G]HHRF
?
? tF|,,&!!!Sf ););)=)=%>%>>!!!!
? C<< KKMMM
2 2 1 1 2 2 28N8N
) !*d2hqk : :DOO (DO @ @ @ @ (( @ @ @
:+ !a000 1 1GAt#&7S=DN4 > = !$."5"5"7"7 8 81 . s6 KKK1K!B%4;?2J2J!2J2J2Jr0 r) r r rB rC )r rW
startswith)r? rf re my_tocr) r titlelvls ` r. rr zTocHeaders.get_header_id s 2KKKKTXKKK 2 F|!!## ' 'AaDJJLLEA$Cu%%
')9)9$)?)?
'Sy3&&&
' rr0 rs )rt ru rv rw rx rj r" rr r3 r0 r. r| r| s`
C
$ c r0 r| c e Zd ZdS )
ParametersN)rt ru rv r3 r0 r. r r s Dr0 r c t | | ||f}g }| dd }|r|d
|z }d}|rXd}t t | dz
dd D ]2}| || j r||| z }||= d}3|X| | |d= |t t
| d }|S ) a Join any rectangles with a pairwise non-empty overlap.
Accepts and returns a list of Rect items.
Note that rectangles that only "touch" each other (common point or edge)
are not considered as overlapping.
Use a positive "enlarge" parameter to enlarge rectangle by these many
points in every direction.
TODO: Consider using a sweeping line algorithm for this.
Nr TFr r< c | j | j fS rs )x0y0rs r. r9 z refine_boxes.. s adAD\ r0 r: )rF rV
intersectsirectappendrZ set)boxesenlargedelta new_rectsprectsr repeatr8 s r. refine_boxesr s Xx' 2EI
111XF
AYJ "F3v;;?Ar22
"
"<<q 00 "NA q !F
" 1I
s9~~+A+ABBBI r0 c P j j k r j dz }n
j dz } ||| | fz } fd|D }t d |D t j hz }t d |D t j hz }t | dk st | dk rdS |D ]}|d }t | s t t
|| s dS |j |j k r@|j |j cxk r |j k r$n n!|j |j
k r|j
|j k r |j |j
k r@|j |j cxk r |j
k r$n |j |j k r|j |j k r dS ) a Check whether the rectangle "box" contains 'signifiant' drawings.
This means that some path is contained in the "interior" of box.
To this end, we build a sub-box of 90% of the original box and check
whether this still contains drawing paths.
g?c @ g | ]}|d v |d k |S rectr3 )r* pboxs r. r/ z"is_significant.. s5 JJJaAfI$4$46c9I9I9I9I9Ir0 c 3 J K | ] }t |d j V dS r N)rU widthr* r s r. z!is_significant.. s0 ::Aqy''::::::r0 c 3 J K | ] }t |d j V dS r )rU heightr s r. r z!is_significant.. s1 <N>N=O
OF<<8<<<<<cj@Q@Q?RRG
6{{a3w<<1,,u
y $ #0 t1L1L#M#M 44
47""47----dg-----$'!!$'!!
47""47----dg-----$'!!$'!! 5r0 c t d )Nz;Function 'to_json' is only available in PyMuPDF-Layout modeNotImplementedErrorargskwargss r. to_jsonr 8
E r0 c t d )Nz=Function 'to_chunks' is only available in PyMuPDF-Layout moder r s r. to_chunksr > s
G r0 c t d )Nz;Function 'to_text' is only available in PyMuPDF-Layout moder r s r. to_textr D r r0 c $ ddl m} || S )zReturn the ``(bbox, html, rows, cols, cells, extract)`` payload for each
table on ``page``. Imported lazily so table_html loads only when
table_output="html" is requested.r )page_html_tables) pymupdf4llm.helpers.table_htmlr )re r s r. _reconstruct_html_tablesr S s( @????? D ! !!r0 c v |dk r| j | d S | j | d S )zRender table ``i`` as markdown (default) or reconstructed HTML.
In HTML mode ``i`` indexes ``parms.html_tables`` directly, so no bbox
matching is needed.
htmlr F)clean) html_tablestabs to_markdown)parmsr8 table_outputs r.
_table_stringr \ s? v #A&& :a= $ $5 $ 1 11r0 Frn pngg? id lines_strictmarkdown )r hdr_info write_images embed_images
ignore_imagesignore_graphicsdetect_bg_color
image_path image_formatimage_size_limitfilename
force_text page_chunkspage_separatorsmarginsdpi
page_width page_heighttable_strategyr graphics_limitfontsize_limit ignore_code
extract_words
show_progress
use_glyphs ignore_alphark c
+,-./0123456789:;<=>?@ | r2t dt | d du rdu r du rt d du rdd}d
cxk rdk sn t d |,|0| 1|--du rd}
d}|d
vrt d |3|22r9du r5t j 2 st j 2d t | t j
st j | } | | j n| .|/|} |} |} |+| j
s| j r(| r| | j rbt% |d r| || n:| |d | j } d| z }!| ||! |!t+ t- | j }t% |d r|gdz }t/ | dk rd|d d|d f}t/ | dk rt d t1 d |D st d t3 | r|6nKt% |d r t3 |j r|j 6n |du rd-d6nt7 | }|j 66fd7d =,12
fd>dd fdt j f-037:=>@fd
@d 4d :d }"-3fd 9>@fd!8d" ;d# 5.fd$}#d%t* d&t* fd'?+-/34 5
89:;>?@fd(}$|
du rd}%ng }%| }&dt j z t j z }'dt j z t j z t j z t j! _" |r|'tF j$ z }'|r"t d). d* tK | }|D ]s< |$| <||'.| | }(|
du r |%|(j& z
}%nP via pymupdf4llm.helpers.table_html.
graphics_limit: (int) if vector graphics count exceeds this, ignore all.
ignore_code: (bool) suppress code-like formatting (mono-space fonts)
extract_words: (bool, False) include "words"-like output in page chunks
show_progress: (bool, False) print progress as each page is processed.
use_glyphs: (bool, False) replace the Invalid Unicode by glyph numbers.
ignore_alpha: (bool, True) ignore text with alpha = 0 (transparent).
z,Warning - arguments ignored in legacy mode: .Fz4Images and text on images cannot both be suppressed.Trn r r z8'image_size_limit' must be non-negative and less than 1.)r r z,'table_output' must be 'markdown' or 'html'.)exist_okN __float__)r r i z'margins must be one, two or four floatsc 3 6 K | ]}t |d V dS )r N)hasattr)r* ms r. r z to_markdown.. s, ::QK((::::::r0 z margin values must be floatsrr c dS )Nrn r3 )r- re s r. r9 z to_markdown.. s R r0 c t d t fd| D D }|sdS d|d dz
z dz S )Nc g | ]
}|d k | S r r3 r* r, s r. r/ z6to_markdown..max_header_id.. s$ X X X1RSVWRWRWQRWRWRWr0 c D g | ] }t | S )re )rV )r* r- rr re s r. r/ z6to_markdown..max_header_id.. s0 M M M!S qt!.max_header_id sk X X M M M M Mu M M MNN X X X
2gaj1n%++r0 c N t j |d }| D ]}|d }|j |j z dz }||vr!|d }|d }dD ]5}| |dt
t | }6d| d | d
}|c S d S ) a Accept a span and return a markdown link string.
Args:
links: a list as returned by page.get_links()
span: a span dictionary as returned by page.get_text("dict")
Returns:
None or a string representing the link in MD format.
bboxfromr r) uriz()
%[z]()N)rH RecttlbrrW replacehexord) linksrf r linkhotmiddler) r cs r.
resolve_linksz"to_markdown..resolve_links s |DL)) Dv,Cfsvo*FT>><%%''Du+C
8
8kk!%6SVV%6%677%t%%s%%%DKKK r0 c
| j }|j |j j z k s|j |j j z k rdS
du s du r| | }ndS |j dk s |j dk rdS
du rt
j | j dd }t
j
| d|j d| d }| | | dd S du r?t | }d
d |z }|S dS ) zOptionally render the rect part of a page.
We will ignore images that are empty or that have an edge smaller
than x% of the corresponding page edge.rn T)clipr r rC -r \/z data:image/z;base64,)re r r r
get_pixmapospathbasenamer r joinr saver tobytesdecode)r r r8 re pixr image_filenamedataDPI
IMG_EXTENSIONIMG_PATHr r r s r.
save_imagez to_markdown..save_image
sp
z J +;; ; ;{TY -0@ @@@2 4 <4#7#7//t/55CC2 :??ci1nn2 4 w''77??SIIH W\\XII IIaII-II N
HH^ $ $ $!))$44 4
T
!
! ckk- 8 899@@BBD8 8884?DKrr0 r c
@ * | j }d}t j |d j }/ fd|D } j d |D d}d}d} d}
|D ]\ *} /* j s|r* fd j D } | D ]\ }
}|d
t |
- z d
z z
}+rq-d k rkt t d j |
j
j j |
j z D d
} j | j |
d}
|r.write_text..L sN
!3!3AaD%/:P:P:R:R!S!S
r0 c g | ]
}|d S r r3 r s r. r/ z3to_markdown..write_text..P s 6 6 6!1 6 6 6r0 r< Fc " g | ]\ }}|j j k rv|j vrm j |j cxk r j k sKn j |j cxk r j k s-n |j j cxk rj cxk r |j k n n||fS r )r r written_tablesr r )r* r8 tab_rectlrectr s r. r/ z3to_markdown..write_text..` s " " "#8 {eh..!555 6 !8x{====UX==== 8hk====UX====#;%(LLLLUXLLLLLLLLL M MLLr0
r c : g | ]}|t j | S rs rH r r* r s r. r/ z3to_markdown..write_text..t s/ !" !" !"()'(} %,LOO (5}}r0 c | j | j fS rs r r r s r. r9 z1to_markdown..write_text..{ s 14, r0 r: r Ttablesimagesr r g ?rC c g | ]
}|d S )r) r3 r* r- s r. r/ z3to_markdown..write_text.. s 6 6 61QvY 6 6 6r0 c $ g | ]
}|d dz S )
char_flagsr r3 r9 s r. r/ z3to_markdown..write_text.. s! D D D< 1!4 D D Dr0 c 8 g | ]}|d t j z S r# )rH TEXT_FONT_ITALICr9 s r. r/ z3to_markdown..write_text.. s$ S S Saj7+C C S S Sr0 c l g | ]1}|d t j z p|d t j j z 2S )r$ r; )rH TEXT_FONT_BOLDr
FZ_STEXT_BOLDr9 s r. r/ z3to_markdown..write_text.. sM wZ'"88 G,'-*EE r0 c 8 g | ]}|d t j z S r# )rH TEXT_FONT_MONOSPACEDr9 s r. r/ z3to_markdown..write_text.. s$ UUU!AgJ)E EUUUr0 r `_z**z~~z```
r1 ?blockr) r r$ r; zzzzzzz- z r r g z
z
z
)5r r textpageaccept_invisible
line_rectsextend img_rectsr% r[ r rZ r r headercellsr* r rF rV written_imagesr\ r r_ r r r
GRAPHICS_TEXTr r r rW allre endswithrE r r r^ rH TEXT_FONT_SUPERSCRIPTrB r r? r r@ r= FZ_STEXT_STRIKEOUTFZ_STEXT_UNDERLINEFZ_STEXT_HIGHLIGHTreversedr r rU )3r r r5 r6 r
out_stringnlines
prev_lrectprev_bnocodeprev_hdr_stringr( tab_candidatesr8 rD rM r pathnameimg_txtr)
all_strikeout
all_italicall_boldall_mono
hdr_stringr indentspan0bnor- superscriptmonobolditalic strikeout underline highlightprefixsuffixltextdistcwidthr,
EXTRACT_WORDS IGNORE_CODE TABLE_OUTPUTr r' r r
write_texts3 ` @r. rw z to_markdown..write_text- s
( <:D
N!&!7 7
6 6v 6 6 6777
" g # g #LE5%%eU_==
+ " " " " "','<'<'>'> " " " + + +DAq $ ua )N)N"NQU"UUJ %
7 )?)? & !" !"-2Z]-A-G&+jm&9.:!" !" !" !7 6
!
!
! (//666(//222&*OO
/ s5? 3 344 / /AE 000 *A14**Sux-@-@@@ A 8qt6666eh6666 8ad6666eh6666 458>>>>eh>>>>!$>>>>>#-:eQ#:#:# C &-(*B BJ &--&0j % !',',+/ ' ' ' G $,G#4#4 6 *g 5
,33A666*. # #E * * * E$%%))Hu / 3 66 9KKKd"
88 6 6 6 6 677==??D D De D D D E EM S SU S S STTJ # H UUuUUUVVH ' u5:>>>J
, :+D , :+D . $;-D . $;-D 00 *t"3d"::JJ %--d33 5%/_
%--d33 5 #*t"33J", (O
')J D UX /E!HV4Ds4J KLL uftmd 22
H
g%
!HE.Chd"
#Hz},u|c/AAA=++C00 B$U6]33 B >A% B
d"
J
g%
!%(( B
# B
#1 j7+H H gJ!== + 'NN*
gJ!77 E )DD 7g&>> lOgm.N N lOgm.N N lOgm.N N ,MM'***MM(+++ (MM$'''MM$''' 'MM#&&&MM#&&& (MM$'''MM$''' *MM%(((MM&))) -MM(+++MM),,, 'MM#&&&MM#&&& &!1!122 &
ek155 O (B&B%BBBBDD (N&N!F)//2C2CNVNNND$T** B $qrr(?D <<c22D =+dg5D #FmA .v q1A ASv EWEW WF }}!&v !4 U4&=%9%9!:!::TADd"
#d"
d
' !JDf
ud + + 3 3D# > > F FxQW X X
r0 c : t fd|D S )z1Check if rect is contained in a rect of the list.c 3 8 K | ]}t | V d S rs )r
r* r r s r. r z8to_markdown..bbox_in_any_bbox..J - <.bbox_in_any_bboxH & <<<<)<<<<<.outside_all_bboxes..N r{ r0 )rP r~ s ` r. r' z'to_markdown..outside_all_bboxesL r r0 c | d dz
| d dz
| d dz | d dz gt fd|D S )z;Check if middle of rect is contained in a rect of the list.r r r r c 3 < K | ]}t |d V dS )rE )portionN)r )r* r enlargeds r. r z:to_markdown..almost_in_any_bbox..T s2 OO>(As;;;OOOOOOr0 r| )r r r s @r. almost_in_any_bboxz'to_markdown..almost_in_any_bboxP sV GaKa1d1gk47Q;GOOOOYOOOOOOr0 c d}t fd| j D d D ]\ }}|| j v r|t | | dz z
}rqdk rkt t d | j | j j | j | j z D d }| j
| | j | n| j D ]\ }}|| j v r|t | | dz z
}rqdk rkt t d
| j | j j | j | j z D d }| j
| | j | |S ) z)Output tables above given text rectangle.rn Nc @ g | ]}|d j j k |S r )r r )r* j text_rects r. r/ z6to_markdown..output_tables..[ s+ OOOqqtw),7N7N7N7N7Nr0 c 6 | d j | d j fS )Nr r2 )r s r. r9 z4to_markdown..output_tables..\ s qtw! 0 r0 r: r- r c : g | ]}|t j | S rs r/ r0 s r. r/ z6to_markdown..output_tables..e / $%#$= !( Q $1==r0 c | j | j fS rs r2 r3 s r. r9 z4to_markdown..output_tables..l qtQTl r0 c : g | ]}|t j | S rs r/ r0 s r. r/ z6to_markdown..output_tables..z r r0 c | j | j fS rs r2 r3 s r. r9 z4to_markdown..output_tables.. r r0 ) rZ r% r[ r* r r r rL rM rI rJ r )r r this_mdr8 trectrM rt rv s ` r.
output_tablesz"to_markdown..output_tablesV s1 "OOOOEO 1 1 3 3OOO00
/
/5 ,,, =<@@4GG
3\V%;%; " ).A )=)C"'*Q-"5*6 3 2
E $++E222$++A....+
/0 "O 1 1 3 3
/
/5,,, =<@@4GG
3\V%;%; " ).A )=)C"'*Q-"5*6 3 2
E $++E222$++A....r0 c | j sdS d}|t | j D ]\ }}|j |j k r|j |j k s|j |j k r7|| j v rA | || }| j | |r
|t |z z
}|r$ | |ddd }t | s||z
}nt | j D ]k\ }}|| j v r | || }| j | |r
|t |z z
}|r$ | |ddd }t | s||z
}l|S )z0Output images and graphics above text rectangle.rn NFTr4 ) rK r^ r r r rN r rO r )
r r r r r8 img_rectr^ r_ r rw s
r.
output_imagesz"to_markdown..output_images s 2 ( 9 9
+
+ 8; -- ;),..(+ 2M2M ,,, %:eXq99$++A... 8}x 77G + (j $ $#' G $G,, + 7*)
+, ) 9 9
+
+ 8,,, %:eXq99$++A... 8}x 77G + (j $ $#' G $G,, + 7*r0 c t d | D }|dhk rdS n# Y nxY wdS )z~Check if page exclusivley contains OCR text.
For this to be true, all text must be written as "ignore-text".
c 2 g | ]}d |d v |d S )r) r r3 r* r+ s r. r/ z4to_markdown..page_is_ocr.. s% Q Q Qq&AaD..ad...r0 z ignore-textTF)r get_bboxlog)re
text_typess r. page_is_ocrz to_markdown..page_is_ocr sY
Q QD,<,<,>,> Q Q QRRJm_,,t - Dus 27 ;c | | j j | j j | j j dz | j j dz f }|j r|j sdS | dd }| | j j dz
| j j | j j | j j dz f }|j r|j sdS | dd }||k sdS | | j j | j j dz
| j j dz | j j f }|j r|j sdS | dd }||k sdS | | j j dz
| j j dz
| j j | j j f }|j r|j sdS | dd }||k sdS |d dz |d dz |d dz fS )a, Determine the background color of the page.
The function returns a PDF RGB color triple or None.
We check the color of 10 x 10 pixel areas in the four corners of the
page. If they are unicolor and of the same color, we assume this to
be the background color.
)r Nr r r ) r r r r samples is_unicolorpixelr r )re r pixel_ulpixel_urpixel_llpixel_lrs r. get_bg_colorz!to_markdown..get_bg_color s oo), dilR.? PRARS
{ #/ 499Q??oo),#TY\49< PRARS
{ #/ 499Q??8##4oo), r 149<"3DdilS
{ #/ 499Q??8##4oo),#TY\B%6 dilS
{ #/ 499Q??8##4 c!8A;#4hqkC6GHHr0 c h | j }|d< | j |d< |dz |d< |S )N file_pathrQ r re )metadatacopyrQ )r rd metaFILENAMEs r. get_metadataz!to_markdown..get_metadata s> | "" $[ ^\QwV r0 wordsrk c : | sg S g }| d g}t j | d dd }| dd D ]}t |d |j z
dk s!t |d |j z
dk r#| | ||dd z }g| d | | |g}t j |dd }| d | | |S ) a Reorder words in lines.
The argument list must be presorted by bottom, then left coordinates.
Words with similar top / bottom coordinates are assumed to belong to
the same line and will be sorted left to right within that line.
r Nr r r c | d S Nr r3 ws r. r9 z1to_markdown..sort_words.. s
! r0 r: c | d S r r3 r s r. r9 z1to_markdown..sort_words.. s
! r0 )rH r absr r r sortrJ )r nwordsliner, r s r.
sort_wordsz to_markdown..sort_words s# Iaz U1Xbqb\**qrr , ,A1Q4%(?##q((C!ux,@,@A,E,E A2A2 nn ---
d###s QrrU++ nn %%%
d
r0 c
4 +,- | | }| t ,|,_ |,_ d,_ g ,_ g ,_ g ,_ g ,_ g ,_ :| p5,_
.sdn
4| ,_ |\ }} }
} |j || |
| fz ,_
d | D ,_ d | D ,_ | |,j
,_ |s| } ng } t+ t- | D ]+}
t/ j | |
d | |
d< ,6,fd| D } | d d
| rtt5 ,j
d z ++,fd | D }t- | t- | k r.|} <,,j
d
}|r,xj t6 |z z
c_ | dd } t+ t- | dz
dd D ]B}
| |
d }|j r| |
= t+ |
D ]}|| | d v r| |
= nC| ,_ d ,j D ,_ t- d | D }0r |0k rd }d }nd}g ,_ g ,_ g ,_! i }|s>sn1dk rtE | ,_! tG ,j! D ]T\ }
\ }}}}}}t/ j | ||
< ,j $ tK ||
||d Un|& ,j
> }|j D ]3} | j' dk s | j( dk r,j $ | 4,j d tG ,j D ]{\ }
} t/ j | j) t/ j | j* j) z ||
< ,j $ tK ||
| j' | j( d ||,_+ tY |- ,_. |s"9,fd|/ D } ng } |rg } g -|0 | } | D ]'} tc | | r-$ | (2-fd | D ,_2 -3 ,j ,j 3 - ti tk ,j d ,_ g ,_6 to - ,_8 ts d tG ,j8 D ,_: t- ,j ; } | dk r*t- ,j < | z }!nd}!|r|!d k r ,j
g}"n@t{ ,j ,j2 3 ,j ,j. ,j8 z |d! |d |" }" |"D ]U}#,xj 8,|# z
c_ ,xj 7,|#3 z
c_ ,xj ?,|#3d d # z
c_ V,j > d$d% > d&d ,_ ,xj 8,d z
c_ ,xj 7,d3 z
c_ ,j ? d' r.,j dd ,_ ,j ? d' .,j > t d t ,_ /d u r,j B }$|$ d( g }%,j D ]\}&g }'|$D ]7}(t/ j |(dd) })|)|&v r|'$ |( 8|%3 =|' ]g }*|%D ]}(|(|*vr|*$ |( |*}%ng }%|%,_ ;r ,xj d*,j jC d+z
c_ ,S ),a Process one page.
Args:
doc: pymupdf.Document
pno: 0-based page number
textflags: text extraction flag bits
Returns:
Markdown string of page content and image, table and vector
graphics information.
rn Nc > g | ]}|d t j k |S )kind)rH LINK_URIr s r. r/ z8to_markdown..get_page_output..1 s) TTTQai7CS6S6Sq6S6S6Sr0 c g | ] }|j
S r3 r )r* as r. r/ z8to_markdown..get_page_output..4 s ; ; ;QV ; ; ;r0 )r$ r r c g | ]}|d j j j z k rb|d j j j z k rD|d j r$|d j dk r|d j dk |S )r r )r r r r )r* r8 r r s r. r/ z8to_markdown..get_page_output..B s
y"2UZ5E"EEE& $4uz7H$HHH& $$UZ00 I& !##& 1$$
%$$r0 c , t | d S Nr r r7 s r. r9 z6to_markdown..get_page_output..M s C& NN r0 Tr; r@ g?c V g | ]%}t |d j z k #|&S r )r r )r* r8 img_max_sizer s r. r/ z8to_markdown..get_page_output..R s6 VVV!3qy5:/E+F+F +U+UA+U+U+Ur0 full r r r< c g | ]
}|d S r r3 )r* r8 s r. r/ z8to_markdown..get_page_output..g s ;;;1V9;;;r0 c & g | ]}d |d v |S )r r r3 r s r. r/ z8to_markdown..get_page_output..j s! N N NAv1~~a~~~r0 Fr )r rowscolumns)r strategyr c 6 | j d | j d fS )Nr r r )r s r. r9 z6to_markdown..get_page_output.. s 16!9afQi*@ r0 r: c x g | ]}|d j v r|d j j j k r|d j j j k ro|d j dk s|d j dk rM|d dk r|d j k |d j |d j |S )r r typer> fill)r r r bg_color
tab_rects0 annot_rects)r* r r' r s r. r/ z8to_markdown..get_page_output.. s
V9
**fIOej&666fI$uz'888vY_q((AfI,.I.I&&qy%2BCC /J&&qy%2CDD /J
/J.I.Ir0 )drawingsc 8 g | ]} |d |S r r3 )r* r r vg_clusters0s r. r/ z8to_markdown..get_page_output.. s@
0 06L I I
r0 c | j | j fS rs r2 r s r. r9 z6to_markdown..get_page_output.. s adAD\ r0 c 3 $ K | ] \ }}||fV d S rs r3 )r* r8 r s r. r z7to_markdown..get_page_output.. s* R RDAq!Q R R R R R Rr0 r )r
no_image_textrG avoid
footer_margin
header_marginr )r r6 r5 z ,,z-
r- c " | d | d fS )Nr r r3 r s r. r9 z6to_markdown..get_page_output.. s 1qt r0 r z
--- end of page=z ---
)Dremove_rotationr re r md_stringr6 r5 graphicsr rI rH r r r get_linksr annotsr get_textpagerG get_image_inforF rV rH r r r rO is_emptyrK r r* r r r r^ r tuple find_tables row_count col_countr rL r% ry r& r get_drawingscluster_drawingsr actual_pathsrJ rZ r rN r r r" vg_clusters
extractBLOCKS extractTEXTr r r chrr
extractWORDSr )@r rd r textflagsr
IGNORE_IMAGESIGNORE_GRAPHICSre lefttoprightbottomimg_infor8 sanenamer r graphics_counttoo_many_graphicsr% r _htmlr cols_cells_extractr r r clustersr block_count char_density
text_rectsr rawwordsr r, lwordsr wrectr r r r DETECT_BG_COLORrt GRAPHICS_LIMITrv r r r r r r r r' r r r r r rw s@ @@@r. get_page_outputz$to_markdown..get_page_output
s 3x
! K -
&5L,,t:L:L#* c5&Y$eVfW!==
UT$.."2"2TTT < ;T[[]] ; ; ; ***LL **,,HHHs8}}%% D DA"),x{6/B"C"CHQK
22D
AAA < uz??S0LVVVVVxVVVD4yy3x==(( !z%V<< <OO}t';;OOCRC=s8}}q(!R00 A F#Az
QK1XX
F+++ E , ;;el;;; N N)9)9);); N N NOO &n~ = = "O $ % "
# . #
V
#
# !9 > >E BKEL]B^B^
>>D%tVX &|D 1 1 ! ## "9Q< 0 0$4PP
##n#MMD[
%
%;??akAoo
!!!$$$$ JOO @ @O A A A "%*--
1 &|AF 3 3gl18=6Q6Q Q ! ## %il 3 3 ! #$; $ 0 0 2 233
**,,
EE E E ((%(88 * *D dE**
* ##D)))
EO,,,
|,,, U_!5!5;Q;QRRR ! *, 7 7 R RIe>P4Q4Q R R R R R%.668899 ?? u~ 9 9 ; ;<<{JLL L !2!2 * JJ%
(",n &);; %aj %aj + J $ I OO}}UI > > >OO OO}}UIz J J JOO
OOzz%
OOO /11$<<DDUBOO ==555
==jAAAo((.. 2 #oabb 1EO o((.. 2 /11#a&&:OPP D ~2244H MM 4 4M 5 5 5E )
1
1! ) )A #L2A2 / /E ~~
a((( ZZ / /0000 F
%
%F??MM!$$$EE E R OO Qej6G Q Q Q QOO r0 z Processing z...c 2 g | ]}|d dz k |S )r< r r3 )r* r rd s r. r/ z to_markdown..M s+ <<**++w $$ )a-
7||qgaj!WQZ0
7||qBCCC
::':::
:
: 97888 /
? + + /9O0P0P / .
U ///
"3'' .
, , , , , 4 F Y
Y
lY
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
v= = == = =P P P / / / / / /b. . . . . .` &I &I &IP $ 4 4R R R R R R R R R R R R R R R R R R R R R Rh e ++--C
$ %
% &
!
% &
$ % M @U?? #
)H)))***E""
% u .OO =<<.j AfI r0 Tr r r r< )r rF rV rH r r r r )re r r r r8 itemr r s r. extract_images_on_page_simpler- ` s
""$$H
3x==
!
! { |DL11EJ>V
MM..M===
3x==1$a
,
, QK : q AHQK''' QK ( Or0 c 0 g }| D ]~}t j |d |j z }|j s8t |j | j j z |j | j j z |k rd||d< | | |S r )
r rH r r r r\ r r r r )re r r r r, r s r. filter_small_imagesr/ y s H##%% Lf & & 3 : $)/)18di6F+F G G
V Or0 c t | || }| d d t t | dz
dd D ]B}|| d }|j r||= t | D ]}||| d v r||= nC|S )Nc , t | d S r r r7 s r. r9 z4extract_images_on_page_simple_drop.. r+ r0 Tr r r r< r )r/ r rF rV r )re r r r r8 r r s r. "extract_images_on_page_simple_dropr2 s "40@AAH
MM..M===
3x==1$a
,
, QK : q AHQK''' QK ( Or0 __main__r zUsage:
python z
input.pdfr z-pagesNr r c 8 g | ]}|t j k |d z S r )r rQ )r* ns r. r/ r/ s( GGGQ13F3F1q53F3F3Fr0 r zPage number(s) z not in 'z'.)r z.mdz$Markdown creation time for FILENAME=rC z sec.r )[rw r restringbinasciir collectionsr dataclassesr rH r "pymupdf4llm.helpers.get_text_linesr pymupdf4llm.helpers.multi_columnr pymupdf4llm.helpers.utilsr r
r r r
r r r r r r r ImportError pymupdf4llm.helpers.progressTOOLSunset_quad_correctionsrO r r| r r r r r r r r rx r r- r/ r2 rt pathlibsystimeargvr
IndexErrorr r r __file__exit perf_countert0rJ r r rF rQ r rV r
pages_specsplitspecmaprE rA endrJ r r wrong_pagesr r r outnamePath write_bytesencodet1rU r3 r0 r. rW s # #J
# # # # # # ! ! ! ! ! ! < < < < < < 9 9 9 9 9 9 9((((((( 9 9 9888888889
$ $T * * *
e e e e e e e eP. . . . . . . .d F+ + +\ " " "2 2 2
!;v v v> ?v v v vr 2 & zNNNJJJKKK8A;
F 0 0 : :FFFGGG
B
',x
C HQRRLE E#. ! !E
s5zzQ58x// 1X%%ccn+>??
$$S)) , ,Dd{{ Sdjjoo 6 6
s UU519c 2 23333 SSYY]++++ cGG%GGGKLL ##%% CHD{DDSDDD E E E I xH GGL%%i&6&6&8&8999 B E
I
I
IEE"R%NN
I
I
IJJJJJW s$ A A%$A%
D ;EE