j1xdZddlZddlmZmZmZmZgdZdZ dZ deDZ d Z d Z d ZGd d ZdS)a ImageFeatureExtractorV1 Purpose: ONNX-based image feature extraction and segmentation-based bbox detection. CCL execution strategy (lazy): predict() -> ONNX inference only. CCL is NOT run. is_image_page(page_img) -> runs predict() + non-picture CCL only. get_picture_detections()-> runs picture CCL on first call after predict(). This minimises redundant work across three usage patterns: Pattern A is_image_page() -> False (common case) ONNX: 1, CCL: 1 (non-picture only) Pattern B is_image_page() -> True -> predict() -> get_picture_detections() ONNX: 1, CCL: 2 (non-picture + picture, each once) Pattern C predict() -> get_feature_map() only ONNX: 1, CCL: 0 Cache protocol (single-use): mark_cached() / consume_cache() -> let BoxRFDGNN.is_image_page() signal that ONNX inference has already run, so the next image_feature_extraction_task() call skips predict(). ONNX output contract (matches ImageFeatureExtractorV2): 'combined' -- (1, 5*F, H, W) concatenated decoder feature maps 'logits' -- (1, C, H, W) raw per-class segmentation logits (no softmax baked into the graph; CCL uses argmax, which does not need it, and softmax is computed in numpy elsewhere only where an actual probability is required). Both outputs are read by name via out_map, never by output position, so the extraction code here is robust to output ordering the same way V2 is. get_feature_map() / get_class_logits() contract: These two outputs are kept SEPARATE (not concatenated) because they have different statistical character: 'combined' is a continuous embedding space best summarized by mean/max pooling, while 'logits' is a per-class score that should be turned into a probability (softmax) before pooling with ops like mean/min/max/entropy/margin. Callers that need both (e.g. GNN node/edge image features) pool each separately with roi_pooling.extract_bbox_features_by_roi_pooling and concatenate the results themselves; see roi_pooling.DEFAULT_FEATURE_MAP_POOLING_OPS and DEFAULT_CLASS_LOGITS_POOLING_OPS. N) resize_imageto_gray extract_bboxes_from_segmentation&extract_bboxes_from_segmentation_numpy) backgroundtexttitlepicturetablez list-itemz page-headerz page-footerzsection-headerfootnotecaptionformularr c2g|]}|ttfv|S)_BACKGROUND_CLASS_PICTURE_CLASS).0cs /Users/tedpeters/Documents/Projects/Claude Projects/Misc Tools/notename-sheets/.venv/lib/python3.11/site-packages/pymupdf/layout/onnx/ImageFeatureExtractorV1.py rBs5IIIa%6$GGGGGGg? cPeZdZdZdZdZdZd dZdZdZ d Z d Z d Z d Z dS)ImageFeatureExtractorV1ap Wraps an ONNX inference session to produce: - a feature map - picture-class bbox detections (lazy, via get_picture_detections()) - non-picture-class detections (lazy, via is_image_page()) All CCL work is deferred until the result is actually needed, and cached so repeated calls within the same predict() cycle are free. c||_d|_d|_d|_d|_d|_d|_d|_d|_d|_ d|_ dS)z Args: onnx_session: onnxruntime.InferenceSession (or any object with get_inputs() / run() matching the ORT interface) NFr) _session _combined_logits _raw_outputs_cached_detections_picture_detections_non_picture_page_h_page_w _target_h _target_w)self onnx_sessions r__init__z ImageFeatureExtractorV1.__init__Us\ * !! !" (, '+$  rcd|_dS)z4Signal that predict() results should be reused once.TNr"r)s r mark_cachedz#ImageFeatureExtractorV1.mark_cachedos  rc&|jr d|_dSdS)z Return True (and clear the flag) if a cached result exists. Called by image_feature_extraction_task() to skip predict(). FTr-r.s r consume_cachez%ImageFeatureExtractorV1.consume_cachess <  DL4urNc|jdd\|_|_|jdj}|d|dc|_|_t||j|jf}t| tj }| | }}||kr ||z ||z z }n tj|tj }tj|d}|tj||gd}tj|d}|jdj} d|jD} |j| | |i} t)t+| | } | d|_| d |_|j|_d|_d|_dS) a Run ONNX inference and store the outputs. CCL is NOT performed here -- results are computed lazily on demand. Calling predict() invalidates all previously cached CCL results so that get_picture_detections() and is_image_page() always reflect the current page. Args: page_img: np.ndarray (H, W, C), uint8 aug_fetmap: optional extra channel map concatenated before inference Side effects: self._combined / self._logits <- set from ONNX outputs by name self._raw_outputs <- alias for self._logits (CCL input) self._detections_picture <- reset to None self._detections_non_picture <- reset to None Nrrrdtype)axiscg|] }|j Sr)name)ros rrz3ImageFeatureExtractorV1.predict..sDDD1DDDrcombinedlogits)shaper%r&r get_inputsr'r(rrastypenpfloat32minmax zeros_like expand_dims concatenater7 get_outputsrundictziprr r!r#r$) r)page_img aug_fetmap input_shape img_resizedimg_graymin_valmax_valnn_input input_name output_names ort_outputsout_maps rpredictzImageFeatureExtractorV1.predicts&&.^BQB%7" dlm..0039 )4QQ&#8dndn-MNN k**11"*==#<<>>8<<>> W   7*w/@AHH}XRZ@@@H>(333  !~x&<1EEEH>(333 }//11!49 DD (A(A(C(CDDD }(( H7MNN C k::;; , *  L(, '+$$$rcr|jdSt|jttgt |_dS)zDRun picture CCL if not yet computed for the current predict() cycle.N seg_logits class_names target_classmin_component_area)r#rr _CLASS_NAMESr_MIN_COMPONENT_AREAr.s r_ensure_picture_detectionsz2ImageFeatureExtractorV1._ensure_picture_detectionss>  # / F#I|$()2 $ $ $    rcp|jdSt|jttt |_dS)zHRun non-picture CCL if not yet computed for the current predict() cycle.NrW)r$rr r\_NON_PICTURE_CLASSESr]r.s r_ensure_non_picture_detectionsz6ImageFeatureExtractorV1._ensure_non_picture_detectionss;  ' 3 F'M|$-2 ( ( ( $$$rc|jS)aY Return the decoder feature map ('combined') from the last predict() call. This is the continuous embedding output only -- it no longer includes the class logits (see get_class_logits() for those). Returns: np.ndarray of shape (1, 5*F, H, W), or None if predict() not yet called. )rr.s rget_feature_mapz'ImageFeatureExtractorV1.get_feature_maps ~rc|jS)a Return the per-class segmentation logits ('logits') from the last predict() call. Raw (pre-softmax) scores; apply softmax before pooling ops that expect a probability distribution (e.g. entropy, margin -- see roi_pooling.extract_bbox_features_by_roi_pooling). Returns: np.ndarray of shape (1, C, H, W), or None if predict() not yet called. )r r.s rget_class_logitsz(ImageFeatureExtractorV1.get_class_logitss |rc|sdS|drdS|}t j|jtj}||j |j |j }| || |td|jDS)a Determine whether the page is an image-only PDF page that requires OCR. Returns True only when ALL three conditions hold: 1. The page contains at least one raster image (pymupdf). 2. The page contains no embedded selectable text (pymupdf). 3. The segmentation model detects at least one non-picture region (text, table, header, etc.) inside the page image. Condition 1 + 2 confirm the PDF structure is image-only. Condition 3 confirms there is recoverable content inside the image. All three must hold for OCR to be both necessary and worthwhile. Calls predict() internally and marks the result as cached so the immediately following BoxRFDGNN.predict() skips re-running inference. picture CCL is deferred to get_picture_detections() if needed. Args: page: PyMuPDF page object Returns: bool Fr r3c38K|]}|dtkVdS)scoreN)_DETECTION_SCORE_THRESHOLD)rds r z8ImageFeatureExtractorV1.is_image_page..s@   gJ3 3      r)get_image_infoget_textstrip get_pixmapr> frombuffersamplesuint8reshapeheightwidthnrUr/raanyr$)r)pagepix bytes_datarIs r is_image_pagez%ImageFeatureExtractorV1.is_image_pages2""$$ 5 == & & ( ( 5__&&]3;bh??? '' CIsuEE X  ++---  1      rcZ|jtd||j|jz }|j|jz }g}|jD]X}|dtkrE|d}| |d|z|d|z|d|z|d|zgY|S) a Return picture-class detections from the last predict() call, rescaled from model resolution to page pixel coordinates. Runs picture CCL on the first call after each predict() (lazy). Subsequent calls within the same predict() cycle are free. Returns: list of [x1, y1, x2, y2] in page pixel space. Raises: RuntimeError if predict() has not been called. Nz8predict() must be called before get_picture_detections()rhbboxrrr) r RuntimeErrorr^r&r(r%r'r#riappend)r)resize_xresize_ybboxesdetbs rget_picture_detectionsz.ImageFeatureExtractorV1.get_picture_detectionss > !YZZ Z '')))<$.0<$.0+  C7|888K aD8OaD8OaD8OaD8O   r)N)__name__ __module__ __qualname____doc__r+r/r1rUr^rarcrer{rrrrrrJs45,5,5,5,v              . . . `     rr)rnumpyr> common_utilrrrrr\rrr`rir]_MORPHOLOGY_KERNEL_SIZErrrrrs,,\ $ II<III!uuuuuuuuuur