AjiJddlZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl Z ddl mZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZmZmZmZmZmZdd l m!Z!m"Z"dd l#m$Z%ddl#m&Z&m'Z'm(Z(e)e&e'ze(zZ*e*+ddZ,dZ-dZ.gdZ/gdZ0d6dZ1d6dZ2Gdde3Z4dZ5dZ6dZ7dZ8dZ9dZ:d Z;d!ed!e?fd$Z@d!e?fd%ZAd7d'ZBd8d(ZCd8d)ZDd9d+ZE d:d,ZFd-ZGd.ZH d;d3ZId>H K!!! OsBB<<CC)columnsedge_tolrow_tol column_tol) process_background line_scale copy_text shift_textline_tol joint_tolthreshold_blocksizethreshold_constant iterations resolutionlatticecffd}dkr|t|dS|t|dS)Nct|t|}|r5tdt |dddS)N,z cannot be used with flavor='')set intersectionkeys ValueErrorr:sorted) parser_kwargs input_kwargsisecflavors rcheck_intersectionz*validate_input..check_intersectionps|=!!..s<3D3D3F3F/G/GHH  88F4LL))QQQQQ   r,rU) stream_kwargslattice_kwargs)kwargsrbrcs ` rvalidate_inputrgos]3=&11111>622222r,c|dkr6|D] }|tvr||!n5|D] }|tvr||!|S)NrU)r\rdpopre)rfrbkeys r remove_extrark}s  ;;==  Cm#  3 ;;==  Cn$  3 Mr,ceZdZdZdZdS)TemporaryDirectoryc@tj|_|jSN)r1mkdtempr<)selfs r __enter__zTemporaryDirectory.__enter__s$&& yr,c8tj|jdSro)r=rmtreer<)rqexc_type exc_value tracebacks r__exit__zTemporaryDirectory.__exit__s di     r,N)__name__ __module__ __qualname__rrrxr,rrmrms2!!!!!r,rmc||z }|S)zTranslates x2 by x1. Parameters ---------- x1 : float x2 : float Returns ------- x2 : float r|)x1x2s r translaters"HB Ir,c||z}|S)zScales x by scaling factor s. Parameters ---------- x : float s : float Returns ------- x : float r|)xss rscalersFA Hr,c|\}}}}|\}}}t||}ttt| ||}t||}ttt| ||}t|t|t|t|f} | S)ayTranslates and scales pdf coordinate space to image coordinate space. Parameters ---------- k : tuple Tuple (x1, y1, x2, y2) representing table bounding box where (x1, y1) -> lt and (x2, y2) -> rb in PDFMiner coordinate space. factors : tuple Tuple (scaling_factor_x, scaling_factor_y, pdf_y) where the first two elements are scaling factors and pdf_y is height of pdf. Returns ------- knew : tuple Tuple (x1, y1, x2, y2) representing table bounding box where (x1, y1) -> lt and (x2, y2) -> rb in OpenCV coordinate space. )rabsrint) kfactorsr~y1ry2scaling_factor_xscaling_factor_ypdf_yknews r scale_pdfrs.NBB07-& r# $ $B s9eVR(())+; < zscale_image..s$777auQ())777r,c hg|].}ttt |/Sr|)rrr)rrimg_yrs rrzscale_image..s8OOOauSE61--..0@AAOOOr,rr")r\rrrzipappend)tables v_segments h_segmentsr tables_newrr~rrrj_xj_yjointsv_segments_newvh_segments_newhrrrs @@@r scale_imagers|218-&J [[]] . .BB 2' ( ( 3y%,,--/? @ @ 2' ( ( 3y%,,--/? @ @q ?S77773777OOOOO3OOOS#'- BB#$$N 00qt-..ad.(s1CCCaajjll.@.@.B.BCCCCr,c^g|]*}|(|+Sr|rrs rrz get_rotation..)s1AAAaAJJLL,>,>,@,@AAAAr,c3ZK|]&}|jddko|jddkV'dSr"rrNmatrixrs r zget_rotation..+s;KK a;AHQK!OKKKKKKr,c3ZK|]&}|jddko|jddkV'dSrrrs rrzget_rotation..,s;OOAAHQK!O? aOOOOOOr, anticlockwise clockwise)lensum)charshorizontal_text vertical_textrotationhlenvlenrrs r get_rotationrs*H CC?CCC D DD AA=AAA B BD d{QKKUKKKKK OOOOOOO &/-&?P??[ Or,c|d|df|d|dffd|D}fd|D}||fS)a6Returns all line segments present inside a bounding box. Parameters ---------- bbox : tuple Tuple (x1, y1, x2, y2) representing a bounding box where (x1, y1) -> lb and (x2, y2) -> rt in PDFMiner coordinate space. v_segments : list List of vertical line segments. h_segments : list List of vertical horizontal segments. Returns ------- v_s : list List of vertical line segments that lie inside table. h_s : list List of horizontal line segments that lie inside table. rr"rrcg|]V}|dddz k|dddzk,ddz |dcxkrddzkQnn|WS)r"rrrr|)rrlbrts rrz$segments_in_bbox..I    Q4"Q%!)  !"!r!uqy 0 68UQY!A$    KMQ%RS)         r,cg|]V}|dddz k|dddzk,ddz |dcxkrddzkQnn|WSrrr"r|)rrrrs rrz$segments_in_bbox..Nrr,r|)bboxrrv_sh_srrs @@rsegments_in_bboxr1s, q'47 B q'47 B        C         C 8Or,c|d|df|d|dffd|D}d|D}|D]y}|D]b}||kr t||rIt||t|z dkr%t ||r||czt |}|S)aReturns all text objects present inside a bounding box. Parameters ---------- bbox : tuple Tuple (x1, y1, x2, y2) representing a bounding box where (x1, y1) -> lb and (x2, y2) -> rt in the PDF coordinate space. text : List of PDFMiner text objects. Returns ------- t_bbox : list List of PDFMiner text objects that lie inside table, discarding the overlapping ones rr"rrcg|]h}ddz |j|jzdz cxkrddzk1nn4ddz |j|jzdz cxkrddzkcnn|iS)rr@r")x0r~y0r)rrrrs rrz text_in_bbox..is a519+02!uqy qEAI!$+, 131  r,ch|]}|Sr|r|rs r ztext_in_bbox..qs   !A   r,g?)copybbox_intersectbbox_intersection_area bbox_area bbox_longerdiscardlist) rtextt_bboxrestbabb unique_boxesrrs @@r text_in_bboxrVs" q'47 B q'47 BF  v   D))))++ ) )BRx b"%% )*2r22Yr]]BcI)"2r**) R((( )::L r,returnct|j|j}t|j|j}t|j|j}t|j|j}||ks||krdS||z ||z z}|S)a.Returns area of the intersection of the bounding boxes of two PDFMiner objects. Parameters ---------- ba : PDFMiner text object bb : PDFMiner text object Returns ------- intersection_area : float Area of the intersection of the bounding boxes of both objects )maxrminrr~r)rrx_lefty_topx_righty_bottomintersection_areas rrrs  F ru  E"%G25"%  H8e+s 6)eh.>? r,c@|j|jz |j|jz zS)zReturns area of the bounding box of a PDFMiner object. Parameters ---------- bb : PDFMiner text object Returns ------- area : float Area of the bounding box of the object r~rrr)rs rrrs EBEMbebem ,,r,c|j|jko/|j|jko|j|jko|j|jkS)aReturns True if the bounding boxes of two PDFMiner objects intersect. Parameters ---------- ba : PDFMiner text object bb : PDFMiner text object Returns ------- overlaps : bool True if the bounding boxes intersect rrrs rrrs9 5BE> Rberun R"% RBERUNRr,cB|j|jz |j|jz kS)a3Returns True if the bounding box of the first PDFMiner object is longer or equal to the second. Parameters ---------- ba : PDFMiner text object bb : PDFMiner text object Returns ------- longer : bool True if the bounding box of the first object is longer or equal )r~rrs rrrs EBEMruru} --r,rcg}|D]\}|s|||d}tj|||r||zdz }||d<G||]|S)zMerges lines which are within a tolerance by calculating a moving mean, based on their x or y axis projections. Parameters ---------- ar : list line_tol : int, optional (default: 2) Returns ------- ret : list )atolr)rnpisclose)arrOr*atemps rmerge_close_linesrs C     JJqMMMMr7Dz$111 qC'B 1 Jr,c |s|Stjddttj|dd|tj}|S)aStrips any characters in `strip` that are present in `text`. Parameters ---------- text : str Text to process and strip. strip : str, optional (default: '') Characters that should be stripped from `text`. Returns ------- stripped : str [r])flags)resubr:mapescapeUNICODE)rrstrippeds r text_striprs\  v.RWWSE** + +...D H Or,ch|dkr d|D}n|dkr d|D}d|D}tt|dkr2g}t|}t|t dD]\}}||krd|D} d| rS| d d | d |d| d |D} d| r(|d| d|} ndd |D} t| |S)aFlags super/subscripts in text by enclosing them with . May give false positives. Parameters ---------- textline : list List of PDFMiner LTChar objects. direction : string Direction of the PDFMiner LTTextLine object. strip_text : str, optional (default: '') Characters that should be stripped from a string before assigning it to a cell. Returns ------- fstring : string horizontalcg|]F}t|t|tj|jdfGSr.)decimals) isinstancer rrroundheightrs rrz"flag_font_size..sV   a(( ZZ\\28AHq999 :   r,verticalcg|]F}t|t|tj|jdfGSr)r r rrr widthrs rrz"flag_font_size..sV   a(( ZZ\\28AGa888 9   r,c@g|]\}}tj|dSr)rr )rrsizes rrz"flag_font_size..!s+888 d$ # # #888r,r"cg|] }|d Srr|rs rrz"flag_font_size..'...1!A$...r,rrzzcg|] }|d Srr|rs rrz"flag_font_size..-rr,c6g|]}|Sr|rrs rrz"flag_font_size..2s :::A1::<<:::r,) rrZrrrr:rinsertrr) textline direction strip_textdlflistmin_sizerjrfcharsfstrings rflag_font_sizer!s&L        j        98a888A 3q66{{Q<q66!!Z]]33 2 2JCh 2.....776??((**2MM!U+++MM&)))LL111.....776??((**2LL111''%..'':::::;; gz * **r,Fc ,d}g}|j |dkr|svfdtjD}fdtjD}|dfd|D} | s!|djdjfg} |jD]} j} | D]} t| tr| d| j | j zdz cxkr | dkr?nn<| j | j zdz | dkr || d| fnb| | dkr!|| ddz| ft| tr|| d| fn|d kr|stfd tjD} fd tjD}|dfd | D}|s!| djdj fg}|jD]} j}|D]} t| tr|d| j | j zdz cxkr |dkr?nn<| j | j zdz | dkr || d| fnb| |dkr!|| ddz | ft| tr|| d| fn(#t $rdd|fgcYSwxYwg}t%|t'ddD]\}}|r?||d|dt)d |D||fFd|D}||d|dt+d||f|S)a9Splits PDFMiner LTTextLine into substrings if it spans across multiple rows/columns. Parameters ---------- table : camelot.core.Table textline : object PDFMiner LTTextLine object. direction : string Direction of the PDFMiner LTTextLine object. flag_size : bool, optional (default: False) Whether or not to highlight a substring using if its size is different from rest of the string. (Useful for super and subscripts.) strip_text : str, optional (default: '') Characters that should be stripped from a string before assigning it to a cell. Returns ------- grouped_chars : list List of tuples of the form (idx, text) where idx is the index of row/column and text is the an lttextline substring. rrcbg|]+\}}|ddkd|dk)|,Srr|)rirrs rrz"split_textline..UUAqQ447?(,Aw!A$r,cvg|]5\}}|dddzdz cxkr |dk0nn|6S)r"rrrr|)rrrrs rrz"split_textline..Z|AqQ4DGd1g-278dr,ctg|]4}j|j|j|jf5Sr|)cellsrightr)rcr'tables rrz"split_textline..`sO./ Aq@Q@WEKN1%()r,rr"rr cbg|]+\}}|ddkd|dk)|,S)r"rrr|)rryrs rrz"split_textline..vr%r,cvg|]5\}}|dddzdz cxkr |dk0nn|6Srr|)rr$r,rs rrz"split_textline..{r(r,ctg|]4}j|j|j|jf5Sr|)r*bottomr)rr'r,r-s rrz"split_textline..sO./ Aq@Q@XEKN1%()r,cg|] }|d Srr|rs rrz"split_textline..s---!1---r,rcBg|]}|dSr4rrs rrz"split_textline..s$555!admmoo555r,r)ris_empty enumeratecolsrowsr*r_objsr rrrrr~rr IndexErrorrrrr!rr:)r-rr flag_sizeridxcut_text x_overlapr_idxx_cutsrCrowcut y_overlapc_idxy_cutscol grouped_charsrjrgcharsrr,r's` @@@rsplit_textlinerK6s4 CH =DD/  $A :X->->-@-@A :%ej11I %ej11E aA3<F A$Q<Q);)>?@~ : :jm! : :C!#v.. :Fsv!&; F F F F?B1v F F F F F!$#&A 5Q ? F%OOQA,<===!E #fRj0F (CFQJ0D E E E#C00: CFC(8999 : * $ :X->->-@-@ :%ej11I %ej11E aA3<F A$Q<R);)>?@~ : :jm! : :C!#v.. :Fsv!&; F F F F?B1v F F F F F!$#&A 5Q ? F%OOSVQ,<===!E #fRj0F (Q!Q0D E E E#C00: QC(8999 ///R**,,-..../Mh 1a(8(899 U    FF"--u---yZ    65u555F  QQBGGFOOZ!H!HI     sL"L44"MMc Fdgdz\}}tt|jD]}|j|jzdz |j|dkr|j|jzdz |j|dkrg} |jD]} | d|jkr| d|jkr| d|jkr|jn| d} | d|jkr|jn| d} | t| | z t| d| dz z | dtttd| dkrw| d} |j|jf}|jdd|jddf}tj| d|d ||}| t#| }ndgd z\}}}}|j|j|dkr(t|j|j|dz }|j|j|dkr(t|j|j|dz }|j|j|dkr(t|j|j|dz }|j|j|dkr(t|j|j|dz }t|j|jz d krd nt|j|jz }t|j|jz d krd nt|j|jz }||z}|||zz|||zzz|z }|rt%||||| |fS|r||t'|j||fg|fS||t+| |fg|fS)a}Gets indices of the table cell where given text object lies by comparing their y and x-coordinates. Parameters ---------- table : camelot.core.Table t : object PDFMiner LTTextLine object. direction : string Direction of the PDFMiner LTTextLine object. split_text : bool, optional (default: False) Whether or not to split a text line if it spans across multiple cells. flag_size : bool, optional (default: False) Whether or not to highlight a substring using if its size is different from rest of the string. (Useful for super and subscripts) strip_text : str, optional (default: '') Characters that should be stripped from a string before assigning it to a cell. Returns ------- indices : list List of tuples of the form (r_idx, c_idx, text) where r_idx and c_idx are row and column indices. error : float Assignment error, percentage of text area that lies outside a cell. +-------+ | | | [Text bounding box] | | +-------+ rrrrr"c|dkS)Nrr|)rs rz!get_table_index..s br,  z does not lie in column range r?)r=rr5)rangerr:rrr9r~rrrrfilterrrwarningswarnindexrrKr!r;r)r-rr split_textr=rrArFr'lt_col_overlapr,leftr+r text_range col_range y0_offset y1_offset x0_offset x1_offsetXYchareaerrors rget_table_indexresKN4!8LE5 3uz?? # # D14K3 Aq!1 1 qtad{c6IEJ M M 7   NZ . .Q414<.AaDADL.#$Q414<9144QqTD$%aDADL:ADDadE"))#dUl*;*;c!A$1+>N>N*NOOOO"))"----400.AABBCCqH zz||))$//dAD\ "Z]1-uz"~a/@A  SSjSS SSE"((^)<)<==E E34q.Iy)Ytej""5uz%03344 tej""5uz%03344 tej""5uz%03344 tej""5uz%03344 14!$;3&<Cqt ,<,.$s..."1...r,z&Sum of weights should be equal to 100.r")rr]rZeroDivisionError) error_weights SCORE_VALscoreriweighterror_percentages rcompute_accuracyrps I  .. ... / /9 < GEFF F 9 9BUSAZZ'F$&qE 9 9 1'7#788 9 9  LsA'A,, A;:A;c d}gg}}|D]$}|D]}|dkr|dz } %d|tt|t|dzz z}|S)zCalculates the percentage of empty strings in a two-dimensional list. Parameters ---------- d : list Returns ------- whitespace : float Percentage of empty cells. rrr"rg)rfloatr)r whitespacer_nempty_cellsc_nempty_cellsr$rs rcompute_whitespacerv/sJ%'NN     AwwyyB a   U3q66C!II+=%>%>>?J r,?rR皙?Tc t|d5}t|} t| } | jst d|t |||||||} t } t| | } t| | }tj | D]I}| || }|j d}|j d}||f}J||fcdddS#1swxYwYdS)aMReturns a PDFMiner LTPage object and page dimension of a single page pdf. To get the definitions of kwargs, see https://pdfminersix.rtfd.io/en/latest/reference/composable.html. Parameters ---------- filename : string Path to pdf file. line_overlap : float char_margin : float line_margin : float word_margin : float boxes_flow : float detect_vertical : bool all_texts : bool Returns ------- layout : object PDFMiner LTPage object. dim : tuple Dimension of pdf page in the form (width, height). rbz Text extraction is not allowed: ) line_overlap char_margin line_margin word_margin boxes_flowdetect_vertical all_texts)laparamsrrN)openrris_extractablerr r r r r create_pages process_page get_resultr)r?r{r|r}r~rrrr@parserdocumentrrsrcmgrdevice interpreterpagelayoutrr dims rget_page_layoutrGsvD h  1v&&& -=8== %###!+   %&&"7X>>>(&99 (22 " "D  $ $T * * *&&((FKNE[^F&/CCs{3sCC<<DDcharc(|dkrt}n)|dkrt}n|dkrt}n |dkrt}|g} |jD]<}t ||r||(|t||z }=n#t$rYnwxYw|S)aRecursively parses pdf layout to get a list of PDFMiner text objects. Parameters ---------- layout : object PDFMiner LTPage object. ltype : string Specify 'char', 'lh', 'lv' to get LTChar, LTTextLineHorizontal, and LTTextLineVertical objects respectively. t : list Returns ------- t : list List of PDFMiner text objects. rimagerrN)ltype) rrrrr;r rget_text_objectsAttributeError)rrrLTObjectrCs rrrs& & ' & # #&' / !&%  < 8 8C#x(( 8 %c7777  8      HsAB BB)rUr4)r)Fr)FFr)rwrRrwrxrwTT)rN)Kr8rr#r=r%r1rU itertoolsroperatorrnumpyrpdfminer.pdfparserrpdfminer.pdfdocumentrpdfminer.pdfpagerrpdfminer.pdfinterpr r pdfminer.converterr pdfminer.layoutr r rrrrurllib.requestrr urllib.parserrrrrrZrrr r+rFrdrergrkobjectrmrrrrrrrrrrrboolrrrrr!rKrerprvrrr|r,rrs ((((((,,,,,,$$$$$$888888111111111111000000,+++++++......@@@@@@@@@@c-+- ;<< B&6A@@     3 3 3 3    !!!!!!!!"   "B969696x>"""J'''Te4 -U - - - - SdSSSS".4...."841+1+1+1+hssssnHJbQbQbQbQJ<4;;;;|% % % % % % r,