o ji@s<ddlZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl Z ddl mZddlmZddlmZddlmZddlmZdd lmZdd lmZdd lmZmZmZmZmZmZdd l m!Z!m"Z"dd l#m$Z%ddl#m&Z&m'Z'm(Z(e)e&e'e(Z*e*+dddZ,ddZ-ddZ.gdZ/gdZ0dOddZ1dOddZ2Gddde3Z4dd Z5d!d"Z6d#d$Z7d%d&Z8d'd(Z9d)d*Z:d+d,Z;d-ed-e?fd2d3Z@d-e?fd4d5ZAdPd7d8ZBdQd9d:ZCdQd;d<ZDdRd>d?ZE dSd@dAZFdBdCZGdDdEZH F G F H F I IdTdJdKZIdUdMdNZJdS)VN)groupby) itemgetter) PDFParser) PDFDocument)PDFPage)PDFTextExtractionNotAllowed)PDFResourceManager)PDFPageInterpreter)PDFPageAggregator)LAParamsLTAnnoLTCharLTTextLineHorizontalLTTextLineVerticalLTImage)Requesturlopen)urlparse) uses_relative uses_netloc uses_paramscCs&zt|jtvWStyYdSw)zCheck to see if a URL has a valid protocol. Parameters ---------- url : str or unicode Returns ------- isurl : bool If url has a valid protocol return True otherwise False. F) parse_urlscheme _VALID_URLS Exception)urlrS/var/www/html/fyndo/pharma/fyndo/venv/lib/python3.10/site-packages/camelot/utils.pyis_url(s   rcCs4d}|r|ttjtjtj7}|d8}|s|S)Nr)randomchoicestringdigitsascii_lowercaseascii_uppercase)lengthretrrr random_string;sr)cCstdd}tjddd+}ddi}t|d|}t|}|}|d kr+td || Wdn1s.check_intersectionrQ) stream_kwargslattice_kwargs)kwargsr]r^rr\rvalidate_inputos rbcCsP|dkr|D] }|tvr||q|S|D] }|tvr%||q|S)NrQ)rVr_popr`)rar]keyrrr remove_extra}s    rec@seZdZddZddZdS)TemporaryDirectorycCst|_|jSN)r-mkdtempr8)selfrrr __enter__s zTemporaryDirectory.__enter__cCst|jdSrg)r9rmtreer8)riexc_type exc_value tracebackrrr__exit__szTemporaryDirectory.__exit__N)__name__ __module__ __qualname__rjrorrrrrfs rfcCs ||7}|S)zTranslates x2 by x1. Parameters ---------- x1 : float x2 : float Returns ------- x2 : float r)x1x2rrr translate rucCs ||9}|S)zScales x by scaling factor s. Parameters ---------- x : float s : float Returns ------- x : float r)xsrrrscalervryc Csv|\}}}}|\}}}t||}ttt| ||}t||}ttt| ||}t|t|t|t|f} | S)ayTranslates and scales pdf coordinate space to image coordinate space. Parameters ---------- k : tuple Tuple (x1, y1, x2, y2) representing table bounding box where (x1, y1) -> lt and (x2, y2) -> rb in PDFMiner coordinate space. factors : tuple Tuple (scaling_factor_x, scaling_factor_y, pdf_y) where the first two elements are scaling factors and pdf_y is height of pdf. Returns ------- knew : tuple Tuple (x1, y1, x2, y2) representing table bounding box where (x1, y1) -> lt and (x2, y2) -> rb in OpenCV coordinate space. )ryabsruint) kfactorsrsy1rty2scaling_factor_xscaling_factor_ypdf_yknewrrr scale_pdfs    rcs|\i}|D]P}|\}}}} t|}ttt |}t|}ttt | } t||\} } fdd| D} fdd| D} t| | } | ||||| f<q g} |D]5}t|dt|d}}ttt |dttt |d}} | |||| fq`g}|D]5}t|dt|d}}ttt |dttt |d}} ||||| fq|| |fS)avTranslates and scales image coordinate space to pdf coordinate space. Parameters ---------- tables : dict Dict with table boundaries as keys and list of intersections in that boundary as value. v_segments : list List of vertical line segments. h_segments : list List of horizontal line segments. factors : tuple Tuple (scaling_factor_x, scaling_factor_y, img_y) where the first two elements are scaling factors and img_y is height of image. Returns ------- tables_new : dict v_segments_new : dict h_segments_new : dict csg|]}t|qSr)ry.0j)rrr szscale_image..cs"g|] }ttt |qSr)ryrzrur)img_yrrrrs"rr )rVryrzruzipappend)tables v_segments h_segmentsr} tables_newr|rsr~rtrj_xj_yjointsv_segments_newvh_segments_newhr)rrrr scale_images8       rcCshd}tdd|D}tdd|D}||kr2tdd|D}tdd|D}||kr0dnd }|S) aCDetects if text in table is rotated or not using the current transformation matrix (CTM) and returns its orientation. Parameters ---------- horizontal_text : list List of PDFMiner LTTextLineHorizontal objects. vertical_text : list List of PDFMiner LTTextLineVertical objects. ltchar : list List of PDFMiner LTChar objects. Returns ------- rotation : string '' if text in table is upright, 'anticlockwise' if rotated 90 degree anticlockwise and 'clockwise' if rotated 90 degree clockwise. rcSg|] }|r|qSrget_textstriprtrrrr(z get_rotation..cSrrrrrrrr)rcss,|]}|jddko|jddkVqdSr rrNmatrixrrrr +*zget_rotation..css,|]}|jddko|jddkVqdSrrrrrrr,r anticlockwise clockwise)lensum)charshorizontal_text vertical_textrotationhlenvlenrrrrr get_rotationsrcsP|d|df|d|dffdd|D}fdd|D}||fS)a6Returns all line segments present inside a bounding box. Parameters ---------- bbox : tuple Tuple (x1, y1, x2, y2) representing a bounding box where (x1, y1) -> lb and (x2, y2) -> rt in PDFMiner coordinate space. v_segments : list List of vertical line segments. h_segments : list List of vertical horizontal segments. Returns ------- v_s : list List of vertical line segments that lie inside table. h_s : list List of horizontal line segments that lie inside table. rr rrcsdg|].}|dddkr|dddkrdd|dkr,ddkrnn|qS)r rrrr)rrlbrtrrrI Tz$segments_in_bbox..csdg|].}|dddkr|dddkrdd|dkr,ddkrnn|qSrrr r)rrrrrrNrr)bboxrrv_sh_srrrsegments_in_bbox1s  rcs|d|df|d|dffdd|D}dd|D}|D](}|D]!}||kr0q)t||rJt||t|d krJt||rJ||q)q#t|}|S) aReturns all text objects present inside a bounding box. Parameters ---------- bbox : tuple Tuple (x1, y1, x2, y2) representing a bounding box where (x1, y1) -> lb and (x2, y2) -> rt in the PDF coordinate space. text : List of PDFMiner text objects. Returns ------- t_bbox : list List of PDFMiner text objects that lie inside table, discarding the overlapping ones rr rrcsxg|]8}dd|j|jdkrddkr:nndd|j|jdkr6ddkrnn|qS)rr@r )x0rsy0r~rrrrris 44z text_in_bbox..cSsh|]}|qSrrrrrr qsztext_in_bbox..g?)copybbox_intersectbbox_intersection_area bbox_area bbox_longerdiscardlist)rtextt_bboxrestbabb unique_boxesrrr text_in_bboxVs$     rreturncCs`t|j|j}t|j|j}t|j|j}t|j|j}||ks$||kr&dS||||}|S)a.Returns area of the intersection of the bounding boxes of two PDFMiner objects. Parameters ---------- ba : PDFMiner text object bb : PDFMiner text object Returns ------- intersection_area : float Area of the intersection of the bounding boxes of both objects )maxrminr~rsr)rrx_lefty_topx_righty_bottomintersection_arearrrrsrcCs|j|j|j|jS)zReturns area of the bounding box of a PDFMiner object. Parameters ---------- bb : PDFMiner text object Returns ------- area : float Area of the bounding box of the object rsrr~r)rrrrrs rcCs0|j|jko|j|jko|j|jko|j|jkS)aReturns True if the bounding boxes of two PDFMiner objects intersect. Parameters ---------- ba : PDFMiner text object bb : PDFMiner text object Returns ------- overlaps : bool True if the bounding boxes intersect rrrrrrrs0rcCs|j|j|j|jkS)a3Returns True if the bounding box of the first PDFMiner object is longer or equal to the second. Parameters ---------- ba : PDFMiner text object bb : PDFMiner text object Returns ------- longer : bool True if the bounding box of the first object is longer or equal )rsrrrrrrsrrcCsZg}|D]&}|s||q|d}tj|||dr%||d}||d<q||q|S)zMerges lines which are within a tolerance by calculating a moving mean, based on their x or y axis projections. Parameters ---------- ar : list line_tol : int, optional (default: 2) Returns ------- ret : list )atolr)rnpisclose)arrKr(atemprrrmerge_close_liness    rcCs6|s|Stjddttj|dd|tjd}|S)aStrips any characters in `strip` that are present in `text`. Parameters ---------- text : str Text to process and strip. strip : str, optional (default: '') Characters that should be stripped from `text`. Returns ------- stripped : str [r])flags)resubr6mapescapeUNICODE)rrstrippedrrr text_strips   rc Cs|dkr dd|D}n |dkrdd|D}dd|D}tt|dkrzg}t|}t|tdD]@\}}||kr]dd|D} d | r\| d d | d |d | q3d d|D} d | rs|d | q3d |} n d dd|D} t | |S)aFlags super/subscripts in text by enclosing them with . May give false positives. Parameters ---------- textline : list List of PDFMiner LTChar objects. direction : string Direction of the PDFMiner LTTextLine object. strip_text : str, optional (default: '') Characters that should be stripped from a string before assigning it to a cell. Returns ------- fstring : string horizontalcS.g|]}t|ts|tj|jddfqSr*)decimals) isinstancer rrroundheightrrrrr z"flag_font_size..verticalcSrr)rr rrrwidthrrrrrrcSsg|] \}}tj|ddqSr)rr)rrsizerrrr!sr cSg|]}|dqSrrrrrrr'rrzzcSrrrrrrrr-rcSsg|]}|qSrrrrrrr2r) rrTrrrr6rinsertrr) textline direction strip_textdlflistmin_sizerdrfcharsfstringrrrflag_font_sizes6    rFc s`d}g}|jzO|dkr|sfddtjD}fddtjD}|dfdd|D} | sF|djdjfg} |jD]a} j} | D]W} t| t r| d| j | j d krm| dkrnn| j | j d | dkr|| d| fn#| | dkr|| dd| fqRt| tr|| d| fqRqIn|d krV|sVfd dtjD} fd dtjD}|dfd d| D}|s| djdj fg}|jD]g} j}|D]]} t| t rD|d| j | j d kr|dkr0nn| j | j d | dkr0|| d| fn%| |dkrC|| dd| fqt| trT|| d| fqqWntyidd|fgYSwg}t|tddD]9\}}|r||d|dtdd|D||dfqtdd|D}||d|dtd||fqt|S)a9Splits PDFMiner LTTextLine into substrings if it spans across multiple rows/columns. Parameters ---------- table : camelot.core.Table textline : object PDFMiner LTTextLine object. direction : string Direction of the PDFMiner LTTextLine object. flag_size : bool, optional (default: False) Whether or not to highlight a substring using if its size is different from rest of the string. (Useful for super and subscripts.) strip_text : str, optional (default: '') Characters that should be stripped from a string before assigning it to a cell. Returns ------- grouped_chars : list List of tuples of the form (idx, text) where idx is the index of row/column and text is the an lttextline substring. rrc4g|]\}}|ddkrd|dkr|qSrr)rirwrrrrU  z"split_textline..csDg|]\}}|ddddkr|dkrnn|qS)r rrrr)rrrrrrrZ 0cs0g|]}j|jr|j|jfqSr)cellsrightrt)rc)rtablerrr` rr rrcr)r rrr)rryrrrrvrcsDg|]\}}|ddddkr|dkrnn|qSrr)rrr rrrr{rcs0g|]}j|jr|j|jfqSr)r bottomr~)rr)r r rrrr cSrrrrrrrrrrcSsg|]}|dqSrrrrrrrsr)ris_empty enumeratecolsrowsr rt_objsrr rr~rrsrr IndexErrorrrrrrr6)r rr flag_sizeridxcut_text x_overlapr_idxx_cutsr?rowcut y_overlapc_idxy_cutscol grouped_charsrdrgcharsr)rr rr rsplit_textline6s      ,        0  r&c Cs dgd\}}tt|jD]}|j|jd|j|dkr|j|jd|j|dkrg} |jD]E} | d|jkrs| d|jkrs| d|jkrM|jn| d} | d|jkr[|jn| d} | t | | t | d| dq3| dq3tt t dd| dkr| d} |j|jf}|jdd|jddf}t| d |d ||}| t| }nqdgd \}}}}|j|j|dkrt |j|j|d}|j|j|dkrt |j|j|d}|j|j|dkrt |j|j|d}|j|j|dkrt |j|j|d}t |j|jd kr*d nt |j|j}t |j|jd kr?d nt |j|j}||}|||||||}|rgt|||||d|fS|rx||t|j||dfg|fS||t| |fg|fS)a}Gets indices of the table cell where given text object lies by comparing their y and x-coordinates. Parameters ---------- table : camelot.core.Table t : object PDFMiner LTTextLine object. direction : string Direction of the PDFMiner LTTextLine object. split_text : bool, optional (default: False) Whether or not to split a text line if it spans across multiple cells. flag_size : bool, optional (default: False) Whether or not to highlight a substring using if its size is different from rest of the string. (Useful for super and subscripts) strip_text : str, optional (default: '') Characters that should be stripped from a string before assigning it to a cell. Returns ------- indices : list List of tuples of the form (r_idx, c_idx, text) where r_idx and c_idx are row and column indices. error : float Assignment error, percentage of text area that lies outside a cell. +-------+ | | | [Text bounding box] | | +-------+ rrrrr cSs|dkS)Nrr)rwrrrsz!get_table_index..  z does not lie in column range r?)rrr)rangerrrr~rrsrrrzrfilterrrwarningswarnindexrr&rrr)r rr split_textrrrr!rlt_col_overlapr leftr r text_range col_range y0_offset y1_offset x0_offset x1_offsetXYchareaerrorrrrget_table_indexsj'0 (  **  r>cCsd}z2d}tdd|D|krtd|D]}|dt|d}|dD] }||d|7}q&qW|Sty@d}Y|Sw)awCalculates a score based on weights assigned to various parameters and their error percentages. Parameters ---------- error_weights : list Two-dimensional list of the form [[p1, e1], [p2, e2], ...] where pn is the weight assigned to list of errors en. Sum of pn should be equal to 100. Returns ------- score : float drcSrrr)rewrrrr$rz$compute_accuracy..z&Sum of weights should be equal to 100.r )rrWrZeroDivisionError) error_weights SCORE_VALscorer@weighterror_percentagerrrcompute_accuracys   rGcCsZd}gg}}|D]}|D] }|dkr|d7}q q d|tt|t|d}|S)zCalculates the percentage of empty strings in a two-dimensional list. Parameters ---------- d : list Returns ------- whitespace : float Percentage of empty cells. rrr r?)rfloatr)r whitespacer_nempty_cellsc_nempty_cellsrrrrrcompute_whitespace/s   rL?r+皙?Tc Cst|dV}t|} t| } | jstd|t|||||||d} t} t| | d} t| | }t | D]}| || }|j d}|j d}||f}q6||fWdS1s^wYdS)aMReturns a PDFMiner LTPage object and page dimension of a single page pdf. To get the definitions of kwargs, see https://pdfminersix.rtfd.io/en/latest/reference/composable.html. Parameters ---------- filename : string Path to pdf file. line_overlap : float char_margin : float line_margin : float word_margin : float boxes_flow : float detect_vertical : bool all_texts : bool Returns ------- layout : object PDFMiner LTPage object. dim : tuple Dimension of pdf page in the form (width, height). rbz Text extraction is not allowed: ) line_overlap char_margin line_margin word_margin boxes_flowdetect_vertical all_texts)laparamsrrN)openrris_extractablerr rr r r create_pages process_page get_resultr)r;rPrQrRrSrTrUrVr<parserdocumentrWrsrcmgrdevice interpreterpagelayoutrrdimrrrget_page_layoutGs6 "      $recharcCs|dkrt}n|dkrt}n |dkrt}n|dkrt}|dur!g}z|jD]}t||r2||q%|t||d7}q%W|StyGY|Sw)aRecursively parses pdf layout to get a list of PDFMiner text objects. Parameters ---------- layout : object PDFMiner LTPage object. ltype : string Specify 'char', 'lh', 'lv' to get LTChar, LTTextLineHorizontal, and LTTextLineVertical objects respectively. t : list Returns ------- t : list List of PDFMiner text objects. rfimagerrN)ltype) r rrrrrrget_text_objectsAttributeError)rcrhrLTObjectr?rrrris*    ri)rQr)r)Fr)FFr)rMr+rMrNrMTT)rfN)Kr4rr!r9r#r-r. itertoolsroperatorrnumpyrpdfminer.pdfparserrpdfminer.pdfdocumentrpdfminer.pdfpagerrpdfminer.pdfinterprr pdfminer.converterr pdfminer.layoutr r r rrrurllib.requestrr urllib.parserrrrrrTrrrr)rBr_r`rbreobjectrfruryrrrrrrHrrboolrrrrrr&r>rGrLrerirrrrsv                 !<%*    4w e >