U /j/ @sUddlmZddlZddlZddlZddlmZddlmZddl m Z ddl m Z ddl mZddlmZdd lmZd d lmZmZmZmZmZmZmZmZmZmZmZmZm Z m!Z!m"Z"m#Z#m$Z$e ed d ddddZ%e ed d ddddZ&e ed d d dddZ'e(dde)DZ*de+d<dde*DZ,de+d<e ed d ddd d!Z-e ed d ddd"d#Z.e ed d ddd$d%Z/e ed d ddd&d'Z0e ed d ddd(d)Z1e ed d ddd*d+Z2e ed d ddd,d-Z3e ed d ddd.d/Z4e ed d ddd0d1Z5e ed d ddd2d3Z6e ed d ddd4d5Z7e ed d ddd6d7Z8e ed d ddd8d9Z9e ed d ddd:d;Z:e ed d dddd?d@Z=e ed d dddAdBZ>didDdddEdFdGZ?e dHd d ddIdJdKZ@dDdLdMdNdOZAd ddPdQdRZBdjd dd dTdUdVZCd d dWdXdYdZZDd d ddXd[d\ZEd]ejFd^fd dd d_d`dadbZGdkdDd dcddddddddedf dgdhZHdS)l) annotationsN) bisect_right)IncrementalDecoder)aliases) lru_cache)findall) Generator)MultibyteIncrementalDecoder)ENCODING_MARKSIANA_SUPPORTED_SIMILARRE_POSSIBLE_ENCODING_INDICATIONUNICODE_RANGES_COMBINEDUNICODE_SECONDARY_RANGE_KEYWORDUTF8_MAXIMAL_ALLOCATIONCOMMON_CJK_CHARACTERS_LATIN_CJK_HANGUL _KATAKANA _HIRAGANA_THAI_ARABIC_ARABIC_ISOLATED_FORM_ACCENT_KEYWORDS _ACCENTUATED)maxsizestrint) characterreturncCszt|}Wntk r$YdSXd}d|kr:|tO}d|krJ|tO}d|krZ|tO}d|krj|tO}d|krz|tO}d|kr|tO}d|kr|t O}d |kr|t O}t D]}||kr|t O}qq|S) zRCompute all name-based classification flags with a single unicodedata.name() call.rZLATINCJKZHANGULZKATAKANAZHIRAGANAZTHAIZARABICz ISOLATED FORM) unicodedataname ValueErrorrrrrrrrrrr)rdescflagskwr([sr9zlist[tuple[int, int, str]]_UNICODE_RANGES_SORTEDcCsg|] }|dqS)rr()r8er(r(r) _sr<z list[int]_UNICODE_RANGE_STARTSz str | NonecCs<t|}tt|d}|dkr8t|\}}}||kr8|SdS)zK Retrieve the Unicode range official name from a single character. r rN)ordrr=r:)rZ character_ordidxr6r7r#r(r(r) unicode_rangebsr@cCstt|t@Sr,)r+r*rr-r(r(r)is_latinssrAcCs2t|}d|krdSt|}|dkr*dSd|kS)NPTFZ Punctuationr"categoryr@rcharacter_categorycharacter_ranger(r(r)is_punctuationxs rHcCsBt|}d|ksd|krdSt|}|dkr2dSd|ko@|dkS)NSNTFZFormsZLorCrEr(r(r) is_symbols rKcCs$t|}|dkrdSd|kp"d|kS)NFZ EmoticonsZ Pictographs)r@)rrGr(r(r) is_emoticonsrLcCs.|s|dkrdSt|}d|kp,|dkS)N>+｜<>TZ>PoPdPc)isspacer"rD)rrFr(r(r) is_separators rVcCs||kSr,)islowerisupperr-r(r(r)is_case_variablesrYcCstt|t@Sr,)r+r*rr-r(r(r)is_cjksrZcCstt|t@Sr,)r+r*rr-r(r(r) is_hiraganasr[cCstt|t@Sr,)r+r*rr-r(r(r) is_katakanasr\cCstt|t@Sr,)r+r*rr-r(r(r) is_hangulsr]cCstt|t@Sr,)r+r*rr-r(r(r)is_thaisr^cCstt|t@Sr,)r+r*rr-r(r(r) is_arabicsr_cCstt|t@Sr,)r+r*rr-r(r(r)is_arabic_isolated_formsr`cCs|tkSr,)rr-r(r(r)is_cjk_uncommonsra) range_namer cstfddtDS)Nc3s|]}|kVqdSr,r()r8keywordrbr(r)r9sz-is_unicode_range_secondary..)anyrrdr(rdr)is_unicode_range_secondarysrfcCs(|dko&|dko&|dko&|dkS)NFu)rU isprintabler-r(r(r)is_unprintables  ri zbytes | bytearray)sequence search_zoner cCst|ttfstt|}tt|dt||jddd}t|dkrLdS|D]N}| dd}t D]0\}}||kr|S||krl|SqlqPdS)zW Extract using ASCII-only decoder any specified encoding in the first n-bytes. Nasciiignoreerrorsr-_) isinstancebytes bytearray TypeErrorlenrr mindecodelowerreplaceritems)rkrlZseq_lenresultsZspecified_encodingencoding_alias encoding_ianar(r(r)any_specified_encodings"  r)r#r cCs |dkpttd|jtS)zQ Verify is a specific encoding is a multi byte one based on it IANA name > utf_8_sig utf_16_leutf_32 utf_16_be utf_32_leutf_8utf_16utf_7 utf_32_be encodings.) issubclass importlib import_modulerr )r#r(r(r)is_multi_byte_encoding s  rztuple[str | None, bytes])rkr cCsJtD]@}t|}t|tr |g}|D]}||r$||fSq$qdS)z9 Identify and extract SIG/BOM in given sequence. )N)r rsrt startswith)rk iana_encodingZmarksmarkr(r(r)identify_sig_or_boms  r)rr cCs|dkS)N>rrr()rr(r(r)should_strip_sig_or_bom0srT)cp_namestrictr cCsN|dd}tD]\}}|||fkr|Sq|rJtd|d|S)zIReturns the Python normalized encoding name (Not the IANA official name).rqrrzUnable to retrieve IANA for '')rzr{rr|r$)rrr~rr(r(r) iana_name4s  rfloat) iana_name_a iana_name_br c Cst|st|rdStd|j}td|j}|dd}|dd}d}tdD]*}t|g}||||krX|d7}qX|dS)Ngrrnrorr )rrrrrangertry) rrZ decoder_aZ decoder_bZid_aZid_bZcharacter_match_countiZ to_be_decodedr(r(r) cp_similarityEs     rcCs|tko|t|kS)z Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using the function cp_similarity. )r )rrr(r(r) is_cp_similarYs rZcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)sNone)r#level format_stringr cCs:t|}||t}|t|||dSr,)logging getLoggersetLevel StreamHandler setFormatter Formatter addHandler)r#rrloggerhandlerr(r(r)set_logging_handlerds   rrrtzGenerator[str, None, None]) sequencesroffsets chunk_sizebom_or_sig_availablestrip_sig_or_bom sig_payloadis_multi_byte_decoderdecoded_payloadr c cs*|r6|dkr6|D]"} || | |} | s,q4| Vqn|D]} | |} | t|dkrXq:|| | |} |r||dkr||| } | j||rdndd} |r| dkrt|d} |r| d| |krt| | dd D]H}||| } |r|dkr|| } | j|dd} | d| |krqq| Vq:dS) NFrnrrorr0)rwryrxr)rrrrrrrrrrchunkZ chunk_endZ cut_sequenceZchunk_partial_size_chkjr(r(r)cut_sequence_chunksqs>       r)rj)T)N)I __future__rrrr"bisectrcodecsrZencodings.aliasesr functoolsrrertypingrZ_multibytecodecr Zconstantr r r rrrrrrrrrrrrrrr*r.r5sortedr|r:__annotations__r=r@rArHrKrLrVrYrZr[r\r]r^r_r`rarwrfrirrrrrrrINFOrrr(r(r(r)s        L"     "