U ÀÒ.eöã@s0ddlZddlZddlmZGdd„deƒZdS)éNé)Ú ProbingStatec@sneZdZdZddd„Zdd„Zedd„ƒZd d „Zed d „ƒZ d d„Z e dd„ƒZ e dd„ƒZ e dd„ƒZdS)Ú CharSetProbergffffffî?NcCsd|_||_t t¡|_dS©N)Ú_stateÚ lang_filterÚloggingZ getLoggerÚ__name__Zlogger)Úselfr©r úE/usr/lib/python3.8/site-packages/pip/_vendor/chardet/charsetprober.pyÚ__init__'szCharSetProber.__init__cCs tj|_dSr)rZ DETECTINGr©r r r r Úreset,szCharSetProber.resetcCsdSrr rr r r Ú charset_name/szCharSetProber.charset_namecCsdSrr )r Úbufr r r Úfeed3szCharSetProber.feedcCs|jSr)rrr r r Ústate6szCharSetProber.statecCsdS)Ngr rr r r Úget_confidence:szCharSetProber.get_confidencecCst dd|¡}|S)Ns([-])+ó )ÚreÚsub)rr r r Úfilter_high_byte_only=sz#CharSetProber.filter_high_byte_onlycCs\tƒ}t d|¡}|D]@}| |dd…¡|dd…}| ¡sL|dkrLd}| |¡q|S)u9 We define three types of bytes: alphabet: english alphabets [a-zA-Z] international: international characters [€-ÿ] marker: everything else [^a-zA-Z€-ÿ] The input buffer can be thought to contain a series of words delimited by markers. This function works to filter all words that contain at least one international character. All contiguous sequences of markers are replaced by a single space ascii character. This filter applies to all scripts which do not use English characters. s%[a-zA-Z]*[€-ÿ]+[a-zA-Z]*[^a-zA-Z€-ÿ]?Néÿÿÿÿó€r)Ú bytearrayrÚfindallÚextendÚisalpha)rÚfilteredZwordsZwordZ last_charr r r Úfilter_international_wordsBsÿ  z(CharSetProber.filter_international_wordscCs¤tƒ}d}d}tt|ƒƒD]n}|||d…}|dkr characters. Also retains English alphabet and high byte characters immediately before occurrences of >. This filter can be applied to all scripts which contain both English characters and extended ASCII characters, but is currently only used by ``Latin1Prober``. Frró>ós