관리-도구
편집 파일: lexer.cpython-39.pyc
a P}<b t � @ s U d Z ddlZddlZddlmZ ddlmZ ddlm Z ddl m Z ddl mZ dd lmZ ejrxddlZdd lmZ ed �Zejejd f ed < e�d�Ze�d�Z e�dej �Z e�dej ej B �Z!e�dej ej B �Z"e d�Z#e d�Z$e d�Z%e d�Z&e d�Z'e d�Z(e d�Z)e d�Z*e d�Z+e d �Z,e d �Z-e d �Z.e d �Z/e d �Z0e d!�Z1e d"�Z2e d#�Z3e d$�Z4e d%�Z5e d&�Z6e d'�Z7e d(�Z8e d)�Z9e d*�Z:e d+�Z;e d,�Z<e d-�Z=e d.�Z>e d/�Z?e d0�Z@e d1�ZAe d2�ZBe d3�ZCe d4�ZDe d5�ZEe d6�ZFe d7�ZGe d8�ZHe d9�ZIe d:�ZJe d;�ZKe d<�ZLe d=�ZMe d>�ZNe d?�ZOe d@�ZPe dA�ZQe dB�ZRe dC�ZSe#e;e'e*e3e2e6e<e.e8e/e9e-e7e)e4e+e,e0e1e$e(e%e5e&e:dD�ZTdEdF� eT�U� D �ZVeWeT�eWeV�k�s�J dG��e�dHdI�XdJdK� eYeTdLdM� dN�D ��� dO��ZZe[eIeKeJe=eNeOePg�Z\e[e=eQeKePg�Z]e^e^dP�dQdR�Z_dSe^dT�dUdV�Z`e^e^dW�dXdY�Zae^ebdZ�d[d\�Zcd]ejdeje^e^f d^�d_d`�ZeG dadb� db�ZfG dcdS� dSejg�ZhG ddde� de�ZiG dfdg� dg�Zjd]d d^�dhdi�ZkG djdk� dkel�ZmG dldm� dmejg�ZnG dnd � d �ZodS )oz�Implements a Jinja / Python combination lexer. The ``Lexer`` class is used to do some preprocessing. It filters out invalid operators like the bitshift operators we don't allow in templates. It separates template code and python code in expressions. � N)� literal_eval)�deque)�intern� )�pattern)�TemplateSyntaxError)�LRUCache)� Environment�2 �Lexer� _lexer_cachez\s+z (\r\n|\r|\n)z7('([^'\\]*(?:\\.[^'\\]*)*)'|"([^"\\]*(?:\\.[^"\\]*)*)")z� ( 0b(_?[0-1])+ # binary | 0o(_?[0-7])+ # octal | 0x(_?[\da-f])+ # hex | [1-9](_?\d)* # decimal | 0(_?0)* # decimal zero ) z� (?<!\.) # doesn't start with a . (\d+_)*\d+ # digits, possibly _ separated ( (\.(\d+_)*\d+)? # optional fractional part e[+\-]?(\d+_)*\d+ # exponent part | \.(\d+_)*\d+ # required fractional part ) �addZassign�colonZcommaZdiv�dot�eq�floordiv�gtZgteqZlbraceZlbracketZlparen�ltZlteq�mod�mul�ne�pipe�pow�rbraceZrbracketZrparenZ semicolon�sub�tildeZ whitespace�floatZinteger�name�string�operatorZ block_beginZ block_endZvariable_beginZ variable_endZ raw_beginZraw_endZ comment_beginZ comment_end�commentZlinestatement_beginZlinestatement_endZlinecomment_beginZlinecomment_endZ linecomment�data�initial�eof)�+�-�/z//�*�%z**�~�[�]�(�)�{�}z==z!=�>z>=�<z<=�=�.�:�|�,�;c C s i | ]\}}||�qS � r8 )�.0�k�vr8 r8 �@/usr/lib/python3.9/site-packages/ansible/_vendor/jinja2/lexer.py� <dictcomp>� � r= zoperators droppedr, r5 c c s | ]}t �|�V qd S �N)�re�escape�r9 �xr8 r8 r< � <genexpr>� r> rD c C s t | � S r? )�len�rC r8 r8 r< �<lambda>� r> rG )�keyr- )� token_type�returnc C sL | t v rt | S tdtdtdtdtdtdtdtdt dt d t d t d i � | | �S ) Nzbegin of commentzend of commentr zbegin of statement blockzend of statement blockzbegin of print statementzend of print statementzbegin of line statementzend of line statementztemplate data / textzend of template)�reverse_operators�TOKEN_COMMENT_BEGIN�TOKEN_COMMENT_END� TOKEN_COMMENT�TOKEN_LINECOMMENT�TOKEN_BLOCK_BEGIN�TOKEN_BLOCK_END�TOKEN_VARIABLE_BEGIN�TOKEN_VARIABLE_END�TOKEN_LINESTATEMENT_BEGIN�TOKEN_LINESTATEMENT_END� TOKEN_DATA� TOKEN_EOF�get)rI r8 r8 r< �_describe_token_type� s" � �rY �Token��tokenrJ c C s | j tkr| jS t| j �S )z#Returns a description of the token.)�type� TOKEN_NAME�valuerY )r\ r8 r8 r< �describe_token� s r` ��exprrJ c C s2 d| v r&| � dd�\}}|tkr*|S n| }t|�S )z0Like `describe_token` but for token expressions.r4 r )�splitr^ rY )rb r] r_ r8 r8 r< �describe_token_expr� s rd �r_ rJ c C s t t�| ��S )zsCount the number of newline characters in the string. This is useful for extensions that filter a stream. )rE � newline_re�findall)r_ r8 r8 r< �count_newlines� s rh r �� environmentrJ c C s� t j}t| j�t|| j�ft| j�t|| j�ft| j�t|| j�fg}| j durp|� t| j �t d|| j � f� | j dur�|� t| j �t d|| j � f� dd� t|dd�D �S )zACompiles all the rules from the environment into a list of rules.Nz ^[ \t\v]*z(?:^|(?<=\S))[^\S\r\n]*c S s g | ]}|d d� �qS )r Nr8 rB r8 r8 r< � <listcomp>� r> z!compile_rules.<locals>.<listcomp>T)�reverse)r@ rA rE �comment_start_stringrL �block_start_stringrP �variable_start_stringrR �line_statement_prefix�appendrT �line_comment_prefix�TOKEN_LINECOMMENT_BEGIN�sorted)rj �e�rulesr8 r8 r< � compile_rules� s: ���� �� ��rw c @ s> e Zd ZdZefeeje dd�dd�Ze edd�dd �Z dS ) �FailurezjClass that raises a `TemplateSyntaxError` if called. Used by the `Lexer` to specify known errors. N)�message�clsrJ c C s || _ || _d S r? )ry � error_class)�selfry rz r8 r8 r< �__init__ s zFailure.__init__z te.NoReturn)�lineno�filenamerJ c C s | � | j||��d S r? )r{ ry )r| r~ r r8 r8 r< �__call__ s zFailure.__call__) �__name__� __module__� __qualname__�__doc__r �str�t�Typer} �intr� r8 r8 r8 r< rx � s � � rx c @ sT e Zd ZU eed< eed< eed< ed�dd�Zeed�dd �Zeed �d d �Z d S )rZ r~ r] r_ �rJ c C s t | �S r? )r` �r| r8 r8 r< �__str__ s z Token.__str__ra c C s2 | j |krdS d|v r.|�dd�| j | jgkS dS )z�Test a token against a token expression. This can either be a token type or ``'token_type:token_value'``. This can only test against string values and types. Tr4 r F)r] rc r_ �r| rb r8 r8 r< �test s z Token.test)�iterablerJ c s t � fdd�|D ��S )z(Test against multiple token expressions.c 3 s | ]}� � |�V qd S r? )r� )r9 rb r� r8 r< rD $ r> z!Token.test_any.<locals>.<genexpr>)�any)r| r� r8 r� r< �test_any" s zToken.test_anyN) r� r� r� r� �__annotations__r� r� �boolr� r� r8 r8 r8 r< rZ s c @ s< e Zd ZdZddd�dd�Zd d�dd �Zed�d d �ZdS ) �TokenStreamIteratorz`The iterator for tokenstreams. Iterate over the stream until the eof token is reached. � TokenStreamN)�streamrJ c C s || _ d S r? )r� )r| r� r8 r8 r< r} , s z TokenStreamIterator.__init__r� c C s | S r? r8 r� r8 r8 r< �__iter__/ s z TokenStreamIterator.__iter__c C s. | j j}|jtu r | j �� t�t| j � |S r? )r� �currentr] rW �close� StopIteration�next�r| r\ r8 r8 r< �__next__2 s z TokenStreamIterator.__next__)r� r� r� r� r} r� rZ r� r8 r8 r8 r< r� ' s r� c @ s� e Zd ZdZeje eje eje d�dd�Z e d�dd�Z e d�dd �Z ee d�d d ��Zed d �dd�Zed�dd�Zd!ed d�dd�Zeeje d�dd�Zee d�dd�Zed�dd �Zd d�d d �Zeed�d d �Zd S )"r� z�A token stream is an iterable that yields :class:`Token`\s. The parser however does not iterate over it but calls :meth:`next` to go one token ahead. The current active token is stored as :attr:`current`. )� generatorr r c C s> t |�| _t� | _|| _|| _d| _tdtd�| _ t | � d S )NFr � ) �iter�_iterr �_pushedr r �closedrZ � TOKEN_INITIALr� r� )r| r� r r r8 r8 r< r} C s zTokenStream.__init__r� c C s t | �S r? )r� r� r8 r8 r< r� Q s zTokenStream.__iter__c C s t | j�p| jjtuS r? )r� r� r� r] rW r� r8 r8 r< �__bool__T s zTokenStream.__bool__c C s | S )z Are we at the end of the stream?r8 r� r8 r8 r< �eosW s zTokenStream.eosNr[ c C s | j �|� dS )z Push a token back to the stream.N)r� rq r� r8 r8 r< �push\ s zTokenStream.pushc C s" t | �}| j}| �|� || _|S )zLook at the next token.)r� r� r� )r| Z old_token�resultr8 r8 r< �look` s zTokenStream.lookr )�nrJ c C s t |�D ] }t| � qdS )zGot n tokens ahead.N)�ranger� )r| r� �_r8 r8 r< �skiph s zTokenStream.skipra c C s | j �|�rt| �S dS )zqPerform the token test and return the token if it matched. Otherwise the return value is `None`. N)r� r� r� r� r8 r8 r< �next_ifm s zTokenStream.next_ifc C s | � |�duS )z8Like :meth:`next_if` but only returns `True` or `False`.N)r� r� r8 r8 r< �skip_ifv s zTokenStream.skip_ifc C sV | j }| jr| j�� | _ n8| j jturRzt| j�| _ W n tyP | �� Y n0 |S )z|Go one token ahead and return the old one. Use the built-in :func:`next` instead of calling this directly. ) r� r� �popleftr] rW r� r� r� r� )r| �rvr8 r8 r< r� z s zTokenStream.__next__c C s& t | jjtd�| _td�| _d| _dS )zClose the stream.r� r8 TN)rZ r� r~ rW r� r� r� r� r8 r8 r< r� � s zTokenStream.closec C sn | j �|�sft|�}| j jtu r>td|�d�| j j| j| j��td|�dt | j ���| j j| j| j��t | �S )z}Expect a given token type and return it. This accepts the same argument as :meth:`jinja2.lexer.Token.test`. z%unexpected end of template, expected r3 zexpected token z, got ) r� r� rd r] rW r r~ r r r` r� r� r8 r8 r< �expect� s ��zTokenStream.expect)r )r� r� r� r� r� �IterablerZ �Optionalr� r} r� r� r� r� �propertyr� r� r� r� r� r� r� r� r� r� r8 r8 r8 r< r� = s" � r� c C sZ | j | j| j| j| j| j| j| j| j| j | j | j f }t � |�}|du rVt| � t |< }|S )z(Return a lexer which is probably cached.N)rn �block_end_stringro �variable_end_stringrm �comment_end_stringrp rr � trim_blocks� lstrip_blocks�newline_sequence�keep_trailing_newliner rX r )rj rH Zlexerr8 r8 r< � get_lexer� s" � r� c s$ e Zd ZdZdZ� fdd�Z� ZS )�OptionalLStripzWA special tuple for marking a point in the state that can have lstrip applied. r8 c s t � �| |�S r? )�super�__new__)rz �members�kwargs�� __class__r8 r< r� � s zOptionalLStrip.__new__)r� r� r� r� � __slots__r� � __classcell__r8 r8 r� r< r� � s r� c @ sN e Zd ZU eje ed< ejeejedf eje f ed< ej e ed< dS )�_Ruler .�tokensZcommandN) r� r� r� r� �Patternr� r� �Union�Tuplerx r� r8 r8 r8 r< r� � s $r� c @ s� e Zd ZdZddd�dd�Zeed�dd �Zdeeje eje eje e d �d d �Z dej ej e eef eje eje eje d �dd�Zdeeje eje eje ejej e eef d �dd�ZdS )r a Class that implements a lexer for a given environment. Automatically created by the environment class, usually you don't have to do that. Note that the lexer is not automatically bound to an environment. Multiple environments can share the same lexer. r Nri c C s$ t j}ttjt d�dd�}tttd �ttt d �tt t d �tt t d �tttd �tttd �g}t|�}||j�}||j�}||j�}||j�} |jr�dnd} |j| _|j| _|j| _d|� d|� d|� d �} d �| gd d � |D � �} d t|d| � d��t t d�d�t|d�t d �gt t|d|� d|� d|� | � d ��t t fd�t|d�t!d�fd �gt"t|d|� d|� d|� | � d��t#d�g| t$t|d| � d| � ��t%d�g| t&t|d|� d|� d|� d|� | � d � �t t t'�d�t|d�t!d�fd �gt(t|d �t)d�g| t*t|d �t+t,fd�gi| _-d S ) N)rC rJ c S s t �| t jt jB �S r? )r@ �compile�M�SrF r8 r8 r< �c� s zLexer.__init__.<locals>.cz\n?r� z(?P<raw_begin>z(\-|\+|)\s*raw\s*(?:\-z\s*|z))r5 c S s" g | ]\}}d |� d|� d��qS )z(?P<r0 z (\-|\+|))r8 )r9 r� �rr8 r8 r< rk r> z"Lexer.__init__.<locals>.<listcomp>�rootz(.*?)(?:r- �#bygroupz.+z (.*?)((?:\+z|\-�#popz(.)zMissing end of comment tagz(?:\+z\-z (.*?)((?:z(\-|\+|))\s*endraw\s*(?:\+z Missing end of raw directivez \s*(\n|$)z(.*?)()(?=\n|$)).r@ rA r� r� r� r� � whitespace_re�TOKEN_WHITESPACE�float_re� TOKEN_FLOAT� integer_re� TOKEN_INTEGER�name_rer^ � string_re� TOKEN_STRING� operator_re�TOKEN_OPERATORrw rn r� r� r� r� r� r� r� �joinr� rV rL rN rM rx rP rQ rR rS �TOKEN_RAW_BEGIN� TOKEN_RAW_ENDrT rU rs rO �TOKEN_LINECOMMENT_ENDrv ) r| rj ru r� Z tag_rulesZroot_tag_rulesZblock_start_reZ block_end_reZcomment_end_reZvariable_end_reZblock_suffix_reZ root_raw_reZ root_parts_rer8 r8 r< r} � s� � ������ ����� ����� � ��� ������ � �����zLexer.__init__re c C s t �| j|�S )z`Replace all newlines with the configured sequence in strings and template data. )rf r r� )r| r_ r8 r8 r< �_normalize_newlinesT s zLexer._normalize_newlines)�sourcer r �staterJ c C s& | � ||||�}t| �|||�||�S )z:Calls tokeniter + tokenize and wraps it in a token stream.)� tokeniterr� �wrap)r| r� r r r� r� r8 r8 r< �tokenizeZ s zLexer.tokenize)r� r r rJ c c st |D �]h\}}}|t v rq|}|tkr.t}�n2|tkr>t}�n"|ttfv rPq�n|tkrd| �|�}n�|dkrr|}n�|t kr�|}|� � s�t d|||��n�|t k�rz$| �|dd� �� dd��d�}W nL t�y } z2t|��d�d �� } t | |||�|�W Y d }~n d }~0 0 nL|tk�r2t|�d d �d �}n.|tk�rNt|�d d ��}n|tk�r`t| }t|||�V qd S ) z�This is called with the stream as returned by `tokenize` and wraps every token in a :class:`Token` and converts the value. �keywordz Invalid character in identifierr ����ascii�backslashreplacezunicode-escaper4 Nr� r� r )�ignored_tokensrT rP rU rQ r� r� rV r� r^ � isidentifierr r� �encode�decode� Exceptionr� rc �stripr� r� �replacer� r r� � operatorsrZ ) r| r� r r r~ r\ Z value_strr_ ru �msgr8 r8 r< r� e sL � ���( z Lexer.wrapc c s@ t �|�ddd� }| js,|d dkr,|d= d�|�}d}d}dg}|durr|dkrr|d v sdJ d ��|�|d � | j|d } t|�} g } d} d } | D �]r\}}}|�||�}|du r�q�| r�|tt t fv r�q�t |t ��rz|� � }t |t��r�|d }td d� |ddd� D ��}|dk�rN|�� }|t|�d� �d�} |g|dd� �}nh|dk�r�| j�r�|�� �t��s�|�d�d }|dk�s�| �r�t�||��r�|d|� g|dd� �}t|�D ]�\}}|jtu �r�|||��n�|dk�r:|�� � � D ]4\}}|du�r�|||fV ||�d�7 } �qt�q�t |�d���n:|| }|�sR|t v�r^|||fV ||�d�| 7 }d} �q�n�|� � }|t k�r"|dk�r�| �d� n�|dk�r�| �d� nj|dk�r�| �d� nT|dv �r"| �s�t!d|� d�|||��| �"� } | |k�r"t!d|� d | � d�|||��|�s2|t v�r>|||fV ||�d�7 }|� � dd� dk} |�#� } |du�r�|d k�r�|�"� nV|dk�r�|�� � � D ]$\}}|du�r�|�|� �qܐq�t |�d ���n |�|� | j|d } n| |k�rt |�d ���| } q�q�|| k�r dS t!d || �d!|� �|||��q�dS )"a This method tokenizes the text and returns the tokens in a generator. Use this method if you just want to tokenize a template. .. versionchanged:: 3.0 Only ``\n``, ``\r\n`` and ``\r`` are treated as line breaks. N� r� r� � r r r� )�variable�blockz invalid stateZ_beginTc s s | ]}|d ur|V qd S r? r8 )r9 �gr8 r8 r< rD � r> z"Lexer.tokeniter.<locals>.<genexpr>r% r$ r� z= wanted to resolve the token dynamically but no group matchedr. r/ r, r- r* r+ )r/ r- r+ z unexpected '�'z ', expected 'r� zA wanted to resolve the new state dynamically but no group matchedz* yielded empty string without stack changezunexpected char z at )$rf rc r� r� rq rv rE �matchrS rQ rU � isinstance�tuple�groupsr� r� �rstrip�countr� � groupdictrX rR �rfindr� � fullmatch� enumerater� rx �items� RuntimeError�ignore_if_empty�groupr� r �pop�end) r| r� r r r� �lines�posr~ �stackZ statetokensZ source_lengthZbalancing_stackZnewlines_strippedZ line_startingZregexr� Z new_state�mr� �textZ strip_sign�strippedZl_pos�idxr\ rH r_ r! Z expected_opZpos2r8 r8 r< r� � s� � �� � � � � � � �zLexer.tokeniter)NNN)NN)NN)r� r� r� r� r} r� r� r� r� r� r� r� r� r� �IteratorrZ r� r� r8 r8 r8 r< r � s< w �� �� : ��)pr� r@ �typingr� �astr � collectionsr �sysr Z _identifierr r� � exceptionsr Zutilsr � TYPE_CHECKINGZtyping_extensions�terj r r �MutableMappingr� r� r� r� rf r� r� � IGNORECASE�VERBOSEr� r� Z TOKEN_ADDZ TOKEN_ASSIGNZ TOKEN_COLONZ TOKEN_COMMAZ TOKEN_DIVZ TOKEN_DOTZTOKEN_EQZTOKEN_FLOORDIVZTOKEN_GTZ TOKEN_GTEQZ TOKEN_LBRACEZTOKEN_LBRACKETZ TOKEN_LPARENZTOKEN_LTZ TOKEN_LTEQZ TOKEN_MODZ TOKEN_MULZTOKEN_NEZ TOKEN_PIPEZ TOKEN_POWZ TOKEN_RBRACEZTOKEN_RBRACKETZ TOKEN_RPARENZTOKEN_SEMICOLONZ TOKEN_SUBZ TOKEN_TILDEr� r� r� r^ r� r� rP rQ rR rS r� r� rL rM rN rT rU rs r� rO rV r� rW r� r rK rE r� rt r� � frozensetr� r r� rY r` rd r� rh �Listrw rx � NamedTuplerZ r� r� r� r� r� r� r r8 r8 r8 r< �<module> s � � �� &��� � ) m