관리-도구
편집 파일: concrete_artifact_manager.cpython-39.pyc
a �)gKs � @ s d Z ddlmZmZmZ eZddlZddlZddl Z ddl Z ddl Z ddl mZ ddlmZ ddlmZ ddlmZ ddlmZ dd lmZ e jr�dd lmZm Z dd l m Z dd l m Z dd l!m"Z" ddl#m$Z$ ddlm%Z% ddl&m'Z' ddl(m)Z)m*Z*m+Z+ ddl,m-Z- ddl,m.Z. ddl/m0Z0 ddl1m2Z2 ddl3m4Z4 ddl5m6Z6 ddl7m8Z8 ddl9m:Z: ddl;m<Z< ddl=Z=e:� Z>dZ?G d d � d �Z@d d � ZAd d!� ZBe-e.d"d#d$d%�e$d&�d;d(d)��ZCd<d*d+�ZDd=d-d.�ZEd>d/d0�ZFd?d1d2�ZGd3d4� ZHd5d6� ZId7d8� ZJed9d:� �ZKdS )@z7Concrete collection candidate management helper module.� )�absolute_import�division�print_functionN)�contextmanager)�sha256)�URLError)� urldefrag)�rmtree)�mkdtemp)� Candidate� Requirement)� GalaxyToken)� AnsibleError)� get_collections_galaxy_meta_info)�should_retry_error)� _GALAXY_YAML)� user_agent)�to_bytes� to_native�to_text)� retry_with_delays_and_condition)�generate_jittered_backoff)� get_bin_path)�MutableMapping)� yaml_load)� raise_from)�open_url)�Display)�Sentinelz MANIFEST.jsonc @ s� e Zd ZdZd&dd�Zedd� �Zed d � �Zed d � �Zed d� �Z e j dd� �Z dd� Z dd� Z dd� Z dd� Zdd� Zdd� Zd d � Zd d � Zd d!� Zd"d#� Zeed'd$d%���ZdS )(�ConcreteArtifactsManagera Manager for on-disk collection artifacts. It is responsible for: * downloading remote collections from Galaxy-compatible servers and direct links to tarballs or SCM repositories * keeping track of local ones * keeping track of Galaxy API tokens for downloads from Galaxy'ish as well as the artifact hashes * keeping track of Galaxy API signatures for downloads from Galaxy'ish * caching all of above * retrieving the metadata out of the downloaded artifacts TN�< c C sR || _ i | _i | _i | _i | _i | _|| _i | _|| _|| _ || _ || _ d| _ dS )z:Initialize ConcreteArtifactsManager caches and costraints.TN) �_validate_certs�_artifact_cache�_galaxy_artifact_cache�_artifact_meta_cache�_galaxy_collection_cache� _galaxy_collection_origin_cache�_b_working_directoryZ _supplemental_signature_cache�_keyring�timeout�_required_signature_count�_ignore_signature_errors�_require_build_metadata)�selfZb_working_directory�validate_certs�keyringr) �required_signature_count�ignore_signature_errors� r2 �W/usr/lib/python3.9/site-packages/ansible/galaxy/collection/concrete_artifact_manager.py�__init__A s z!ConcreteArtifactsManager.__init__c C s | j S �N)r( �r- r2 r2 r3 r/ R s z ConcreteArtifactsManager.keyringc C s | j S r5 )r* r6 r2 r2 r3 �#required_successful_signature_countV s z<ConcreteArtifactsManager.required_successful_signature_countc C s | j d u rg S | j S r5 )r+ r6 r2 r2 r3 r1 Z s z0ConcreteArtifactsManager.ignore_signature_errorsc C s | j S r5 �r, r6 r2 r2 r3 �require_build_metadata` s z/ConcreteArtifactsManager.require_build_metadatac C s || _ d S r5 r8 )r- �valuer2 r2 r3 r9 e s c C s| |j j}z | j| d }| j| \}}W n6 ty^ } z tdj|d��|�W Y d }~n d }~0 0 d|j|j|j ||||d�S )Nr �#The is no known source for {coll!s}��collz1.0.0)Zformat_version� namespace�name�version�serverZ version_url� download_url� signatures) �src� api_serverr% r& �KeyError� RuntimeError�formatr> r? �ver)r- � collectionrA rB �signatures_urlrC �key_errr2 r2 r3 � get_galaxy_artifact_source_infoj s* ����z8ConcreteArtifactsManager.get_galaxy_artifact_source_infoc C sb z | j | W S ty Y n0 z| j| \}}}W n: tyn } z"ttdj|d��|� W Y d}~n d}~0 0 t�dj|d�� zt|| j || j |d�}W n� t y� } z*t djt |j�t |�d��|�W Y d}~n~d}~0 t�y } z*t d jt |j�t |�d��|�W Y d}~n<d}~0 0 t�d j||j�p6d |jdu�rJ|jjnd d �� || j |< |S )z�Given a Galaxy-stored collection, return a cached path. If it's not yet on disk, this method downloads the artifact first. r; r< NzFFetching a collection tarball for '{collection!s}' from Ansible Galaxy�rJ )� expected_hashr. �token�GFailed to download collection tar from '{coll_src!s}': {download_err!s}�Zcoll_srcZ download_errzmFailed to download collection tar from '{coll_src!s}' due to the following unforeseen error: {download_err!s}z=Collection '{coll!s}' obtained from server {server!s} {url!s}ZGalaxy� )r= rA �url)r# rF r% r rG rH �display�vvvv�_download_filer' r! r r r rD � ExceptionZvvvrE )r- rJ rT � sha256_hashrP rL �b_artifact_path�errr2 r2 r3 �get_galaxy_artifact_path� sn ������ ������ �� z1ConcreteArtifactsManager.get_galaxy_artifact_pathc C s� z| j |j W S ty Y n0 |jr�t�dj|jd�� z t|j| j d| j | j d�}W q� t y� } z.t tdjt|j�t|�d��|� W Y d}~q�d}~0 0 nJ|jr�t|j|j| j �}n0|js�|js�|jr�t|j�}ntdj|jd���|| j |j< |S ) z�Given a concrete collection pointer, return a cached path. If it's not yet on disk, this method downloads the artifact first. zBCollection requirement '{collection!s}' is a URL to a tar artifactrN N)rO r. r) rQ rR z2The artifact is of an unexpected type {art_type!s})Zart_type)r" rD rF �is_urlrU rV rH ZfqcnrW r' r! r) rX r r r �is_scm� _extract_collection_from_gitrI �is_file�is_dir� is_subdirsr rG �type)r- rJ rZ r[ r2 r2 r3 �get_artifact_path� sV ��� ��� � �� z*ConcreteArtifactsManager.get_artifact_pathc C s | � |�d S )Nr> ��get_direct_collection_meta�r- rJ r2 r2 r3 � _get_direct_collection_namespace s z9ConcreteArtifactsManager._get_direct_collection_namespacec C s | � |�d S )Nr? re rg r2 r2 r3 �_get_direct_collection_name s z4ConcreteArtifactsManager._get_direct_collection_namec C s$ |j r dS d�| �|�| �|�f�S )z�Extract FQCN from the given on-disk collection artifact. If the collection is virtual, ``None`` is returned instead of a string. N�.)Z is_virtual�joinrh ri rg r2 r2 r3 �get_direct_collection_fqcn s �z3ConcreteArtifactsManager.get_direct_collection_fqcnc C s | � |�d S )z;Extract version from the given on-disk collection artifact.r@ re rg r2 r2 r3 � get_direct_collection_version s z6ConcreteArtifactsManager.get_direct_collection_versionc C s | � |�d }|du ri }|S )z8Extract deps from the given on-disk collection artifact.� dependenciesNre )r- rJ Zcollection_dependenciesr2 r2 r3 �"get_direct_collection_dependencies# s z;ConcreteArtifactsManager.get_direct_collection_dependenciesc C s� z| j |j W S ty* | �|�}Y n0 |js8|jrBt|�}n�|jr�zt|| j �}W q� t y� } z&t t dj t|�d��|� W Y d}~q�d}~0 0 nH|jr�ddt|�didd�}n*|jr�ddt�tt|j�d�dd�}nt�|| j |j< |S )z8Extract meta from the given on-disk collection artifact.z/Failed to find the collection dir deps: {err!s})r[ N�*)r? r> rn r@ )r$ rD rF rd r] r` �_get_meta_from_tarra �_get_meta_from_dirr9 � LookupErrorr r rH r r^ rb �dict�fromkeys�mapZnamespace_collection_pathsrG )r- rJ rZ Zcollection_metaZ lookup_errr2 r2 r3 rf + sH ��� � �� z3ConcreteArtifactsManager.get_direct_collection_metac C s" |||f| j |< ||f| j|< dS )z�Store collection URL, SHA256 hash and Galaxy API token. This is a hook that is supposed to be called before attempting to download Galaxy-based collections with ``get_galaxy_artifact_path()``. N)r% r& )r- rJ rT rY rP rK rC r2 r2 r3 �save_collection_sourceY s z/ConcreteArtifactsManager.save_collection_sourcec c sN t t|dd�d�}t|dd�}z | |||||d�V W t|� n t|� 0 dS )a Custom ConcreteArtifactsManager constructor with temp dir. This method returns a context manager that allocates and cleans up a temporary directory for caching the collection artifacts during the dependency resolution process. �surrogate_or_strict��errors)�dir)r/ r0 r1 N)r r r ) �clsZ temp_dir_baser. r/ r0 r1 r9 Z temp_pathZ b_temp_pathr2 r2 r3 � under_tmpdirc s � � z%ConcreteArtifactsManager.under_tmpdir)TNr NN)TNNNT)�__name__� __module__� __qualname__�__doc__r4 �propertyr/ r7 r1 r9 �setterrM r\ rd rh ri rl rm ro rf rw � classmethodr r} r2 r2 r2 r3 r 4 s< CA. �r c C s� d| v r| � dd�\} }n|dks&|s*d}| �d�rB| dd� }n| }t|�\}}|�tjj�}|�tjjd �r�|� tjj�d }n:d |vr�d |vr�|}n$|� d �d }|�d�r�|dd� }||||fS )z>Extract name, version, path and subdir out of the SCM pointer.�,� rp �HEADzgit+� Nz.git���z://�@�/������)�split� startswithr �strip�os�path�sep�endswith)rJ r@ r� �fragmentr? r2 r2 r3 � parse_scm� s" r� c C sp t | |�\}}}}t|t|dd�d�}z td�}W n: tyl } z"tdjt|�d��| �W Y d } ~ n d } ~ 0 0 |dkr�|dd |t|�f} n|d|t|�f} zt � | � W n@ t j y� } z&t td jt|�d��| � W Y d } ~ n d } ~ 0 0 |d t|�f} zt j | |d � W nH t j �yR } z,t td jt|�t|�d��| � W Y d } ~ n d } ~ 0 0 |�rlt j�|t|��S |S )Nrx ry �r{ �prefix�gitz_Could not find git executable to extract the collection from the Git repository `{repo_url!s}`.)�repo_urlr� Zclonez --depth=1z5Failed to clone a Git repository from `{repo_url!s}`.Zcheckout)�cwdz\Failed to switch a cloned Git repo `{repo_url!s}` to the requested revision `{commitish!s}`.)Z commitishr� )r� r r r � ValueErrorr rH r r � subprocessZ check_callZCalledProcessErrorr r� r� rk ) r� Zcoll_ver�b_pathr? r@ Zgit_urlr� Zb_checkout_pathZgit_executabler[ Z git_clone_cmdZproc_errZgit_switch_cmdr2 r2 r3 r_ � s\ � �����������r_ � � �( )ZretriesZ delay_baseZdelay_threshold)Zbackoff_iteratorr r c C s4 t | �dd�d dd�}|d td� � }t|d�|df�d�}tj�||�} t�d | t|�f � z4t t | dd�||d u r�d n|� � d gt � |d �} W n4 t y� } z tt | �| d ��W Y d } ~ n d } ~ 0 0 t| d �� } t| | d�} W d � n1 s�0 Y |�r0t�dj| |d�� || k�r0td��| S )Nr� r� rx ry z.tar.gz� -� r� zDownloading %s to %sZ Authorization)r. �headersZunredirected_headersZ http_agentr) )Zorig_exc�wb)�write_tozTValidating downloaded file hash {actual_hash!s} with expected hash {expected_hash!s})� actual_hashrO z+Mismatch artifact hash with downloaded file)r �rsplit�lenr rk r� r� rU r r r r� r rX r �open� _consume_filerV rH )rT r� rO r. rP r) Zb_tarball_nameZ b_file_nameZ b_tarball_dirZ b_file_pathZrespr[ Z download_filer� r2 r2 r3 rW � s> � � � & *�� rW c C sP d}t � }| �|�}|rH|d ur2|�|� |�� |�|� | �|�}q|�� S )Ni )r �read�write�flush�updateZ hexdigest)Z read_fromr� �bufsizeZ sha256_digest�datar2 r2 r3 r� s r� Tc C s� t � }t� }t� }t� }t� }t� }|D ]F} | �dd�rF|�| d � ||||d�| �dd� } | �| d � q(t||B |B |B |B �} t| �� �} |�| �} | r�dt|�d�t | ��f }|r�t |��t � |� t |��| �| �}t|�d k�rt � d t|�d�|�f � |D ]}|| v�rd | |< �q|D ]<}| �|d �}|d u �rTg | |< nt|t��s0|g| |< �q0|D ]}|| v�rri | |< �qr|D ]}|| v�r�t| |< �q�| �d ��s�d | d < | S ) N�requiredF�key)�str�listrt �sentinelrc r� zMThe collection galaxy.yml at '%s' is missing the following mandatory keys: %sz, r z7Found unknown keys in collection galaxy.yml at '%s': %sr@ rp )r �set�get�add� frozenset�keys� differencer rk �sortedr rU �warningr� r� r � isinstancer� r )� galaxy_ymlZb_galaxy_yml_pathr9 Zgalaxy_yml_schemaZmandatory_keysZ string_keysZ list_keys� dict_keysZ sentinel_keys�infoZ key_list_typeZall_keysZset_keysZ missing_keys�msgZ extra_keysZoptional_stringZ optional_listZlist_valZ optional_dictZoptional_sentinelr2 r2 r3 � _normalize_galaxy_yml_manifest sj � � � �� � r� c C s, z t | �W S ty& t| |� Y S 0 d S r5 )� _get_meta_from_installed_dirrs �_get_meta_from_src_dir)r� r9 r2 r2 r3 rr g s rr c C s t j�| t�}t j�|�s.tdjt|�d���t|d��f}z t |�}W nH t j j y� } z,t tdjt|�t|�d��|� W Y d }~n d }~0 0 W d � n1 s�0 Y t|t�s�|r�tdt|�� d���t�dt|�� d�� tdt|�� d���t|||�S )Nz9The collection galaxy.yml path '{path!s}' does not exist.�r� �rbzRFailed to parse the galaxy.yml at '{path!s}' with the following error: {err_txt!s})r� Zerr_txtz The collection galaxy.yml at 'z' is incorrectly formatted.)r� r� rk r �isfilers rH r r� r �yaml�errorZ YAMLErrorr r r� rt rU r� r� r� )r� r9 r� Zmanifest_file_obj�manifestZyaml_errr2 r2 r3 r� q s4 �� ���8 r� c C s� t j�| t|dd��}z6t|d��}|�� }W d � n1 s@0 Y W n, ttfyx tdj |t |�d���Y n0 t |dd�}zt � |�}W n" ty� tdj |d���Y n0 |S )Nrx ry r� z;The collection {manifest!s} path '{path!s}' does not exist.)r� r� �KCollection tar file member {member!s} does not contain a valid json string.��member)r� r� rk r r� r� �IOError�OSErrorrs rH r r �json�loadsr� r )r� �filenameZb_json_filepathZ manifest_fdZ b_json_textZ manifest_txtr� r2 r2 r3 � _get_json_from_installed_dir� s* *�� �� r� c C sD t | t�}|d }|�d�}|s@tdjtt| �tt|��d���|S )N�collection_infor@ z�Collection metadata file `{manifest_filename!s}` at `{meta_file!s}` is expected to have a valid SemVer version value but got {version!s})Zmanifest_filenameZ meta_filer@ )r� �MANIFEST_FILENAMEr� r rH r �repr)r� r� r� r@ r2 r2 r3 r� � s �� r� c C s* t �| �s tdjt| �d���t j| dd���}z|�t�}W n( tyd tdjt| �td���Y n0 t ||���\}}|d u r�tdjtd���t |� � d d �}zt � |�}W n" ty� td jtd���Y n0 |d W d � W d � S 1 � s�0 Y W d � n1 �s 0 Y d S ) Nz:Collection artifact at '{path!s}' is not a valid tar file.r� �r)�modezNCollection at '{path!s}' does not contain the required file {manifest_file!s}.)r� Z manifest_filez6Collection tar file does not contain member {member!s}r� rx ry r� r� )�tarfileZ is_tarfiler rH r r� Z getmemberr� rF �_tarfile_extractr r� r� r� r� )r� Zcollection_tarr� Z_memberZ member_objZ text_contentr� r2 r2 r3 rq � sH �� �� ��� �� rq c c s@ | � |�}z ||fV W |d ur<|�� n|d ur:|�� 0 d S r5 )Z extractfile�close)�tarr� Ztar_objr2 r2 r3 r� � s �r� )Nr )N)T)T)T)Lr� Z __future__r r r rc Z __metaclass__r� r� r� r� �typing�t� contextlibr Zhashlibr Z urllib.errorr � urllib.parser �shutilr Ztempfiler � TYPE_CHECKINGZ0ansible.galaxy.dependency_resolution.dataclassesr r Zansible.galaxy.tokenr Zansible.errorsr Zansible.galaxyr Zansible.galaxy.apir r Zansible.galaxy.user_agentr Zansible.module_utils._textr r r Zansible.module_utils.apir r Z#ansible.module_utils.common.processr Z/ansible.module_utils.common._collections_compatr Z ansible.module_utils.common.yamlr Zansible.module_utils.sixr Zansible.module_utils.urlsr Zansible.utils.displayr Zansible.utils.sentinelr r� rU r� r r� r_ rW r� r� rr r� r� r� rq r� r2 r2 r2 r3 �<module> sp T8 � , � Q � � %!-