/
usr
/
libexec
/
kcare
/
python
/
kcarectl
/
__pycache__
/
Upload File
HOME
3 �;j _ � . @ s d Z ddlZddlZddlZddlZddlZddlmZmZm Z m Z m Z m Z m Z mZmZ ddlmZ ddlmZmZmZ dZdZdZd Zd Zd Zd Z d Z dZ d�Z d�Z d�Z!G dd� de j"�Z#G dd� de$�Z%d d � Z&d d � Z'd d!� Z(d"d#� Z)G d$d%� d%e$�Z*d&d'� Z+d(d)� Z,d�d+d,�Z-d-d.� Z.ej/d/d0� �Z0d1d2� Z1d3d4� Z2d5d6� Z3d7d8� Z4d9d:� Z5d;d<� Z6d=d>� Z7d?d@� Z8dAdB� Z9dCdD� Z:dEdF� Z;dGdH� Z<dIdJ� Z=dKdL� Z>e*dMe.�e*dNe-dOgdPdQ��e*dRe0�e*dSe1�e*dTe2�e*dUe-dUg��e*dVe3�e*dWe+dX��e*dYe+dZ��e*d[e+d\��e*d]e+d^��e*d_e+d`��e*dae+db��e*dce+dd��e*dee+df��e*dge+dh��e*die,djdk��e*dle4�e*dme,dndo��e*dpe5�e*dqe6�gZ?e&drdsj@e ��e&dtdt�e&dudvj@e ��e&dwdx�e'dydz�e&d{d|�e(d}d~�e(dd��e(d�d��e(d�d��e(d�d��e(d�d��e(d�d��e(d�d��e(d�d��e'd�d��e(d�d��e'd�e�e(d�d��e&d�d��e&d�d��e&d�d��e&d�d��e)d�e7�e&d�d��e'd�d��e)d�e9�e&d�d��e&d�d��e)d�e:�e'd�d��e&d�d��e)d�e;�e&d�d�j@e ��e'd�e�e'd�d��e)d�e<�e)d�e=�e&d�d��e&d�d��e&d�d��e)d�e>�e&d�d��e&d�d��g,ZAejBe jCd��d�d�� �ZDd�d�d��ZEdS )�aO Doctor v2 vitals collector (KPT-5984). Python port of the kcdoctor.sh (1.0-8) vitals collection: gathers the same data set into a DataPackage-based archive processed by the patch server generic uploads pipeline. Scalar and short values are consolidated into a single ``doctor.json`` (the eportal doctor precedent), one field each, instead of a tiny archive entry per value; bulk data (logs, command dumps, package lists, sysfs tunables, crash artifacts) stays as separate archive entries. The 1:1 inventory of collected items lives in docs/features/KPT-5984-doctor-v2/parity-inventory.md. � N) �auth�config� delivery_kit� http_utils�kcare� log_utils�platform_utils�serverid�utils)� run_command)� HTTPError�URLError�httplibz2.0-1z/boot/grub2/grub.cfgz/etc/kdump.confz/var/log/dpkg.logz/etc/apt/sources.list*z/etc/yum.repos.d/*z/var/log/libcare/*.logi' z/sys�enable*�nr_*�max_*�*cnt*� /sys/kernel/debug/tracing/events�/sys/kernel/tracing/events� /sys/devices� [0-9]*.log�kmsg*logc @ s e Zd ZdZdZedd� �ZdS )�KcarectlDoctorPackagezkcarectl-doctorz/upload/kcarectl-doctor/c C s t jS )N)r � DOCTOR_REPORT_MAX_SIZE_BYTES)�self� r �,/usr/libexec/kcare/python/kcarectl/doctor.py�max_size; s z KcarectlDoctorPackage.max_sizeN)�__name__� __module__� __qualname__� data_type� upload_uri�propertyr r r r r r 7 s r c @ s e Zd ZdZdd� Zdd� ZdS )�CollectionSpecz�A single bulk vitals item ported from kcdoctor.sh. Collection failures are recorded in errors.log and never abort the report: an unhandled exception inside the package context manager would remove the whole archive. c C s || _ || _d S )N)�arcname�_collect)r r% �collectr r r �__init__L s zCollectionSpec.__init__c C sN y| j || j� W n6 tk rH } z|jdj| j|�� W Y d d }~X nX d S )Nzfailed to collect {0}: {1})r& r% � Exception� log_error�format)r � data_package�er r r �applyQ s zCollectionSpec.applyN)r r r �__doc__r( r. r r r r r$ D s r$ c s � fdd�}t | |�S )z4`run "<cmd>"` kcdoctor.sh helper (no shell features)c s | j |� � d S )N)� add_stdout)r, r% )�cmdr r r' ] s zrun_spec.<locals>.collect)r$ )r% r1 r' r )r1 r �run_specY s r2 c s � fdd�}t | |�S )z4`dump "<path>"` kcdoctor.sh helper for regular filesc s | j |� d� d S )N)�src_path)�add_file)r, r% )�pathr r r' h s zdump_spec.<locals>.collect)r$ )r% r5 r' r )r5 r � dump_specd s r6 c s � fdd�}t | |�S )z�`dump "<path>"` for /proc and /sys files. tar reports them as 0-size so they are first read into memory (see the same approach in anomaly.prepare_kernel_anomaly_report). c sL t jj� �s | jdj� �� d S t� d��}| j||j� d� W d Q R X d S )Nzfile not found: {0}�rb)� data_bytes)�osr5 �existsr* r+ �openr4 �read)r, r% �f)r5 r r r' w s z proc_dump_spec.<locals>.collect)r$ )r% r5 r' r )r5 r �proc_dump_speco s r> c C s t | |�S )N)r$ )r% �fnr r r � callable_spec� s r@ c @ s e Zd ZdZdd� Zdd� ZdS )� FieldSpecz�A single doctor.json field ported from kcdoctor.sh. Collection failures are recorded in errors.log and never abort the report; a collector returning None omits the field (e.g. a distro-specific source file absent on this system). c C s || _ || _d S )N)�keyr& )r rB r' r r r r( � s zFieldSpec.__init__c C sX y | j � }W n4 tk r@ } z|jdj| j|�� d S d }~X nX |d k rT||| j< d S )Nzfailed to collect {0}: {1})r& r) r* r+ rB )r r, �report�valuer- r r r r. � s zFieldSpec.applyN)r r r r/ r( r. r r r r rA � s rA c s � fdd�}|S )z�`dump "<path>"` as a doctor.json string field. Read in binary (non-utf8 safe) and decoded leniently; an absent distro-specific source file simply omits the field (None). c s: t jj� �sd S t� d��} | j� jdd�j� S Q R X d S )Nr7 zutf-8�replace)r9 r5 r: r; r< �decode�strip)r= )r5 r r r' � s zread_field.<locals>.collectr )r5 r' r )r5 r � read_field� s rH c s" t jtj|���� �fdd�}|S )z�`grep <pattern> <path>` as a doctor.json list-of-lines field. Match on bytes (like collect_apt_sources) so a non-utf8 byte cannot lose a line; the matched lines are decoded leniently for the JSON. c sD t jj� �sd S t� d��} | j� j� }W d Q R X �fdd�|D �S )Nr7 c s" g | ]}� j |�r|jd d��qS )zutf-8rE )�searchrF )�.0�line)�regexr r � <listcomp>� s z/grep_field.<locals>.collect.<locals>.<listcomp>)r9 r5 r: r; r< � splitlines)r= �lines)r5 rL r r r' � s zgrep_field.<locals>.collect)�re�compiler �bstr)r5 �patternr' r )r5 rL r � grep_field� s rT Fc s � �fdd�}|S )z@`run "<cmd>"` captured as a doctor.json field instead of a file.c s0 t � ddd�\} }} |pdj� }�r,|j� S |S )NT)� catch_stdout� catch_stderr� )r rG rN )�_�stdout�text)�argv�as_linesr r r' � s zcmd_field.<locals>.collectr )r[ r\ r' r )r[ r\ r � cmd_field� s r] c C s t S )N)�DOCTOR_VERSIONr r r r �collect_doctor_version� s r_ c C s< y"t jtjd��} tj| j� �j� S tk r6 dS X dS )z�the public IP of the machine as seen by the patch server. Cached: the main_ip and server_id fields share one HTTP roundtrip per run. ZmyipZNAN)r �urlopenr �get_patch_server_url�nstrr< rG r) )�responser r r � get_main_ip� s rd c C s$ t j� } | st� jdd�} tj| �S )N�.rX )r � get_serveridrd rE r rb )� server_idr r r �collect_server_id� s rh c C s t j� S )N)r �get_kernel_hashr r r r �collect_kernel_id� s rj c C sB i } x8d D ]0}t dd| gddd �\}}}|p0d j� | |< q W | S ) N�a�r�m�p�o�uname�-T)rU rV rW )rk rl rm rn ro )r rG )�result�flagrX rY r r r � collect_uname� s rt c C sH t jjt�sdS ttd��} | j� jdd�}W dQ R X dd� |j� D �S )z7`grep vmlinuz /boot/grub2/grub.cfg | sed 's/root=.*//'`Nr7 zutf-8rE c S s" g | ]}d |krt jdd|��qS )Zvmlinuzzroot=.*rW )rP �sub)rJ rK r r r rM s z)collect_grub2_entries.<locals>.<listcomp>)r9 r5 r: � GRUB2_CFGr; r< rF rN )r= rZ r r r �collect_grub2_entries s rw c C s t tjt��S )N)�sorted�glob�YUM_REPOS_GLOBr r r r �collect_yum_repos s r{ c C s dd� t jD �} | t j� d�S )z8presence-only port of the kcdoctor.sh detect_cp() probesc S s g | ]\}}|� r|�qS r r )rJ �name�prober r r rM s z)collect_control_panel.<locals>.<listcomp>)�cpZ softaculous)r �CONTROL_PANEL_PROBES�has_softaculous)Zpanelsr r r �collect_control_panel s r� c C s@ t ddgddd�\}}}| j|tj|p(djdd�dd �d � d S ) z`ipcs -m | sed -e s/-/=/g`�ipcsz-mT)rU rV rW rq �=zutf-8)�encoding)r8 N)r r4 r rR rE )r, r% rX rY r r r � collect_ipcs% s r� c C sh t jjt�s | jdjt�� dS ttd�� }dd� |j� j� D �}W dQ R X | j |dj |�d d� dS )z$`grep ' install ' /var/log/dpkg.log`zfile not found: {0}Nr7 c S s g | ]}d |kr|�qS )s install r )rJ rK r r r rM 4 s z,collect_dpkg_install_log.<locals>.<listcomp>� )r8 ) r9 r5 r: �DPKG_LOGr* r+ r; r< rN r4 �join)r, r% r= rO r r r �collect_dpkg_install_log, s r� c C s� t jjd�st jjd�r d}nXt jjd�rrd}yt| d� W qv tk rn } z| jdj|�� W Y d d }~X qvX nd}| j||� d S ) Nz /usr/bin/rpmz/bin/rpmzFrpm -q -a --queryformat="%{N}|%{V}-%{R}|%{arch}|%{INSTALLTIME:date}\n"z /usr/bin/dpkgzJ/usr/bin/dpkg-query -W -f "${binary:Package}|${Version}|${Architecture}\n"zdpkg.logz failed to collect dpkg.log: {0}z echo "unknown package manager")r9 r5 r: r� r) r* r+ r0 )r, r% � packages_cmdr- r r r �collect_packages9 s $r� c sF t jd��g }xdttjt��D ]R}tjj|�rfx@tj|�D ]&\�}}|j �fdd�t|�D �� q:W q |j |� q W |s�| j dj t�� dS g }x�|D ]�}y"t |d��}|j� }W dQ R X W n6 tk r� } z| j dj || �� w�W Y dd} ~ X nX tjdj |�d d �� |j � �fd d�|j� D �� q�W | j|d j|�d d � dS )z/`grep -rE '^(deb|URIs)' /etc/apt/sources.list*`s ^(deb|URIs)c 3 s | ]}t jj� |�V qd S )N)r9 r5 r� )rJ r| )�rootr r � <genexpr>U s z&collect_apt_sources.<locals>.<genexpr>zfile not found: {0}Nr7 zfailed to read {0}: {1}z{0}:zutf-8)r� c 3 s | ]}�j |�r� | V qd S )N)rI )rJ rK )�prefixrL r r r� j s r� )r8 )rP rQ rx ry �APT_SOURCES_GLOBr9 r5 �isdir�walk�extend�appendr* r+ r; r< r) r rR rN r4 r� ) r, r% �filesr5 rX �namesrO r= �contentr- r )r� rL r� r �collect_apt_sourcesL s* " "r� c C s� t tjt��}|s&| jdjt�� dS g }x�|D ]�}y0t|d�� }|j� j� t d� }W dQ R X W n6 t k r� } z| jdj||�� w0W Y dd}~X nX t j dj|�dd�}|j |dj |� � q0W | j|dj |�d d � dS ) z%`tail -n10000 /var/log/libcare/*.log`zfile not found: {0}Nr7 zfailed to read {0}: {1}z ==> {0} <== zutf-8)r� r� )r8 )rx ry �LIBCARE_LOGS_GLOBr* r+ r; r< rN �LOG_TAIL_LINESr) r rR r� r� r4 ) r, r% �paths�chunksr5 r= rO r- �headerr r r �collect_libcare_logso s $r� c s� t jjtj�sdS | j|djtj�� xtt jtj�D ]d\}}}xXt|�D ]L� t � fdd�t D ��rHt jj |� �}t jj |tj�}| j dj|�|d� qHW q4W dS )z3crashreporter artifacts from /var/cache/kcare/dumpsNz ls -lR {0}c 3 s | ]}t j � |�V qd S )N)�fnmatch)rJ rS )r| r r r� � s z&collect_crash_dumps.<locals>.<genexpr>zcrashreporter/{0})r3 )r9 r5 r� r � KDUMPS_DIRr0 r+ r� rx �any�CRASH_DUMP_PATTERNSr� �relpathr4 )r, r% r� rX r� r5 Zrel_pathr )r| r �collect_crash_dumps� s r� c C s� t jjt�sd S tj� }t jj|�s(d S | j|dj|�� xTt t j t jj |d���D ]8}t jj|�sfqT| j djt jj |��t jj |d�d� qTW d S )Nz ls -lR {0}�*zkdump/{0}/vmcore-dmesg.txtzvmcore-dmesg.txt)r3 )r9 r5 r: � KDUMP_CONFr �get_kdump_rootr� r0 r+ rx ry r� r4 �basename)r, r% � kdump_rootZevtr r r � collect_kdump� s r� c s� g }x�t jt�D ]�\}}}|jt�r2g |dd�< qx�t|�D ]�� t� fdd�tD ��r<t jj |� �}y"t |d��}|j � }W dQ R X W n t k r� d}Y nX |j tjdj|��| d � q<W qW | j|dj |�d� dS ) zGtunables gathered by the kcdoctor.sh get_sysfs_info() `find /sys` callsNc 3 s | ]}t j � |�V qd S )N)r� )rJ rS )r| r r r� � s z'collect_sysfs_config.<locals>.<genexpr>r7 � zcat {0} r� )r8 )r9 r� � SYSFS_ROOT� startswith�SYSFS_EXCLUDE_PREFIXESrx r� �SYSFS_NAME_PATTERNSr5 r� r; r< r) r� r rR r+ r4 ) r, r% r� r� �dirsr� r5 r= r� r )r| r �collect_sysfs_config� s &r� Zdoctor_versionZ virt_whatz /usr/libexec/kcare/virt-whatT)r\ Zmain_iprg Z kernel_id�daterp Zredhat_releasez/etc/redhat-releaseZdebian_versionz/etc/debian_versionZ os_releasez/etc/os-releaseZissuez /etc/issueZsysconfig_kernelz/etc/sysconfig/kernelZ proc_uptimez /proc/uptimeZ proc_loadavgz /proc/loadavgZ proc_cmdlinez /proc/cmdline� proc_versionz /proc/versionZ default_grubz/etc/default/grub�DEFAULTZ grub2_entriesZ sshd_portz/etc/ssh/sshd_configZPortZ yum_repos_dZ control_panel�syslogztail -n{0} /var/log/syslog�dmesg�messagesz tail -n{0} /var/log/messages�ls_var_cache_kcarezls -lR /var/cache/kcare/z kcare.confz /etc/sysconfig/kcare/kcare.conf�cpuinfozcat /proc/cpuinfoZ proc_vmstatz /proc/vmstatZ proc_devicesz /proc/devicesZproc_diskstatsz/proc/diskstatsZ proc_mdstatz /proc/mdstatZ proc_meminfoz /proc/meminfoZ proc_swapsz /proc/swapsZproc_filesystemsz/proc/filesystemsZ proc_mountsz /proc/mountsZproc_interruptsz/proc/interruptsz grub.confz/boot/grub/grub.conf� proc_modulesz /proc/modulesz grub2.cfgZ proc_zoneinfoz/proc/zoneinfoZls_boot_configsz:ls /etc/grub.conf /boot/grub/grub.conf /boot/grub/menu.lstZls_bootz ls -l /bootZprintenvZ dmidecoder� �sysctlz sysctl -az sysctl.confz/etc/sysctl.confz packages.listZlspciz lspci -vvZdpkg_lzdpkg -lZ apt_sourceszyum.confz /etc/yum.confZ yum_repolistz yum repolistZ libcare_logsz kcarectl.logz tail -n{0} /var/log/kcarectl.logz kdump.confz kcare-cronz/etc/cron.d/kcare-cronzcrashreporter/lszkdump/lsZ aa_statusz aa-statusZsestatusZ kprobes_listz"cat /sys/kernel/debug/kprobes/listZ sysfs_configZlsblkzlsblk -fZdfzdf -h)�loggerc C s\ t � } | �H i }xtD ]}|j| |� qW | jd|� xtD ]}|j| � q<W W d Q R X | S )Nz doctor.json)r � FIELD_SPECSr. �add_json� FILE_SPECS)r, rC �field�specr r r �prepare_doctor_report s r� c C s |s t jr4|rdnd}tjdj|�dd� | � dS tj� dkrXtjddd� | � dS t� }|dkrztjd� | � dS zzy0|j � }tjd j|�dd� t j d j|�� W nD t t tjtjfk r� } z tjd j|�� | � W Y dd}~X nX W d|j� X dS ) a� Collect and upload the doctor report via the generic uploads proxy. PUTs the archive to ``/upload/kcarectl-doctor/`` -- the same modern pipeline as ``kernel-anomaly`` -- which ePortal proxies transparently to the patch server, so no ePortal change is required. Any 2xx reply is success, including the ``REDUCED_REPORT`` ``204`` where the proxy intentionally drops the report (``send()`` returns without raising, so we do not fall back). Falls back to the legacy ``kcdoctor.sh`` flow (``fallback``) when: * ``force_fallback`` (the ``--doctor --fallback`` CLI flag) or ``config.FORCE_DOCTOR_FALLBACK`` is set -- the rollout kill-switch. ``--fallback`` is the one-off, per-invocation form; the config knob is settable per host in kcare.conf or pushed fleet-wide via the ``KC-Flag-Force-Doctor-Fallback`` feature flag. Either forces the legacy flow on every host, registered or not, before the new path is even considered; * the host is not registered, i.e. there is no auth string (OQ-4: the proxy upload requires auth, so the new path is not even attempted); * report collection failed (``prepare_doctor_report`` returned None); * the new route is unavailable on an older patch server / ePortal -- a 404 / 405 / 403, surfaced as ``HTTPError``, or a connection-level failure. ``send()`` drives ``http_utils.upload_file`` (raw ``httplib``, not ``urllib``), so transport failures surface as an ``OSError`` subclass -- ``socket.timeout``, ``ConnectionRefused``, ``socket.gaierror`` (DNS), ``ssl.SSLError`` -- or an ``httplib.HTTPException``, *not* ``URLError``; all of these mean the new path is unusable. ``upload_file`` raises only on status >= 400, so a redirect is not treated as a failure; the proxy and patch server never redirect uploads. :param fallback: legacy ``kcdoctor.sh`` runner, injected by the caller (KPT-6065) to keep this module free of an ``__init__`` import cycle. :param force_fallback: when True (``--doctor --fallback``), skip the v2 path and run ``fallback`` directly; the per-invocation form of ``config.FORCE_DOCTOR_FALLBACK``. z--fallback requestedz FORCE_DOCTOR_FALLBACK is setz.doctor: {0}; using the legacy kcdoctor.sh flowF)� print_msgNzAdoctor: host is not registered; using the legacy kcdoctor.sh flowzCdoctor: report collection failed; using the legacy kcdoctor.sh flowz doctor: report uploaded as {0}z,Please provide this filename to support: {0}zJdoctor: report upload unavailable ({0}); using the legacy kcdoctor.sh flow)r �FORCE_DOCTOR_FALLBACKr �loginfor+ r �get_http_auth_stringr� �logwarn�sendr � print_wrapperr r r � HTTPException�socket�error�remove_archive)�fallback�force_fallback�srcr, � upload_name�errr r r �send_doctor_report- s4 ) r� )r r r r )r r r )r r )F)F)Fr/ r� ry r9 rP r� �kcarectlr r r r r r r r r Zkcarectl.process_utilsr Z kcarectl.py23r r r r^ rv r� r� r� rz r� r� r� r� r� r� � DataPackager �objectr$ r2 r6 r> r@ rA rH rT r] r_ �cachedrd rh rj rt rw r{ r� r� r� r� r� r� r� r� r� r� r+ r� � catch_errorsr� r� r� r r r r �<module> s� , #