Python 3.5: ‘ERROR! A worker was found in a dead state’ showing up a lot more often

Summary

In several collection CIs, @briantist and me noticed that ERROR! A worker was found in a dead state started showing up a lot more often since < 24 h, but only with Python 3.5.

In the CIs I’ve observed this in, ansible-test is invoked with ansible-test integration -v --color --retry-on-error --continue-on-error --diff --python 3.5 --docker and running itself under Python 3.8.

Issue Type

Bug Report

Component Name

core / ansible-test

Ansible Version

stable-2.11
devel

Configuration

OS / Environment

Steps to Reproduce

Expected Results

Actual Results

A run from felixfontein/ansible-tools:

2021-04-06T05:13:49.6948640Z ##[group]Run actions/setup-python@v2
2021-04-06T05:13:49.6949245Z with:
2021-04-06T05:13:49.6949727Z   python-version: 3.8
2021-04-06T05:13:49.6950709Z   token: ***
2021-04-06T05:13:49.6951172Z ##[endgroup]
2021-04-06T05:13:49.7678082Z Successfully setup CPython (3.8.8)
2021-04-06T05:13:49.7747685Z ##[group]Run pip install https://github.com/ansible/ansible/archive/stable-2.11.tar.gz --disable-pip-version-check
2021-04-06T05:13:49.7749339Z �[36;1mpip install https://github.com/ansible/ansible/archive/stable-2.11.tar.gz --disable-pip-version-check�[0m
2021-04-06T05:13:49.7796866Z shell: /usr/bin/bash -e {0}
2021-04-06T05:13:49.7797401Z env:
2021-04-06T05:13:49.7798043Z   pythonLocation: /opt/hostedtoolcache/Python/3.8.8/x64
2021-04-06T05:13:49.7799087Z   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.8/x64/lib
2021-04-06T05:13:49.7799732Z ##[endgroup]
2021-04-06T05:13:50.3521030Z Collecting https://github.com/ansible/ansible/archive/stable-2.11.tar.gz
2021-04-06T05:13:50.3939817Z   Downloading https://github.com/ansible/ansible/archive/stable-2.11.tar.gz (5.9 MB)
2021-04-06T05:13:53.1193924Z Collecting jinja2
2021-04-06T05:13:53.1433511Z   Downloading Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
2021-04-06T05:13:53.2652915Z Collecting PyYAML
2021-04-06T05:13:53.2697213Z   Downloading PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl (662 kB)
2021-04-06T05:13:53.8364433Z Collecting cryptography
2021-04-06T05:13:53.8426076Z   Downloading cryptography-3.4.7-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB)
2021-04-06T05:13:54.0112801Z Collecting packaging
2021-04-06T05:13:54.0154167Z   Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
2021-04-06T05:13:54.0489830Z Collecting resolvelib<0.6.0,>=0.5.3
2021-04-06T05:13:54.0530164Z   Downloading resolvelib-0.5.4-py2.py3-none-any.whl (12 kB)
2021-04-06T05:13:54.9601955Z Collecting cffi>=1.12
2021-04-06T05:13:54.9613930Z   Downloading cffi-1.14.5-cp38-cp38-manylinux1_x86_64.whl (411 kB)
2021-04-06T05:13:54.9615051Z Collecting pycparser
2021-04-06T05:13:54.9616413Z   Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
2021-04-06T05:13:54.9618703Z Collecting MarkupSafe>=0.23
2021-04-06T05:13:54.9621238Z   Downloading MarkupSafe-1.1.1-cp38-cp38-manylinux2010_x86_64.whl (32 kB)
2021-04-06T05:13:54.9624670Z Collecting pyparsing>=2.0.2
2021-04-06T05:13:54.9625789Z   Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
2021-04-06T05:13:54.9627021Z Using legacy 'setup.py install' for ansible-core, since package 'wheel' is not installed.
2021-04-06T05:13:54.9628563Z Installing collected packages: pycparser, pyparsing, MarkupSafe, cffi, resolvelib, PyYAML, packaging, jinja2, cryptography, ansible-core
2021-04-06T05:13:55.5541871Z     Running setup.py install for ansible-core: started
2021-04-06T05:13:57.5371824Z     Running setup.py install for ansible-core: finished with status 'done'
2021-04-06T05:13:57.5845493Z Successfully installed MarkupSafe-1.1.1 PyYAML-5.4.1 ansible-core-2.11.0rc1.post0 cffi-1.14.5 cryptography-3.4.7 jinja2-2.11.3 packaging-20.9 pycparser-2.20 pyparsing-2.4.7 resolvelib-0.5.4
2021-04-06T05:13:57.8352266Z ##[group]Run git clone --depth=1 --single-branch https://github.com/ansible-collections/community.general.git ./ansible_collections/community/general
2021-04-06T05:13:57.8354189Z �[36;1mgit clone --depth=1 --single-branch https://github.com/ansible-collections/community.general.git ./ansible_collections/community/general�[0m
2021-04-06T05:13:57.8399226Z shell: /usr/bin/bash -e {0}
2021-04-06T05:13:57.8399628Z env:
2021-04-06T05:13:57.8400399Z   pythonLocation: /opt/hostedtoolcache/Python/3.8.8/x64
2021-04-06T05:13:57.8401183Z   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.8/x64/lib
2021-04-06T05:13:57.8401733Z ##[endgroup]
2021-04-06T05:13:57.8489367Z Cloning into './ansible_collections/community/general'...
2021-04-06T05:13:58.4851582Z ##[group]Run ansible-test integration -v --color --retry-on-error --continue-on-error --diff --python 3.5 --docker --coverage
2021-04-06T05:13:58.4853106Z �[36;1mansible-test integration -v --color --retry-on-error --continue-on-error --diff --python 3.5 --docker --coverage�[0m
2021-04-06T05:13:58.4901623Z shell: /usr/bin/bash -e {0}
2021-04-06T05:13:58.4902051Z env:
2021-04-06T05:13:58.4902616Z   pythonLocation: /opt/hostedtoolcache/Python/3.8.8/x64
2021-04-06T05:13:58.4903359Z   LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.8.8/x64/lib
2021-04-06T05:13:58.4903918Z ##[endgroup]
2021-04-06T05:13:58.7271498Z �[32mFalling back to tests in "tests/integration/targets/" because "roles/test/" was not found.�[0m
2021-04-06T05:13:58.7302044Z �[32mRun command: docker images quay.io/ansible/default-test-container:3.1.0 --format '{{json .}}'�[0m
2021-04-06T05:13:59.0199281Z �[32mRun command: docker pull quay.io/ansible/default-test-container:3.1.0�[0m
2021-04-06T05:13:59.2784910Z 3.1.0: Pulling from ansible/default-test-container
2021-04-06T05:13:59.2810454Z f22ccc0b8772: Pulling fs layer
[...]
2021-04-06T05:14:30.2046776Z b799b5892cf0: Pull complete
2021-04-06T05:14:30.2099406Z Digest: sha256:a88f080db38e6aefc446f45ebfcc861f34e71d278b97c47fdc653f63062b8f12
2021-04-06T05:14:30.2118420Z Status: Downloaded newer image for quay.io/ansible/default-test-container:3.1.0
2021-04-06T05:14:30.2161129Z quay.io/ansible/default-test-container:3.1.0
2021-04-06T05:14:30.2400654Z �[32mScanning collection root: /home/runner/work/ansible-tools/ansible-tools/ansible_collections�[0m
2021-04-06T05:14:30.2402148Z �[32mRun command: git ls-files -z --cached --others --exclude-standard�[0m
2021-04-06T05:14:30.2573188Z �[32mRun command: git ls-files -z --deleted�[0m
2021-04-06T05:14:30.2772983Z �[32mRun command: git submodule status --recursive�[0m
2021-04-06T05:14:30.3141641Z �[32mIncluding collection: community.general (2745 files)�[0m
2021-04-06T05:14:30.3143310Z �[32mIncluding collection: felixfontein.tools (60 files)�[0m
2021-04-06T05:14:30.3282673Z �[32mCreating a payload archive containing 3614 files...�[0m
2021-04-06T05:14:31.2530291Z �[32mCreated a 3671942 byte payload archive containing 3614 files in 0 seconds.�[0m
2021-04-06T05:14:31.2535617Z �[32mAssuming Docker is available on localhost.�[0m
2021-04-06T05:14:31.2539218Z �[32mRun command: docker run --detach --volume /sys/fs/cgroup:/sys/fs/cgroup:ro --privileged=false --security-opt seccomp=unconfined --volume /var/run/docker.sock:/var/run/docker.sock quay.io/ansible/default-test-container:3.1.0�[0m
2021-04-06T05:14:34.0402998Z �[32mRun command: ssh-keygen -m PEM -q -t rsa -N '' -f /home/runner/.ansible/test/id_rsa�[0m
2021-04-06T05:14:34.6922757Z �[32mRun command: docker exec -i 56324fe634d11b359151a72528ec6fa065bdd45405216bbbed205a08ac59e959 /bin/sh�[0m
2021-04-06T05:14:34.8058311Z �[32mRun command: docker exec -i 56324fe634d11b359151a72528ec6fa065bdd45405216bbbed205a08ac59e959 dd of=/root/test.tgz bs=65536�[0m
2021-04-06T05:14:34.9300143Z �[32mRun command: docker exec 56324fe634d11b359151a72528ec6fa065bdd45405216bbbed205a08ac59e959 tar oxzf /root/test.tgz -C /root�[0m
2021-04-06T05:14:35.2485284Z �[32mRun command: docker exec 56324fe634d11b359151a72528ec6fa065bdd45405216bbbed205a08ac59e959 /usr/bin/env ANSIBLE_TEST_CONTENT_ROOT=/root/ansible_collections/felixfontein/tools LC_ALL=en_US.UTF-8 /usr/bin/python3.5 /root/ansible/bin/ansible-test integration -v --retry-on-error --continue-on-error --diff --python 3.5 --coverage --metadata tests/output/.tmp/metadata-l3bovzqn.json --truncate 0 --redact --color yes --requirements --coverage-label docker-default --allow-destructive�[0m
2021-04-06T05:14:35.6415061Z �[32mFalling back to tests in "tests/integration/targets/" because "roles/test/" was not found.�[0m
2021-04-06T05:14:35.6432923Z �[32mRun command: /usr/bin/python3.5 /root/ansible/test/lib/ansible_test/_data/quiet_pip.py install --disable-pip-version-check -r /root/ansible/test/lib/ansible_test/_data/requirements/ansible-test.txt�[0m
2021-04-06T05:14:36.3863248Z �[32mRun command: /usr/bin/python3.5 /root/ansible/test/lib/ansible_test/_data/quiet_pip.py install --disable-pip-version-check setuptools -c /root/ansible/test/lib/ansible_test/_data/requirements/constraints.txt�[0m
2021-04-06T05:14:37.1962822Z �[32mRun command: /usr/bin/python3.5 -c 'import setuptools; print(setuptools.__version__)'�[0m
2021-04-06T05:14:37.4053429Z �[32mRun command: /usr/bin/python3.5 /root/ansible/test/lib/ansible_test/_data/sslcheck.py�[0m
2021-04-06T05:14:37.4528938Z �[32mDetected OpenSSL version 1.1.1 under Python 3.5.�[0m
2021-04-06T05:14:37.4534211Z �[32mRun command: /usr/bin/python3.5 /root/ansible/test/lib/ansible_test/_data/quiet_pip.py install --disable-pip-version-check 'cryptography < 3.4' -c /root/ansible/test/lib/ansible_test/_data/cryptography-constraints.txt�[0m
2021-04-06T05:14:38.1937829Z �[32mRun command: /usr/bin/python3.5 /root/ansible/test/lib/ansible_test/_data/quiet_pip.py install --disable-pip-version-check -r /root/ansible/test/lib/ansible_test/_data/requirements/integration.txt coverage -c /root/ansible/test/lib/ansible_test/_data/requirements/constraints.txt�[0m
2021-04-06T05:14:39.0080553Z �[32mRun command: /usr/bin/python3.5 /root/ansible/test/lib/ansible_test/_data/quiet_pip.py check --disable-pip-version-check�[0m
2021-04-06T05:14:39.5189977Z �[32mRun command: /usr/bin/python3.5 /root/ansible/test/lib/ansible_test/_data/yamlcheck.py�[0m
2021-04-06T05:14:40.1024105Z Running filter_dict integration test role
2021-04-06T05:14:40.1145356Z �[32mInjecting "/tmp/python-9p9xd2z3-ansible/python" as a execv wrapper for the "/usr/bin/python3.5" interpreter.�[0m
2021-04-06T05:14:40.1149207Z �[32mRun command: ansible-playbook filter_dict-y0kg6mvg.yml -i inventory --diff -v�[0m
2021-04-06T05:14:41.2119324Z �[0;34mUsing /root/ansible_collections/felixfontein/tools/tests/output/.tmp/integration/filter_dict-drb3a1sm-ÅÑŚÌβŁÈ/tests/integration/integration.cfg as config file�[0m
2021-04-06T05:14:41.6551038Z /root/ansible/lib/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release of cryptography. Please upgrade your Python.
2021-04-06T05:14:41.6552634Z   from cryptography.exceptions import InvalidSignature
2021-04-06T05:14:41.6554204Z �[1;35m[WARNING]: running playbook inside collection felixfontein.tools�[0m
2021-04-06T05:14:41.8362461Z 
2021-04-06T05:14:41.8363128Z PLAY [testhost] ****************************************************************
2021-04-06T05:14:41.8781439Z 
2021-04-06T05:14:41.8782271Z TASK [Gathering Facts] *********************************************************
2021-04-06T05:14:43.0374223Z �[0;32mok: [testhost]�[0m
2021-04-06T05:14:43.0789754Z 
2021-04-06T05:14:43.0791127Z TASK [filter_dict : Test list_to_dict filter] **********************************
2021-04-06T05:14:43.4940300Z �[0;32mok: [testhost] => {�[0m
2021-04-06T05:14:43.4940995Z �[0;32m    "changed": false,�[0m
2021-04-06T05:14:43.4941998Z �[0;32m    "msg": "All assertions passed"�[0m
2021-04-06T05:14:43.4942559Z �[0;32m}�[0m
2021-04-06T05:14:43.5134985Z 
2021-04-06T05:14:43.5135725Z TASK [filter_dict : Test dict filter] ******************************************
2021-04-06T05:14:43.9143949Z �[0;32mok: [testhost] => {�[0m
2021-04-06T05:14:43.9144634Z �[0;32m    "changed": false,�[0m
2021-04-06T05:14:43.9145300Z �[0;32m    "msg": "All assertions passed"�[0m
2021-04-06T05:14:43.9145856Z �[0;32m}�[0m
2021-04-06T05:14:43.9692771Z 
2021-04-06T05:14:43.9693487Z PLAY RECAP *********************************************************************
2021-04-06T05:14:43.9694972Z �[0;32mtesthost�[0m                   : �[0;32mok=3   �[0m changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
2021-04-06T05:14:43.9695532Z 
2021-04-06T05:14:44.6182759Z Running filter_domain_suffix integration test role
2021-04-06T05:14:44.6206329Z �[32mRun command: ansible-playbook filter_domain_suffix-mg3h6oek.yml -i inventory --diff -v�[0m
2021-04-06T05:14:45.5306057Z �[0;34mUsing /root/ansible_collections/felixfontein/tools/tests/output/.tmp/integration/filter_domain_suffix-nppzzuha-ÅÑŚÌβŁÈ/tests/integration/integration.cfg as config file�[0m
2021-04-06T05:14:45.9180313Z /root/ansible/lib/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release of cryptography. Please upgrade your Python.
2021-04-06T05:14:45.9181828Z   from cryptography.exceptions import InvalidSignature
2021-04-06T05:14:45.9183590Z �[1;35m[WARNING]: running playbook inside collection felixfontein.tools�[0m
2021-04-06T05:14:46.0064933Z 
2021-04-06T05:14:46.0065754Z PLAY [testhost] ****************************************************************
2021-04-06T05:14:46.0250739Z 
2021-04-06T05:14:46.0251719Z TASK [Gathering Facts] *********************************************************
2021-04-06T05:14:47.9723817Z �[0;32mok: [testhost]�[0m
2021-04-06T05:14:47.9737138Z 
2021-04-06T05:14:47.9738013Z TASK [filter_domain_suffix : Test dns_zone filter] *****************************
2021-04-06T05:14:47.9739170Z �[0;32mok: [testhost] => {�[0m
2021-04-06T05:14:47.9740135Z �[0;32m    "changed": false,�[0m
2021-04-06T05:14:47.9741174Z �[0;32m    "msg": "All assertions passed"�[0m
2021-04-06T05:14:47.9742073Z �[0;32m}�[0m
2021-04-06T05:14:47.9742584Z 
2021-04-06T05:14:47.9743343Z TASK [filter_domain_suffix : Test dns_zone_prefix filter] **********************
2021-04-06T05:14:48.2293291Z �[0;32mok: [testhost] => {�[0m
2021-04-06T05:14:48.2293949Z �[0;32m    "changed": false,�[0m
2021-04-06T05:14:48.2294719Z �[0;32m    "msg": "All assertions passed"�[0m
2021-04-06T05:14:48.2295288Z �[0;32m}�[0m
2021-04-06T05:14:48.2496559Z 
2021-04-06T05:14:48.2497247Z TASK [filter_domain_suffix : Test get_domain_suffix filter] ********************
2021-04-06T05:14:48.7374839Z �[0;32mok: [testhost] => {�[0m
2021-04-06T05:14:48.7375466Z �[0;32m    "changed": false,�[0m
2021-04-06T05:14:48.7376098Z �[0;32m    "msg": "All assertions passed"�[0m
2021-04-06T05:14:48.7376855Z �[0;32m}�[0m
2021-04-06T05:14:48.7567646Z 
2021-04-06T05:14:48.7568414Z TASK [filter_domain_suffix : Test remove_domain_suffix filter] *****************
2021-04-06T05:14:49.2268037Z �[0;32mok: [testhost] => {�[0m
2021-04-06T05:14:49.2268712Z �[0;32m    "changed": false,�[0m
2021-04-06T05:14:49.2269471Z �[0;32m    "msg": "All assertions passed"�[0m
2021-04-06T05:14:49.2270026Z �[0;32m}�[0m
2021-04-06T05:14:49.2463827Z 
2021-04-06T05:14:49.2464469Z TASK [filter_domain_suffix : Test registrable_domain filter] *******************
2021-04-06T05:14:49.7611216Z �[0;32mok: [testhost] => {�[0m
2021-04-06T05:14:49.7611967Z �[0;32m    "changed": false,�[0m
2021-04-06T05:14:49.7612631Z �[0;32m    "msg": "All assertions passed"�[0m
2021-04-06T05:14:49.7613199Z �[0;32m}�[0m
2021-04-06T05:14:49.8152567Z 
2021-04-06T05:14:49.8153305Z PLAY RECAP *********************************************************************
2021-04-06T05:14:49.8161623Z �[0;32mtesthost�[0m                   : �[0;32mok=6   �[0m changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
2021-04-06T05:14:49.8163251Z 
2021-04-06T05:14:50.4611385Z Running filter_path_join_shim integration test role
2021-04-06T05:14:50.4634889Z �[32mRun command: ansible-playbook filter_path_join_shim-n0es6gwd.yml -i inventory --diff -v�[0m
2021-04-06T05:14:51.4093789Z �[0;34mUsing /root/ansible_collections/felixfontein/tools/tests/output/.tmp/integration/filter_path_join_shim-06p9xgfj-ÅÑŚÌβŁÈ/tests/integration/integration.cfg as config file�[0m
2021-04-06T05:14:51.8222028Z /root/ansible/lib/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release of cryptography. Please upgrade your Python.
2021-04-06T05:14:51.8223585Z   from cryptography.exceptions import InvalidSignature
2021-04-06T05:14:51.8225626Z �[1;35m[WARNING]: running playbook inside collection felixfontein.tools�[0m
2021-04-06T05:14:51.9091083Z 
2021-04-06T05:14:51.9091902Z PLAY [testhost] ****************************************************************
2021-04-06T05:14:51.9390767Z 
2021-04-06T05:14:51.9391508Z TASK [Gathering Facts] *********************************************************
2021-04-06T05:14:53.1231842Z �[0;32mok: [testhost]�[0m
2021-04-06T05:14:53.1644200Z 
2021-04-06T05:14:53.1645785Z TASK [filter_path_join_shim : Test path_join filter] ***************************
2021-04-06T05:14:53.3062185Z �[0;32mok: [testhost] => {�[0m
2021-04-06T05:14:53.3062847Z �[0;32m    "changed": false,�[0m
2021-04-06T05:14:53.3063607Z �[0;32m    "msg": "All assertions passed"�[0m
2021-04-06T05:14:53.3064161Z �[0;32m}�[0m
2021-04-06T05:14:53.3620476Z 
2021-04-06T05:14:53.3622373Z PLAY RECAP *********************************************************************
2021-04-06T05:14:53.3623877Z �[0;32mtesthost�[0m                   : �[0;32mok=2   �[0m changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
2021-04-06T05:14:53.3624738Z 
2021-04-06T05:14:53.9806394Z Running lookup_dependent integration test role
2021-04-06T05:14:53.9852850Z �[32mRun command: ansible-playbook lookup_dependent-vjzxhqmn.yml -i inventory --diff -v�[0m
2021-04-06T05:14:54.8912378Z �[0;34mUsing /root/ansible_collections/felixfontein/tools/tests/output/.tmp/integration/lookup_dependent-ks276ivk-ÅÑŚÌβŁÈ/tests/integration/integration.cfg as config file�[0m
2021-04-06T05:14:55.2949931Z /root/ansible/lib/ansible/parsing/vault/__init__.py:44: CryptographyDeprecationWarning: Python 3.5 support will be dropped in the next release of cryptography. Please upgrade your Python.
2021-04-06T05:14:55.2951594Z   from cryptography.exceptions import InvalidSignature
2021-04-06T05:14:55.2953161Z �[1;35m[WARNING]: running playbook inside collection felixfontein.tools�[0m
2021-04-06T05:14:55.3926942Z 
2021-04-06T05:14:55.3932048Z PLAY [testhost] ****************************************************************
2021-04-06T05:14:55.4154241Z 
2021-04-06T05:14:55.4155440Z TASK [Gathering Facts] *********************************************************
2021-04-06T05:14:56.5738357Z �[0;32mok: [testhost]�[0m
2021-04-06T05:14:56.6180053Z 
2021-04-06T05:14:56.6181844Z TASK [lookup_dependent : Test 1: using with with_dependent] ********************
2021-04-06T05:14:56.8097679Z �[0;32mok: [testhost] => (item={0: 1, 1: 4, 2: 41}) => {�[0m
2021-04-06T05:14:56.8098356Z �[0;32m    "msg": "1 4 41"�[0m
2021-04-06T05:14:56.8098836Z �[0;32m}�[0m
2021-04-06T05:14:56.8185068Z �[0;32mok: [testhost] => (item={0: 1, 1: 7, 2: 71}) => {�[0m
2021-04-06T05:14:56.8185700Z �[0;32m    "msg": "1 7 71"�[0m
2021-04-06T05:14:56.8186194Z �[0;32m}�[0m
2021-04-06T05:14:56.8209276Z �[0;32mok: [testhost] => (item={0: 2, 1: 5, 2: 52}) => {�[0m
2021-04-06T05:14:56.8209986Z �[0;32m    "msg": "2 5 52"�[0m
2021-04-06T05:14:56.8210507Z �[0;32m}�[0m
2021-04-06T05:14:56.8232524Z �[0;32mok: [testhost] => (item={0: 2, 1: 8, 2: 82}) => {�[0m
2021-04-06T05:14:56.8233198Z �[0;32m    "msg": "2 8 82"�[0m
2021-04-06T05:14:56.8233698Z �[0;32m}�[0m
2021-04-06T05:14:56.8253803Z �[0;31mERROR! A worker was found in a dead state�[0m
2021-04-06T05:14:57.4619523Z �[35mWARNING: Retrying test target "lookup_dependent" with maximum verbosity.�[0m
(there is no more output with higher verbosity, so I'm skipping the remainder)

3 thoughts on “Python 3.5: ‘ERROR! A worker was found in a dead state’ showing up a lot more often

  1. This was introduced in #74099.

    I can reproduce the issue by, for example, ansible-test integration handlers --docker --python 3.5 but I guess any playbook execution on Python 3.5 is a reproducer.

    I have not looked into why the issue manifests itself only on Python 3.5 but perhaps the original change should have been:

    diff --git a/lib/ansible/executor/process/worker.py b/lib/ansible/executor/process/worker.py
    index 983df45bca..510928c0ed 100644
    --- a/lib/ansible/executor/process/worker.py
    +++ b/lib/ansible/executor/process/worker.py
    @@ -145,8 +145,7 @@ class WorkerProcess(multiprocessing_context.Process):
                 # TODO: Evaluate overhauling ``Display`` to not write directly to stdout
                 # and evaluate migrating away from the ``fork`` multiprocessing start method.
                 if sys.version_info[0] >= 3:
    -                sys.stdout = os.devnull
    -                sys.stderr = os.devnull
    +                sys.stdout = sys.stderr = open(os.devnull, 'w')
    
         def _run(self):
             '''

    so assigning a file-like object to sys.stdout/sys.stderr instead of a file path (str).

    The above diff fixes this issue, I can’t comment on whether it still fixes the original issue.

    cc @sivel

  2. OK, so the reason why this is only an issue on Python 3.5 is because of this change in CPython:
    python/cpython@069b8d2#diff-95e01a1db1003c313b93ab726870bc6ddaa916f28479bb17fea2331fb8b5ac7f

    Since Python 3.6 AttributeError is ignored when attempting sys.stdout.flush() and sys.stderr.flush() in Process._bootstrap().

    So on Python 3.5 because we have sys.stdout = os.devnull which is sys.stdout = '/dev/null' on my machine, sys.stdout.flush() in Process._bootstrap raises AttributeError which kills the worker. On Python 3.6+ the AttributeError is ignored.