DOCKER_HOST=ssh://example.com works with docker 19.03.8 but not docker-compose 1.26

Description of the issue

Seems like same problem in #6655 where ssh:// protocol does not work with docker-compose, even though it works with docker. That bug was closed as fixed in 1.26 but I still see the problem.

Context information (for bug reports)

Output of docker-compose version

$ docker-compose --version
docker-compose version 1.25.5, build 8a1c60f6
$ docker-compose-1.26.0 --version
docker-compose version 1.26.0, build d4451659

Output of docker version

$ docker version
Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b
 Built:             Wed Mar 11 01:21:11 2020
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b
  Built:            Wed Mar 11 01:29:16 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683
 Kubernetes:
  Version:          v1.16.6-beta.0
  StackAPI:         v1beta2

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

$ docker-compose config
services:
  hello:
    image: hello-world
version: '3.7'

Steps to reproduce the issue

  1. Make sure you can ssh to the docker server with this in ~/.ssh/config.
Host 192.168.1.50 mini.local foobar.example.com
 User jamshid
 IdentityFile ~/.ssh/id_mini
$ ssh mini.local
Last login: Fri Jun  5 17:39:51 2020 from fe...

The default interactive shell is now zsh.
To update your account to use zsh, please run `chsh -s /bin/zsh`.
For more details, please visit https://support.apple.com/kb/HT208050.
mini:~ jamshid$ 
  1. Make sure docker itself works with that DOCKER_HOST:
$ DOCKER_HOST=ssh://mini.local docker ps
CONTAINER ID        IMAGE                                            COMMAND                  CREATED             STATUS                  PORTS                                                                              NAMES
47d5447cfa6e        a4a90686c3db                                     "/config-and-run.sh …"   5 days ago          Up 5 days                                                                                                  suspicious_vaughan
...
  1. docker-compose ps does not work even though docker does.

Observed result

paramiko.ssh_exception.SSHException: No authentication methods available

Expected result

I can access other (linux) docker servers via DOCKER_HOST=ssh://servername, not sure why this isn’t working. Also I’d swear I had this working yesterday before I applied the macOS supplemental update (10.15.5 (19F101)).

Stacktrace / full error message

$ docker-compose --verbose ps
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.utils.config.find_config_file: Trying paths: ['/Users/jamshid/.docker/config.json', '/Users/jamshid/.dockercfg']
docker.utils.config.find_config_file: Found file at path: /Users/jamshid/.docker/config.json
docker.auth.load_config: Found 'auths' section
docker.auth.parse_auth: Auth data for https://index.docker.io/v1/ is absent. Client might be using a credentials store instead.
docker.auth.load_config: Found 'credsStore' section
paramiko.transport._log: starting thread (client mode): 0xa8e0090
paramiko.transport._log: Local version/idstring: SSH-2.0-paramiko_2.7.1
paramiko.transport._log: Remote version/idstring: SSH-2.0-OpenSSH_8.1
paramiko.transport._log: Connected (version 2.0, client OpenSSH_8.1)
paramiko.transport._log: kex algos:['curve25519-sha256', 'curve25519-sha256@libssh.org', 'ecdh-sha2-nistp256', 'ecdh-sha2-nistp384', 'ecdh-sha2-nistp521', 'diffie-hellman-group-exchange-sha256', 'diffie-hellman-group16-sha512', 'diffie-hellman-group18-sha512', 'diffie-hellman-group14-sha256', 'diffie-hellman-group14-sha1'] server key:['rsa-sha2-512', 'rsa-sha2-256', 'ssh-rsa', 'ecdsa-sha2-nistp256', 'ssh-ed25519'] client encrypt:['chacha20-poly1305@openssh.com', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com'] server encrypt:['chacha20-poly1305@openssh.com', 'aes128-ctr', 'aes192-ctr', 'aes256-ctr', 'aes128-gcm@openssh.com', 'aes256-gcm@openssh.com'] client mac:['umac-64-etm@openssh.com', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] server mac:['umac-64-etm@openssh.com', 'umac-128-etm@openssh.com', 'hmac-sha2-256-etm@openssh.com', 'hmac-sha2-512-etm@openssh.com', 'hmac-sha1-etm@openssh.com', 'umac-64@openssh.com', 'umac-128@openssh.com', 'hmac-sha2-256', 'hmac-sha2-512', 'hmac-sha1'] client compress:['none', 'zlib@openssh.com'] server compress:['none', 'zlib@openssh.com'] client lang:[''] server lang:[''] kex follows?False
paramiko.transport._log: Kex agreed: curve25519-sha256@libssh.org
paramiko.transport._log: HostKey agreed: ecdsa-sha2-nistp256
paramiko.transport._log: Cipher agreed: aes128-ctr
paramiko.transport._log: MAC agreed: hmac-sha2-256
paramiko.transport._log: Compression agreed: none
paramiko.transport._log: kex engine KexCurve25519 specified hash_algo <built-in function openssl_sha256>
paramiko.transport._log: Switch to new keys ...
Traceback (most recent call last):
  File "docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 72, in main
  File "compose/cli/main.py", line 125, in perform_command
  File "compose/cli/command.py", line 65, in project_from_options
  File "compose/cli/command.py", line 149, in get_project
  File "compose/cli/command.py", line 120, in get_client
  File "compose/cli/docker_client.py", line 127, in docker_client
  File "site-packages/docker/api/client.py", line 166, in __init__
  File "site-packages/docker/transport/sshconn.py", line 84, in __init__
  File "site-packages/docker/transport/sshconn.py", line 94, in _connect
  File "site-packages/paramiko/client.py", line 446, in connect
  File "site-packages/paramiko/client.py", line 765, in _auth
paramiko.ssh_exception.SSHException: No authentication methods available
[12219] Failed to execute script docker-compose
$ docker-compose-1.26.0 --verbose ps
compose.config.config.find: Using configuration files: ./docker-compose.yml
docker.utils.config.find_config_file: Trying paths: ['/Users/jamshid/.docker/config.json', '/Users/jamshid/.dockercfg']
docker.utils.config.find_config_file: Found file at path: /Users/jamshid/.docker/config.json
docker.auth.load_config: Found 'auths' section
docker.auth.parse_auth: Auth data for https://index.docker.io/v1/ is absent. Client might be using a credentials store instead.
docker.auth.load_config: Found 'credsStore' section
Traceback (most recent call last):
  File "docker-compose", line 6, in <module>
  File "compose/cli/main.py", line 72, in main
  File "compose/cli/main.py", line 125, in perform_command
  File "compose/cli/command.py", line 76, in project_from_options
  File "compose/cli/command.py", line 142, in get_project
  File "compose/cli/docker_client.py", line 47, in get_client
  File "compose/cli/docker_client.py", line 169, in docker_client
  File "site-packages/docker/api/client.py", line 166, in __init__
  File "site-packages/docker/transport/sshconn.py", line 111, in __init__
  File "site-packages/docker/transport/sshconn.py", line 119, in _connect
  File "site-packages/paramiko/client.py", line 446, in connect
  File "site-packages/paramiko/client.py", line 765, in _auth
paramiko.ssh_exception.SSHException: No authentication methods available
[12234] Failed to execute script docker-compose

Additional information

Docker for Desktop updated to latest which is 1.25 but I also tried downloading the latest docker-compose 1.26.0 binary because the other bug implied this error was fixed. Unfortunately it is not.

My ssh key is RSA generated June 2017, maybe that’s related.

-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAyzAARS1tfwCG/VpIti/37WaYssU7esuMwUX08v9g1r4TZWd9
...
POVazhiwWEvJfu8HbVmZaI7F58r2ypPOGIK12DQ3lfRRVB2Y8sNmDQ==
-----END RSA PRIVATE KEY-----
ssh-rsa AAAAB3NzaC1...FCu41l jamshid@mini.local

1 possible answer(s) on “DOCKER_HOST=ssh://example.com works with docker 19.03.8 but not docker-compose 1.26

  1. Can confirm the comment quoted has not been resolved as of:
    OS: Windows 10 Pro Version 10.0.18363 Build 18363
    Docker: Docker version 19.03.13, build 4484c46d9d
    Docker-compose: docker-compose version 1.27.4, build 4052419

    I’m using ssh-agent on windows and it works well with docker.
    I had the same error message “paramiko.ssh_exception.SSHException: No authentication methods available” and then I renamed my ssh key file id_rsa in .ssh folder and I got this message :

    PS C:\Users\myuser\> docker-compose.exe -c remote_ssh_context ps
    Traceback (most recent call last):
      File "docker-compose", line 6, in <module>
      File "compose\cli\main.py", line 72, in main
      File "compose\cli\main.py", line 125, in perform_command
      File "compose\cli\command.py", line 76, in project_from_options
      File "compose\cli\command.py", line 142, in get_project
      File "compose\cli\docker_client.py", line 47, in get_client
      File "compose\cli\docker_client.py", line 174, in docker_client
      File "site-packages\docker\api\client.py", line 166, in __init__
      File "site-packages\docker\transport\sshconn.py", line 111, in __init__
      File "site-packages\docker\transport\sshconn.py", line 119, in _connect
      File "site-packages\paramiko\client.py", line 446, in connect
      File "site-packages\paramiko\client.py", line 764, in _auth
      File "site-packages\paramiko\client.py", line 735, in _auth
      File "site-packages\paramiko\client.py", line 586, in _key_from_filepath
      File "site-packages\paramiko\pkey.py", line 235, in from_private_key_file
      File "site-packages\paramiko\rsakey.py", line 55, in __init__
      File "site-packages\paramiko\rsakey.py", line 175, in _from_private_key_file
      File "site-packages\paramiko\pkey.py", line 308, in _read_private_key_file
      File "site-packages\paramiko\pkey.py", line 334, in _read_private_key
      File "site-packages\paramiko\pkey.py", line 386, in _read_private_key_pem
    paramiko.ssh_exception.PasswordRequiredException: Private key file is encrypted
    [27312] Failed to execute script docker-compose
    

    I did this tutorial to make docker context work with ssh Remote SSH & Docker in VS code
    Windows 10 Pro 1909
    Docker Desktop 2.3.0.4
    docker-compose version 1.26.2, build eefe0d3
    Docker version 19.03.12, build 48a66213fe

    Edit: potential solution

    Upon further investigation I found that paramiko as of commit 432819766 on windows checks only for the pageant agent. Following the instructions from this post to convert an openssh key to the pageant format, and subsequently adding that converted (ppk) key to pageant I was able to get docker-compose to properly connect to the my remote docker engine over ssh.