I recently ran into trouble pulling from my git repository on my linux machine:
$ git pull Enter passphrase for key '/home/<user>/.ssh/<name_of_my_key>': Connection to bitbucket.org closed by remote host. fatal: Konnte nicht vom Remote-Repository lesen. Bitte stellen Sie sicher, dass die korrekten Zugriffsberechtigungen bestehen und das Repository existiert.
Which was kind of confusing, because I could remember it working just a while ago without knowingly changing anything related to git. After suspecting my ssh keypairs or IPv4 / IPv6 issues, I narrowed it down to a problem with openssh, which of course is used under the hood of git pull.
As the help pages of Bitbucket and Github suggest, you can test your connection to your repo server using:ssh -Tv bitbucket.org
If you have the same problem as I did, the output should look something like this:
$ ssh -Tv bitbucket.org OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n 7 Dec 2017 debug1: Reading configuration data /home/<user>/.ssh/config debug1: /home/<user>/.ssh/config line 12: Applying options for bitbucket.org debug1: /home/<user>/.ssh/config line 15: Applying options for bitbucket.org debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug1: Connecting to bitbucket.org [2406:da00:ff00::22c0:3470] port 22. debug1: Connection established. debug1: key_load_public: No such file or directory debug1: identity file /home/<user>/.ssh/<name_of_my_key> type -1 debug1: key_load_public: No such file or directory debug1: identity file /home/<user>/.ssh/<name_of_my_key>-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3 debug1: Remote protocol version 2.0, remote software version conker_405f5f426f-dirty app-149 debug1: no match: conker_405f5f426f-dirty app-149 debug1: Authenticating to bitbucket.org:22 as 'git' debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug1: kex: algorithm: curve25519-sha256@libssh.org debug1: kex: host key algorithm: ssh-rsa debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: ssh-rsa SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A debug1: Host 'bitbucket.org' is known and matches the RSA host key. debug1: Found key in /home/<user>/.ssh/known_hosts:1 debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug1: SSH2_MSG_NEWKEYS received debug1: rekey after 134217728 blocks debug1: SSH2_MSG_SERVICE_ACCEPT received debug1: Authentications that can continue: publickey debug1: Next authentication method: publickey debug1: Trying private key: /home/<user>/.ssh/<name_of_my_key> Enter passphrase for key '/home/<user>/.ssh/<name_of_my_key>': debug1: Authentication succeeded (publickey). Authenticated to bitbucket.org ([2406:da00:ff00::22c0:3470]:22). debug1: channel 0: new [client-session] debug1: Entering interactive session. debug1: pledge: network debug1: Sending environment. debug1: Sending env LANG = de_DE.UTF-8 debug1: channel 0: free: client-session, nchannels 1 Connection to bitbucket.org closed by remote host. Transferred: sent 2808, received 1332 bytes, in 20.2 seconds Bytes per second: sent 139.2, received 66.0 debug1: Exit status -1
With execution freezing just after debug1: Sending env.
Looking at this transaction using Wireshark, we can spot the reason why:
So apparently, after the keys have been exchanged, openssh sets in its TCP packets a field called “ToS”, “QoS” or in wireshark’s terms “DSCP (differentiated services code point)” or “DiffServ”. This is used to prioritize some traffic over other, less important traffic and can be set per packet. And here the problem arises, because some routers are known to choke on packets with this field set to anything but the default 0x00.
You can see all communication at first happening using DSCP set to CS0, which is the default priority for packets. The first packet showing DSCP=2 (which is DiffServ=0x08) never gets through. Once communication is established, your openssh client tries to send packets with DiffServ byte set to 0x08, which then the router drops.
And so apparently did mine, a TP-Link Archer vr2600v. Quite a shame for such an expensive piece of kit, honestly. But once I narrowed down what caused the problem, I actually found 2 ways to fix it:
1. Using ssh config and the IPQoS tag
You can prevent openssh from using anything but the default QoS by adding the following lines to your ssh config:
Host * IPQoS 0x00
On Ubuntu, this file is located at ~/.ssh/config. If this file does not exist yet, simply create it.
And a quick reminder while we’re at it: You can use these lines to use a ssh key with non-standard file name / location with git:
Host bitbucket.org HostName bitbucket.org User git IdentityFile ~/.ssh/<name_of_my_key>
2. If you have a TP-Link router
Because this is what the QoS flag is actually used for, I got an idea what might have caused it with my router. I’m using a TP-Link Archer VR2600v, and it has a setting called “Bandwith Control”.
I found out that if you enable Bandwith Control, the problem disappears. You don’t actually have to set any rules and you can set your total bandwith limits to your max available (stated just above as “Current Upstream/Downstream Rate”). Just doing so seems to make the router not drop packets with non-default QoS flag.
This is quite a stupid bug, especially since TP-Link has known about this for at least 3 years as you can see from the post in their forum (see below).
References
Here are some accounts of people having the same issue:
https://stackoverflow.com/questions/2247782/ssh-example-com-hangs-but-ssh-example-com-bash-i-does-not
https://community.tp-link.com/en/home/forum/topic/94412
Here is an explanation what the DiffServ codes are supposed to mean:
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus1000/sw/4_0/qos/configuration/guide/nexus1000v_qos/qos_6dscp_val.pdf