I recently ran into trouble pulling from my git repository on my linux machine:

$ git pull
Enter passphrase for key '/home/<user>/.ssh/<name_of_my_key>': 
Connection to bitbucket.org closed by remote host.
fatal: Konnte nicht vom Remote-Repository lesen.

Bitte stellen Sie sicher, dass die korrekten Zugriffsberechtigungen bestehen
und das Repository existiert.

Which was kind of confusing, because I could remember it working just a while ago without knowingly changing anything related to git. After suspecting my ssh keypairs or IPv4 / IPv6 issues, I narrowed it down to a problem with openssh, which of course is used under the hood of git pull.

As the help pages of Bitbucket and Github suggest, you can test your connection to your repo server using:
ssh -Tv bitbucket.org

If you have the same problem as I did, the output should look something like this:

$ ssh -Tv bitbucket.org
OpenSSH_7.6p1 Ubuntu-4ubuntu0.3, OpenSSL 1.0.2n  7 Dec 2017
debug1: Reading configuration data /home/<user>/.ssh/config
debug1: /home/<user>/.ssh/config line 12: Applying options for bitbucket.org
debug1: /home/<user>/.ssh/config line 15: Applying options for bitbucket.org
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to bitbucket.org [2406:da00:ff00::22c0:3470] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/<user>/.ssh/<name_of_my_key> type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/<user>/.ssh/<name_of_my_key>-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.6p1 Ubuntu-4ubuntu0.3
debug1: Remote protocol version 2.0, remote software version conker_405f5f426f-dirty app-149
debug1: no match: conker_405f5f426f-dirty app-149
debug1: Authenticating to bitbucket.org:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ssh-rsa SHA256:zzXQOXSRBEiUtuE8AikJYKwbHaxvSc0ojez9YXaGp1A
debug1: Host 'bitbucket.org' is known and matches the RSA host key.
debug1: Found key in /home/<user>/.ssh/known_hosts:1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/<user>/.ssh/<name_of_my_key>
Enter passphrase for key '/home/<user>/.ssh/<name_of_my_key>': 
debug1: Authentication succeeded (publickey).
Authenticated to bitbucket.org ([2406:da00:ff00::22c0:3470]:22).
debug1: channel 0: new [client-session]
debug1: Entering interactive session.
debug1: pledge: network
debug1: Sending environment.
debug1: Sending env LANG = de_DE.UTF-8
debug1: channel 0: free: client-session, nchannels 1
Connection to bitbucket.org closed by remote host.
Transferred: sent 2808, received 1332 bytes, in 20.2 seconds
Bytes per second: sent 139.2, received 66.0
debug1: Exit status -1

With execution freezing just after debug1: Sending env.

Looking at this transaction using Wireshark, we can spot the reason why:

QoS Wireshark Screenshot

So apparently, after the keys have been exchanged, openssh sets in its TCP packets a field called “ToS”, “QoS” or in wireshark’s terms “DSCP (differentiated services code point)” or “DiffServ”. This is used to prioritize some traffic over other, less important traffic and can be set per packet. And here the problem arises, because some routers are known to choke on packets with this field set to anything but the default 0x00.

You can see all communication at first happening using DSCP set to CS0, which is the default priority for packets. The first packet showing DSCP=2 (which is DiffServ=0x08) never gets through. Once communication is established, your openssh client tries to send packets with DiffServ byte set to 0x08, which then the router drops.

And so apparently did mine, a TP-Link Archer vr2600v. Quite a shame for such an expensive piece of kit, honestly. But once I narrowed down what caused the problem, I actually found 2 ways to fix it:

1. Using ssh config and the IPQoS tag

You can prevent openssh from using anything but the default QoS by adding the following lines to your ssh config:

Host *
  IPQoS 0x00

On Ubuntu, this file is located at ~/.ssh/config. If this file does not exist yet, simply create it.

And a quick reminder while we’re at it: You can use these lines to use a ssh key with non-standard file name / location with git:

Host bitbucket.org
  HostName bitbucket.org
  User git
  IdentityFile ~/.ssh/<name_of_my_key>

2. If you have a TP-Link router

Because this is what the QoS flag is actually used for, I got an idea what might have caused it with my router. I’m using a TP-Link Archer VR2600v, and it has a setting called “Bandwith Control”.

I found out that if you enable Bandwith Control, the problem disappears. You don’t actually have to set any rules and you can set your total bandwith limits to your max available (stated just above as “Current Upstream/Downstream Rate”). Just doing so seems to make the router not drop packets with non-default QoS flag.

This is quite a stupid bug, especially since TP-Link has known about this for at least 3 years as you can see from the post in their forum (see below).


Here are some accounts of people having the same issue:

Here is an explanation what the DiffServ codes are supposed to mean:

Leave a Reply

Your email address will not be published. Required fields are marked *

Post Navigation