The problem
Under Manjaro, I can’t clone a reposity from Gitlab with git.
$ git clone [email protected]/fhusson/xxxx.git
Cloning into 'xxxx'...
Connection to gitlab.com closed by remote host.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
My ssh key are ok :) and I have the access rights !
TL;DR : The solution with IPQoS
In the file /etc/ssh/ssh_config add the following configuration :
Host *
IPQoS reliability
Full debug story
Chapter 1 : The fix
If I try directly with ssh :
$ ssh -T [email protected]
Connection to gitlab.com closed by remote host.
Searching similar problem with Google I have found this article that suggest using the ssh option IPQoS=0
$ ssh -o IPQoS=0 -T [email protected]
Welcome to GitLab, @fhusson!
and it works !
\o/
In the OpenSSH article of the Arch Wiki I have found where to set it globally
In the file /etc/ssh/ssh_config add the following configuration :
Host *
IPQoS reliability
Chapter 2 : Manjaro vs Ubuntu (Debian inside)
Ok but now it works but why did I have this problem with Manjaro and not with Ubuntu ?
If I run the ssh command with more verbose log (-vvv) I have this interesting line :
debug3: ssh_packet_set_tos: set IP_TOS 0x20
I have done the same test in Ubuntu and I have this line :
debug3: ssh_packet_set_tos: set IP_TOS 0x08
To be sure under Manjaro I tested IPQoS=0x08 and it works too and with IPQoS=0x20 if fail.
Ok next why did I have a different default value for the IPQoS in Ubuntu and in Manjaro.
With man ssh_config for the IPQoS option we have :
- In Manjaro : The default is af21 (Low-Latency Data) for interactive sessions and cs1 (Lower Effort) for non-interactive sessions.
- In Ubuntu : The default is lowdelay for interactive sessions and throughput for non-interactive sessions.
I have found this mail/patch from ‘Job Snijders’ where we learn about this change in OpenSSH and here is the link to the commit message
Package version for Manjaro is openssh 8.4p1-2 and ubuntu is openssh-client (1:8.3p1-1)
The change were made in the OpenSSH version 7.8 and we can see in the ubuntu changelog
openssh (1:7.9p1-10) unstable; urgency=medium
- Temporarily revert IPQoS defaults to pre-7.8 values until issues with “iptables -m tos” and VMware have been fixed (closes: #923879, #926229; LP: #1822370).
– Colin Watson [email protected] Mon, 08 Apr 2019 11:13:04 +0100
In the debian/patches/revert-ipqos-defaults.patch we have :
- options->ip_qos_bulk = IPTOS_DSCP_CS1;
+ options->ip_qos_bulk = IPTOS_THROUGHPUT;
and in the openssh/defines.h we have
/*
* Definitions for IP type of service (ip_tos)
*/
IPTOS_THROUGHPUT 0x08
/*
* Definitions for DiffServ Codepoints as per RFC2474
*/
IPTOS_DSCP_CS1 0x20
More information from the debian patch :
The IPQoS default changes have some unfortunate interactions with iptables (see https://bugs.debian.org/923880) and VMware, so I’m temporarily reverting them until those have been fixed. Bug-Debian: https://bugs.debian.org/923879 Bug-Debian: https://bugs.debian.org/926229 Bug-Ubuntu: https://bugs.launchpad.net/bugs/1822370 Last-Update: 2019-04-08
Chapter 3 : More tests
I still need to understand why ‘cs1’ is not working.
- I have changed my network links with no success :
- My Computer -> My router with OpenWrt -> The Internet provider box : Failed
- My Computer -> The Internet provider box : Failed
- My Computer -> My router with OpenWrt : Failed
- I have the same error for github.
- Test with my mobile network over USB : Success
Chapter 4 : To be continued
The next step is to try to open a ticket at my internet provider ^_^