Problem when sending multiple commands via jumpSSH using multithreading

[davidoiknine]

Hello,

I created a script where I send 15 000 commands to 70 switchs, so around 200 commands per switch. For each switch, I create one SSH connection via jumpSSH (via a jump server) this way:

# establish ssh connection between your local machine and the jump server
gateway_session = SSHSession('gateway.example.com',
                             'my_user', password='my_password').open()

# from jump server, establish connection with a remote server
remote_session = gateway_session.get_remote_session('remote.example.com',
                                                    password='my_password2')

Once the connection is established, I'm sending 200 commands using multithreading:

def main_loop():
   gateway_session = SSHSession('gateway.example.com',
                            'my_user', password='my_password').open()
   for switch in result.keys(): 
       remote_session = gateway_session.get_remote_session(switch["ip"],
                                                   password='my_password2')
       with ThreadPoolExecutor(max_workers=8) as executor:
           for command in result[switch]["commands"]:
               executor.submit(launch_command, command, remote_session)

def launch_command(command, remote_session):
   process = remote_session.get_cmd_output(command)           

The problem is that if I'm using 7 or 8 workers, it's not working properly (some information I get is missing) and I'm getting this error:

Secsh channel 55 open FAILED: open failed: Connect failed
Secsh channel 60 open FAILED: open failed: Connect failed
Secsh channel 113 open FAILED: open failed: Connect failed
Secsh channel 170 open FAILED: open failed: Connect failed
Secsh channel 213 open FAILED: open failed: Connect failed
Secsh channel 44 open FAILED: open failed: Connect failed...

If I'm using 6 workers or less, I have no any error and everything is working correctly.

I have the impression that it's creating more than one SSH connection per leaf and so that's why there is this error, because maybe I'm reaching the number max of sessions allowed in switches, but according to my code, only one connection per switch is set up.

Do you know why I'm experiencing this issue?
Thank you,

[teodor1746]

Hi @davidoiknine
Did you manage to find out why those secsh channel errors appear? Im currently facing the same problem with 8 workers and threading. My code doesn't fail or exit but I do see a lot of these if I have to go through 300+ hosts.

[davidoiknine]

Hello @teodor1746 ,
Sorry for my late reply.

I spent a lot of time trying to solve this problem and I finally used the asyncssh module which allows to do asynchronous over SSH. The creator of this module (ronf) helped me a lot and I put you the link of our conversation in which finally leads to a result that works well and quickly.

ronf/asyncssh#475