You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Added CI scripts for building system packages and binary wheels.
Migrated ssh2 client line parsing code to Cython.
Updated readme.
Updated test imports.
Added monkey patching to paramiko single host client as well as parallel client
As of version ``1.2.0``, a new client is supported in ``ParallelSSH`` which offers much greater performance and reduced overhead than the current default client (paramiko). Binary wheel packages with ``libssh2`` included are provided for Linux, OSX and Windows platforms and all supported Python versions.
70
69
71
-
Standard output buffers are available in output object. Iterating on them can be used to get output as it becomes available. Iteration ends *only when command has finished*, though it may be interrupted and resumed at any point.
70
+
The new client is based on ``libssh2`` via the ``ssh2-python`` extension library and supports non-blocking mode natively. In addition, SFTP push/pull operations in the new client have also been implemented in native code without Python's GIL, allowing for much greater performance and significantly reduced overhead.
72
71
73
-
`Host output <http://parallel-ssh.readthedocs.io/en/latest/output.html>`_ attributes are available in host output object, for example ``output['myhost1'].stdout``.
72
+
See < here > for a performance comparison of the two clients.
73
+
74
+
To make use of this new client, ``ParallelSSHClient`` can be imported from ``pssh.pssh2_client`` instead of ``pssh.pssh_client``. The respective APIs are almost identical, though some things have either not yet been implemented or are not supported in ``libssh2``.
75
+
76
+
Note that the new client will become the default and will replace the current ``pssh.pssh_client`` in a new major version of the library - ``2.x.x`` - once remaining features have been implemented.
77
+
78
+
For example:
74
79
75
80
.. code-block:: python
76
81
77
-
for host in output:
78
-
for line in output[host].stdout:
79
-
pprint("Host %s - output: %s"% (host, line))
82
+
from pprint import pprint
83
+
from pssh.pssh2_client import ParallelSSHClient
80
84
81
-
:Output:
85
+
hosts = ['myhost1', 'myhost2']
86
+
client = ParallelSSHClient(hosts)
87
+
88
+
output = client.run_command('uname')
89
+
for host, host_output in output.items():
90
+
for line in host_output.stdout:
91
+
print(line)
92
+
93
+
94
+
Compared to the current default, the native client currently lacks proxying/tunnelling implementation, as well as SSH agent forwarding. The latter is not currently supported by ``libssh2``.
95
+
96
+
See documentation for more information on how the two clients compare feature
Exit codes become available once output is iterated on to completion *or* ``client.join(output)`` is called.
103
+
* Highest performance and least overhead of any currently available Python SSH libraries
104
+
* Native non-blocking client based on ``libssh2`` via the ``ssh2-python`` wrapper
105
+
* Thread safe - utilises both native threads for blocking calls like authentication and non-blocking network requests
106
+
* Natively non-blocking - **no monkey patching of the Python standard library**
107
+
* Native binary-like SFTP speeds thanks to SFTP and local file read/write operations being implemented in native code
108
+
109
+
110
+
***********
111
+
Exit codes
112
+
***********
113
+
114
+
Once either standard output is iterated on *to completion*, or ``client.join(output)`` is called, exit codes become available in host output. Iteration ends *only when remote command has completed*, though it may be interrupted and resumed at any point.
91
115
92
116
.. code-block:: python
93
117
@@ -100,6 +124,7 @@ Exit codes become available once output is iterated on to completion *or* ``clie
100
124
0
101
125
0
102
126
127
+
103
128
The client's ``join`` function can be used to block and wait for all parallel commands to finish:
104
129
105
130
.. code-block:: python
@@ -112,13 +137,15 @@ Similarly, output and exit codes are available after ``client.join`` is called:
112
137
113
138
output = client.run_command('exit 0')
114
139
115
-
# Block and gather exit codes. Output is updated in-place
140
+
# Wait for commands to complete and gather exit codes.
141
+
# Output is updated in-place.
116
142
client.join(output)
117
143
pprint(output.values()[0].exit_code)
118
144
119
-
# Output is available
120
-
for line in output.values()[0].stdout:
121
-
pprint(line)
145
+
# Output remains available in output generators
146
+
for host, host_output in output.items():
147
+
for line in host_output.stdout:
148
+
pprint(line)
122
149
123
150
:Output:
124
151
.. code-block:: python
@@ -130,12 +157,14 @@ Similarly, output and exit codes are available after ``client.join`` is called:
130
157
131
158
In versions prior to ``1.0.0`` only, ``client.join`` would consume standard output.
132
159
133
-
There is also a built in host logger that can be enabled to log output from remote hosts. The helper function ``pssh.utils.enable_host_logger`` will enable host logging to stdout, for example:
160
+
There is also a built in host logger that can be enabled to log output from remote hosts. The helper function ``pssh.utils.enable_host_logger`` will enable host logging to stdout.
161
+
162
+
To log output without having to iterate over standard output generators, the ``consume_output`` flag can be enabled, for example:
@@ -147,9 +176,9 @@ There is also a built in host logger that can be enabled to log output from remo
147
176
Design And Goals
148
177
*****************
149
178
150
-
``ParallelSSH``'s design goals and motivation are to provide a *library* for running *asynchronous* SSH commands in parallel with little to no load induced on the system by doing so with the intended usage being completely programmatic and non-interactive.
179
+
``ParallelSSH``'s design goals and motivation are to provide a *library* for running *non-blocking* asynchronous SSH commands in parallel with little to no load induced on the system by doing so with the intended usage being completely programmatic and non-interactive.
151
180
152
-
To meet these goals, API driven solutions are preferred first and foremost. This frees up the developer to drive the library via any method desired, be that environment variables, CI driven tasks, command line tools, existing OpenSSH or new configuration files, from within an application et al.
181
+
To meet these goals, API driven solutions are preferred first and foremost. This frees up developers to drive the library via any method desired, be that environment variables, CI driven tasks, command line tools, existing OpenSSH or new configuration files, from within an application et al.
153
182
154
183
********
155
184
Scaling
@@ -178,20 +207,21 @@ Output *generation* is done remotely and has no effect on the event loop until o
178
207
SFTP/SCP
179
208
********
180
209
181
-
SFTP is supported (SCP version 2) natively, no ``scp`` binary required.
210
+
SFTP is supported natively, no ``scp`` binary required.
182
211
183
212
For example to copy a local file to remote hosts in parallel:
``ParallelSSH`` is in other words well suited to be the SSH client tools like Fabric and Ansible and others use to run their commands rather than a direct replacement for.
223
253
224
-
By focusing on providing a well defined, lightweight - actual code is a few hundred lines - library, ``ParallelSSH`` is far better suited for *run this command on X number of hosts* tasks for which frameworks like Fabric, Capistrano and others are overkill and unsuprisignly, as it is not what they are for, ill-suited to and do not perform particularly well with.
254
+
By focusing on providing a well defined, lightweight - actual code is a few hundred lines - library, ``ParallelSSH`` is far better suited for *run this command on these hosts* tasks for which frameworks like Fabric, Capistrano and others are overkill and unsuprisignly, as it is not what they are for, ill-suited to and do not perform particularly well with.
225
255
226
-
Fabric and tools like it are high level deployment frameworks - as opposed to general purpose libraries - for building deployment tasks to perform on hosts matching a role with task chaining, a DSL like syntax and are primarily intended for command line use for which the framework is a good fit for - very far removed from an SSH client *library*.
256
+
Fabric and tools like it are high level deployment frameworks - as opposed to general purpose libraries - for building deployment tasks to perform on hosts matching a role with task chaining, a DSL like syntax and are primarily intended for command line use - very far removed from an SSH client *library*.
227
257
228
258
Fabric in particular is a port of `Capistrano <https://github.com/capistrano/capistrano>`_ from Ruby to Python. Its design goals are to provide a faithful port of Capistrano with its `tasks` and `roles` framework to python with interactive command line being the intended usage.
0 commit comments