SFTP upload vs download performance discrepancy

[agaveapidev]

I've run into some issues recently trying to move roughly 200 files totaling about 27G over SFTP. I don't really expect the performance to match up to the native utilities, but for sake of completeness, in 100 tests of files ranging from 2MB to 2GB, the the discrepancy between the linux sftp command tool and the j2ssh-maverick test class below is about 10x on put operations and well over 80x on get operations. In the tests, both servers are reading and writing to SSD with 8GB of dedicated memory and 2 dedicated CPU. I ran the same experiment over a GigE and 40G network, both in house, with little to no load on them.

My first question is how on earth can read and write performance differ so dramatically in the opposite direction? I've tried changing the SFTPClient blockSize, bufferSize, and the tcp socket preferences, but anything less than loading the entire file into memory causes put to slow down and get to crawl.

Second, we purchased a license to the Maverick Legacy Client a few years back and, because we open sourced our project, switched over to this library. None of the above behavior occurs in the commercial product, despite the current j2ssh-maverick library being a fork of the commercial library. Is the open source library intentionally throttled in some way? That would explain the dramatic increase in failure rates we've seen since switching from the commercial version.

[ludup]

The API is certainly not throttled. As the source is publicly available its something that could be easily discovered and frankly would be very bad form and not something I would do. The API is a genuine branch from our commercial base, refactored with a different package name and logging implementation.

I will run some tests to see if I can reproduce.

[agaveapidev]

I’ve been running benchmarks for a few days. The performance difference is really challenging. I’m happy to share a couple benchmark repositories and provide a test VM and/or docker containers to replicate if it helps.

[ludup]

I've just run some very basic tests and see similar performance between the two APIs.

I assume that in your tests you are using the same JVM for both APIs and that you have confirmed that similar algorithms are in use between them? For example that both are using AES with the same key size?

Could you tell me what sort of performance you are seeing, do you have some stats?

[deardooley]

I built out a couple benchmark applications using Spring Shell here. If you'd like access, just shoot me your bitbucket account or email. Here are the results running against a docker container locally. Each line represents 20 iterations of the benchmark suite using get and put. I can rerun using streams if it's helpful. Either way, similar results are found when the host is external. The primary difference is the oss client gets much, much slower and begins to fail on file sizes over 1GB. The legacy client behaves like a champ, but we go through periods of time when we push over the license server when we get popped with traffic.

J2SSH OSS Results

File Size	Upload time(ms)	Upload Rate(kB/s)	Download time(ms)	Download Rate(kB/s)
2097152	56.85	36203.6151	1822.45	1347.458334
20971520	486.9	42371.31825	25413.8	811.2552085
209715200	4427.2	46334.65	255763.6	801.212598

Legacy Client Results

File Size	Upload time(ms)	Upload Rate(kB/s)	Download time(ms)	Download Rate(kB/s)
2097152	52.4	40443.8427	92.05	25738.1707
20971520	471.9	43645.5317	355.95	57859.1215
209715200	4552.1	45108.47765	3216.9	63762.6524

[deardooley]

I spent some time playing around with the ciphers, buffer size, and streaming disk vs streaming memory vs get/put performance between the two libraries. I'm still seeing a 100x difference on download speed between the two libraries, and upload is often 50-100x faster than download on files larger than available memory in both versions.

It looks like the biggest bottleneck in the OSS library is the SFTPClient.blockSize defaulting to 4096. Small default buffer sizes when reading from the server and writing to disk has a large negative impact on performance as well. At this point, I'm a little unclear about how or if to proceed. Many of the issues I'm seeing in the 1.5.5 release seem to have already been reported and addressed in the 1.6 changelog. Is there a maintenance strategy for the open source version, or was it a one-off release? Does it make sense to make pull requests when they likely won't be merged with the upstream version?

[ludup]

Historically the block size was set at that low size because support for larger sizes was rather fragmented and we had too many support cases where customers found the API incompatible with some types of servers. So it was a case of setting it to a value that all servers would support and iirc OpenSSH had the same value at the time. These days most implementations have come around to supporting larger sizes with larger windows so its not such an issue to increase the default; although if my support queue is anything to go by there will still be plenty of people using old outdated versions of SSH. But if your using this in a controlled environment that will simply not be a problem.

The maintenance strategy for the open source version is that it gets attention when I can, there have been some updates since it was branched so its certainly not static. I'd welcome your pull requests so please submit what changes you need.