sunset-sftp added to Cargo.lock

This was an automatic resolution

error fix on casting SftpNum to u8

Fixing error message `casting `&SftpNum` as `u8` is invalid: needs casting through a raw pointer firstrust-analyzerE0606`

fixing error in sftpmessages macro_rules SSHDecode implementation

I believe that $SSH_MESSAGE_NAME should be $SSH_FXP_NAME for this macro to work and that it was intended for this match ty expansion.

fixing use of  sftp proto

Adding some extra uses in proto.rs

making  proto StatusCode pub

Fixing sftpmessages SftpNum impl issue

providing the correct data type to contains (a reference to a u8)

removing reduntant use in proto.rs

Added variant to proto StatusCode

cleaning redundant comments

the commit 7028e03 covers this

Removing some unused lifetimes that caused issues

They can be added when needed

Chasing SSHEncode,SSHDecode issues

Adding the derive for Filename and Attrs

Adding empty vanilla DirReply struct as ReadReply

Adding Vanilla ChanOut to finish the placeholders for Dir and Read Reply

Extra fix for SftpNum to avoid moving issues

WIP: Replacing sunset::Result with proto::Result

Fixing issues with error handling is far from finished

sunset::Result cannot take the new Errors definded in proto.rs therefore I have added error conversion for sftp errors to Sunset global error.

commenting out sftpmessages macro rules to focus on the foundational SFTP data types

commenting out Result and ane extra SunsetError implementation too

Removing phantomData for the moment to avoid SSHEncode/Decode issues

Implementing SSHEncode/Decode for Attrs

To do so, I introduced the method flags(), and the enum AttrsFlags to serialise and deserialise the flags

Added a response variant "ResponseAttributes" to complete all server responses SFTP packets

Feel free to rename it

Fixed typo and change other variant string to ssh_fx_other

isolating proto.rs issues. Commenting out sftpserver

proto.rs StatusCode SSHEncode expanded version seems incorrect.

I made a StatusCode version with lifetime (to be used by Other) and StatusCode SSHEncode autogenerated enc() function (L584) looks wrong since it is not performing any ::sunset::sshwire::SSHEncode::enc()

Am I using SSHEncode wrong for this enum?

As per the pull request fails because of the edition, I am downgrading it  2024->2021

Clarifying that this implementation will use SFTP version 3

As it is the most common version in use today, ensuring broad compatibility. Support for other versions may be considered in the future.

Uncommenting macro_rules sftpmessages partially until problems arose with `Name`

Implementing SSHEncode and SSHDecode for Name

Name has changed. It no longer contains count, but it Encodes it as the Names length

adding macro_rules! sftpmessages closing curly brackets

StatusCode: SSHEncode and SSHDecode

Adding num_enum to sftp to handle the u32 to enum expansion

Bumping CI minimum version to 1.85 and sftp to 2024 edition

More on this version [here](https://blog.rust-lang.org/2025/02/20/Rust-1.85.0/)

moving sftp edition to 2024

WIP: Starting to implement SftpPacket modifying SftpNum

- modified macro pattern message_num from literal to tt (token tree)
- lifetimes for SftpPacket are back
- Added dependency to crate  paste for pattern modification (to lower case but can be use for many other things)
- I had to duplicate InitVersion to avoid repetitions of implementations in the macro expansion

Adding SSHEncode and SSHDecode for SftpPacket

Updating cargo-expand-sftp.rs too

Added past dependency in Cargo.lock and as use

Restricting SSHDecode for SftpPacket lifetimes so they match

It is convoluted, changing unifying lifetimes for this implementation to `'a` would work as well

cargo check passing and all the code uncommented. However...

I have changed the signature for methods `decode_request` and `decode_response` to avoid the dyn SSHSource which was causing a problen during calls to dec(s)? Since it does not have a known size at compile time.

To do so I have included the `'de` a lifetime for the method parameter s and made the structure SftpPacket lifetime `'a` and `'de` be equal, as I did in the previous commit fab7996

Will this be an issue?

Last, the checks to find out if the decoded packet is a response or a request in `decode_response` and `decode_request` now return WireError::PacketWrong instead of error::SSHProto.fail() or Error::bug(). This might need to change

Fixing Avoidable warnings:

- visibility for `FileHandle`
- removing unused uses
- removing unused local result and error

removing outdated TODOs in sftp/src/proto.rs

Fix from comment @mkj comment on Name

Fixed according to the indications in the next discussion:
mkj#29 (comment)

Adding the expanded version for reference too

Thanks

reordering dependencies in sftp to match other workspace members

Starting a new sftp-std demo taking the basic std demo as model

It has some very basic code proving that when an sftp connection is accepted the version handshake starts SSH_FXP_INIT ->
<- SSH_FXP_VERSION
etc

adding mod sftpserver as I am going to start working on it next

Adding PathInfo SFTP Packet as it is requested by sftp clients after initialisation

adding some Doc comments

exposing sftp entities in lib.rs

fixing impl From<SftpNum> for u8

captures the number in Other SftpNum

Added log to sftp

Big changes: Fixed decoding SftpPacket

This changes the sftpmessge parameters to handle the three classes of SFTP Packets: Initialization, Requests and Responses.

The motivation is to add a mechanism to capture the Request Ids. Since only requests and responses contain that field but they do not include in their "inner" sftp Packets definitions (See Read, Write, etc) I thought that it was a good idea keeping them out of them.

To store them away from the Inner packets I decided adding extra parameters to the SftpPacket enum.

To do so the parameters have been reestructured so each class of sftp packet can be expanded with the required data.

As a side effect, the sftpmessages packets definition readability has been improved.

Certainly this adds complexity to the sftpmessages

adding from str for Filename

changing ranges for is_init, is_request, is_response

fixing decode_request

Fixing the exposed items in SFTP library. More changes will come

added sunset-sftp to sftp demo

adding log dep to sunset-sftp

WIP: Adding demosftpserver.rs

This will be the local implementation to serve the requests from the client. For now just structure and a mock realpath

adding the module demosftpserver to the demo. Reordering uses

Reordering uses to clarify internal and external dependencies

tyding up dependencies

adding trait associated function realpath and struct  ItemHandle

Still deciding on how to use ItemHandle

WIP Added sftphandle.rs

This is the "traffic coordinator" for SFTP.
- Handles incoming packets deserialisation
- Handle the initialisation
- Calls SftpServer trait to handle client requests
- Handles incoming packets serialisation

The current implementation tries limiting its  dependencies to the minimum. Therefore it does not handle the SSH Channels itself but only buffers. This also allow the user to make decisions on the buffers.

the sftp demo has been updated to use this, delegating the sftp details to the sftphandle and demosftpserver

Adding as_str for Filename

Just gets the inner BinString as_str

Handling the conditions where the SFTP protocol fails, setting things up so it can be handle and will wait for a new channel on the pipe from the program loop

removing std uses

Reformed sftpserver

- flags are no longer required
- removed ItemHandle
- Added Into and TryFrom File Handle for SftpServer trait
- Added lifetime 'a to Handle and the trait associated function taking or returning the type

minor changes in proto:
added clone, copy and compare implements to FileHandle and TextString

Bug fix in SftpPacket.encode_request

enconde+request was encoding sftp number and the request id, which is incoded in self.enc(c) call. Fixed and removed req_id from parameter list

Working on sftphandle

- handling unsupported packet
- fixing calls to SftpPacket.encode_request
- adding 'a lifetime to SftpHandler to handle a Vec of FileHandle<'a>

sftp.rs

 - removed ItemHandle;
 + Adding StatusCode
 + FileHandle

minor fix in decode_request return value

changes in demosftpserver very unstable at this point

handling and logging unsupported or uninitialised messages

fix bug encoding_request->encoding_response

Adding trait implementation

- requires instance: WIP
- catches default behaviour: Unsupported

Simplifying SftpServer trait async and the Handle are out for now

I would like to keep those but since I want a structured of SftpServer to keep it's own internal state, I could not use async or a Handle with a trait definition. It is unfortunately unstable.

Still a WIP but now a client can PUT a small file and its content will be displayed in debug.

For now the buffers overflow on long SSH_FXP_WRITE packets and fail after trying to decode in the next loop iteration more of the same packet content.

The previous sftpserver trait has been stored in asyncsftpserver.rs

Simple random data files generation for big write testing

Adding SFTP Packet Header and SSH_FXP_WRITE offset definitions

Simplified Main Demo SFTP loop and reduced the SFTP buffers size

WIP: Working on long WRITE packets processing

WIP: Processing long Write Requests

For now the flow of receiving SFTP Write request longer than the buffer is going in the right direction. I have been able to process up to 64kB of data. Still this is not complete since I need to:

- Write the data received and check that the data sent and received has not been altered
- Issue: Writes with files over 64kB fail
- Reshape the file handle exchanged from the demoserver so it has a fixed length and it is compact
- Refactor sftphandle

Adding basic obscuring of handler

This forces the SftpServer to use obscured handles that are size constrain.

It also introduces the handle manager, that is a helper that the SFTP server implementer can use for dealing with the obscured handle and a private handle, that can contain all the information that the local server requires.

Minimal SFTP Server implementation of write 64kB write pass the test

File sent and written are identical. That is a good start.

Next I will find what is wrong passed 64kB

Troubleshooting the 65kB limit: There are two write requests

I was assuming that the client would wait for a `SSH_FXP_STATUS` Ok, but it does not. I need to handle a second request segment in the same SftpSource

WIP: Refactoring opaque_file_handle -> obscure_file_handle and adding traits

Will not compile ATM

First, I am still working on the details to make the use of traits to define the DemoSftpServer flexible. In this commit there is too much rigidity in the  DemoSftpServer which does not allow my intended vision.

Second, there is an unrelated refactoring here where SFTPSink and SftpSource have been extracted from sftphandle.rs to their own files for clarity

WIP: Fixed DemoSftpServer definition to use a custom implementation

This allows the library user to define both a private and opaque file handle

WIP: Minimal implementation for DemoSftpServer

Still a work in progress, but extract the definition of:

- OpaqueFileHandle trait struct: to allow library users to define the OpaqueFileHandle at will
- OpaqueFileHandle trait struct: A proposed structure trait to allow the users defining its own OpaqueFileHandle and InternalHandle Manager

This increase the flexibility of the library allowing users taking decisions of their application rather than forcing them to use predefined elements

Next step: Handle consecutive requests that might be received in the same buffer

opaque_file_handle not obscure_file_handle

Letting the `sftp_loop` and `prog_loop` finish DemoServer.run()

This way the common server closes the socket and SFTP clients do not get hanged on failure or bye

Tiding up

Added change to Cargo.lock is part of commit a19b0c6

Removed cargo-example-sftp.rs, used for macro_rules troubleshooting not required at the moment

Bact to SftpHandle: Process failing as soon as possible on short packets

Added extra docs to SFTP proto decode_request

Broken: Ugly experimentation with multi part consecutive packets

Somehow I am sending an ok different from the expected by the client

decoding request len in SftpPacket::decode_request

Handing consecutive request in a buffer with some problems

The issue that is left to fix is what happen when the second request is trunked and cannot be decoded.

I am contemplating creating a "truncated" request buffer that would be used when the request cannot be decoded because requires extra data to be received in the next process call.

Apart from this, the current code is able to process fragmented and consecutive SSH_FXP_WRITE for files with occasional problems caused by the issue detailed.

WIP: Modifying an error condition to RanOut and adding peak_packet_len

WIP: Restructuring SftpHandle. Adding a FSM

Not finished, but this structure prevents borrowing issues

WIP: handling ok request in FragmentedRequestState::ProcessingClippedRequest

This does not cover what happens with NoRoom requests such as Write request

WIP: Removing dead code

WIP: replacing WireErrors and minor renaming

WIP: Fixing bugs

RequestHolder
Transition logic

WIP: Successfully received 100MB and 1024MB

It is time to tidy up

SFTP Roadmap added to docs

The crate doc has been rendered and improved.

I propose a roadmap for the development of teh SFTP crate

WIP: Added SftpError::FileServerError(StatusCode) + others

- Reasonable warning removal
- Adding comments
- Removing dead code
- More refactoring

WIP: sftphandle->sftphandler fixing sftperror statuscode casting

WIP: Removing dead code and adding comments

TODO: Use heapless::Vec instead of Vec for Name as it breaks the no_std premisses

WIP: fixed bad tag ids

WIP: More docs

WIP: Refactoring SFTP lib and added comments

from flat structure to a more conceptual structure

Also added a brief header doc to show what is the intent and the status of the module.

Partial functionality and refactor completed

In this commit we have a working server that can receive files of arbitrary size. This can be tested running demo/sftp/std/ and running
`demo/sftp/std/testing/test_long_write_requests.sh`

Significative documentation has been added.

SftpHandler relies on a FSM to process fragmented and long packet.

[ci skip] Adding no_std, removing unnecessary use of String

[ci skip] WIP: Added OpenDir operation

SftpServer trait and DemoSftpServer modified to allow the use of Directories

[skip ci] WIP: Added OpenDir and ReadDir requests

I am at the point where I can start implementing the way I am going to put all the data in the channel.

Not trivial

[skip ci] WIP: moving main loop inside process_loop to have stdio in scope

[skip ci] WIP: commenting out the no_std restriction for now

[skip ci] WIP: I missed out adding items to Cargo.lock

[skip ci] WIP: Exploring the iterator option

[skip ci] WIP: Experimenting with SftpServer
DirReply structure

For now I am only checking borrowings

[skip ci] WIP: SftpSink playload_slice()

[skip ci] WIP: Adding DirReply Visitor to `ReadDir` and `DirEntriesResponseHelpers` to sftpserver.rs

These will help me standardize other implementations. Also, I have proven that the Visitor pattern can be used to process each directory entry without borrowing or lifetime issues

[skip ci] WIP: Refactored constant

[skip ci] WIP: Reworking a wrapper for Sink and ChanOut

I want to use them grouped to later pass them to DirReply

Also I might be able to use better the buffer out

[skip ci] WIP: fixing some comments and warnings

[skip ci] WIP: Ugly explorative refactoring

[skip ci] WIP: Ugly explorative refactoring now working

It is really ugly

[skip ci] WIP: Refactoring. Creating a mod for sftphandler

[skip ci] WIP: Refactoring. Normalizing push function

Unifying around push_status

[skip ci] WIP: Refactoring. Integrating push functions in OutputWrapper

- push_status
- push_packet

[skip ci] WIP: Refactoring. Moving SftpOutputChannelWrapper (aka OutputWrapper) to its own module

cleaning up sftphandle.rs

[skip ci] WIP: Refactoring

- SftpHandler: estate machine restructuring
- SftpServer: WIP for DirReply
- SftpOutputChannelWrapper: WIP refactoring

[skip ci] WIP: Adding Producer and Consumer to avoid mutable references writing to ChanOut

[skip ci] WIP: Removing DirReplay wrong lifetime parameters

[skip ci] WIP: Improved SftpSource readability

[skip ci] WIP: Removing excessive lifetimes to build successfully

[skip ci] WIP: Improved readability in requestholder.rs

[skip ci] WIP: Fixing the Pipe mutex to main crate SunsetRawMutex, solve send_packet bug and added debugging

[skip ci] WIP: Fixing Bug where an error was thrown instead of trying to encode a packet

[skip ci] WIP: Refactor sftphandler.rs and main.rs to accommodate the new Producer-Consumer for ChanOut

It fixes the mutable borrow problems but I am getting the client disconnected with writing request:
```
Received message too long 1572864
Ensure the remote shell produces no output for non-interactive sessions.
```

[skip ci] WIP: Looking for the error. Made some small readability changes

[skip ci] WIP: looking for traces explaining where the length [0,0,0,18] becames [18,0,0,0]

Increasing verbosity level globally Still I cannot find where it is happening if it happens in Sunset SSH

[skip ci] WIP: Tyding up:

Removing deprecated parts of code ,commenting out others and adding comments

[skip ci] WIP: write request fails but will be parked for now

I am going to focus on listing folder files

[skip ci] WIP: Some issues in DemoSftpServer DirEntriesCollection but ready to make the SftpServer trait functions async

The length calculation seems erroneous but I am more worried about the async trait

[skip ci] WIP: Progress with readdir. Needs tyding up Issue with pipe

Looks like the number of sent items in the pipe does not match received items. I am going to add a send/receive total bytes counter. It might also be the cause of the write errors

[skip ci] WIP: Cargo lock update long forgotten. Apologies

[skip ci] WIP: FIX read operation

Looping to write all bytes. The counters confirm that there is no missing bytes as they match.

Tested with a 100MB writing operation: ./test_long_write_requests.sh

I am confident that this the write issue is solved

[skip ci] WIP: Hack readdir response was not adding header field length

packet type + request id + items (1 + 4 + 4 bytes) hardcoded but will refactor

[skip ci] WIP: BUG in sftphandler process keeps responding to the same read dir request

[skip ci] WIP: BUG **Sending loop includes sftp client**

- the ReqId increases

- The loop stop in a client interruption [ctl]+C

I am going to need more visibility on why the client keeps on sending new request for the same files

[skip ci] WIP: BUG I was not following V3 specification as did not send EOF

https://datatracker.ietf.org/doc/html/draft-ietf-secsh-filexfer-02#section-6.7

Next issue is that I am sending the relative path and not just the filename. Will fix next

[skip ci] WIP: BUG The EOF needs to be sent once the readdir has sent all the items and is asked again. Anything else fails

This is point I have not considered. I could have sent one item at a time, with different challenges.

In my opinion I should add this behavior to the SftpServer implementation, but that passes the responsibility to the library user and the idea is making the user not needing to care too much about the protocol implementation

[skip ci] WIP: Typo and cleanup commented out code

[skip ci] WIP: Added a simplistic last read status approach

Found a bug sending back the status code. It is always Ok!

[skip ci] WIP: Temporary ways to track completed Read operations

I will add a test to understand the  unexpected behaviour in response status packet

[skip ci] WIP: Missing StatusCode field in Status derived from SSHEncode implementation for enum

sshwire-derive lib.rs defines the rules to encode enums in `encode_enum` and the encoding decision is not to encode the value. See [implementation here](https://github.com/mkj/sunset/blob/8e5d20916cf7b29111b90e4d3b7bb7827c9be8e5/sshwire-derive/src/lib.rs#L287)

This could be addressed by introducing a parameter for enum fields where  the encoded data type is declared (or not). There is probably a good reason for this behavior.

[skip ci] Fixed Status Encoding. Shall I extend the sshwire-derive to generate encoded values?

To fix the issue with the message type encoding I have done a manual implementation of SSHEncode for `StatusCode`. It is tested in a proto_test mod.

From last commit:
sshwire-derive lib.rs defines the rules to encode enums in `encode_enum` and the encoding decision is not to encode the value. See [implementation here](https://github.com/mkj/sunset/blob/8e5d20916cf7b29111b90e4d3b7bb7827c9be8e5/sshwire-derive/src/lib.rs#L287)

This could be addressed by introducing a parameter for enum fields where  the encoded data type is declared (or not). There is probably a good reason for this behavior.

[skip ci] WIP: First ls command successful

Not ready yet. In order to answer  EOF to a SSH_SXP_READDIR after all the dir items have been already sent. I am storing a `sftpserver::ReadStatus` passed as a result on  `SftpServer.dirread()` with PendingData or EndOfFile.

This would not work well in scenarios where there is more than one read transaction in course.

I have decided that I am going to   make the SftpServer implementators the responsibility to send the EOF status message but I will help them by providing a detailed trait documentation.

To test this listing you may run the demo/sftp/std and run the script testing/test_read_dir.sh

mind that the hard link counter is not recovered as it is not part of SFTP V3 implementation

[skip ci] Implementing EOF responsibility in SftpServer and documenting

[skip ci] Added SunsetSftp Feature: Std

Will be used for helpers only available to Std compilations

[skip ci] Extracted DirEntriesCollection and added to std helpers

This way we keep from std users some of the protocol details

[skip ci] Refactor proto.rs Name structure and
sftpserver trait to remove the `std` dependency

It splits the responsibility of encoding/decoding a SSH_FXP_NAME into Name and NameEntry<'a> avoiding the Vector or any fixed memory allocation for the list NameEntry.

That required refactoring SftpServer.realpath() and SftpHandler::handle_general_request() to adopt this approach of sending a header and after that one or more NameEntry items

Also documenting and reducing the number of warnings

Basic Features Directory read is now complete

sunset-sftp and sunset-demo-sftp-std crates version have been increased and now match.

Documentation has been updated,

An Producer Consumer subsystem has been introduced to allow sending responses exceeding the output buffer length without introducing lifetimes and borrowing issues.

proto.rs has been refactored to eliminate all std dependencies.

helper structures and modules have been added to assist the SftpServer` implementer.

a new crate feature `std` has been introduced to allow accessing std helper structures that reduce the exposure of sftp protocol details to the `SftpServer` implementer.

The std example has been refactored. While still quite complex it delegates some pretty specific tasks to the std helpers.

[skip ci] Extending SftpSource and Tests

Now we can peak the total packet length and also if the contained packet fits within the buffer.

Tests for all of that

[skip ci] SftpOutputPipe checks if already splits

removed TODO, not a good idea

[skip ci] intercepting incomplete UnknownPacket in decode request to help flushing it

It is important that we flush the rest of an incomplete unknown packet. So until it is completed, we will return a RanOut so we can read it full and then flush it

Handling properly unknown packets

This way:

- We send the right ReqId for Unsupported status
- Flush out the complete packet to avoid desynchronizing packet decoding

One more time I missed the Cargo.lock changes

refactored std helper to expose get_file_attrs

completed getting file stats

Tested with `test_get.sh`

Ticked off from the roadmap

[skip ci] Fixed misrepresentation of SSH_FXP_OPEN packet

also fixing an error with no_std for std helper function

Removed doc orphan document comments

Fixing SftpTrait and implementations to use the proper Open mode (PFlags)

Tested with `test_get.sh`

Next step: Implement SSH_FXP_READ

WIP: Added SFTP Get capability. Chasing unexpected locks

After sending a full SSH_FXA_DATA response, sometimes the process stalls and there is no further request from the client. Looks like this happens when the server answer one response behind the last SSH_FXP_READ request.

See client verbose output

```
sftp> get ./65kB_random
debug3: Looking up ./demo/sftp/std/testing/out/./65kB_random
debug3: Sent message fd 3 T:7 I:21
debug3: Received stat reply T:105 I:21 F:0x000f M:100644
Fetching ./demo/sftp/std/testing/out/./65kB_random to 65kB_random
debug2: do_download: download remote "./demo/sftp/std/testing/out/./65kB_random" to local "65kB_random"
debug2: Sending SSH2_FXP_STAT "./demo/sftp/std/testing/out/./65kB_random"
debug3: Sent message fd 3 T:17 I:22
debug3: Received stat reply T:105 I:22 F:0x000f M:100644
debug2: Sending SSH2_FXP_OPEN "./demo/sftp/std/testing/out/./65kB_random"
debug3: Sent remote message SSH2_FXP_OPEN I:23 P:./demo/sftp/std/testing/out/./65kB_random M:0x0001
65kB_random                             0%    0     0.0KB/s   --:-- ETAdebug3: Request range 0 -> 32767 (0/1)
debug2: channel 0: window 1998336 sent adjust 98816
debug3: Received reply T:103 I:24 R:1
debug3: Received data 0 -> 32767
debug3: Request range 32768 -> 65535 (0/2)
debug3: Request range 65536 -> 98303 (1/2)
debug3: Received reply T:103 I:25 R:2
debug3: Received data 32768 -> 65535
debug3: Finish at 98304 ( 1)
```

SFTP Read stuck: Failing to receive a a read request?

Anyone can take a look at this?

Following the analysis in the previous commit, it happens that from time to time there is  no more data to be read from the SSH Chan_In while the client is waiting for a response. I have documented this with some logs extracts

See files failing-get-client.log, failing-get-server.log (sucess-get{client, server}.log provided for reference). In those failing examples, the server looks stuck waiting to receive new bytes from the channel (SFTP: About to read bytes from SSH Channel)

During another run, while in the deadlock, I have manually killed the sftp client (`pkill sftp`) and the server logs receives an rx complete and listen again for new connections (get-deadlock-client-killed.log)

[skip ci] small changes to make visible SftpOutputPipe total bytes communicated

There is no data lost in these communications

[skip ci] Removing outdated check in SftpHandler.process

The fragmented packet handling will take care of that situation

[skip ci] Removed nested enum in Sftp process FSM

[skip ci]some tiding up

adding large file 2MB to test_get.sh

[skip ci] Bug? Large write buffers cause for Chan_out cause ssh.server.progress fail: NoRoom

Running `sunset_demo_sftp_std` and calling test_get.sh triggers this error from time to time

[skip ci] Debugging runner.read_channel and .input

Still haven't found where the missing requests are.

If I am sending bad data, I am still expecting to receive the data already sent according to the SFTP client

New logs can be found running sftp_std demo  and ./test_get.sh

[skip ci] RequestHolder function visibility reduced and capacity() added

I added capacity for uses when I want to check if a given slice fits in the holder

[skip ci] Removing outdated logs

[skip ci] Adding test_get_single.sh and simplifying test_get.sh

The upload process has been removed to reduce verbosity and test time. Also keeps the test focus on dowloading items.

test_get_single.sh only performs a get transaction of a long file (4MB)

Tests will keep failing check files for further inspection

[skip ci] Refactor packet handling: Packets will have fixed length part and extra data

This normalize the way of treating packets while decoding and completing fragments.

Processing long write requests is simplified with a new state `HandlerState::ProcessWriteRequest`, that keeps the progress and simply consume data for the the extra data that is  no longer part of the proto definition of Write.

requestholder.rs has new methods that simplify its usage and makes it generic.

sftpsource.rs has been simplified

most warnings have been corrected

[skip ci] Adding more unit tests. Noted that it would be nice improving encoding of SSH_FXP_DATA and SSH_FXP_NAME packages

[skip ci] Clearing RequestHolder on reset() and
updating field len on each SftpSink push call

clearing the reset holder makes it easier to debug it

finalizing every time data is serialized in the sink avoid usability by not relying on users finalizing the sink

[skip ci] caught a condition where an unknown packet would not be flush

[skip ci] Checking the errors getting files

Results of a simple test, repeated 10 times. Blocked receiving data, with our without error.

Next I will add more visibility to the base sunset library

[skip ci] Fixed test to accommodate that sinks don't need to be finalized

[skip ci] Adding checks around sftpserver read and readdir

Since in production the library will not be in control of the way the SftpServer trait implementer will do with the {Dir/Data}Reply parameters checking that the data announced to be sent and the data really sent would prevent running into invalid packet.

Until further changes data length sent != data length announced will trigger a runtime bug.

The trait documentation has been updated accordingly

[skip ci] some changes to the testing sh. -o LogLevel=[Error->Debug]

[skip ci] extra verbosity strace runs in sftp get tests

I have rocorded packets (encrypted) and straces for exec and test script finding:

- invalid packets reading from the channel
- No data reading from channel after sending EOF
- demo-sftp-std server stucked reading from file

[skip ci] Starting logging stuck communication

The next files have been updated to facilitate the collection of data.

The data collected will include:
- Output of trace level of the sunset-demo-sftp-std
- strace for sunset-demo-sftp-std
- Output for log_get_single.sh script

[skip ci] Minor change regarding buffer names

Author: jubeormk1

.github/workflows/ci.yml

@@ -13,8 +13,8 @@ jobs:
   all:
     strategy:
       matrix:
-        # 1.87 is an arbitrary minimum, tested to notice when it bumps
-        rust_version: [stable, nightly, 1.87]
+        # 1.83 is an arbitrary minimum, tested to notice when it bumps
+        rust_version: [stable, nightly, 1.85]
     runs-on: ubuntu-latest
     env:
       RUSTUP_TOOLCHAIN: ${{ matrix.rust_version }}

Cargo.toml

@@ -14,6 +14,7 @@ members = [
     "demo/picow",
     "demo/std", "fuzz",
     "sftp",
+    "demo/sftp/std",
     "stdasync",
     # workspace.dependencies paths are automatic
 ]

demo/sftp/std/src/demosftpserver.rs

@@ -228,7 +228,7 @@ impl SftpServer<'_, DemoOpaqueFileHandle> for DemoSftpServer {
         opaque_file_handle: &DemoOpaqueFileHandle,
         offset: u64,
         len: u32,
-        reply: &ReadReply<'_, N>,
+        reply: &mut ReadReply<'_, N>,
     ) -> SftpResult<()> {
         if let PrivatePathHandle::File(private_file_handle) = self
             .handles_manager
@@ -369,7 +369,7 @@ impl SftpServer<'_, DemoOpaqueFileHandle> for DemoSftpServer {
     async fn readdir<const N: usize>(
         &mut self,
         opaque_dir_handle: &DemoOpaqueFileHandle,
-        reply: &DirReply<'_, N>,
+        reply: &mut DirReply<'_, N>,
     ) -> SftpOpResult<()> {
         debug!("read dir for  {:?}", opaque_dir_handle);
 
@@ -402,7 +402,7 @@ impl SftpServer<'_, DemoOpaqueFileHandle> for DemoSftpServer {
                 let name_entry_collection = DirEntriesCollection::new(dir_iterator);
 
                 let response_read_status =
-                    name_entry_collection.send_response(reply).await?;
+                    name_entry_collection.send_response( reply).await?;
 
                 dir.read_status = response_read_status;
                 return Ok(());

demo/sftp/std/src/main.rs

@@ -95,16 +95,8 @@ impl DemoServer for StdDemo {
                 let ev = match serv.progress(&mut ph).await {
                     Ok(event) => event,
                     Err(e) => {
-                        match e {
-                            Error::NoRoom {} => {
-                                warn!("NoRoom triggered. Trying again");
-                                continue;
-                            }
-                            _ => {
-                                error!("server  progress failed: {:?}", e); // NoRoom: 2048 Bytes Output buffer
-                                return Err(e);
-                            }
-                        }
+                        error!("server  progress failed: {:?}", e); // NoRoom: 2048 Bytes Output buffer
+                        return Err(e);
                     }
                 };
 
@@ -163,7 +155,7 @@ impl DemoServer for StdDemo {
 
                 // TODO Do some research to find reasonable default buffer lengths
                 let mut buffer_in = [0u8; 512];
-                let mut incomplete_request_buffer = [0u8; 512];
+                let mut request_buffer = [0u8; 512];
 
                 match {
                     let stdio = serv.stdio(ch).await?;
@@ -173,7 +165,7 @@ impl DemoServer for StdDemo {
 
                     SftpHandler::<DemoOpaqueFileHandle, DemoSftpServer, 512>::new(
                         &mut file_server,
-                        &mut incomplete_request_buffer,
+                        &mut request_buffer,
                     )
                     .process_loop(stdio, &mut buffer_in)
                     .await?;
@@ -219,39 +211,25 @@ async fn listen(
 
 #[embassy_executor::main]
 async fn main(spawner: Spawner) {
-    env_logger::builder()
-        .filter_level(log::LevelFilter::Info)
-        .filter_module(
-            "sunset_sftp::sftphandler::sftphandler",
-            log::LevelFilter::Debug,
-        )
-        // .filter_module(
-        //     "sunset_demo_sftp_std::demosftpserver",
-        //     log::LevelFilter::Debug,
-        // )
-        // .filter_module("sunset_sftp::sftphandler", log::LevelFilter::Trace)
-        // .filter_module("sunset_sftp", log::LevelFilter::Trace)
-        // .filter_module("sunset_sftp::sftpsource", log::LevelFilter::Debug)
-        // .filter_module(
-        //     "sunset_sftp::sftphandler::sftpoutputchannelhandler",
-        //     log::LevelFilter::Info,
-        // )
-        // .filter_module("sunset::channel", log::LevelFilter::Trace)
-        // .filter_module("sunset_async::async_sunset", log::LevelFilter::Trace)
-        // .filter_module("sunset::runner", log::LevelFilter::Debug)
-        // .filter_module("sunset::traffic", log::LevelFilter::Trace)
-        // .filter_module("sunset_sftp::sftpsink", log::LevelFilter::Info)
-        // .filter_module("sunset_sftp::sftpsource", log::LevelFilter::Info)
-        // .filter_module("sunset_sftp::sftpserver", log::LevelFilter::Info)
-        // .filter_module("sunset::runner", log::LevelFilter::Info)
-        // .filter_module("sunset::encrypt", log::LevelFilter::Info)
-        // .filter_module("sunset::conn", log::LevelFilter::Info)
-        // .filter_module("sunset::kex", log::LevelFilter::Info)
-        // .filter_module("async_io", log::LevelFilter::Info)
-        // .filter_module("polling", log::LevelFilter::Info)
-        // .filter_module("embassy_net", log::LevelFilter::Info)
-        .format_timestamp_nanos()
-        .init();
+    let log_path = std::env::var("RUST_LOG_FILE").ok();
+
+    let mut builder = env_logger::Builder::new();
+
+    builder.filter_level(log::LevelFilter::Trace);
+    // if std::env::var("RUST_LOG").is_err() {
+    // } else {
+    //     builder.filter_level(log::LevelFilter::Debug);
+    // }
+    builder.format_timestamp_nanos();
+
+    if let Some(path) = log_path {
+        let file = std::fs::File::create(path).expect("Failed to create log file");
+        builder.target(env_logger::Target::Pipe(Box::new(file)));
+    } else {
+        builder.target(env_logger::Target::Stdout);
+    }
+
+    builder.init();
 
     spawner.spawn(main_task(spawner)).unwrap();
 }

demo/sftp/std/testing/log_get_single.sh

@@ -0,0 +1,67 @@
+#!/bin/bash
+
+echo "Launching latest build demo with log and strace..."
+# Create logs directory if it doesn't exist
+LOG_DIR="$PWD/logs"
+mkdir -p "$LOG_DIR"
+
+CURRENT_DIR=$(pwd)
+cd ../../../../
+RUST_LOG_FILE="$LOG_DIR/log_demo_get_single.log"  RUST_LOG="trace" strace-opt $LOG_DIR target/debug/sunset-demo-sftp-std &
+# Store the process ID
+
+echo "Sleeping for 3 seconds to let the server start..."
+sleep 3 
+
+cd "$CURRENT_DIR"
+echo "Testing Single long GETs..."
+
+echo "Cleaning up previous run files"
+rm -f -r ./*_random ./out/*_random
+
+
+# Set remote server details
+REMOTE_HOST="192.168.69.2"
+REMOTE_USER="any"
+
+# Define test files
+# FILES=("512B_random" "16kB_random" "64kB_random" "65kB_random" "2048kB_random")
+FILES=("4096kB_random")
+
+echo "Generating random data files..."
+# dd if=/dev/random bs=512 count=1 of=./512B_random 2>/dev/null
+# dd if=/dev/random bs=1024 count=16 of=./16kB_random 2>/dev/null
+# dd if=/dev/random bs=1024 count=64 of=./64kB_random 2>/dev/null
+# dd if=/dev/random bs=1024 count=65 of=./65kB_random 2>/dev/null
+# dd if=/dev/random bs=1024 count=256 of=./256kB_random 2>/dev/null
+# dd if=/dev/random bs=1024 count=1024 of=./1024kB_random 2>/dev/null
+dd if=/dev/random bs=1024 count=4096 of=./4096kB_random 2>/dev/null
+echo "Uploading files to ${REMOTE_USER}@${REMOTE_HOST}..."
+
+echo "Moving to the server folder..."
+for file in "${FILES[@]}"; do
+    mv "./${file}" "./out/${file}"
+done
+
+
+echo "Downloading files..."
+sftp -vvvvv -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=DEBUG ${REMOTE_USER}@${REMOTE_HOST} << EOF
+$(printf 'get ./%s\n' "${FILES[@]}")
+bye
+EOF
+
+echo "DOWNLOAD Test Results:"
+echo "============="
+# Test each file
+for file in "${FILES[@]}"; do
+    if diff "./${file}" "./out/${file}" >/dev/null 2>&1; then
+        echo "Download PASS: ${file}. Cleaning it"
+        rm -f -r ./${file} ./out/${file}        
+    else
+        echo "Download FAIL: ${file}". Keeping for inspection
+    fi
+done
+
+# At the end of your script, you might want to clean up:
+echo "Tests complete, shutting down the server..."
+pkill sunset-demo-sftp-std