Skip to content

feat(sftp): add deferrable mode to SFTPOperator#68520

Open
venkata-pavani wants to merge 2 commits into
apache:mainfrom
venkata-pavani:sftp-deferrable-65475
Open

feat(sftp): add deferrable mode to SFTPOperator#68520
venkata-pavani wants to merge 2 commits into
apache:mainfrom
venkata-pavani:sftp-deferrable-65475

Conversation

@venkata-pavani

Copy link
Copy Markdown

Summary

  • Add deferrable parameter to SFTPOperator, allowing get and put transfers of a single file (or list of single-file pairs, when concurrency == 1) to be deferred to a trigger instead of blocking a worker slot while the transfer is in progress.
  • Add SFTPOperatorTrigger (in airflow.providers.sftp.triggers.sftp) which performs the async file transfer via SFTPHookAsync and yields a TriggerEvent on success or failure.
  • deferrable=True raises ValueError at execute-time for unsupported configurations:
    • operation == "delete"
    • concurrency > 1
    • no ssh_conn_id available (neither explicit ssh_conn_id nor an sftp_hook with one set)

This re-implements the design from #65480 (closed unmerged due to a rebase issue), scoped down per discussion on #65475.

Closes #65475

Test plan

  • New unit tests added in providers/sftp/tests/unit/sftp/triggers/test_sftp.py (TestSFTPOperatorTrigger) covering serialization, get/put success, intermediate directory creation, and failure events.
  • New unit tests added in providers/sftp/tests/unit/sftp/operators/test_sftp.py covering deferral behavior, the ValueError guard rails, and execute_complete.
  • CI test suite (could not run pytest locally on this Windows checkout due to POSIX-only dependencies in airflow-core: fcntl, os.register_at_fork, and symlinked _shared packages).

Allow SFTPOperator to defer to a trigger for single-file get/put
transfers, freeing the worker slot while the transfer is in
progress. Directory transfers, delete operations, and
concurrency > 1 are not supported and raise a ValueError when
deferrable=True.

Closes apache#65475

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@boring-cyborg

boring-cyborg Bot commented Jun 13, 2026

Copy link
Copy Markdown

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

  • Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
  • In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
  • Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
  • Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
  • Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
  • Be sure to read the Airflow Coding style.
  • Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
    Apache Airflow is a community-driven project and together we are making it better 🚀.
    In case of doubts contact the developers at:
    Mailing List: dev@airflow.apache.org
    Slack: https://s.apache.org/airflow-slack

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add deferrable mode to SFTPOperator

1 participant