Skip to content

9.3.1 Flows #86

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
davramov wants to merge 28 commits into
base: main
Choose a base branch
from
Draft

9.3.1 Flows #86

davramov wants to merge 28 commits into from

Conversation

@davramov
Copy link
Contributor

@davramov davramov commented Sep 10, 2025 •
edited
Loading

Preparing prefect flows for BL 9.3.1

Asana Project

  • Draft 9.3.1 globus endpoints in config.yml
  • Copying boilerplate code from 7.3.3, and making it 9.3.1-specific

davramov added 24 commits January 14, 2026 11:54
@davramov
init commit for bl 9.3.1
bac02a4
@davramov
Adjusting 9.3.1 endpoint configuration to use the compute-dtn globus …
3738106 
…collection
@davramov
Adding pytests for bl931 flows
8d6e834
@davramov
Making Config931 optional, since it is initialized within the methods…
9b35bd3 
… if it is set to None
@davramov
Adding documentation for bl 9.3.1 flows
4da35bf
@davramov
changing the data description in the documentation
19128a8
@davramov
Making the move flow call the move task, and updating the pytest/disp…
62fd035 
…atcher/prefect.yaml reflect the change
@davramov
Adjusting endpoint names in the test movement flow
8948b1d
@davramov
Ensuring get_run_logger() is set in the flight check
560259a
@davramov
if the transfer test fails, it should throw a runtime error
d395dcd
@davramov
updating init_work_pools to handle when beamline id is a number (e.g.…
ffcb3b4 
… 733 -> bl733 is the folder name) or a word (dichroism -> dichroism is the folder name)
@davramov
adjusting the test directory path for 931
0058581
@davramov
Pinning fastapi==0.116.1; breaks with fastapi==0.124.0
965df42
@davramov
Switching JSON Blocks to Variable Blocks
4832eb8
@davramov
Adding flow diagram to 9.3.1 documentation
d925c89
@davramov
Updating flight scheck to move globus_test/test.txt
4236517
@davramov
removing commented out code
fc437c6
@davramov
adding _sync=True to Variable.get() call and added missing flow_run_n…
c1689fc 
…ame from schedule_prefect_flow
@davramov
Adding verbose logging to process_new_931_file_task
ccaad74
@davramov
Adding transfer_success RuntimeError if False
79f280f
@davramov
Fixing the deployment name when scheduling prune_globus_endpoint
4b24dcb
@davramov
Adding beegfs endpoint for 931
8fa4939
@davramov
Adding placeholder for transfer to beegfs
b596ea0
@davramov
reorganizing config.yml to be in numerical order by beamline number
1f6d976
@davramov
Copy link
Contributor Author

davramov commented Jan 20, 2026

Some information from Johannes about the medium/long-term data movement strategy.

Current workflow:

  1. Collect scan
  2. Save the tpx3 file and the metadata file
  3. convert tpx3 file to h5
  4. embed metadata into h5 file

Right now, steps 2-4 run on the beamline computer. So at the moment, we just need to worry about moving a single h5 file per scan.

However, in the future, the flow will be a little different.
Step 2 will still happen on the beamline computer, but steps 3+4 will happen on Lamar after transferring to beegfs.
This means the bl931 dispatcher will eventually need to listen for a list of files, and the move flow will need to trigger the file format conversion and metadata embedding codes on Lamar.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@davramov