add guidance section for developers using bps at USDF

Author: jchiang87

usdf/batch.rst

@@ -35,10 +35,28 @@ In general, using BPS is preferred to running ``pipetask`` directly since many c
 
 Running LSST Pipelines with BPS
 ===============================
-The LSST Batch Processing Service (`BPS <https://github.com/lsst/ctrl_bps>`__) is the standard execution framework for running LSST pipelines using batch resources.  There are a few different plugins to BPS that are available that can be used for running BPS on various computing systems:
+The LSST Batch Processing Service (`BPS <https://github.com/lsst/ctrl_bps>`__) is the standard execution framework for running LSST pipelines using batch resources.
 
-- :ref:`ctrl_bps_htcondor <ctrl_bps_htcondor>` 
-- ctrl_bps_panda
+Guidance for developers running pipelines
+=========================================
+- For running at USDF, we recommend using ctrl_bps_htcondor, the HTCondor-based plugin.  For questions regarding its use, please post in `#usdf-support <https://app.slack.com/client/T02SVMGU4/C082C6R9JQ1>`__ for USDF-specific issues or in `#dm-middleware-support <https://app.slack.com/client/T02SVMGU4/C08CANY4B6H>`__ for more general BPS-related questions.  Both of those channels are in the `rubin-obs <https://rubin-obs.slack.com>`__ Slack space.
+- Please use the software distributions in
+  ``/cvmfs/sw.lsst.eu/almalinux-x86_64/lsst_distrib``, if possible.  The weekly versions are made available there each Thursday, and the following script will set up the most recent version::
+
+   weekly=$(ls -rd /cvmfs/sw.lsst.eu/almalinux-x86_64/lsst_distrib/w* | head -1)
+   source $weekly/loadLSST.sh
+   setup lsst_distrib -t w_latest
+
+  The shared stack distributions in ``/sdf/group/rubin/sw`` can also be used, but we've seen general slowness for people using those distributions when a lot of jobs are running the software in that area.
+- Use BPS clustering to avoid many, e.g., >10s of thousands, of short <1 min jobs.  A common use case would be running ``isr`` on a night's worth of LSSTCam exposures.  Clustering will combine pipeline jobs into single pipetask calls resulting in fewer overall jobs and thus fewer jobs trying to set up the stack, import all the python modules, etc., at the same time.  In addition to mitigating disk contention issues, clustering makes it easier for the workflow systems to manage the execution of the overall pipeline.  Instructions for using bps clustering are available in the `BPS documentation <https://pipelines.lsst.io/modules/lsst.ctrl.bps/quickstart.html#clustering>`__.
+
+
+BPS Plugins
+===========
+A few different plugins to BPS are available that use various workflow systems for running BPS on the USDF Slurm batch system:
+
+- :ref:`ctrl_bps_htcondor <ctrl_bps_htcondor>`
+- `ctrl_bps_panda <https://panda.lsst.io/>`__  The PanDA base system can also be used for running remotely at the UK and French Data facilities.
 - :ref:`ctrl_bps_parsl <ctrl_bps_parsl>`
 
 .. _ctrl_bps_htcondor: