From 546d3ba94c96c5e2afa0ece0a75e1eda93eca9f4 Mon Sep 17 00:00:00 2001 From: Sara Burns Date: Wed, 25 Feb 2026 14:45:48 -0500 Subject: [PATCH 1/9] docs: rewriting dbt extension docs --- docs/Makefile | 3 + .../how-tos/dbt_extensions.rst | 234 +++++++----------- .../technical_documentation/how-tos/index.rst | 14 +- .../how-tos/install_aspects.rst | 2 +- .../how-tos/production_configuration.rst | 9 +- .../how-tos/xapi_transforms.rst | 2 +- 6 files changed, 106 insertions(+), 158 deletions(-) diff --git a/docs/Makefile b/docs/Makefile index d4bb2cb..47cbb95 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -14,6 +14,9 @@ help: .PHONY: help Makefile +serve_docs: ## serve the built docs locally to preview the site in the browser + sphinx-autobuild source $(BUILDDIR)/html --ignore '**/_tags/*' + # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile diff --git a/docs/technical_documentation/how-tos/dbt_extensions.rst b/docs/technical_documentation/how-tos/dbt_extensions.rst index b59a7b0..c8fdeb7 100644 --- a/docs/technical_documentation/how-tos/dbt_extensions.rst +++ b/docs/technical_documentation/how-tos/dbt_extensions.rst @@ -3,20 +3,17 @@ Extending dbt ************* -As noted in :ref:`dbt`, you can install your own custom dbt package to apply your own transforms to the event data -in Aspects. +As noted in :ref:`dbt`, you can install your own custom dbt package to apply your own transforms to the event data in Aspects. -This guide demonstrates how to create and use a custom dbt package in Aspects by building the `aspects-dbt-example`_ -repo. See `aspects-dbt-example#1`_ to follow along with each of these steps. +This guide demonstrates how to create and use a custom dbt package in Aspects by building the `sample-aspects-dbt `_ +repo. -See `Building dbt packages`_ and `Best practice guides`_ for dbt's official documentation. - -Step 0. Install dbt core -======================== +Step 1. Install dbt core (if needed) +==================================== The easiest way to install dbt core is to use pip in a python3 virtual environment. -See `aspects-dbt requirements.txt`_ for the specific package versions used by Aspects. +See `aspects-dbt requirements.txt `_ for the specific package versions used by Aspects. .. code-block:: bash @@ -26,71 +23,74 @@ See `aspects-dbt requirements.txt`_ for the specific package versions used by As pip install --upgrade pip # Install the dbt package versions used by Aspects - pip install dbt-clickhouse==1.7.2 dbt-core==1.7.0 - -See `Install dbt`_ for more ways to install dbt. - -Step 1. Initialize your dbt package -=================================== - -Create a new dbt package by following the prompts given by the ``dbt init`` tool. - -.. code-block:: bash + pip install dbt-clickhouse==x.x.x dbt-core==x.x.x - dbt init - # Enter a name for your project (letters, digits, underscore): aspects_dbt_example - # Which database would you like to use? [1] clickhouse +See `Install dbt `_ for more ways to install dbt. - ls aspects_dbt_example - # analyses/ dbt_project.yml macros/ models/ README.md seeds/ snapshots/ tests/ +Step 2. Set up new dbt package +=============================== +#. Create a new repository for your custom dbt package, and clone it to your local machine. -See `About dbt init`_ for more options. +#. In the root of your local repository, create a new dbt package by following the prompts given by the ``dbt init`` tool. -Step 2. Use the Aspects profile -=============================== + * .. code-block:: bash -Update the generated ``dbt_project.yml`` to use the ``aspects`` profile: + # Use the profile name from your ``dbt/profiles.yml`` file + dbt init --profile aspects + + # Enter a name for your project (letters, digits, underscore): + sample_aspects_dbt -.. code-block:: yaml + ls sample_aspects_dbt + # analyses/ dbt_project.yml macros/ models/ README.md seeds/ snapshots/ tests/ - # This setting configures which "profile" dbt uses for this project. - profile: 'aspects' + * ``dbt_project.yml`` should have the same `profile name `_ as your local profiles file + * See `About dbt init `_ for more options. -Also set the location for compiled SQL to match that used by ``aspects``: +#. In ``dbt_project.yml``, set the location for compiled SQL to match that used by ``aspects``: -.. code-block:: yaml + * .. code-block:: yaml - # directory which will store compiled SQL files - target-path: "target" + # directory which will store compiled SQL files + target-path: "target" Step 3. Link to aspects-dbt =========================== -Aspects charts depend on the transforms in `aspects-dbt`_, so it's important that your dbt package also installs -the same version of `aspects-dbt`_ as your version of the Aspects Tutor plugin. +Aspects charts depend on the transforms in `aspects-dbt`_, so it's important that your dbt package also installs the same version of aspects-dbt as your version of the Aspects Tutor plugin. -To do this, add a ``packages.yml`` file to your dbt package at the top level where: +To do this, add a `packages.yml `_ file to your dbt package at the top level where: -* ``git`` url matches the default value of ``DBT_REPOSITORY`` in `tutor-contrib-aspects plugin.py`_ -* ``revision`` matches the default value of ``DBT_BRANCH`` in `tutor-contrib-aspects plugin.py`_ +* ``git`` url matches the default value of ``DBT_REPOSITORY`` in `tutor-contrib-aspects plugin.py `_ +* ``revision`` matches the default value of ``DBT_BRANCH`` in `tutor-contrib-aspects plugin.py `_ .. code-block:: yaml packages: - git: "https://github.com/openedx/aspects-dbt.git" - revision: v3.4.1 + revision: vX.X.X -Step 4. Add your custom transforms -================================== + +Step 4. Test dbt connection +=========================== + +Before adding your custom transforms, it's a good idea to test that your dbt package can connect to the Clickhouse database and run the base transforms from aspects-dbt. + +#. Run ``dbt debug`` to test the connection to your database and the validity of your dbt project. +#. You might need to run ``dbt deps`` to install the dependencies for your package. +#. Run ``dbt run`` to run the base transforms from aspects-dbt. +#. You may need to run ``dbt run --full-refresh`` if the previous step fails. + +Step 5. Create your custom transforms +===================================== Here is where you will need an understanding of dbt, Clickhouse, Aspects' data schemas, and the specific transforms you want to create. -If you need any python dependencies beyond what is provided by aspects-dbt, add these to a ``requirements.txt`` file at -the top level of your repository. +If you need any python dependencies beyond what is provided by aspects-dbt, add these to a ``requirements.txt`` file at the top level of your repository. .. note:: You can use Aspects to debug your custom SQL: @@ -98,94 +98,57 @@ the top level of your repository. #. Using the menus at the top of the page, navigate to the "SQL -> SQL Lab" UI. #. Browse the schemas and run read-only SQL queries on your data. -For this tutorial, we added a new model which will be materialized by dbt into a view in Clickhouse. -Our new model calculates the average number of attempts made by users on each problem by referencing the -`int_problem_results` model created by the base aspects-dbt package (see `dbt ref`_): - -.. code-block:: - - select - problem_id, AVG(attempts) as average_attempts - from - ( - select - problem_id, - max(attempts) as attempts - from - {{ ref('int_problem_results') }} - group by - actor_id, - problem_id - ) - group by problem_id - -Next, make sure your model is configured, for example in the ``db_project.yml``. If you forget this step, dbt will warn -you when running your package. - -.. code-block:: yaml - - models: - problem_responses: - # Config indicated by + and applies to all files under models/problem_responses/ - +materialized: view - +For this tutorial, we created two new models - `course_enrollments`_ `learner_responses `_ and which will be materialized by dbt into a view and materialized view in Clickhouse. (more information on `materialized views `_.) -See `Configure dbt models`_ to learn more. -Step 5. Add tests -================= +Step 6. Add dbt tests +===================== -Writing tests for your transforms is important: not only can tests validate and document your intended changes, they -can be used to guard against data edge cases and regressions from future code changes. +Writing tests for your transforms is important to validate and document your intended changes, and guard against data edge cases and regressions from future code changes. -dbt generic tests are defined as SQL files, where the goal of the SQL statement is to return zero records. +There are two types of dbt tests; `data tests `_ and `unit tests `_. -Because our new `average_attempts` model aggregates on `actor_id` and `problem_id`, it should only have 1 entry for each -`problem_id`. So our test can be: +Run ``dbt test`` to run both the data and unit tests for your package. -.. code-block:: +Data tests +------------ +Data tests can be defined in the `schema.yml `_ file for each model, and are used to validate properties of the data such as types, accepted values, uniqueness, and relationships between tables. - -- average_attempts should only have one record for each problem_id. - select - count(*) as num_rows - from - {{ ref('average_attempts') }} - group by - problem_id - having num_rows > 1 +Data tests can also be defined in a `SQL file `_, where the goal of the SQL statement is to return zero records. +Unit tests +---------- +Unit tests are used to validate the logic of your dbt models in isolation from the underlying data. They are defined in a `unit_tests.yaml `_ file within the ``models`` directory. -See `Writing data tests`_ for more examples. +Other unit test resources: +- `dbt Unit Testing: Why You Need Them, Tutorial & Best Practices `_ +- `dbt unit testing best practices `_ -Step 6. Install and use your dbt package +Step 7. Install and use your dbt package ======================================== Once you've pushed all the changes to your custom dbt package repo, now we're ready to use it. -Use ``tutor config save`` to update the following Tutor variables to use your custom package instead of the Aspects -default. +Use ``tutor config save`` to update the following Tutor variables to use your custom package instead of the Aspects default. -- ``DBT_REPOSITORY``: A git repository URL to clone and use as the dbt project. + - ``DBT_REPOSITORY``: The git repository URL of your custom dbt package. - Set this to the URL for your custom dbt package. + Default: ``https://github.com/openedx/aspects-dbt`` - Default: ``https://github.com/openedx/aspects-dbt`` -- ``DBT_BRANCH``: The branch to use when cloning the dbt project. + - ``DBT_BRANCH``: The hash/branch/tag of your custom dbt package that you wish to use. - Set this to the hash/branch/tag of the custom dbt package that you wish to use. + Default: the latest tagged version of aspects-dbt - Default: varies between versions of Aspects. -- ``EXTRA_DBT_PACKAGES``: Add any python packages that your dbt project requires here. + - ``EXTRA_DBT_PACKAGES``: Add any python packages that your dbt project requires here. - Default: ``[]`` -- ``DBT_PROFILE_*``: variables used in the Aspects ``dbt/profiles.yml`` file, including several Clickhouse connection settings. + - ``DBT_PROFILE_*``: variables used in the Aspects ``dbt/profiles.yml`` file, including several Clickhouse connection settings. -- ``DBT_SSH_KEY``: The private SSH key to use when cloning the dbt project. Only necessary if you are using a private repository. + - ``DBT_SSH_KEY``: The private SSH key to use when cloning the dbt project. Only necessary if you are using a private repository. Once your package is configured in Tutor, you can run dbt commands directly on your deployment. -See `dbt commands`_ for a full list of available commands. +See `dbt commands `_ for a full list of available commands. .. code-block:: bash @@ -195,49 +158,30 @@ See `dbt commands`_ for a full list of available commands. # Deploy your customizations tutor dev do dbt -c "run" - # Run tests on the data + # Run data tests on the data tutor dev do dbt -c "test" + # Run unit tests on the data + tutor dev do dbt -c "test --selector unit_tests" -Step 7. Troubleshooting -======================= + # To push your new transformations to Superset SQL Lab + tutor dev do import-assets -You may need to repeat steps 4-6 a few times to resolve any warnings or errors that dbt reports with your package. -Don't forget to push your changes to your repo before running the tutor dbt command: it fetches a clean copy of your -configured package repo + branch each time it runs. +Troubleshooting +=============== +- Tutor commands may need to be run with ``--only_changed False`` to force a full dbt run if you have made changes to your dbt package that are not being picked up. -See `dbt debugging`_ and `The missing guide to debug() in dbt`_ for more information on how to debug issues with your package. +- Don't forget to push your changes to your repo before running the tutor dbt command: it fetches a clean copy of your configured package repo + branch each time it runs. References -########## - -* `Building dbt packages`_: dbt's guide to building packages -* `Best practice guides`_: dbt's guidelines on project structure, style, and setup. -* `About dbt models`_: dbt's guide to creating SQL or Python model transforms -* `Writing data tests`_: dbt's guide to writing package tests -* `dbt commands`_: list of all dbt commands -* `dbt debugging`_: guide for debugging issues with dbt -* `The missing guide to debug() in dbt`_: detailed advice for debugging issues with dbt -* `aspects-dbt`_: Aspects' dbt package -* `aspects-dbt-example`_: the demo custom dbt package used in this tutorial. -* `eduNEXT/dbt-aspects-unidigital`_: a real custom dbt package running in production Aspects - -.. _aspects-dbt: https://github.com/openedx/aspects-dbt -.. _aspects-dbt-example: https://github.com/open-craft/aspects-dbt-example -.. _aspects-dbt-example#1: https://github.com/open-craft/aspects-dbt-example/pull/1 -.. _aspects-dbt requirements.txt: https://github.com/openedx/aspects-dbt/blob/main/requirements.txt -.. _About dbt init: https://docs.getdbt.com/reference/commands/init -.. _About dbt models: https://docs.getdbt.com/docs/build/models -.. _Configure dbt models: https://docs.getdbt.com/reference/model-configs -.. _Best practice guides: https://docs.getdbt.com/best-practices -.. _dbt commands: https://docs.getdbt.com/reference/dbt-commands -.. _dbt debugging: https://docs.getdbt.com/guides/debug-errors -.. _dbt ref: https://docs.getdbt.com/reference/dbt-jinja-functions/ref -.. _eduNEXT/dbt-aspects-unidigital: https://github.com/eduNEXT/dbt-aspects-unidigital -.. _Building dbt packages: https://docs.getdbt.com/guides/building-packages -.. _Install dbt: https://docs.getdbt.com/docs/core/installation-overview -.. _Writing data tests: https://docs.getdbt.com/best-practices/writing-custom-generic-tests -.. _tutor-contrib-aspects plugin.py: https://github.com/openedx/tutor-contrib-aspects/blob/main/tutoraspects/plugin.py -.. _The missing guide to debug() in dbt: https://docs.getdbt.com/blog/guide-to-jinja-debug +========== + +* `aspects-dbt `_: Aspects' dbt package +* `sample-aspects-dbt `_: the demo custom dbt package used in this tutorial +* `Building dbt packages `_: dbt's guide to building packages +* `Best practice guides `_: dbt's guidelines on project structure, style, and setup +* `About dbt models `_: dbt's guide to creating SQL or Python model transforms +* `dbt debugging `_: guide for debugging issues with dbt +* `The missing guide to debug() in dbt `_: detailed advice for debugging issues with dbt diff --git a/docs/technical_documentation/how-tos/index.rst b/docs/technical_documentation/how-tos/index.rst index e661e9c..c13b025 100644 --- a/docs/technical_documentation/how-tos/index.rst +++ b/docs/technical_documentation/how-tos/index.rst @@ -2,15 +2,19 @@ How-Tos ####### .. toctree:: - :maxdepth: 2 + :maxdepth: 1 :caption: Content: - Aspects Plugin for Tutor + Install Aspects Plugin for Tutor Upgrade Aspects Configure Aspects for Production - Backfill old or missing data Troubleshooting Aspects + Running Aspects without Tutor Connect to external Clickhouse database + Run Aspects in a ClickHouse cluster + Backfill old or missing data + Extending dbt + Customizing xAPI Transforms Changing the xAPI actor identifier Superset extra assets Superset language settings @@ -19,7 +23,3 @@ How-Tos Superset custom roles Superset custom config settings Clickhouse extra SQL - Extending dbt - Run Aspects in a ClickHouse cluster - xAPI Transforms - Running Aspects without Tutor diff --git a/docs/technical_documentation/how-tos/install_aspects.rst b/docs/technical_documentation/how-tos/install_aspects.rst index 9ebafa2..8dfd0c7 100644 --- a/docs/technical_documentation/how-tos/install_aspects.rst +++ b/docs/technical_documentation/how-tos/install_aspects.rst @@ -1,4 +1,4 @@ -Aspects Plugin for Tutor +Install Aspects Plugin for Tutor ************************ Aspects is implemented as a Tutor plugin. To learn more about how to install it, please refer to the plugin GitHub site: diff --git a/docs/technical_documentation/how-tos/production_configuration.rst b/docs/technical_documentation/how-tos/production_configuration.rst index da5d31f..e5b7699 100644 --- a/docs/technical_documentation/how-tos/production_configuration.rst +++ b/docs/technical_documentation/how-tos/production_configuration.rst @@ -199,13 +199,13 @@ By default, Aspects enables plugin functionality in the LMS that embeds a define - ``ASPECTS_LEARNER_GROUPS_HELP_MARKDOWN`` controls the content of the "Help" tab in the At-Risk Learners dashboard - ``ASPECTS_OPERATOR_HELP_MARKDOWN`` controls the content of the "Help" tab in the Operator dashboard -In-context Metrics +In-Context Metrics ------------------ -Starting in the Teak Open edX release, Aspects provides in-context metrics in Studio. The following settings controls this functionality. +Starting in the Teak Open edX release, Aspects provides :ref:`In-Context Dashboards` in Studio. Reference the `plugin ` for setup instructions. -- ``ASPECTS_ENABLE_STUDIO_IN_CONTEXT_METRICS`` - Enables or disables in-context metrics. -- ``ASPECTS_IN_CONTEXT_DASHBOARDS`` - A dictionary mapping block types to in-context dashboards. You can use this option to remove or replace the in-context dashboard for a block type. The key `course` defines the in-context dashboard for course overview. +For further customization, the setting ``ASPECTS_IN_CONTEXT_DASHBOARDS`` (in `platform-plugin-aspects`_) is a dictionary mapping block types to in-context dashboards. +You can use this setting to remove or replace the in-context dashboard for a block type. The key `course` defines the in-context dashboard for course overview. Ralph Accessibility @@ -396,3 +396,4 @@ These are also captured in the Aspects Operator Dashboard as well as a filterabl .. _Ralph via Helm chart: https://openfun.github.io/ralph/latest/tutorials/helm/ .. _dump_data_to_clickhouse arguments: https://github.com/openedx/platform-plugin-aspects/blob/951ed84de01dda6bec9923c60fcd96bf80d6fa54/platform_plugin_aspects/management/commands/dump_data_to_clickhouse.py#L91 .. _transform_tracking_logs: https://event-routing-backends.readthedocs.io/en/latest/technical_documentation/how-tos/how_to_bulk_transform.html +.. _platform-plugin-aspects: https://github.com/openedx/platform-plugin-aspects/blob/main/platform_plugin_aspects/settings/common.py#L112 diff --git a/docs/technical_documentation/how-tos/xapi_transforms.rst b/docs/technical_documentation/how-tos/xapi_transforms.rst index 54d3599..000b72c 100644 --- a/docs/technical_documentation/how-tos/xapi_transforms.rst +++ b/docs/technical_documentation/how-tos/xapi_transforms.rst @@ -1,6 +1,6 @@ .. _xapi_transforms: -xAPI Transforms +Customizing xAPI Transforms ############### Aspects converts raw Open edX tracking event JSON into :ref:`xapi-concepts` for storage and analysis. This conversion process is called "transformation". From 133e3c2897231c8b2b3cbdbbf49d1152040d9629 Mon Sep 17 00:00:00 2001 From: Sara Burns Date: Wed, 25 Feb 2026 15:01:37 -0500 Subject: [PATCH 2/9] docs: format updates --- docs/technical_documentation/how-tos/dbt_extensions.rst | 2 +- docs/technical_documentation/how-tos/install_aspects.rst | 2 +- docs/technical_documentation/how-tos/xapi_transforms.rst | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/technical_documentation/how-tos/dbt_extensions.rst b/docs/technical_documentation/how-tos/dbt_extensions.rst index c8fdeb7..b36602d 100644 --- a/docs/technical_documentation/how-tos/dbt_extensions.rst +++ b/docs/technical_documentation/how-tos/dbt_extensions.rst @@ -98,7 +98,7 @@ If you need any python dependencies beyond what is provided by aspects-dbt, add #. Using the menus at the top of the page, navigate to the "SQL -> SQL Lab" UI. #. Browse the schemas and run read-only SQL queries on your data. -For this tutorial, we created two new models - `course_enrollments`_ `learner_responses `_ and which will be materialized by dbt into a view and materialized view in Clickhouse. (more information on `materialized views `_.) +For this tutorial, we created two new models - `course_enrollments `_ and `learner_responses `_ and which will be materialized by dbt into a view and materialized view in Clickhouse. (more information on `materialized views `_.) Step 6. Add dbt tests diff --git a/docs/technical_documentation/how-tos/install_aspects.rst b/docs/technical_documentation/how-tos/install_aspects.rst index 8dfd0c7..a491b05 100644 --- a/docs/technical_documentation/how-tos/install_aspects.rst +++ b/docs/technical_documentation/how-tos/install_aspects.rst @@ -1,5 +1,5 @@ Install Aspects Plugin for Tutor -************************ +******************************** Aspects is implemented as a Tutor plugin. To learn more about how to install it, please refer to the plugin GitHub site: ``_ diff --git a/docs/technical_documentation/how-tos/xapi_transforms.rst b/docs/technical_documentation/how-tos/xapi_transforms.rst index 000b72c..b2c43df 100644 --- a/docs/technical_documentation/how-tos/xapi_transforms.rst +++ b/docs/technical_documentation/how-tos/xapi_transforms.rst @@ -1,7 +1,7 @@ .. _xapi_transforms: Customizing xAPI Transforms -############### +########################### Aspects converts raw Open edX tracking event JSON into :ref:`xapi-concepts` for storage and analysis. This conversion process is called "transformation". From 81045cb3366132be9ff5d89362a5aa94144f42df Mon Sep 17 00:00:00 2001 From: Sara Burns Date: Fri, 27 Feb 2026 15:08:15 -0500 Subject: [PATCH 3/9] docs: superset asset plugin --- .../how-tos/dbt_extensions.rst | 12 ++- .../how-tos/superset_extra_assets.rst | 86 ++++++------------- 2 files changed, 32 insertions(+), 66 deletions(-) diff --git a/docs/technical_documentation/how-tos/dbt_extensions.rst b/docs/technical_documentation/how-tos/dbt_extensions.rst index b36602d..8b9f3e1 100644 --- a/docs/technical_documentation/how-tos/dbt_extensions.rst +++ b/docs/technical_documentation/how-tos/dbt_extensions.rst @@ -56,6 +56,9 @@ Step 2. Set up new dbt package # directory which will store compiled SQL files target-path: "target" +#. Create a `requirements.txt `_ file at the top level of your repository with the same version of ``dbt-core`` as in Step 1. This is needed when building the aspects image. + + * If you need any python dependencies beyond what is provided by aspects-dbt, add these to the requirements file. Step 3. Link to aspects-dbt =========================== @@ -87,10 +90,7 @@ Before adding your custom transforms, it's a good idea to test that your dbt pac Step 5. Create your custom transforms ===================================== -Here is where you will need an understanding of dbt, Clickhouse, Aspects' data schemas, and the specific transforms you -want to create. - -If you need any python dependencies beyond what is provided by aspects-dbt, add these to a ``requirements.txt`` file at the top level of your repository. +Here is where you will need an understanding of dbt, Clickhouse, Aspects' data schemas, and the specific transforms you want to create. .. note:: You can use Aspects to debug your custom SQL: @@ -174,6 +174,10 @@ Troubleshooting - Don't forget to push your changes to your repo before running the tutor dbt command: it fetches a clean copy of your configured package repo + branch each time it runs. +Next Steps +========== +With your custom dbt package, you can create new charts and dashboards in Superset using the new models created by your dbt transforms. See :ref:`superset-extra-assets` for more information. + References ========== diff --git a/docs/technical_documentation/how-tos/superset_extra_assets.rst b/docs/technical_documentation/how-tos/superset_extra_assets.rst index f2e828e..734b8aa 100644 --- a/docs/technical_documentation/how-tos/superset_extra_assets.rst +++ b/docs/technical_documentation/how-tos/superset_extra_assets.rst @@ -3,8 +3,7 @@ Superset extra assets ********************* -Developers can use the `superset-extra-assets` patch to add extra assets to Superset and those -will be imported at initialization. +Developers can use the `superset-extra-assets` patch to add extra assets to Superset and those will be imported at initialization. An example a tutor inline plugin using the patch is the following: @@ -28,82 +27,45 @@ An example a tutor inline plugin using the patch is the following: ... -The patch is expected to be a list of assets with an extra attribute called **_file_name**, -which uniquely identifies the asset entry. Each asset is expected to be a valid yaml object -with the attributes that Superset expects for each asset type. See -`assets `_ -for examples of asset yaml declarations. +The patch is expected to be a list of assets with an extra attribute called ``_file_name``, which uniquely identifies the asset entry. Each asset is expected to be a valid yaml object with the attributes that Superset expects for each asset type. See +`assets `_ for examples of asset yaml declarations. -In most cases, however, you will want to develop Superset dashboards, charts, datasets, and -sometimes even databases in the Superset UI and import them into a plugin that can be managed -in source control. As of Aspects 1.0.3 use the same command to import assets into a Tutor -plugin that we use to maintain Aspects itself. +In most cases, however, you will want to develop Superset dashboards, charts, datasets, and sometimes even databases in the Superset UI and import them into a plugin that can be managed in source control. As of Aspects 1.0.3, use the same command to import assets into a Tutor plugin that we use to maintain Aspects itself. -You can use the `Tutor Plugin Cookiecutter `_ -to create a skeleton plugin. Then you can create a patch file that will include all of your -Superset assets into Aspects when the plugin is enabled. +Creating a Tutor plugin +======================= -The file name is the patch name ``/patches/superset-extra-assets``: +#. Use the `Tutor Plugin Cookiecutter `_ to create a skeleton plugin. See `tutor-contrib-aspects-sample `_ for an example. -.. code-block:: +#. Create a `patch file `_ that will recursively include all of your Superset assets from ``/templates/build/assets`` into Aspects when the plugin is enabled and ``tutor config save`` is run. - {% for file in "/superset-assets/"|walk_templates %} - --- - _file_name: {{ file.split('/')[-1] }} - {% include file %} - {% endfor %} + * The directory containing ``assets`` must be included in `ENV_TEMPLATE_TARGETS `_. Since this defaults to ``build`` and ``apps``, you can either add your asset files to one of these directories, or add an additional target to the plugin file. +#. Create a `requirements.txt `_ with ``ruff`` for formatting. -This recursively includes each file in the superset-assets directory of your extension's -tempates directory, when ``tutor config save`` is run. For example -``/templates/superset-assets/*``. +Importing assets from Superset +============================== -Now you can export your Superset dashboard, which will download to your local computer -as a .zip file. As long as you are running Aspects 1.0.3 or newer you can use the following -command to unzip the file to your plugin directory: +#. From the Superset UI, you can export your dashboard, which will download to your local computer as a .zip file. + +#. As long as you are running Aspects 1.0.3 or newer you can use this command to unzip the file to your plugin directory + ``tutor aspects import_superset_zip .zip --base_assets_path /templates/build/assets/`` -.. code-block:: + * This command does several things to make import safer and easier, including checking for some required security fields, safely naming files so that charts with duplicate names don't overwrite each other, and checking that certain fields use Tutor settings instead of hard coded values. It also adds a special ``_file_name`` key that tells Aspects what name to use for the file when importing back to Superset. - tutor aspects import_superset_zip /.zip --base_assets_path //templates//superset-assets/ +#. Before importing a dashboard back into Superset, you will need to manually add a `_roles `_ key, which sets the permissions for who can view the dashboard. - -This command does several things to make import safer and easier, including checking for some -required security fields, safely naming files so that charts with duplicate names don't -overwrite each other, and checking that certain fields use Tutor settings instead of hard -coded values. It also adds a special ``_file_name`` key that tells Aspects what name to use -for the file when importing back to Superset. + * In addition to the default roles (student, instructor, operator, admin) you can also add custom roles in a ``superset_roles`` patch. .. warning:: - When importing a dashboard you will need to manually add a ``_roles`` key, which - sets the permissions for who can view the dashboard. In addition to the default roles - (student, instructor, operator, admin) you can also add custom roles . - - -An example roles key looks like this: - -.. code-block:: - - _roles: - - '{{ SUPERSET_ROLES_MAPPING.instructor }}' - - -.. warning:: - - When a Superset zip is exported it will contain any datasets and databases needed for - all of the charts in that dashboard. This may include the default xapi and MySQL - databases or Aspects datasets that you probably do not want to overwrite, and can - include their passwords! - - Use caution when importing datasets or databases your project and make sure that they - are only ones you have created yourself unless you truly intend to overwrite defaults. + When a Superset zip is exported it will contain any datasets and databases needed for all of the charts in that dashboard. This may include the default xapi and MySQL databases or Aspects datasets that you probably do not want to overwrite, and can include their passwords! + Use caution when importing datasets or databases in your project and make sure that they are only ones you have created yourself unless you truly intend to overwrite defaults. -A simple example showing the final directory structure is available -`here `_. +Importing assets into Superset +============================== -Once you have your files imported to the plugin, make sure it is installed and enabled -in your Tutor environment, then you should just be able to do ``tutor config save`` and -``tutor import-assets`` to update your Superset assets. +Once you have your files imported to the plugin, make sure it is installed and enabled in your Tutor environment, then run ``tutor config save`` and ``tutor import-assets`` to update your Superset assets. From 6d18bc9ea1f9afdf16f01d02688b9ff486356db6 Mon Sep 17 00:00:00 2001 From: Sara Burns Date: Fri, 27 Feb 2026 15:13:09 -0500 Subject: [PATCH 4/9] docs: superset asset plugin --- .../how-tos/superset_extra_assets.rst | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/technical_documentation/how-tos/superset_extra_assets.rst b/docs/technical_documentation/how-tos/superset_extra_assets.rst index 734b8aa..1da82c2 100644 --- a/docs/technical_documentation/how-tos/superset_extra_assets.rst +++ b/docs/technical_documentation/how-tos/superset_extra_assets.rst @@ -43,12 +43,12 @@ Creating a Tutor plugin #. Create a `requirements.txt `_ with ``ruff`` for formatting. -Importing assets from Superset -============================== +Import assets into plugin +========================= #. From the Superset UI, you can export your dashboard, which will download to your local computer as a .zip file. -#. As long as you are running Aspects 1.0.3 or newer you can use this command to unzip the file to your plugin directory +#. As long as you are running Aspects 1.0.3 or newer you can use this command to unzip the file to your plugin directory. ``tutor aspects import_superset_zip .zip --base_assets_path /templates/build/assets/`` @@ -64,8 +64,8 @@ Importing assets from Superset Use caution when importing datasets or databases in your project and make sure that they are only ones you have created yourself unless you truly intend to overwrite defaults. -Importing assets into Superset -============================== +Update Superset assets +====================== Once you have your files imported to the plugin, make sure it is installed and enabled in your Tutor environment, then run ``tutor config save`` and ``tutor import-assets`` to update your Superset assets. From fa8040ea3099d872f528d10cb4a134187cd7ad7c Mon Sep 17 00:00:00 2001 From: Sara Burns Date: Wed, 4 Mar 2026 09:30:00 -0500 Subject: [PATCH 5/9] docs: Apply suggestion from @sarina Co-authored-by: Sarina Canelake --- docs/technical_documentation/how-tos/dbt_extensions.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/technical_documentation/how-tos/dbt_extensions.rst b/docs/technical_documentation/how-tos/dbt_extensions.rst index 8b9f3e1..821c20a 100644 --- a/docs/technical_documentation/how-tos/dbt_extensions.rst +++ b/docs/technical_documentation/how-tos/dbt_extensions.rst @@ -128,7 +128,7 @@ Other unit test resources: Step 7. Install and use your dbt package ======================================== -Once you've pushed all the changes to your custom dbt package repo, now we're ready to use it. +Once you've pushed all the changes to your custom dbt package repo, you're ready to use it. Use ``tutor config save`` to update the following Tutor variables to use your custom package instead of the Aspects default. From 3a5f2d2d00c48c6a6610fc7ffc028e10730bb73e Mon Sep 17 00:00:00 2001 From: Sara Burns Date: Thu, 5 Mar 2026 10:58:59 -0500 Subject: [PATCH 6/9] docs: addressing pr comments --- .../how-tos/dbt_extensions.rst | 122 ++++++++++++------ .../how-tos/superset_extra_assets.rst | 2 +- 2 files changed, 85 insertions(+), 39 deletions(-) diff --git a/docs/technical_documentation/how-tos/dbt_extensions.rst b/docs/technical_documentation/how-tos/dbt_extensions.rst index 821c20a..3447c19 100644 --- a/docs/technical_documentation/how-tos/dbt_extensions.rst +++ b/docs/technical_documentation/how-tos/dbt_extensions.rst @@ -8,12 +8,10 @@ As noted in :ref:`dbt`, you can install your own custom dbt package to apply you This guide demonstrates how to create and use a custom dbt package in Aspects by building the `sample-aspects-dbt `_ repo. -Step 1. Install dbt core (if needed) +Step 1. Install dbt ==================================== -The easiest way to install dbt core is to use pip in a python3 virtual environment. - -See `aspects-dbt requirements.txt `_ for the specific package versions used by Aspects. +The easiest way to install dbt is to use pip in a python3 virtual environment. .. code-block:: bash @@ -22,6 +20,10 @@ See `aspects-dbt requirements.txt `_ for the specific package versions used by Aspects. Both dbt-core and dbt-clickhouse are required. + +.. code-block:: bash + # Install the dbt package versions used by Aspects pip install dbt-clickhouse==x.x.x dbt-core==x.x.x @@ -32,11 +34,11 @@ Step 2. Set up new dbt package #. Create a new repository for your custom dbt package, and clone it to your local machine. -#. In the root of your local repository, create a new dbt package by following the prompts given by the ``dbt init`` tool. +#. In the root of your local repository, create a new dbt package by following the prompts given by the ``dbt init`` tool. See `About dbt init `_ for more options. - * .. code-block:: bash + .. code-block:: bash - # Use the profile name from your ``dbt/profiles.yml`` file + # Use the profile name from your ``dbt/profiles.yml`` file or 'aspects' dbt init --profile aspects # Enter a name for your project (letters, digits, underscore): @@ -45,20 +47,38 @@ Step 2. Set up new dbt package ls sample_aspects_dbt # analyses/ dbt_project.yml macros/ models/ README.md seeds/ snapshots/ tests/ - * ``dbt_project.yml`` should have the same `profile name `_ as your local profiles file + .. attention :: + + Possible error when running ``dbt init``: + + .. code-block:: bash + + No sample profile found for clickhouse + + Make sure the `profile `_ matches your local ``dbt/profiles.yml`` profile name. - * See `About dbt init `_ for more options. + .. code-block:: yaml -#. In ``dbt_project.yml``, set the location for compiled SQL to match that used by ``aspects``: + # This setting configures which "profile" dbt uses for this project. + profile: "aspects" - * .. code-block:: yaml +#. In ``dbt_project.yml``, set the location for compiled SQL to match the `aspects target path `_: + + .. code-block:: yaml # directory which will store compiled SQL files target-path: "target" -#. Create a `requirements.txt `_ file at the top level of your repository with the same version of ``dbt-core`` as in Step 1. This is needed when building the aspects image. +#. Create a `requirements.txt `_ file at the top level of your repository with the same version of dbt-core as in Step 1. This is needed when building the aspects image. + + .. code-block:: + + # requirements.txt + dbt-core~=x.x.x + + .. note :: - * If you need any python dependencies beyond what is provided by aspects-dbt, add these to the requirements file. + If you need any python dependencies beyond what is provided by aspects-dbt, add these to the requirements file. Step 3. Link to aspects-dbt =========================== @@ -82,23 +102,35 @@ Step 4. Test dbt connection Before adding your custom transforms, it's a good idea to test that your dbt package can connect to the Clickhouse database and run the base transforms from aspects-dbt. -#. Run ``dbt debug`` to test the connection to your database and the validity of your dbt project. -#. You might need to run ``dbt deps`` to install the dependencies for your package. -#. Run ``dbt run`` to run the base transforms from aspects-dbt. -#. You may need to run ``dbt run --full-refresh`` if the previous step fails. +* Run ``dbt debug`` to test the connection to your database and the validity of your dbt project. +* Run ``dbt run`` to run the base transforms from aspects-dbt. + + * You may need to run ``dbt run --full-refresh`` if the previous step fails. + +.. attention :: + + Possible error when running dbt commands: + + .. code-block:: bash + + Compilation Error + dbt found 1 package(s) specified in packages.yml, but only 0 package(s) installed in dbt_packages. + Run "dbt deps" to install package dependencies. + + Run ``dbt deps`` to install the dependencies for your package, including aspects-dbt. Step 5. Create your custom transforms ===================================== Here is where you will need an understanding of dbt, Clickhouse, Aspects' data schemas, and the specific transforms you want to create. -.. note:: You can use Aspects to debug your custom SQL: +.. tip:: You can use Aspects to debug your custom SQL: #. Login to Superset as an Open edX superuser. #. Using the menus at the top of the page, navigate to the "SQL -> SQL Lab" UI. #. Browse the schemas and run read-only SQL queries on your data. -For this tutorial, we created two new models - `course_enrollments `_ and `learner_responses `_ and which will be materialized by dbt into a view and materialized view in Clickhouse. (more information on `materialized views `_.) +For this tutorial, we created two new models - `course_enrollments `_ and `learner_responses `_ and which will be materialized by dbt into a view and materialized view in Clickhouse. More information on materialized views `here `_. Step 6. Add dbt tests @@ -106,23 +138,38 @@ Step 6. Add dbt tests Writing tests for your transforms is important to validate and document your intended changes, and guard against data edge cases and regressions from future code changes. -There are two types of dbt tests; `data tests `_ and `unit tests `_. - -Run ``dbt test`` to run both the data and unit tests for your package. +There are two types of dbt tests; `data tests `_ and `unit tests `_. Run ``dbt test`` to run all tests for your package. Data tests ------------ -Data tests can be defined in the `schema.yml `_ file for each model, and are used to validate properties of the data such as types, accepted values, uniqueness, and relationships between tables. +Data tests can be defined in the `schema.yml `_ file for each model, and are used to validate properties of the data such as types, accepted values, uniqueness, and relationships between tables. + +.. code-block:: yaml + + - name: enrollment_mode + description: "The enrollment mode of the user" + data_tests: + - accepted_values: + values: ["audit", "honor", "verified"] Data tests can also be defined in a `SQL file `_, where the goal of the SQL statement is to return zero records. +.. code-block:: + + select count(*) as num_rows + from {{ ref('enrollment_mode') }} + group by org, course_key, enrollment_mode, course_name, course_run + having num_rows > 1 + Unit tests ---------- Unit tests are used to validate the logic of your dbt models in isolation from the underlying data. They are defined in a `unit_tests.yaml `_ file within the ``models`` directory. -Other unit test resources: -- `dbt Unit Testing: Why You Need Them, Tutorial & Best Practices `_ -- `dbt unit testing best practices `_ +.. seealso:: Other unit test resources: + + * `dbt Unit Testing: Why You Need Them, Tutorial & Best Practices `_ + + * `dbt unit testing best practices `_ Step 7. Install and use your dbt package @@ -132,24 +179,23 @@ Once you've pushed all the changes to your custom dbt package repo, you're ready Use ``tutor config save`` to update the following Tutor variables to use your custom package instead of the Aspects default. - - ``DBT_REPOSITORY``: The git repository URL of your custom dbt package. +* ``DBT_REPOSITORY``: The git repository URL of your custom dbt package. - Default: ``https://github.com/openedx/aspects-dbt`` + (Default: ``https://github.com/openedx/aspects-dbt``) - - ``DBT_BRANCH``: The hash/branch/tag of your custom dbt package that you wish to use. +* ``DBT_BRANCH``: The hash/branch/tag of your custom dbt package that you wish to use. - Default: the latest tagged version of aspects-dbt + (Default: the latest tagged version of aspects-dbt) - - ``EXTRA_DBT_PACKAGES``: Add any python packages that your dbt project requires here. +Optional + * ``EXTRA_DBT_PACKAGES``: Add any python packages that your dbt project requires here. - - ``DBT_PROFILE_*``: variables used in the Aspects ``dbt/profiles.yml`` file, including several Clickhouse connection settings. + * ``DBT_PROFILE_*``: variables used in the Aspects ``dbt/profiles.yml`` file, including several Clickhouse connection settings. - - ``DBT_SSH_KEY``: The private SSH key to use when cloning the dbt project. Only necessary if you are using a private repository. + * ``DBT_SSH_KEY``: The private SSH key to use when cloning the dbt project. Only necessary if you are using a private repository. Once your package is configured in Tutor, you can run dbt commands directly on your deployment. -See `dbt commands `_ for a full list of available commands. - .. code-block:: bash # Build and test your package @@ -167,18 +213,18 @@ See `dbt commands `_ for a full # To push your new transformations to Superset SQL Lab tutor dev do import-assets +See `dbt commands `_ for a full list of available commands. Troubleshooting =============== -- Tutor commands may need to be run with ``--only_changed False`` to force a full dbt run if you have made changes to your dbt package that are not being picked up. +* Tutor dbt commands may need to be run with ``--only_changed False`` to force a full dbt run if you have made changes to your dbt package that are not being picked up. -- Don't forget to push your changes to your repo before running the tutor dbt command: it fetches a clean copy of your configured package repo + branch each time it runs. +* Don't forget to push your changes to your repo before running the tutor dbt command: it fetches a clean copy of your configured package repo + branch each time it runs. Next Steps ========== With your custom dbt package, you can create new charts and dashboards in Superset using the new models created by your dbt transforms. See :ref:`superset-extra-assets` for more information. - References ========== diff --git a/docs/technical_documentation/how-tos/superset_extra_assets.rst b/docs/technical_documentation/how-tos/superset_extra_assets.rst index 1da82c2..354c85d 100644 --- a/docs/technical_documentation/how-tos/superset_extra_assets.rst +++ b/docs/technical_documentation/how-tos/superset_extra_assets.rst @@ -24,7 +24,7 @@ An example a tutor inline plugin using the patch is the following: ... - _file_name: my-dataset.yaml table_name: "..." - ... + ... The patch is expected to be a list of assets with an extra attribute called ``_file_name``, which uniquely identifies the asset entry. Each asset is expected to be a valid yaml object with the attributes that Superset expects for each asset type. See From dd9b5d6b0178339c6b5f847cbd82bafca0d14a0c Mon Sep 17 00:00:00 2001 From: Sara Burns Date: Mon, 9 Mar 2026 10:41:22 -0400 Subject: [PATCH 7/9] docs: addressing pr comments --- .../how-tos/dbt_extensions.rst | 4 +- .../how-tos/superset_extra_assets.rst | 74 ++++++++++++++----- 2 files changed, 57 insertions(+), 21 deletions(-) diff --git a/docs/technical_documentation/how-tos/dbt_extensions.rst b/docs/technical_documentation/how-tos/dbt_extensions.rst index 3447c19..6594637 100644 --- a/docs/technical_documentation/how-tos/dbt_extensions.rst +++ b/docs/technical_documentation/how-tos/dbt_extensions.rst @@ -62,7 +62,7 @@ Step 2. Set up new dbt package # This setting configures which "profile" dbt uses for this project. profile: "aspects" -#. In ``dbt_project.yml``, set the location for compiled SQL to match the `aspects target path `_: +#. In ``dbt_project.yml``, set the location for compiled SQL to match the `aspects target path `_: .. code-block:: yaml @@ -142,7 +142,7 @@ There are two types of dbt tests; `data tests `_ file for each model, and are used to validate properties of the data such as types, accepted values, uniqueness, and relationships between tables. +Data tests can be defined in the `schema.yml `_ file for each model, and are used to validate properties of the data such as types, accepted values, uniqueness, and relationships between tables. .. code-block:: yaml diff --git a/docs/technical_documentation/how-tos/superset_extra_assets.rst b/docs/technical_documentation/how-tos/superset_extra_assets.rst index 354c85d..84994f7 100644 --- a/docs/technical_documentation/how-tos/superset_extra_assets.rst +++ b/docs/technical_documentation/how-tos/superset_extra_assets.rst @@ -3,9 +3,12 @@ Superset extra assets ********************* -Developers can use the `superset-extra-assets` patch to add extra assets to Superset and those will be imported at initialization. +Developers can use the ``superset-extra-assets`` patch to add extra assets (charts, datasets, dashboards, databases) to Superset that will be imported at initialization. -An example a tutor inline plugin using the patch is the following: +The patch is expected to be a list of assets (with an extra attribute called ``_file_name`` to uniquely identify each asset). Each asset should be a valid yaml object with the attributes that Superset expects for each asset type. See +`aspects assets `_ for examples of asset yaml declarations. + +Assets in a tutor inline plugin using the patch would look like: .. code-block:: yaml @@ -26,46 +29,79 @@ An example a tutor inline plugin using the patch is the following: table_name: "..." ... - -The patch is expected to be a list of assets with an extra attribute called ``_file_name``, which uniquely identifies the asset entry. Each asset is expected to be a valid yaml object with the attributes that Superset expects for each asset type. See -`assets `_ for examples of asset yaml declarations. - -In most cases, however, you will want to develop Superset dashboards, charts, datasets, and sometimes even databases in the Superset UI and import them into a plugin that can be managed in source control. As of Aspects 1.0.3, use the same command to import assets into a Tutor plugin that we use to maintain Aspects itself. +| +In most cases, however, you will want to develop Superset dashboards, charts, datasets, and sometimes even databases in the Superset UI, and import them into a plugin that can be managed in source control. Creating a Tutor plugin ======================= -#. Use the `Tutor Plugin Cookiecutter `_ to create a skeleton plugin. See `tutor-contrib-aspects-sample `_ for an example. +#. Use the `Tutor Plugin Cookiecutter `_ to create a skeleton plugin. We have created `tutor-contrib-aspects-sample `_ as an example. + +#. Create a `patch file `_ that will recursively include all of your Superset assets from ``templates/build/assets`` into Aspects when the plugin is enabled and ``tutor config save`` is run. -#. Create a `patch file `_ that will recursively include all of your Superset assets from ``/templates/build/assets`` into Aspects when the plugin is enabled and ``tutor config save`` is run. + .. code-block:: - * The directory containing ``assets`` must be included in `ENV_TEMPLATE_TARGETS `_. Since this defaults to ``build`` and ``apps``, you can either add your asset files to one of these directories, or add an additional target to the plugin file. + {% for file in "aspects-sample/build/assets"|walk_templates %} + --- + _file_name: {{ file.split('/')[-1] }} + {% include file %} + {% endfor %} + + .. attention:: + + The directory containing ``assets`` must be included in `ENV_TEMPLATE_TARGETS `_. Since this defaults to ``build`` and ``apps``, you can either add your asset files to one of these directories, or add an additional target to the plugin file. #. Create a `requirements.txt `_ with ``ruff`` for formatting. Import assets into plugin ========================= -#. From the Superset UI, you can export your dashboard, which will download to your local computer as a .zip file. - -#. As long as you are running Aspects 1.0.3 or newer you can use this command to unzip the file to your plugin directory. +#. From the Superset UI, you can export your dashboard, which will download to your local computer as a ``.zip`` file. + +#. As long as you are running Aspects 1.0.3 or newer, you can use this command to unzip the file to your plugin directory. + + .. code-block:: bash - ``tutor aspects import_superset_zip .zip --base_assets_path /templates/build/assets/`` + tutor aspects import_superset_zip .zip --base_assets_path /templates/build/assets/ - * This command does several things to make import safer and easier, including checking for some required security fields, safely naming files so that charts with duplicate names don't overwrite each other, and checking that certain fields use Tutor settings instead of hard coded values. It also adds a special ``_file_name`` key that tells Aspects what name to use for the file when importing back to Superset. + This command does several things to make import safer and easier including: -#. Before importing a dashboard back into Superset, you will need to manually add a `_roles `_ key, which sets the permissions for who can view the dashboard. + * Check for some required security fields + * Safely name files so that charts with duplicate names don't overwrite each other + * Check that certain fields use Tutor settings instead of hard coded values + * Add a special ``_file_name`` key that tells Aspects what name to use for the file when importing back to Superset - * In addition to the default roles (student, instructor, operator, admin) you can also add custom roles in a ``superset_roles`` patch. +#. Before importing a dashboard back into Superset, you will need to manually add a `_roles `_ key, which sets the permissions for who can view the dashboard. + + .. code-block:: yaml + + _roles: + - '{{ SUPERSET_ROLES_MAPPING.instructor }}' + + In addition to the default roles (student, instructor, operator, admin) you can also add custom roles in a ``superset_roles`` patch. .. warning:: When a Superset zip is exported it will contain any datasets and databases needed for all of the charts in that dashboard. This may include the default xapi and MySQL databases or Aspects datasets that you probably do not want to overwrite, and can include their passwords! - Use caution when importing datasets or databases in your project and make sure that they are only ones you have created yourself unless you truly intend to overwrite defaults. + Use caution when importing datasets or databases into your plugin - only ones you have created yourself should exist in your plugin. + + If you truly mean to overwrite the default databases or datasets, make sure to remove any sensitive information such as passwords from the yaml files before importing. + + If you accidentally overwrite a default database, you can reset it by: + + #. Disabling your plugin (``tutor plugins disable ``) + #. Running ``tutor config save`` and ``tutor do import-assets`` to reset the assets + #. Re-enabling your plugin Update Superset assets ====================== -Once you have your files imported to the plugin, make sure it is installed and enabled in your Tutor environment, then run ``tutor config save`` and ``tutor import-assets`` to update your Superset assets. +Once you have your files imported to the plugin, make sure it is `installed and enabled `_ in your Tutor environment, then use ``import-assets`` to update your Superset assets. + +.. code-block:: bash + pip install + tutor plugins enable + tutor config save + tutor import-assets From 96cfc95722e918688ec211fbbd153af840c585b2 Mon Sep 17 00:00:00 2001 From: Sara Burns Date: Mon, 9 Mar 2026 10:56:20 -0400 Subject: [PATCH 8/9] docs: addressing pr comments --- .../how-tos/backfill.rst | 2 ++ .../technical_documentation/how-tos/index.rst | 20 ++----------------- .../how-tos/install_aspects.rst | 2 ++ ....rst => running_aspects_without_tutor.rst} | 0 .../how-tos/superset_extra_assets.rst | 2 +- 5 files changed, 7 insertions(+), 19 deletions(-) rename docs/technical_documentation/how-tos/{aspects_without_tutor.rst => running_aspects_without_tutor.rst} (100%) diff --git a/docs/technical_documentation/how-tos/backfill.rst b/docs/technical_documentation/how-tos/backfill.rst index 418fd4a..bb4313b 100644 --- a/docs/technical_documentation/how-tos/backfill.rst +++ b/docs/technical_documentation/how-tos/backfill.rst @@ -1,3 +1,5 @@ +.. _backfill: + Backfill old or missing data **************************** diff --git a/docs/technical_documentation/how-tos/index.rst b/docs/technical_documentation/how-tos/index.rst index c13b025..6b5941d 100644 --- a/docs/technical_documentation/how-tos/index.rst +++ b/docs/technical_documentation/how-tos/index.rst @@ -3,23 +3,7 @@ How-Tos .. toctree:: :maxdepth: 1 + :glob: :caption: Content: - Install Aspects Plugin for Tutor - Upgrade Aspects - Configure Aspects for Production - Troubleshooting Aspects - Running Aspects without Tutor - Connect to external Clickhouse database - Run Aspects in a ClickHouse cluster - Backfill old or missing data - Extending dbt - Customizing xAPI Transforms - Changing the xAPI actor identifier - Superset extra assets - Superset language settings - Superset extra row level security - Superset extra jinja filters - Superset custom roles - Superset custom config settings - Clickhouse extra SQL + * diff --git a/docs/technical_documentation/how-tos/install_aspects.rst b/docs/technical_documentation/how-tos/install_aspects.rst index a491b05..c9c6d4d 100644 --- a/docs/technical_documentation/how-tos/install_aspects.rst +++ b/docs/technical_documentation/how-tos/install_aspects.rst @@ -1,3 +1,5 @@ +.. _install_aspects: + Install Aspects Plugin for Tutor ******************************** diff --git a/docs/technical_documentation/how-tos/aspects_without_tutor.rst b/docs/technical_documentation/how-tos/running_aspects_without_tutor.rst similarity index 100% rename from docs/technical_documentation/how-tos/aspects_without_tutor.rst rename to docs/technical_documentation/how-tos/running_aspects_without_tutor.rst diff --git a/docs/technical_documentation/how-tos/superset_extra_assets.rst b/docs/technical_documentation/how-tos/superset_extra_assets.rst index 84994f7..c2c6f39 100644 --- a/docs/technical_documentation/how-tos/superset_extra_assets.rst +++ b/docs/technical_documentation/how-tos/superset_extra_assets.rst @@ -30,7 +30,7 @@ Assets in a tutor inline plugin using the patch would look like: ... | -In most cases, however, you will want to develop Superset dashboards, charts, datasets, and sometimes even databases in the Superset UI, and import them into a plugin that can be managed in source control. +| In most cases, however, you will want to develop Superset dashboards, charts, datasets, and sometimes even databases in the Superset UI, and import them into a plugin that can be managed in source control. Creating a Tutor plugin ======================= From 1fc426a57a22db39c440764474e19e33dc4a669f Mon Sep 17 00:00:00 2001 From: Sara Burns Date: Mon, 9 Mar 2026 11:13:31 -0400 Subject: [PATCH 9/9] docs: new clickhouse and superset sections --- .../{install_aspects.rst => 01_install_aspects.rst} | 0 .../how-tos/{upgrade.rst => 02_upgrade.rst} | 2 +- ...iguration.rst => aspects_production_configuration.rst} | 2 +- ...leshooting_aspects.rst => aspects_troubleshooting.rst} | 2 +- ...spects_without_tutor.rst => aspects_without_tutor.rst} | 4 ++-- .../how-tos/changing_actor_identifier.rst | 2 +- .../how-tos/{ => clickhouse}/clickhouse_cluster.rst | 4 ++-- .../clickhouse_external.rst} | 4 ++-- .../how-tos/{ => clickhouse}/clickhouse_sql.rst | 4 ++-- docs/technical_documentation/how-tos/clickhouse/index.rst | 8 ++++++++ docs/technical_documentation/how-tos/dbt_extensions.rst | 4 ++-- docs/technical_documentation/how-tos/index.rst | 2 ++ docs/technical_documentation/how-tos/superset/index.rst | 8 ++++++++ .../how-tos/{ => superset}/superset_config_docker.rst | 4 ++-- .../how-tos/{ => superset}/superset_extra_assets.rst | 4 ++-- .../how-tos/{ => superset}/superset_jinja_filters.rst | 4 ++-- .../how-tos/{ => superset}/superset_language_settings.rst | 4 ++-- .../how-tos/{ => superset}/superset_roles.rst | 4 ++-- .../{ => superset}/superset_row_level_security.rst | 4 ++-- docs/technical_documentation/how-tos/xapi_transforms.rst | 4 ++-- 20 files changed, 46 insertions(+), 28 deletions(-) rename docs/technical_documentation/how-tos/{install_aspects.rst => 01_install_aspects.rst} (100%) rename docs/technical_documentation/how-tos/{upgrade.rst => 02_upgrade.rst} (99%) rename docs/technical_documentation/how-tos/{production_configuration.rst => aspects_production_configuration.rst} (99%) rename docs/technical_documentation/how-tos/{troubleshooting_aspects.rst => aspects_troubleshooting.rst} (99%) rename docs/technical_documentation/how-tos/{running_aspects_without_tutor.rst => aspects_without_tutor.rst} (99%) rename docs/technical_documentation/how-tos/{ => clickhouse}/clickhouse_cluster.rst (95%) rename docs/technical_documentation/how-tos/{remote_clickhouse.rst => clickhouse/clickhouse_external.rst} (98%) rename docs/technical_documentation/how-tos/{ => clickhouse}/clickhouse_sql.rst (91%) create mode 100644 docs/technical_documentation/how-tos/clickhouse/index.rst create mode 100644 docs/technical_documentation/how-tos/superset/index.rst rename docs/technical_documentation/how-tos/{ => superset}/superset_config_docker.rst (90%) rename docs/technical_documentation/how-tos/{ => superset}/superset_extra_assets.rst (99%) rename docs/technical_documentation/how-tos/{ => superset}/superset_jinja_filters.rst (96%) rename docs/technical_documentation/how-tos/{ => superset}/superset_language_settings.rst (95%) rename docs/technical_documentation/how-tos/{ => superset}/superset_roles.rst (95%) rename docs/technical_documentation/how-tos/{ => superset}/superset_row_level_security.rst (91%) diff --git a/docs/technical_documentation/how-tos/install_aspects.rst b/docs/technical_documentation/how-tos/01_install_aspects.rst similarity index 100% rename from docs/technical_documentation/how-tos/install_aspects.rst rename to docs/technical_documentation/how-tos/01_install_aspects.rst diff --git a/docs/technical_documentation/how-tos/upgrade.rst b/docs/technical_documentation/how-tos/02_upgrade.rst similarity index 99% rename from docs/technical_documentation/how-tos/upgrade.rst rename to docs/technical_documentation/how-tos/02_upgrade.rst index a300812..fea16a8 100644 --- a/docs/technical_documentation/how-tos/upgrade.rst +++ b/docs/technical_documentation/how-tos/02_upgrade.rst @@ -1,6 +1,6 @@ .. _upgrade-aspects: -How-to Upgrade Aspects +Upgrade Aspects ********************** Aspects is intended to have a faster upgrade cycle than Open edX named releases since operators might want to upgrade between releases to get features as they become available. The upgrade process is the same whether it's happening as part of a named release upgrade or in between. diff --git a/docs/technical_documentation/how-tos/production_configuration.rst b/docs/technical_documentation/how-tos/aspects_production_configuration.rst similarity index 99% rename from docs/technical_documentation/how-tos/production_configuration.rst rename to docs/technical_documentation/how-tos/aspects_production_configuration.rst index e5b7699..c9e72bb 100644 --- a/docs/technical_documentation/how-tos/production_configuration.rst +++ b/docs/technical_documentation/how-tos/aspects_production_configuration.rst @@ -1,6 +1,6 @@ .. _production_configuration: -Configure Aspects for Production +Configure Aspects for production ******************************** Choosing an xAPI Pipeline diff --git a/docs/technical_documentation/how-tos/troubleshooting_aspects.rst b/docs/technical_documentation/how-tos/aspects_troubleshooting.rst similarity index 99% rename from docs/technical_documentation/how-tos/troubleshooting_aspects.rst rename to docs/technical_documentation/how-tos/aspects_troubleshooting.rst index 913c5d2..b950f7f 100644 --- a/docs/technical_documentation/how-tos/troubleshooting_aspects.rst +++ b/docs/technical_documentation/how-tos/aspects_troubleshooting.rst @@ -1,6 +1,6 @@ .. _troubleshooting_aspects: -Troubleshooting Aspects +Troubleshoot Aspects ####################### Most common problems with Aspects can be fixed by making sure everything is up to date by deleting diff --git a/docs/technical_documentation/how-tos/running_aspects_without_tutor.rst b/docs/technical_documentation/how-tos/aspects_without_tutor.rst similarity index 99% rename from docs/technical_documentation/how-tos/running_aspects_without_tutor.rst rename to docs/technical_documentation/how-tos/aspects_without_tutor.rst index 295d9e7..6d3f3b6 100644 --- a/docs/technical_documentation/how-tos/running_aspects_without_tutor.rst +++ b/docs/technical_documentation/how-tos/aspects_without_tutor.rst @@ -1,7 +1,7 @@ .. _aspects-without-tutor: -Running Aspects Without Tutor -***************************** +Run Aspects without Tutor +************************* In cases where Open edX is deployed outside of the Tutor environment, setting up Aspects is quite challenging. This guide aims to present the steps needed to stand up and configure an Aspects environment using Tutor and tutor-contrib-aspects to create the resources needed to configure an Aspects environment with an existing non-Tutor deployment. diff --git a/docs/technical_documentation/how-tos/changing_actor_identifier.rst b/docs/technical_documentation/how-tos/changing_actor_identifier.rst index 7dca456..acf45a4 100644 --- a/docs/technical_documentation/how-tos/changing_actor_identifier.rst +++ b/docs/technical_documentation/how-tos/changing_actor_identifier.rst @@ -1,6 +1,6 @@ .. _changing_actor_identifier: -Changing the xAPI actor identifier +Change the xAPI actor identifier ################################## The xAPI :ref:`actor ` identifier is a unique user identifier provided diff --git a/docs/technical_documentation/how-tos/clickhouse_cluster.rst b/docs/technical_documentation/how-tos/clickhouse/clickhouse_cluster.rst similarity index 95% rename from docs/technical_documentation/how-tos/clickhouse_cluster.rst rename to docs/technical_documentation/how-tos/clickhouse/clickhouse_cluster.rst index f239aaa..d185e32 100644 --- a/docs/technical_documentation/how-tos/clickhouse_cluster.rst +++ b/docs/technical_documentation/how-tos/clickhouse/clickhouse_cluster.rst @@ -1,7 +1,7 @@ .. _clickhouse-cluster: -How To Run Aspects With ClickHouse Cluster -****************************************** +Run Aspects with a ClickHouse cluster +************************************* ClickHouse clusters are an advanced way of running ClickHouse, but offer many benefits for large scale installations. Deciding whether to run in a clustered environment is a big decision with many cost and administrative impacts and should be carefully considered before launching a production environment. In most cases we expect Aspects to perform well without cluster scaling. diff --git a/docs/technical_documentation/how-tos/remote_clickhouse.rst b/docs/technical_documentation/how-tos/clickhouse/clickhouse_external.rst similarity index 98% rename from docs/technical_documentation/how-tos/remote_clickhouse.rst rename to docs/technical_documentation/how-tos/clickhouse/clickhouse_external.rst index 0968df7..e826528 100644 --- a/docs/technical_documentation/how-tos/remote_clickhouse.rst +++ b/docs/technical_documentation/how-tos/clickhouse/clickhouse_external.rst @@ -1,7 +1,7 @@ .. _remote-clickhouse: -Connect to external Clickhouse database -*************************************** +Connect to an external Clickhouse database +****************************************** Connect to Clickhouse Cloud ########################### diff --git a/docs/technical_documentation/how-tos/clickhouse_sql.rst b/docs/technical_documentation/how-tos/clickhouse/clickhouse_sql.rst similarity index 91% rename from docs/technical_documentation/how-tos/clickhouse_sql.rst rename to docs/technical_documentation/how-tos/clickhouse/clickhouse_sql.rst index fa9ffbe..fe41ce4 100644 --- a/docs/technical_documentation/how-tos/clickhouse_sql.rst +++ b/docs/technical_documentation/how-tos/clickhouse/clickhouse_sql.rst @@ -1,7 +1,7 @@ .. _clickhouse-sql: -Clickhouse extra SQL -******************** +Execute extra Clickhouse SQL +**************************** You can execute extra Clickhouse SQL at initialization. To do so, you need to use the patch `clickhouse-extra-sql`. This patch expects valid Clickhouse SQL code: diff --git a/docs/technical_documentation/how-tos/clickhouse/index.rst b/docs/technical_documentation/how-tos/clickhouse/index.rst new file mode 100644 index 0000000..a984f6e --- /dev/null +++ b/docs/technical_documentation/how-tos/clickhouse/index.rst @@ -0,0 +1,8 @@ +ClickHouse +########## + +.. toctree:: + :maxdepth: 1 + :glob: + + * diff --git a/docs/technical_documentation/how-tos/dbt_extensions.rst b/docs/technical_documentation/how-tos/dbt_extensions.rst index 6594637..a920748 100644 --- a/docs/technical_documentation/how-tos/dbt_extensions.rst +++ b/docs/technical_documentation/how-tos/dbt_extensions.rst @@ -1,7 +1,7 @@ .. _dbt-extensions: -Extending dbt -************* +Extend dbt +********** As noted in :ref:`dbt`, you can install your own custom dbt package to apply your own transforms to the event data in Aspects. diff --git a/docs/technical_documentation/how-tos/index.rst b/docs/technical_documentation/how-tos/index.rst index 6b5941d..5f280ce 100644 --- a/docs/technical_documentation/how-tos/index.rst +++ b/docs/technical_documentation/how-tos/index.rst @@ -6,4 +6,6 @@ How-Tos :glob: :caption: Content: + ClickHouse + Superset * diff --git a/docs/technical_documentation/how-tos/superset/index.rst b/docs/technical_documentation/how-tos/superset/index.rst new file mode 100644 index 0000000..f46ffa3 --- /dev/null +++ b/docs/technical_documentation/how-tos/superset/index.rst @@ -0,0 +1,8 @@ +Superset +######## + +.. toctree:: + :maxdepth: 1 + :glob: + + * diff --git a/docs/technical_documentation/how-tos/superset_config_docker.rst b/docs/technical_documentation/how-tos/superset/superset_config_docker.rst similarity index 90% rename from docs/technical_documentation/how-tos/superset_config_docker.rst rename to docs/technical_documentation/how-tos/superset/superset_config_docker.rst index 07f26d8..5f5fe43 100644 --- a/docs/technical_documentation/how-tos/superset_config_docker.rst +++ b/docs/technical_documentation/how-tos/superset/superset_config_docker.rst @@ -1,7 +1,7 @@ .. _superset-config-docker: -Superset Custom Production Config Settings -****************************************** +Use Superset custom production config settings +********************************************** To add or override production config settings in Superset, use the patch ``superset-config-docker``. Changes take effect on LMS/CMS restart. diff --git a/docs/technical_documentation/how-tos/superset_extra_assets.rst b/docs/technical_documentation/how-tos/superset/superset_extra_assets.rst similarity index 99% rename from docs/technical_documentation/how-tos/superset_extra_assets.rst rename to docs/technical_documentation/how-tos/superset/superset_extra_assets.rst index c2c6f39..a6dc676 100644 --- a/docs/technical_documentation/how-tos/superset_extra_assets.rst +++ b/docs/technical_documentation/how-tos/superset/superset_extra_assets.rst @@ -1,7 +1,7 @@ .. _superset-extra-assets: -Superset extra assets -********************* +Add extra Superset assets +************************* Developers can use the ``superset-extra-assets`` patch to add extra assets (charts, datasets, dashboards, databases) to Superset that will be imported at initialization. diff --git a/docs/technical_documentation/how-tos/superset_jinja_filters.rst b/docs/technical_documentation/how-tos/superset/superset_jinja_filters.rst similarity index 96% rename from docs/technical_documentation/how-tos/superset_jinja_filters.rst rename to docs/technical_documentation/how-tos/superset/superset_jinja_filters.rst index 7756c40..e97fab8 100644 --- a/docs/technical_documentation/how-tos/superset_jinja_filters.rst +++ b/docs/technical_documentation/how-tos/superset/superset_jinja_filters.rst @@ -1,7 +1,7 @@ .. _superset-jinja-filters: -Superset extra jinja filters -***************************** +Create Superset extra jinja filters +*********************************** To create extra jinja filters, you can use the patch `superset-jinja-filters` diff --git a/docs/technical_documentation/how-tos/superset_language_settings.rst b/docs/technical_documentation/how-tos/superset/superset_language_settings.rst similarity index 95% rename from docs/technical_documentation/how-tos/superset_language_settings.rst rename to docs/technical_documentation/how-tos/superset/superset_language_settings.rst index 9b69457..8279b28 100644 --- a/docs/technical_documentation/how-tos/superset_language_settings.rst +++ b/docs/technical_documentation/how-tos/superset/superset_language_settings.rst @@ -1,7 +1,7 @@ .. _superset-language-settings: -Superset language settings -************************** +Change Superset language settings +********************************* By changing the following Tutor configuration variables you can change the default language: diff --git a/docs/technical_documentation/how-tos/superset_roles.rst b/docs/technical_documentation/how-tos/superset/superset_roles.rst similarity index 95% rename from docs/technical_documentation/how-tos/superset_roles.rst rename to docs/technical_documentation/how-tos/superset/superset_roles.rst index d002479..d34fdfb 100644 --- a/docs/technical_documentation/how-tos/superset_roles.rst +++ b/docs/technical_documentation/how-tos/superset/superset_roles.rst @@ -1,7 +1,7 @@ .. _superset-roles: -Superset extra roles -********************* +Create extra Superset roles +*************************** Create extra Superset roles, you can use the patch `superset-extra-roles`. This patch expects validJSON objects with the following structure: diff --git a/docs/technical_documentation/how-tos/superset_row_level_security.rst b/docs/technical_documentation/how-tos/superset/superset_row_level_security.rst similarity index 91% rename from docs/technical_documentation/how-tos/superset_row_level_security.rst rename to docs/technical_documentation/how-tos/superset/superset_row_level_security.rst index 617674e..e31fd86 100644 --- a/docs/technical_documentation/how-tos/superset_row_level_security.rst +++ b/docs/technical_documentation/how-tos/superset/superset_row_level_security.rst @@ -1,7 +1,7 @@ .. _superset-row-level-security: -Superset extra row level security -********************************* +Customize Superset row level security +************************************* To apply custom row level security filters to Superset, you can use the patch `superset-row-level-security`. This patch expects a list of python dictionaries diff --git a/docs/technical_documentation/how-tos/xapi_transforms.rst b/docs/technical_documentation/how-tos/xapi_transforms.rst index b2c43df..b3caffe 100644 --- a/docs/technical_documentation/how-tos/xapi_transforms.rst +++ b/docs/technical_documentation/how-tos/xapi_transforms.rst @@ -1,7 +1,7 @@ .. _xapi_transforms: -Customizing xAPI Transforms -########################### +Customize xAPI transforms +######################### Aspects converts raw Open edX tracking event JSON into :ref:`xapi-concepts` for storage and analysis. This conversion process is called "transformation".