diff --git a/docs/Makefile b/docs/Makefile index d4bb2cb..47cbb95 100644 --- a/docs/Makefile +++ b/docs/Makefile @@ -14,6 +14,9 @@ help: .PHONY: help Makefile +serve_docs: ## serve the built docs locally to preview the site in the browser + sphinx-autobuild source $(BUILDDIR)/html --ignore '**/_tags/*' + # Catch-all target: route all unknown targets to Sphinx using the new # "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). %: Makefile diff --git a/docs/technical_documentation/how-tos/install_aspects.rst b/docs/technical_documentation/how-tos/01_install_aspects.rst similarity index 85% rename from docs/technical_documentation/how-tos/install_aspects.rst rename to docs/technical_documentation/how-tos/01_install_aspects.rst index 9ebafa2..c9c6d4d 100644 --- a/docs/technical_documentation/how-tos/install_aspects.rst +++ b/docs/technical_documentation/how-tos/01_install_aspects.rst @@ -1,5 +1,7 @@ -Aspects Plugin for Tutor -************************ +.. _install_aspects: + +Install Aspects Plugin for Tutor +******************************** Aspects is implemented as a Tutor plugin. To learn more about how to install it, please refer to the plugin GitHub site: ``_ diff --git a/docs/technical_documentation/how-tos/upgrade.rst b/docs/technical_documentation/how-tos/02_upgrade.rst similarity index 99% rename from docs/technical_documentation/how-tos/upgrade.rst rename to docs/technical_documentation/how-tos/02_upgrade.rst index a300812..fea16a8 100644 --- a/docs/technical_documentation/how-tos/upgrade.rst +++ b/docs/technical_documentation/how-tos/02_upgrade.rst @@ -1,6 +1,6 @@ .. _upgrade-aspects: -How-to Upgrade Aspects +Upgrade Aspects ********************** Aspects is intended to have a faster upgrade cycle than Open edX named releases since operators might want to upgrade between releases to get features as they become available. The upgrade process is the same whether it's happening as part of a named release upgrade or in between. diff --git a/docs/technical_documentation/how-tos/production_configuration.rst b/docs/technical_documentation/how-tos/aspects_production_configuration.rst similarity index 96% rename from docs/technical_documentation/how-tos/production_configuration.rst rename to docs/technical_documentation/how-tos/aspects_production_configuration.rst index da5d31f..c9e72bb 100644 --- a/docs/technical_documentation/how-tos/production_configuration.rst +++ b/docs/technical_documentation/how-tos/aspects_production_configuration.rst @@ -1,6 +1,6 @@ .. _production_configuration: -Configure Aspects for Production +Configure Aspects for production ******************************** Choosing an xAPI Pipeline @@ -199,13 +199,13 @@ By default, Aspects enables plugin functionality in the LMS that embeds a define - ``ASPECTS_LEARNER_GROUPS_HELP_MARKDOWN`` controls the content of the "Help" tab in the At-Risk Learners dashboard - ``ASPECTS_OPERATOR_HELP_MARKDOWN`` controls the content of the "Help" tab in the Operator dashboard -In-context Metrics +In-Context Metrics ------------------ -Starting in the Teak Open edX release, Aspects provides in-context metrics in Studio. The following settings controls this functionality. +Starting in the Teak Open edX release, Aspects provides :ref:`In-Context Dashboards` in Studio. Reference the `plugin ` for setup instructions. -- ``ASPECTS_ENABLE_STUDIO_IN_CONTEXT_METRICS`` - Enables or disables in-context metrics. -- ``ASPECTS_IN_CONTEXT_DASHBOARDS`` - A dictionary mapping block types to in-context dashboards. You can use this option to remove or replace the in-context dashboard for a block type. The key `course` defines the in-context dashboard for course overview. +For further customization, the setting ``ASPECTS_IN_CONTEXT_DASHBOARDS`` (in `platform-plugin-aspects`_) is a dictionary mapping block types to in-context dashboards. +You can use this setting to remove or replace the in-context dashboard for a block type. The key `course` defines the in-context dashboard for course overview. Ralph Accessibility @@ -396,3 +396,4 @@ These are also captured in the Aspects Operator Dashboard as well as a filterabl .. _Ralph via Helm chart: https://openfun.github.io/ralph/latest/tutorials/helm/ .. _dump_data_to_clickhouse arguments: https://github.com/openedx/platform-plugin-aspects/blob/951ed84de01dda6bec9923c60fcd96bf80d6fa54/platform_plugin_aspects/management/commands/dump_data_to_clickhouse.py#L91 .. _transform_tracking_logs: https://event-routing-backends.readthedocs.io/en/latest/technical_documentation/how-tos/how_to_bulk_transform.html +.. _platform-plugin-aspects: https://github.com/openedx/platform-plugin-aspects/blob/main/platform_plugin_aspects/settings/common.py#L112 diff --git a/docs/technical_documentation/how-tos/troubleshooting_aspects.rst b/docs/technical_documentation/how-tos/aspects_troubleshooting.rst similarity index 99% rename from docs/technical_documentation/how-tos/troubleshooting_aspects.rst rename to docs/technical_documentation/how-tos/aspects_troubleshooting.rst index 913c5d2..b950f7f 100644 --- a/docs/technical_documentation/how-tos/troubleshooting_aspects.rst +++ b/docs/technical_documentation/how-tos/aspects_troubleshooting.rst @@ -1,6 +1,6 @@ .. _troubleshooting_aspects: -Troubleshooting Aspects +Troubleshoot Aspects ####################### Most common problems with Aspects can be fixed by making sure everything is up to date by deleting diff --git a/docs/technical_documentation/how-tos/aspects_without_tutor.rst b/docs/technical_documentation/how-tos/aspects_without_tutor.rst index 295d9e7..6d3f3b6 100644 --- a/docs/technical_documentation/how-tos/aspects_without_tutor.rst +++ b/docs/technical_documentation/how-tos/aspects_without_tutor.rst @@ -1,7 +1,7 @@ .. _aspects-without-tutor: -Running Aspects Without Tutor -***************************** +Run Aspects without Tutor +************************* In cases where Open edX is deployed outside of the Tutor environment, setting up Aspects is quite challenging. This guide aims to present the steps needed to stand up and configure an Aspects environment using Tutor and tutor-contrib-aspects to create the resources needed to configure an Aspects environment with an existing non-Tutor deployment. diff --git a/docs/technical_documentation/how-tos/backfill.rst b/docs/technical_documentation/how-tos/backfill.rst index 418fd4a..bb4313b 100644 --- a/docs/technical_documentation/how-tos/backfill.rst +++ b/docs/technical_documentation/how-tos/backfill.rst @@ -1,3 +1,5 @@ +.. _backfill: + Backfill old or missing data **************************** diff --git a/docs/technical_documentation/how-tos/changing_actor_identifier.rst b/docs/technical_documentation/how-tos/changing_actor_identifier.rst index 7dca456..acf45a4 100644 --- a/docs/technical_documentation/how-tos/changing_actor_identifier.rst +++ b/docs/technical_documentation/how-tos/changing_actor_identifier.rst @@ -1,6 +1,6 @@ .. _changing_actor_identifier: -Changing the xAPI actor identifier +Change the xAPI actor identifier ################################## The xAPI :ref:`actor ` identifier is a unique user identifier provided diff --git a/docs/technical_documentation/how-tos/clickhouse_cluster.rst b/docs/technical_documentation/how-tos/clickhouse/clickhouse_cluster.rst similarity index 95% rename from docs/technical_documentation/how-tos/clickhouse_cluster.rst rename to docs/technical_documentation/how-tos/clickhouse/clickhouse_cluster.rst index f239aaa..d185e32 100644 --- a/docs/technical_documentation/how-tos/clickhouse_cluster.rst +++ b/docs/technical_documentation/how-tos/clickhouse/clickhouse_cluster.rst @@ -1,7 +1,7 @@ .. _clickhouse-cluster: -How To Run Aspects With ClickHouse Cluster -****************************************** +Run Aspects with a ClickHouse cluster +************************************* ClickHouse clusters are an advanced way of running ClickHouse, but offer many benefits for large scale installations. Deciding whether to run in a clustered environment is a big decision with many cost and administrative impacts and should be carefully considered before launching a production environment. In most cases we expect Aspects to perform well without cluster scaling. diff --git a/docs/technical_documentation/how-tos/remote_clickhouse.rst b/docs/technical_documentation/how-tos/clickhouse/clickhouse_external.rst similarity index 98% rename from docs/technical_documentation/how-tos/remote_clickhouse.rst rename to docs/technical_documentation/how-tos/clickhouse/clickhouse_external.rst index 0968df7..e826528 100644 --- a/docs/technical_documentation/how-tos/remote_clickhouse.rst +++ b/docs/technical_documentation/how-tos/clickhouse/clickhouse_external.rst @@ -1,7 +1,7 @@ .. _remote-clickhouse: -Connect to external Clickhouse database -*************************************** +Connect to an external Clickhouse database +****************************************** Connect to Clickhouse Cloud ########################### diff --git a/docs/technical_documentation/how-tos/clickhouse_sql.rst b/docs/technical_documentation/how-tos/clickhouse/clickhouse_sql.rst similarity index 91% rename from docs/technical_documentation/how-tos/clickhouse_sql.rst rename to docs/technical_documentation/how-tos/clickhouse/clickhouse_sql.rst index fa9ffbe..fe41ce4 100644 --- a/docs/technical_documentation/how-tos/clickhouse_sql.rst +++ b/docs/technical_documentation/how-tos/clickhouse/clickhouse_sql.rst @@ -1,7 +1,7 @@ .. _clickhouse-sql: -Clickhouse extra SQL -******************** +Execute extra Clickhouse SQL +**************************** You can execute extra Clickhouse SQL at initialization. To do so, you need to use the patch `clickhouse-extra-sql`. This patch expects valid Clickhouse SQL code: diff --git a/docs/technical_documentation/how-tos/clickhouse/index.rst b/docs/technical_documentation/how-tos/clickhouse/index.rst new file mode 100644 index 0000000..a984f6e --- /dev/null +++ b/docs/technical_documentation/how-tos/clickhouse/index.rst @@ -0,0 +1,8 @@ +ClickHouse +########## + +.. toctree:: + :maxdepth: 1 + :glob: + + * diff --git a/docs/technical_documentation/how-tos/dbt_extensions.rst b/docs/technical_documentation/how-tos/dbt_extensions.rst index b59a7b0..a920748 100644 --- a/docs/technical_documentation/how-tos/dbt_extensions.rst +++ b/docs/technical_documentation/how-tos/dbt_extensions.rst @@ -1,22 +1,17 @@ .. _dbt-extensions: -Extending dbt -************* +Extend dbt +********** -As noted in :ref:`dbt`, you can install your own custom dbt package to apply your own transforms to the event data -in Aspects. +As noted in :ref:`dbt`, you can install your own custom dbt package to apply your own transforms to the event data in Aspects. -This guide demonstrates how to create and use a custom dbt package in Aspects by building the `aspects-dbt-example`_ -repo. See `aspects-dbt-example#1`_ to follow along with each of these steps. +This guide demonstrates how to create and use a custom dbt package in Aspects by building the `sample-aspects-dbt `_ +repo. -See `Building dbt packages`_ and `Best practice guides`_ for dbt's official documentation. +Step 1. Install dbt +==================================== -Step 0. Install dbt core -======================== - -The easiest way to install dbt core is to use pip in a python3 virtual environment. - -See `aspects-dbt requirements.txt`_ for the specific package versions used by Aspects. +The easiest way to install dbt is to use pip in a python3 virtual environment. .. code-block:: bash @@ -25,168 +20,182 @@ See `aspects-dbt requirements.txt`_ for the specific package versions used by As source venv/bin/activate pip install --upgrade pip +Check the `aspects-dbt requirements.txt `_ for the specific package versions used by Aspects. Both dbt-core and dbt-clickhouse are required. + +.. code-block:: bash + # Install the dbt package versions used by Aspects - pip install dbt-clickhouse==1.7.2 dbt-core==1.7.0 + pip install dbt-clickhouse==x.x.x dbt-core==x.x.x -See `Install dbt`_ for more ways to install dbt. +See `Install dbt `_ for more ways to install dbt. -Step 1. Initialize your dbt package -=================================== +Step 2. Set up new dbt package +=============================== -Create a new dbt package by following the prompts given by the ``dbt init`` tool. +#. Create a new repository for your custom dbt package, and clone it to your local machine. -.. code-block:: bash +#. In the root of your local repository, create a new dbt package by following the prompts given by the ``dbt init`` tool. See `About dbt init `_ for more options. - dbt init - # Enter a name for your project (letters, digits, underscore): aspects_dbt_example - # Which database would you like to use? [1] clickhouse + .. code-block:: bash - ls aspects_dbt_example - # analyses/ dbt_project.yml macros/ models/ README.md seeds/ snapshots/ tests/ + # Use the profile name from your ``dbt/profiles.yml`` file or 'aspects' + dbt init --profile aspects + + # Enter a name for your project (letters, digits, underscore): + sample_aspects_dbt + ls sample_aspects_dbt + # analyses/ dbt_project.yml macros/ models/ README.md seeds/ snapshots/ tests/ -See `About dbt init`_ for more options. + .. attention :: -Step 2. Use the Aspects profile -=============================== + Possible error when running ``dbt init``: -Update the generated ``dbt_project.yml`` to use the ``aspects`` profile: + .. code-block:: bash -.. code-block:: yaml + No sample profile found for clickhouse - # This setting configures which "profile" dbt uses for this project. - profile: 'aspects' + Make sure the `profile `_ matches your local ``dbt/profiles.yml`` profile name. + .. code-block:: yaml -Also set the location for compiled SQL to match that used by ``aspects``: + # This setting configures which "profile" dbt uses for this project. + profile: "aspects" -.. code-block:: yaml +#. In ``dbt_project.yml``, set the location for compiled SQL to match the `aspects target path `_: + + .. code-block:: yaml + + # directory which will store compiled SQL files + target-path: "target" - # directory which will store compiled SQL files - target-path: "target" +#. Create a `requirements.txt `_ file at the top level of your repository with the same version of dbt-core as in Step 1. This is needed when building the aspects image. + .. code-block:: + + # requirements.txt + dbt-core~=x.x.x + + .. note :: + + If you need any python dependencies beyond what is provided by aspects-dbt, add these to the requirements file. Step 3. Link to aspects-dbt =========================== -Aspects charts depend on the transforms in `aspects-dbt`_, so it's important that your dbt package also installs -the same version of `aspects-dbt`_ as your version of the Aspects Tutor plugin. +Aspects charts depend on the transforms in `aspects-dbt`_, so it's important that your dbt package also installs the same version of aspects-dbt as your version of the Aspects Tutor plugin. -To do this, add a ``packages.yml`` file to your dbt package at the top level where: +To do this, add a `packages.yml `_ file to your dbt package at the top level where: -* ``git`` url matches the default value of ``DBT_REPOSITORY`` in `tutor-contrib-aspects plugin.py`_ -* ``revision`` matches the default value of ``DBT_BRANCH`` in `tutor-contrib-aspects plugin.py`_ +* ``git`` url matches the default value of ``DBT_REPOSITORY`` in `tutor-contrib-aspects plugin.py `_ +* ``revision`` matches the default value of ``DBT_BRANCH`` in `tutor-contrib-aspects plugin.py `_ .. code-block:: yaml packages: - git: "https://github.com/openedx/aspects-dbt.git" - revision: v3.4.1 + revision: vX.X.X + -Step 4. Add your custom transforms -================================== +Step 4. Test dbt connection +=========================== + +Before adding your custom transforms, it's a good idea to test that your dbt package can connect to the Clickhouse database and run the base transforms from aspects-dbt. + +* Run ``dbt debug`` to test the connection to your database and the validity of your dbt project. +* Run ``dbt run`` to run the base transforms from aspects-dbt. + + * You may need to run ``dbt run --full-refresh`` if the previous step fails. + +.. attention :: + + Possible error when running dbt commands: + + .. code-block:: bash + + Compilation Error + dbt found 1 package(s) specified in packages.yml, but only 0 package(s) installed in dbt_packages. + Run "dbt deps" to install package dependencies. + + Run ``dbt deps`` to install the dependencies for your package, including aspects-dbt. -Here is where you will need an understanding of dbt, Clickhouse, Aspects' data schemas, and the specific transforms you -want to create. +Step 5. Create your custom transforms +===================================== -If you need any python dependencies beyond what is provided by aspects-dbt, add these to a ``requirements.txt`` file at -the top level of your repository. +Here is where you will need an understanding of dbt, Clickhouse, Aspects' data schemas, and the specific transforms you want to create. -.. note:: You can use Aspects to debug your custom SQL: +.. tip:: You can use Aspects to debug your custom SQL: #. Login to Superset as an Open edX superuser. #. Using the menus at the top of the page, navigate to the "SQL -> SQL Lab" UI. #. Browse the schemas and run read-only SQL queries on your data. -For this tutorial, we added a new model which will be materialized by dbt into a view in Clickhouse. -Our new model calculates the average number of attempts made by users on each problem by referencing the -`int_problem_results` model created by the base aspects-dbt package (see `dbt ref`_): +For this tutorial, we created two new models - `course_enrollments `_ and `learner_responses `_ and which will be materialized by dbt into a view and materialized view in Clickhouse. More information on materialized views `here `_. -.. code-block:: - - select - problem_id, AVG(attempts) as average_attempts - from - ( - select - problem_id, - max(attempts) as attempts - from - {{ ref('int_problem_results') }} - group by - actor_id, - problem_id - ) - group by problem_id - -Next, make sure your model is configured, for example in the ``db_project.yml``. If you forget this step, dbt will warn -you when running your package. - -.. code-block:: yaml - models: - problem_responses: - # Config indicated by + and applies to all files under models/problem_responses/ - +materialized: view +Step 6. Add dbt tests +===================== +Writing tests for your transforms is important to validate and document your intended changes, and guard against data edge cases and regressions from future code changes. -See `Configure dbt models`_ to learn more. +There are two types of dbt tests; `data tests `_ and `unit tests `_. Run ``dbt test`` to run all tests for your package. -Step 5. Add tests -================= +Data tests +------------ +Data tests can be defined in the `schema.yml `_ file for each model, and are used to validate properties of the data such as types, accepted values, uniqueness, and relationships between tables. -Writing tests for your transforms is important: not only can tests validate and document your intended changes, they -can be used to guard against data edge cases and regressions from future code changes. +.. code-block:: yaml -dbt generic tests are defined as SQL files, where the goal of the SQL statement is to return zero records. + - name: enrollment_mode + description: "The enrollment mode of the user" + data_tests: + - accepted_values: + values: ["audit", "honor", "verified"] -Because our new `average_attempts` model aggregates on `actor_id` and `problem_id`, it should only have 1 entry for each -`problem_id`. So our test can be: +Data tests can also be defined in a `SQL file `_, where the goal of the SQL statement is to return zero records. .. code-block:: - -- average_attempts should only have one record for each problem_id. - select - count(*) as num_rows - from - {{ ref('average_attempts') }} - group by - problem_id + select count(*) as num_rows + from {{ ref('enrollment_mode') }} + group by org, course_key, enrollment_mode, course_name, course_run having num_rows > 1 +Unit tests +---------- +Unit tests are used to validate the logic of your dbt models in isolation from the underlying data. They are defined in a `unit_tests.yaml `_ file within the ``models`` directory. + +.. seealso:: Other unit test resources: -See `Writing data tests`_ for more examples. + * `dbt Unit Testing: Why You Need Them, Tutorial & Best Practices `_ + * `dbt unit testing best practices `_ -Step 6. Install and use your dbt package + +Step 7. Install and use your dbt package ======================================== -Once you've pushed all the changes to your custom dbt package repo, now we're ready to use it. +Once you've pushed all the changes to your custom dbt package repo, you're ready to use it. -Use ``tutor config save`` to update the following Tutor variables to use your custom package instead of the Aspects -default. +Use ``tutor config save`` to update the following Tutor variables to use your custom package instead of the Aspects default. -- ``DBT_REPOSITORY``: A git repository URL to clone and use as the dbt project. +* ``DBT_REPOSITORY``: The git repository URL of your custom dbt package. - Set this to the URL for your custom dbt package. + (Default: ``https://github.com/openedx/aspects-dbt``) - Default: ``https://github.com/openedx/aspects-dbt`` -- ``DBT_BRANCH``: The branch to use when cloning the dbt project. +* ``DBT_BRANCH``: The hash/branch/tag of your custom dbt package that you wish to use. - Set this to the hash/branch/tag of the custom dbt package that you wish to use. + (Default: the latest tagged version of aspects-dbt) - Default: varies between versions of Aspects. -- ``EXTRA_DBT_PACKAGES``: Add any python packages that your dbt project requires here. +Optional + * ``EXTRA_DBT_PACKAGES``: Add any python packages that your dbt project requires here. - Default: ``[]`` -- ``DBT_PROFILE_*``: variables used in the Aspects ``dbt/profiles.yml`` file, including several Clickhouse connection settings. + * ``DBT_PROFILE_*``: variables used in the Aspects ``dbt/profiles.yml`` file, including several Clickhouse connection settings. -- ``DBT_SSH_KEY``: The private SSH key to use when cloning the dbt project. Only necessary if you are using a private repository. + * ``DBT_SSH_KEY``: The private SSH key to use when cloning the dbt project. Only necessary if you are using a private repository. Once your package is configured in Tutor, you can run dbt commands directly on your deployment. -See `dbt commands`_ for a full list of available commands. - .. code-block:: bash # Build and test your package @@ -195,49 +204,34 @@ See `dbt commands`_ for a full list of available commands. # Deploy your customizations tutor dev do dbt -c "run" - # Run tests on the data + # Run data tests on the data tutor dev do dbt -c "test" + # Run unit tests on the data + tutor dev do dbt -c "test --selector unit_tests" -Step 7. Troubleshooting -======================= + # To push your new transformations to Superset SQL Lab + tutor dev do import-assets -You may need to repeat steps 4-6 a few times to resolve any warnings or errors that dbt reports with your package. +See `dbt commands `_ for a full list of available commands. -Don't forget to push your changes to your repo before running the tutor dbt command: it fetches a clean copy of your -configured package repo + branch each time it runs. +Troubleshooting +=============== +* Tutor dbt commands may need to be run with ``--only_changed False`` to force a full dbt run if you have made changes to your dbt package that are not being picked up. -See `dbt debugging`_ and `The missing guide to debug() in dbt`_ for more information on how to debug issues with your package. +* Don't forget to push your changes to your repo before running the tutor dbt command: it fetches a clean copy of your configured package repo + branch each time it runs. +Next Steps +========== +With your custom dbt package, you can create new charts and dashboards in Superset using the new models created by your dbt transforms. See :ref:`superset-extra-assets` for more information. References -########## - -* `Building dbt packages`_: dbt's guide to building packages -* `Best practice guides`_: dbt's guidelines on project structure, style, and setup. -* `About dbt models`_: dbt's guide to creating SQL or Python model transforms -* `Writing data tests`_: dbt's guide to writing package tests -* `dbt commands`_: list of all dbt commands -* `dbt debugging`_: guide for debugging issues with dbt -* `The missing guide to debug() in dbt`_: detailed advice for debugging issues with dbt -* `aspects-dbt`_: Aspects' dbt package -* `aspects-dbt-example`_: the demo custom dbt package used in this tutorial. -* `eduNEXT/dbt-aspects-unidigital`_: a real custom dbt package running in production Aspects - -.. _aspects-dbt: https://github.com/openedx/aspects-dbt -.. _aspects-dbt-example: https://github.com/open-craft/aspects-dbt-example -.. _aspects-dbt-example#1: https://github.com/open-craft/aspects-dbt-example/pull/1 -.. _aspects-dbt requirements.txt: https://github.com/openedx/aspects-dbt/blob/main/requirements.txt -.. _About dbt init: https://docs.getdbt.com/reference/commands/init -.. _About dbt models: https://docs.getdbt.com/docs/build/models -.. _Configure dbt models: https://docs.getdbt.com/reference/model-configs -.. _Best practice guides: https://docs.getdbt.com/best-practices -.. _dbt commands: https://docs.getdbt.com/reference/dbt-commands -.. _dbt debugging: https://docs.getdbt.com/guides/debug-errors -.. _dbt ref: https://docs.getdbt.com/reference/dbt-jinja-functions/ref -.. _eduNEXT/dbt-aspects-unidigital: https://github.com/eduNEXT/dbt-aspects-unidigital -.. _Building dbt packages: https://docs.getdbt.com/guides/building-packages -.. _Install dbt: https://docs.getdbt.com/docs/core/installation-overview -.. _Writing data tests: https://docs.getdbt.com/best-practices/writing-custom-generic-tests -.. _tutor-contrib-aspects plugin.py: https://github.com/openedx/tutor-contrib-aspects/blob/main/tutoraspects/plugin.py -.. _The missing guide to debug() in dbt: https://docs.getdbt.com/blog/guide-to-jinja-debug +========== + +* `aspects-dbt `_: Aspects' dbt package +* `sample-aspects-dbt `_: the demo custom dbt package used in this tutorial +* `Building dbt packages `_: dbt's guide to building packages +* `Best practice guides `_: dbt's guidelines on project structure, style, and setup +* `About dbt models `_: dbt's guide to creating SQL or Python model transforms +* `dbt debugging `_: guide for debugging issues with dbt +* `The missing guide to debug() in dbt `_: detailed advice for debugging issues with dbt diff --git a/docs/technical_documentation/how-tos/index.rst b/docs/technical_documentation/how-tos/index.rst index e661e9c..5f280ce 100644 --- a/docs/technical_documentation/how-tos/index.rst +++ b/docs/technical_documentation/how-tos/index.rst @@ -2,24 +2,10 @@ How-Tos ####### .. toctree:: - :maxdepth: 2 + :maxdepth: 1 + :glob: :caption: Content: - Aspects Plugin for Tutor - Upgrade Aspects - Configure Aspects for Production - Backfill old or missing data - Troubleshooting Aspects - Connect to external Clickhouse database - Changing the xAPI actor identifier - Superset extra assets - Superset language settings - Superset extra row level security - Superset extra jinja filters - Superset custom roles - Superset custom config settings - Clickhouse extra SQL - Extending dbt - Run Aspects in a ClickHouse cluster - xAPI Transforms - Running Aspects without Tutor + ClickHouse + Superset + * diff --git a/docs/technical_documentation/how-tos/superset/index.rst b/docs/technical_documentation/how-tos/superset/index.rst new file mode 100644 index 0000000..f46ffa3 --- /dev/null +++ b/docs/technical_documentation/how-tos/superset/index.rst @@ -0,0 +1,8 @@ +Superset +######## + +.. toctree:: + :maxdepth: 1 + :glob: + + * diff --git a/docs/technical_documentation/how-tos/superset_config_docker.rst b/docs/technical_documentation/how-tos/superset/superset_config_docker.rst similarity index 90% rename from docs/technical_documentation/how-tos/superset_config_docker.rst rename to docs/technical_documentation/how-tos/superset/superset_config_docker.rst index 07f26d8..5f5fe43 100644 --- a/docs/technical_documentation/how-tos/superset_config_docker.rst +++ b/docs/technical_documentation/how-tos/superset/superset_config_docker.rst @@ -1,7 +1,7 @@ .. _superset-config-docker: -Superset Custom Production Config Settings -****************************************** +Use Superset custom production config settings +********************************************** To add or override production config settings in Superset, use the patch ``superset-config-docker``. Changes take effect on LMS/CMS restart. diff --git a/docs/technical_documentation/how-tos/superset/superset_extra_assets.rst b/docs/technical_documentation/how-tos/superset/superset_extra_assets.rst new file mode 100644 index 0000000..a6dc676 --- /dev/null +++ b/docs/technical_documentation/how-tos/superset/superset_extra_assets.rst @@ -0,0 +1,107 @@ +.. _superset-extra-assets: + +Add extra Superset assets +************************* + +Developers can use the ``superset-extra-assets`` patch to add extra assets (charts, datasets, dashboards, databases) to Superset that will be imported at initialization. + +The patch is expected to be a list of assets (with an extra attribute called ``_file_name`` to uniquely identify each asset). Each asset should be a valid yaml object with the attributes that Superset expects for each asset type. See +`aspects assets `_ for examples of asset yaml declarations. + +Assets in a tutor inline plugin using the patch would look like: + +.. code-block:: yaml + + name: custom-inline-plugin + version: 0.1.0 + patches: + superset-extra-assets: | + - _file_name: my-dashboard.yaml + dashboard_title: "..." + ... + - _file_name: my-chart.yaml + slice_name: "..." + ... + - _file_name: my-database.yaml + database_name: "..." + ... + - _file_name: my-dataset.yaml + table_name: "..." + ... + +| +| In most cases, however, you will want to develop Superset dashboards, charts, datasets, and sometimes even databases in the Superset UI, and import them into a plugin that can be managed in source control. + +Creating a Tutor plugin +======================= + +#. Use the `Tutor Plugin Cookiecutter `_ to create a skeleton plugin. We have created `tutor-contrib-aspects-sample `_ as an example. + +#. Create a `patch file `_ that will recursively include all of your Superset assets from ``templates/build/assets`` into Aspects when the plugin is enabled and ``tutor config save`` is run. + + .. code-block:: + + {% for file in "aspects-sample/build/assets"|walk_templates %} + --- + _file_name: {{ file.split('/')[-1] }} + {% include file %} + {% endfor %} + + .. attention:: + + The directory containing ``assets`` must be included in `ENV_TEMPLATE_TARGETS `_. Since this defaults to ``build`` and ``apps``, you can either add your asset files to one of these directories, or add an additional target to the plugin file. + +#. Create a `requirements.txt `_ with ``ruff`` for formatting. + +Import assets into plugin +========================= + +#. From the Superset UI, you can export your dashboard, which will download to your local computer as a ``.zip`` file. + +#. As long as you are running Aspects 1.0.3 or newer, you can use this command to unzip the file to your plugin directory. + + .. code-block:: bash + + tutor aspects import_superset_zip .zip --base_assets_path /templates/build/assets/ + + This command does several things to make import safer and easier including: + + * Check for some required security fields + * Safely name files so that charts with duplicate names don't overwrite each other + * Check that certain fields use Tutor settings instead of hard coded values + * Add a special ``_file_name`` key that tells Aspects what name to use for the file when importing back to Superset + +#. Before importing a dashboard back into Superset, you will need to manually add a `_roles `_ key, which sets the permissions for who can view the dashboard. + + .. code-block:: yaml + + _roles: + - '{{ SUPERSET_ROLES_MAPPING.instructor }}' + + In addition to the default roles (student, instructor, operator, admin) you can also add custom roles in a ``superset_roles`` patch. + +.. warning:: + + When a Superset zip is exported it will contain any datasets and databases needed for all of the charts in that dashboard. This may include the default xapi and MySQL databases or Aspects datasets that you probably do not want to overwrite, and can include their passwords! + + Use caution when importing datasets or databases into your plugin - only ones you have created yourself should exist in your plugin. + + If you truly mean to overwrite the default databases or datasets, make sure to remove any sensitive information such as passwords from the yaml files before importing. + + If you accidentally overwrite a default database, you can reset it by: + + #. Disabling your plugin (``tutor plugins disable ``) + #. Running ``tutor config save`` and ``tutor do import-assets`` to reset the assets + #. Re-enabling your plugin + +Update Superset assets +====================== + +Once you have your files imported to the plugin, make sure it is `installed and enabled `_ in your Tutor environment, then use ``import-assets`` to update your Superset assets. + +.. code-block:: bash + + pip install + tutor plugins enable + tutor config save + tutor import-assets diff --git a/docs/technical_documentation/how-tos/superset_jinja_filters.rst b/docs/technical_documentation/how-tos/superset/superset_jinja_filters.rst similarity index 96% rename from docs/technical_documentation/how-tos/superset_jinja_filters.rst rename to docs/technical_documentation/how-tos/superset/superset_jinja_filters.rst index 7756c40..e97fab8 100644 --- a/docs/technical_documentation/how-tos/superset_jinja_filters.rst +++ b/docs/technical_documentation/how-tos/superset/superset_jinja_filters.rst @@ -1,7 +1,7 @@ .. _superset-jinja-filters: -Superset extra jinja filters -***************************** +Create Superset extra jinja filters +*********************************** To create extra jinja filters, you can use the patch `superset-jinja-filters` diff --git a/docs/technical_documentation/how-tos/superset_language_settings.rst b/docs/technical_documentation/how-tos/superset/superset_language_settings.rst similarity index 95% rename from docs/technical_documentation/how-tos/superset_language_settings.rst rename to docs/technical_documentation/how-tos/superset/superset_language_settings.rst index 9b69457..8279b28 100644 --- a/docs/technical_documentation/how-tos/superset_language_settings.rst +++ b/docs/technical_documentation/how-tos/superset/superset_language_settings.rst @@ -1,7 +1,7 @@ .. _superset-language-settings: -Superset language settings -************************** +Change Superset language settings +********************************* By changing the following Tutor configuration variables you can change the default language: diff --git a/docs/technical_documentation/how-tos/superset_roles.rst b/docs/technical_documentation/how-tos/superset/superset_roles.rst similarity index 95% rename from docs/technical_documentation/how-tos/superset_roles.rst rename to docs/technical_documentation/how-tos/superset/superset_roles.rst index d002479..d34fdfb 100644 --- a/docs/technical_documentation/how-tos/superset_roles.rst +++ b/docs/technical_documentation/how-tos/superset/superset_roles.rst @@ -1,7 +1,7 @@ .. _superset-roles: -Superset extra roles -********************* +Create extra Superset roles +*************************** Create extra Superset roles, you can use the patch `superset-extra-roles`. This patch expects validJSON objects with the following structure: diff --git a/docs/technical_documentation/how-tos/superset_row_level_security.rst b/docs/technical_documentation/how-tos/superset/superset_row_level_security.rst similarity index 95% rename from docs/technical_documentation/how-tos/superset_row_level_security.rst rename to docs/technical_documentation/how-tos/superset/superset_row_level_security.rst index 93f8c1b..53ac954 100644 --- a/docs/technical_documentation/how-tos/superset_row_level_security.rst +++ b/docs/technical_documentation/how-tos/superset/superset_row_level_security.rst @@ -1,7 +1,7 @@ .. _superset-row-level-security: -Superset extra row level security -********************************* +Customize Superset row level security +************************************* To apply custom row level security filters to Superset, you can use the patch ``superset-row-level-security``. This patch can overwrite the `default list `_ of roles diff --git a/docs/technical_documentation/how-tos/superset_extra_assets.rst b/docs/technical_documentation/how-tos/superset_extra_assets.rst deleted file mode 100644 index f2e828e..0000000 --- a/docs/technical_documentation/how-tos/superset_extra_assets.rst +++ /dev/null @@ -1,109 +0,0 @@ -.. _superset-extra-assets: - -Superset extra assets -********************* - -Developers can use the `superset-extra-assets` patch to add extra assets to Superset and those -will be imported at initialization. - -An example a tutor inline plugin using the patch is the following: - -.. code-block:: yaml - - name: custom-inline-plugin - version: 0.1.0 - patches: - superset-extra-assets: | - - _file_name: my-dashboard.yaml - dashboard_title: "..." - ... - - _file_name: my-chart.yaml - slice_name: "..." - ... - - _file_name: my-database.yaml - database_name: "..." - ... - - _file_name: my-dataset.yaml - table_name: "..." - ... - - -The patch is expected to be a list of assets with an extra attribute called **_file_name**, -which uniquely identifies the asset entry. Each asset is expected to be a valid yaml object -with the attributes that Superset expects for each asset type. See -`assets `_ -for examples of asset yaml declarations. - -In most cases, however, you will want to develop Superset dashboards, charts, datasets, and -sometimes even databases in the Superset UI and import them into a plugin that can be managed -in source control. As of Aspects 1.0.3 use the same command to import assets into a Tutor -plugin that we use to maintain Aspects itself. - -You can use the `Tutor Plugin Cookiecutter `_ -to create a skeleton plugin. Then you can create a patch file that will include all of your -Superset assets into Aspects when the plugin is enabled. - -The file name is the patch name ``/patches/superset-extra-assets``: - -.. code-block:: - - {% for file in "/superset-assets/"|walk_templates %} - --- - _file_name: {{ file.split('/')[-1] }} - {% include file %} - {% endfor %} - - -This recursively includes each file in the superset-assets directory of your extension's -tempates directory, when ``tutor config save`` is run. For example -``/templates/superset-assets/*``. - -Now you can export your Superset dashboard, which will download to your local computer -as a .zip file. As long as you are running Aspects 1.0.3 or newer you can use the following -command to unzip the file to your plugin directory: - - -.. code-block:: - - tutor aspects import_superset_zip /.zip --base_assets_path //templates//superset-assets/ - - -This command does several things to make import safer and easier, including checking for some -required security fields, safely naming files so that charts with duplicate names don't -overwrite each other, and checking that certain fields use Tutor settings instead of hard -coded values. It also adds a special ``_file_name`` key that tells Aspects what name to use -for the file when importing back to Superset. - -.. warning:: - - When importing a dashboard you will need to manually add a ``_roles`` key, which - sets the permissions for who can view the dashboard. In addition to the default roles - (student, instructor, operator, admin) you can also add custom roles . - - -An example roles key looks like this: - -.. code-block:: - - _roles: - - '{{ SUPERSET_ROLES_MAPPING.instructor }}' - - -.. warning:: - - When a Superset zip is exported it will contain any datasets and databases needed for - all of the charts in that dashboard. This may include the default xapi and MySQL - databases or Aspects datasets that you probably do not want to overwrite, and can - include their passwords! - - Use caution when importing datasets or databases your project and make sure that they - are only ones you have created yourself unless you truly intend to overwrite defaults. - - -A simple example showing the final directory structure is available -`here `_. - -Once you have your files imported to the plugin, make sure it is installed and enabled -in your Tutor environment, then you should just be able to do ``tutor config save`` and -``tutor import-assets`` to update your Superset assets. - diff --git a/docs/technical_documentation/how-tos/xapi_transforms.rst b/docs/technical_documentation/how-tos/xapi_transforms.rst index 54d3599..b3caffe 100644 --- a/docs/technical_documentation/how-tos/xapi_transforms.rst +++ b/docs/technical_documentation/how-tos/xapi_transforms.rst @@ -1,7 +1,7 @@ .. _xapi_transforms: -xAPI Transforms -############### +Customize xAPI transforms +######################### Aspects converts raw Open edX tracking event JSON into :ref:`xapi-concepts` for storage and analysis. This conversion process is called "transformation".