diff --git a/src/cookbooks/client_guidelines.md b/src/cookbooks/client_guidelines.md index 0fe1efd96..c8e7345ea 100644 --- a/src/cookbooks/client_guidelines.md +++ b/src/cookbooks/client_guidelines.md @@ -1,161 +1,3 @@ # Client Implementation Guidelines for Experiments -**Note**: This guidance is useful for implementing Normandy experiments. To ship experiments with the Nimbus platform, please see the guidance for engineers at . - -There are three supported approaches for enabling experimental features for Firefox: - -- [Firefox Prefs](#prefs) - - Prefs can be used to control features that **land in-tree**. - [Feature Gates](#feature-gates) provide a wrapper around prefs that can be used from JavaScript. -- [Firefox Extensions](#extensions) AKA "**Add-ons**". - - If the feature being tested should not land in the tree, or if it will ultimately ship as an extension, then an extension should be used. - -New features go through the standard Firefox review, testing, and deployment processes, and are then enabled experimentally in the field using [Normandy][normandy-docs]. - -## Prefs - -Firefox Preferences (AKA "prefs") are commonly used to enable and disable features. However, prefs are more complex to implement correctly than [feature gates](#feature-gates). - -**Each pref should represent a different experimental treatment**. If your experimental feature requires multiple prefs, then Normandy does not currently support this but will soon. In the meantime, an [extension](#extensions) such as [multipreffer][multipreffer-docs] may be used. - -There are three types of Prefs: - -1. Built-in prefs - shipped with Firefox, in `firefox.js`. -2. `user branch` - set by the user, overriding built-in prefs. -3. `default branch` - Overrides both built-in and `user branch` prefs. Only persists until the browser session ends, next restart will revert to either built-in or `user branch` (if set). - -[Normandy][normandy-docs] supports overriding both the `user` and `default` branches, although the latter is preferred as it does not permanently override user settings. `default` branch prefs are simple to reset since they do not persist past a restart. - -**In order for features to be activated experimentally using `default branch` prefs**: - -- The feature must not start up before `final-ui-startup` is observed. - -For instance, to set an observer: - -```js -Services.obs.addObserver(this, "final-ui-startup", true); -``` - -In this example, `this` would implement an `observe(subject, topic, data)` function which will be called when `final-ui-startup` is observed. See the [Observer documentation][observer-docs] for more information. - -- It must be possible to enable/disable the feature at runtime, via a pref change. - -This is similar to the observer pattern above: - -```js -Services.prefs.addObserver("pref_name", this); -``` - -More information is available in the [Preference service documentation][pref-service-docs]. - -- Never use `Services.prefs.prefHasUserValue()`, or any other function specific to `user branch` prefs. - -- Prefs should be set by default in `firefox.js` - -If your feature cannot abide by one or more of these rules (for instance, it needs to run at startup and/or cannot be toggled at runtime) then experimental preferences can be set on the `user branch`. This is more complex than using the methods described above; user branch prefs override the users choice, which is a really complex thing to try to support when flipping prefs experimentally. We also need to be careful to back up and reset the pref, and then figure out how to resolve conflicts if the user has changed the pref in the meantime. - -## Feature Gates - -A new Feature Gate library for Firefox Desktop is now available. - -**Each feature gate should represent a different experimental treatment**. If your experimental feature requires multiple flags, then Normandy will not be able to support this directly and an [extension](#extensions) may be used. - -### Feature Gate caveats - -The current Feature Gate library comes with a few caveats, and may not be appropriate for your situation: - -- Only JS is supported. -- Always asynchronous. - -Future versions of the Feature Gate API will include C++/Rust support and a synchronous API. - -### Using the Feature Gate library - -Read [the documentation][feature-gate-docs] to get started. - -## Extensions - -Firefox currently supports the [Web Extensions API](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions). - -**If new WebExtension APIs are needed, they should land in-tree**. Extensions which are signed by Mozilla can load privileged code using the [WebExtension Experiments](https://firefox-source-docs.mozilla.org/toolkit/components/extensions/webextensions/index.html), but this is not preferred. - -WebExtensions go through the same correctness and performance tests as other features. This is possible using the Mozilla tryserver by dropping your XPI into `testing/profiles/common/extensions` in `mozilla-central` and pushing to Tryserver - see the [Testing Extensions](#testing-extensions) section below. - -NOTE - it is ideal to test against the version of Firefox which the extension will release against, but there is a [bug related to artifact builds on release channels][artifact-bug] which must be worked around. The workaround is pretty simple (modify an `artifacts.py` file), but this bug being resolved will make it much simpler. - -**Each extension can represent a different experimental treatment (preferred), or the extension can choose the branch internally**. - -### SHIELD studies - -The previous version of the experiments program, SHIELD, always bundled privileged code with extensions and would do things such as mock UI features in Firefox. - -This sort of approach is discouraged for new features - land these (or the necessary WebExtension APIs) in-tree instead. - -For the moment, the [SHIELD Study Add-on Utilities](https://github.com/mozilla/shield-studies-addon-utils/) may be used if the extension needs to control the lifecycle of the study, but using one extension per experimental treatment makes this unnecessary and is preferred. The APIs provided by the SHIELD Study Add-on Utilities will be available as privileged APIs shipped with Firefox soon. - -# Development and Testing - -## Testing Built-in Features - -Firefox features go through standard development and testing processes. See the [Firefox developer guide][firefox-dev-docs] for more information. - -## Testing Extensions - -Extensions do not need to go through the same process, but should take advantage of Mozilla CI and bug tracking systems: - -1. Use the Mozilla CI to test changes (tryserver). -2. Performance tests (**this step is required**) - extension XPI files should be placed in `testing/profiles/common/extensions/`, which will cause test harnesses to load the XPI. -3. Custom unit/functional tests (AKA `xpcshell`/`mochitest`) may be placed in `testing/extensions`, although running these tests outside Mozilla CI is acceptable so these are **optional**. -4. Receive reviewer approval. A Firefox peer **must sign off** if this extension contains privileged code, aka WebExtension Experiments. - -- Any [Firefox Peer][firefox-peer-list] should be able to do the review, or point you to someone who can. - -5. Extension is signed. -6. Email to `pi-request@mozilla.com` is sent to request QA -7. QA approval signed off in Bugzilla. -8. Extension is shipped via [Normandy][normandy-docs]. - -## Example Extensions Testing Workflow - -Note that for the below to work you only need [Mercurial][mercurial] installed, but if you want to do local testing you must be set up to [build Firefox][firefox-build-docs]. You don't need to build Firefox from source; [artifact builds][firefox-artifact-build] are sufficient. - -In order to use Mozilla CI (AKA "[Tryserver][try-server-docs]"), you must have a full clone of the `mozilla-central` repository: - -```bash -hg clone https://hg.mozilla.org/mozilla-central -cd mozilla-central -``` - -Copy in unsigned XPI, and commit it to your local Mercurial repo: - -```bash -cp ~/src/my-extension.xpi testing/profiles/common/extensions/ -hg add testing/profiles/common/extensions/my-extension.xpi -hg commit -m "Bug nnn - Testing my extension" testing/profiles/common/extensions/my-extension.xpi -``` - -Push to Try: - -```bash -./mach try -p linux64,macosx64,win64 -b do -u none -t all --artifact -``` - -This will run Mozilla CI tests on all platforms - -Note that you must have Level 1 commit access to use tryserver. If you are interested in interacting with Mozilla CI from Github (which only requires users to be in the Mozilla GitHub org), check out the [Taskcluster Integration proof-of-concept][taskcluster-integration-poc]. - -Also note that this requires an investment time to set up just as CircleCI or Travis-CI would, so it's not really appropriate for short-term projects. Use tryserver directly instead. - -[feature-gate-docs]: https://firefox-source-docs.mozilla.org/toolkit/components/featuregates/featuregates/index.html -[firefox-dev-docs]: https://firefox-source-docs.mozilla.org/ -[mercurial]: https://www.mercurial-scm.org/ -[taskcluster-integration-poc]: https://github.com/biancadanforth/taskcluster-integration-poc/ -[firefox-build-docs]: https://firefox-source-docs.mozilla.org/setup/index.html -[firefox-artifact-build]: https://firefox-source-docs.mozilla.org/contributing/build/artifact_builds.html -[try-server-docs]: https://firefox-source-docs.mozilla.org/tools/try/ -[observer-docs]: https://searchfox.org/mozilla-central/rev/59e797b66f5ce8a27ede0e7677688931be7aed20/xpcom/ds/nsIObserverService.idl#24-39 -[normandy-docs]: https://github.com/mozilla/normandy -[pref-service-docs]: https://searchfox.org/mozilla-central/source/modules/libpref/nsIPrefService.idl -[multipreffer-docs]: https://github.com/nhnt11/multipreffer -[firefox-peer-list]: https://wiki.mozilla.org/Modules/All#Firefox -[artifact-bug]: https://bugzilla.mozilla.org/show_bug.cgi?id=1435403 +To ship experiments with the Nimbus platform, please see the guidance for engineers at . diff --git a/src/cookbooks/normandy_events.md b/src/cookbooks/normandy_events.md index 91bd1be49..85f481179 100644 --- a/src/cookbooks/normandy_events.md +++ b/src/cookbooks/normandy_events.md @@ -1,7 +1,18 @@ # Working with Normandy events -A common request is to count the number of users who have -enrolled or unenrolled from a SHIELD experiment. +
+Normandy and SHIELD are older experimentation platforms +at Mozilla, but are no longer in use. However, Normandy +events are still in use (as of January 2025) by Jetstream +for computing enrollments and exposures. Note that this +will likely change in the near future because these events +have been ported to Glean and legacy telemetry is being +deprecated across the platform. + +For more up to date information on events used by Nimbus, +see . + +
The [`events` table](../datasets/batch_view/events/reference.md) includes Normandy enrollment and unenrollment events @@ -15,12 +26,6 @@ or name (for add-on experiments). Normandy events are described in detail in the [Firefox source tree docs][normandy-doc]. -Note that addon studies do not have branch information in the events table, -since addons, not Normandy, are responsible for branch assignment. -For studies built with the obsolete [add-on utilities][`addon-utils`], -branch assignments are published to the -[shield_study] dataset. - ## Counting pref-flip enrollment events by branch The `event_map_values` column of enroll events contains a `branch` key, @@ -68,5 +73,3 @@ ORDER BY 1, 2, 3 ``` [normandy-doc]: https://firefox-source-docs.mozilla.org/toolkit/components/normandy/normandy/data-collection.html#enrollment -[shield_study]: ../datasets/experiment_telemetry.md#telemetryshield_study -[`addon-utils`]: https://github.com/mozilla/shield-studies-addon-utils diff --git a/src/datasets/experiment_telemetry.md b/src/datasets/experiment_telemetry.md index 3e9291d73..32f271290 100644 --- a/src/datasets/experiment_telemetry.md +++ b/src/datasets/experiment_telemetry.md @@ -1,11 +1,8 @@ -# Analyzing data from SHIELD studies +# Accessing Experiment Telemetry This article introduces the datasets that are useful for analyzing studies in Firefox. -After reading this article, -you should understand how to answer questions about -study enrollment, -identify telemetry from clients enrolled in an experiment, -and locate telemetry from add-on studies. +After reading this article, you should understand how to answer questions about +experiment enrollment, and identify telemetry from clients enrolled in an experiment. ## Table of contents @@ -27,7 +24,34 @@ the slug for add-on experiments is defined in the recipe by a field named `name` You can find the slug associated with an experiment in Experimenter. -## Tables +## Tables (Glean) + +### `experiments` map (Glean) + +Glean tables include a `ping_info` column with `experiments` mapping from +experiment slug to a struct containing information about the experiment, +including `branch`. + +You can query for enrolled clients using a query like: + +```sql +SELECT + -- ... some fields ..., + `mozfun.map.get_key`(ping_info.experiments, '{experiment_slug}').branch +FROM + `moz-fx-data-shared-prod.firefox_desktop.metrics` +WHERE + `mozfun.map.get_key`(ds.ping_info.experiments, '{experiment_slug}') IS NOT NULL + AND DATE(m.submission_timestamp) BETWEEN '' AND '' + AND normalized_channel = 'release' + -- AND ...other filters... +``` + +## Tables (Legacy Telemetry) + +As of January 2025, legacy telemetry is still used for enrollment and exposure +events. However, while Glean adoption is in progress, these docs remain for +the time being as reference. ### `experiments` map (ping tables) @@ -85,47 +109,3 @@ The event schema is described [in the Firefox source tree](https://hg.mozilla.org/mozilla-central/file/tip/toolkit/components/normandy/lib/TelemetryEvents.sys.mjs). The `events` table is updated daily. - -### `telemetry.shield_study_addon` - -The `telemetry.shield_study_addon` table contains SHIELD telemetry from legacy add-on experiments, -i.e. key-value pairs sent with the -`browser.study.sendTelemetry()` method from the -[SHIELD study add-on utilities](https://github.com/mozilla/shield-studies-addon-utils/) -library. - -The `study_name` attribute of the `payload` column will contain the identifier -registered with the SHIELD add-on utilities. -This is set by the add-on; sometimes it takes the value of -`applications.gecko.id` from the add-on's `manifest.json`. -This is often not the same as the Normandy slug. - -The schema for shield-study-addon pings is described in the -[`mozilla-pipeline-schemas` repository](https://github.com/mozilla-services/mozilla-pipeline-schemas/tree/master/schemas/telemetry/shield-study-addon). - -The key-value pairs are present in `data` attribute of the `payload` column. - -The `telemetry.shield_study_addon` table contains only full days of data. -If you need access to data with lower latency, you can use the "live" table -`telemetry_live.shield_study_addon_v4` which should have latency significantly -less than 1 hour. - -### `telemetry.shield_study` - -The `telemetry.shield_study` dataset includes -enrollment and unenrollment events for legacy add-on experiments only, -sent by the [SHIELD study add-on utilities](https://github.com/mozilla/shield-studies-addon-utils/). - -The `study_name` attribute of the `payload` column will contain the identifier -registered with the SHIELD add-on utilities. -This is set by the add-on; sometimes it takes the value of -`applications.gecko.id` from the add-on's `manifest.json`. -This is often not the same as the Normandy slug. - -Normandy also emits its own enrollment and unenrollment events for these studies, -which are available in the `events` table. - -The `telemetry.shield_study` table contains only full days of data. -If you need access to data with lower latency, you can use the "live" table -`telemetry_live.shield_study_v4` which should have latency significantly -less than 1 hour. diff --git a/src/tools/projects.md b/src/tools/projects.md index 1cc16ff03..da4ab825f 100644 --- a/src/tools/projects.md +++ b/src/tools/projects.md @@ -80,9 +80,11 @@ See also [`data-docs`][docs] for documentation on datasets. | [Hardware Report][hwreport_gh] | Firefox Hardware Report, [available here][hwreport] | | [St. Mocli][stmocli] | A command-line interface to [STMO][stmo] | | [probe-scraper] | Scrape and publish Telemetry probe data from Firefox | -| [test-tube] | Compare data across branches in experiments | | [experimenter] | A web application for managing experiments | -| [St. Moab][stmoab] | Automatically generate Redash dashboard for A/B experiments | +| [Jetstream] | Automated analysis for experiments | +| [metric-hub] | Semantic layer for metric definitions | + +See also [What Data Tool Should I Use?][data-tools-wiki] for more information on Data Tools and their uses. [tmo_gh]: https://github.com/mozilla/telemetry-dashboard [glam]: https://github.com/mozilla/glam @@ -91,16 +93,16 @@ See also [`data-docs`][docs] for documentation on datasets. [redashstmo]: https://github.com/mozilla/redash-stmo [taar]: https://github.com/mozilla/taar [ensemble]: https://github.com/mozilla/ensemble -[shield]: https://wiki.mozilla.org/index.php?title=Firefox/Shield [tmo]: https://telemetry.mozilla.org [stmo]: https://sql.telemetry.mozilla.org [hwreport_gh]: https://github.com/mozilla/firefox-hardware-report [hwreport]: https://data.firefox.com/dashboard/hardware [stmocli]: https://github.com/mozilla/stmocli [probe-scraper]: https://github.com/mozilla/probe-scraper -[test-tube]: https://github.com/mozilla/firefox-test-tube [experimenter]: https://github.com/mozilla/experimenter -[stmoab]: https://github.com/mozilla/stmoab +[jetstream]: https://github.com/mozilla/jetstream +[metric-hub]: https://github.com/mozilla/metric-hub +[data-tools-wiki]: https://mozilla-hub.atlassian.net/wiki/spaces/DATA/pages/375750774/Data+Tools ## Legacy projects