diff --git a/README.md b/README.md index 3b35344b021..0ea5b0a135a 100644 --- a/README.md +++ b/README.md @@ -255,7 +255,6 @@ The list below contains the functionality that contributors are planning to deve * [x] [Offline Feature Server (alpha)](https://docs.feast.dev/reference/feature-servers/offline-feature-server) * [x] [Registry server (alpha)](https://github.com/feast-dev/feast/blob/master/docs/reference/feature-servers/registry-server.md) * **Data Quality Management (See [RFC](https://docs.google.com/document/d/110F72d4NTv80p35wDSONxhhPBqWRwbZXG4f9mNEMd98/edit))** - * [x] ~~Data profiling and validation (Great Expectations)~~ (deprecated) * [x] [Feature Quality Monitoring](https://docs.feast.dev/how-to-guides/feature-monitoring) — built-in metrics, drift detection, serving log monitoring, and UI dashboard * **Feature Discovery and Governance** * [x] Python SDK for browsing feature registry diff --git a/docs/README.md b/docs/README.md index 18f9cf8207d..e8588be340f 100644 --- a/docs/README.md +++ b/docs/README.md @@ -71,7 +71,7 @@ Feast helps ML platform/MLOps teams with DevOps experience productionize real-ti * **batch feature engineering**: Feast supports on-demand and streaming transformations. Feast is also investing in supporting batch transformations. * **native streaming feature integration:** Feast enables users to push streaming features, but does not pull from streaming sources or manage streaming pipelines. * **lineage:** Feast helps tie feature values to model versions, but is not a complete solution for capturing end-to-end lineage from raw data sources to model versions. Feast also has community contributed plugins with [DataHub](https://datahubproject.io/docs/generated/ingestion/sources/feast/) and [Amundsen](https://github.com/amundsen-io/amundsen/blob/4a9d60176767c4d68d1cad5b093320ea22e26a49/databuilder/databuilder/extractor/feast\_extractor.py). -* **data quality / drift detection**: Feast now includes built-in [Feature Quality Monitoring](how-to-guides/feature-monitoring.md) that computes statistical metrics (null rates, distributions, percentiles), detects drift across batch data and serving logs, and provides a monitoring UI dashboard. The older Great Expectations integration is deprecated. +* **data quality / drift detection**: Feast includes built-in [Feature Quality Monitoring](how-to-guides/feature-monitoring.md) that computes statistical metrics (null rates, distributions, percentiles), detects drift across batch data and serving logs, and provides a monitoring UI dashboard. ## Example use cases diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 547c88acf68..d35dbf1652b 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -57,7 +57,6 @@ * [Fraud detection on GCP](tutorials/tutorials-overview/fraud-detection.md) * [Real-time credit scoring on AWS](tutorials/tutorials-overview/real-time-credit-scoring-on-aws.md) * [Driver stats on Snowflake](tutorials/tutorials-overview/driver-stats-on-snowflake.md) -* [\[Deprecated\] Validating historical features with Great Expectations](tutorials/validating-historical-features.md) * [Building streaming features](tutorials/building-streaming-features.md) * [Retrieval Augmented Generation (RAG) with Feast](tutorials/rag-with-docling.md) * [RAG Fine Tuning with Feast and Milvus](../examples/rag-retriever/README.md) @@ -206,7 +205,7 @@ * [\[Beta\] On demand feature view](reference/beta-on-demand-feature-view.md) * [\[Alpha\] Static Artifacts Loading](reference/alpha-static-artifacts.md) * [\[Alpha\] Vector Database](reference/alpha-vector-database.md) -* [\[Deprecated\] Data quality monitoring (Great Expectations)](reference/dqm.md) +* [Data Quality Monitoring](reference/dqm.md) * [\[Alpha\] Streaming feature computation with Denormalized](reference/denormalized.md) * [\[Alpha\] Feature View Versioning](reference/alpha-feature-view-versioning.md) * [OpenLineage Integration](reference/openlineage.md) diff --git a/docs/adr/ADR-0011-data-quality-monitoring.md b/docs/adr/ADR-0011-data-quality-monitoring.md index e2e13745a61..657d219c48c 100644 --- a/docs/adr/ADR-0011-data-quality-monitoring.md +++ b/docs/adr/ADR-0011-data-quality-monitoring.md @@ -2,7 +2,7 @@ ## Status -Accepted +Superseded — The original external-library-based validation has been replaced by Feast's native [Feature Quality Monitoring](../how-to-guides/feature-monitoring.md) system (`feast monitor run`). ## Context @@ -12,92 +12,51 @@ Data quality issues can significantly impact ML model performance. Several compl - **Upstream pipeline bugs**: Bugs in upstream pipelines can cause invalid values to overwrite existing valid values in an online store. - **Training/serving skew**: Distribution shift between training and serving data can decrease model performance. -Feast needed a mechanism to validate data at retrieval time to catch these issues before they affect model training or serving. +Feast needed a mechanism to validate data to catch these issues before they affect model training or serving. ## Decision -Introduce a Data Quality Monitoring (DQM) module that validates datasets against user-curated rules, initially targeting historical retrieval (training dataset generation). +Introduce a Data Quality Monitoring (DQM) module that validates datasets against user-curated rules. -### Design +### Original Design (now replaced) -The validation process uses a **reference dataset** and a **profiler** pattern: +The original validation process used a **reference dataset** and a **profiler** pattern: 1. User prepares a reference dataset (saved from a known-good historical retrieval). 2. User defines a profiler function that produces a profile (set of expectations) from a dataset. 3. Validation is performed by comparing the tested dataset against the reference profile. -### Integration with Great Expectations +This approach was limited to historical retrieval only, required additional dependencies, and offered no built-in UI or automation. -The initial implementation uses [Great Expectations](https://greatexpectations.io/) as the validation engine: +### Current Design -```python -from feast.dqm.profilers.ge_profiler import ge_profiler -from great_expectations.dataset import Dataset -from great_expectations.core.expectation_suite import ExpectationSuite +The current system (`feast monitor run`) provides: -@ge_profiler -def my_profiler(dataset: Dataset) -> ExpectationSuite: - dataset.expect_column_max_to_be_between("column", 1, 2) - dataset.expect_column_values_to_not_be_null("important_feature") - return dataset.get_expectation_suite() -``` - -### Usage - -Validation is triggered during historical feature retrieval via a `validation_reference` parameter: - -```python -from feast import FeatureStore - -store = FeatureStore(".") - -job = store.get_historical_features(...) -df = job.to_df( - validation_reference=store - .get_saved_dataset("my_reference_dataset") - .as_reference(profiler=my_profiler) -) -``` - -If validation fails, a `ValidationFailed` exception is raised with details for all expectations that didn't pass. If validation succeeds, the materialized dataset is returned normally. - -### Key Decisions +- Automatic metric computation (null rates, percentiles, histograms) with no external dependencies +- Monitoring across batch data and serving logs +- CLI and REST API for automation +- Built-in UI monitoring dashboard +- Support for all offline store backends via SQL push-down -- **Profiler-based approach**: Users define their own validation rules via profiler functions rather than Feast prescribing fixed validation rules. -- **Great Expectations integration**: Leverages an established data validation framework rather than building custom validation logic. -- **Validation at retrieval time**: Validation is performed when datasets are materialized (`.to_df()` or `.to_arrow()`), not during ingestion. -- **ValidationReference as a registry object**: Saved datasets and their validation references are stored in the Feast registry for reuse. +See [Feature Quality Monitoring](../how-to-guides/feature-monitoring.md) for full documentation. ## Consequences ### Positive - Users can detect data quality issues before they affect model training. -- Flexible profiler pattern allows custom validation rules per use case. -- Integration with Great Expectations provides a rich set of built-in expectations. -- Reference datasets provide a baseline for detecting data drift. +- Native integration requires no extra dependencies. +- Covers both batch data and serving logs. +- Built-in UI provides immediate visibility into feature health. +- Baselines computed automatically on `feast apply`. ### Negative -- Currently limited to historical retrieval; online store write/read validation is planned but not yet implemented. -- Dependency on Great Expectations adds to the install footprint (optional via `feast[ge]`). -- Automatic profiling capabilities are limited; manual expectation crafting is recommended. - -## Superseded - -This ADR documents the original GE-based approach which is now **deprecated**. It has been superseded by Feast's built-in [Feature Quality Monitoring](../how-to-guides/feature-monitoring.md) system (introduced in 2025), which provides: - -- Automatic metric computation (null rates, percentiles, histograms) with no external dependencies -- Monitoring across batch data and serving logs -- CLI (`feast monitor run`) and REST API for automation -- Built-in UI monitoring dashboard -- Support for all offline store backends via SQL push-down - -The GE-based integration may be removed in a future release. +- Migration required from the original profiler-based approach. ## References -- Original RFC: Feast RFC-027: Data Quality Monitoring -- Implementation: `sdk/python/feast/dqm/`, `sdk/python/feast/saved_dataset.py` -- Documentation: [Data Quality Monitoring (deprecated)](../reference/dqm.md) -- **New system:** [Feature Quality Monitoring](../how-to-guides/feature-monitoring.md) +- Original RFC: Feast RFC-027: Data Quality Monitoring +- Implementation: `sdk/python/feast/monitoring/` +- Documentation: [Data Quality Monitoring](../reference/dqm.md) +- [Feature Quality Monitoring guide](../how-to-guides/feature-monitoring.md) diff --git a/docs/blog/feast-0-18-adds-snowflake-support-and-data-quality-monitoring.md b/docs/blog/feast-0-18-adds-snowflake-support-and-data-quality-monitoring.md index 4b4321e3259..8c587bdde9f 100644 --- a/docs/blog/feast-0-18-adds-snowflake-support-and-data-quality-monitoring.md +++ b/docs/blog/feast-0-18-adds-snowflake-support-and-data-quality-monitoring.md @@ -6,7 +6,7 @@ We are delighted to announce the release of Feast [0.18](https://github.com/feas * Snowflake offline store, which allows you to define and use features stored in Snowflake. * [Experimental] Saved Datasets, which allow training datasets to be persisted in an offline store. -* [Experimental] Data quality monitoring, which allows you to validate your training data with Great Expectations. Future work will allow you to detect issues with upstream data pipelines and check for training-serving skew. +* [Experimental] Data quality monitoring, which allows you to validate your training data. This has since been superseded by Feast's native [Feature Quality Monitoring](../how-to-guides/feature-monitoring.md) system. * Python feature server graduation from alpha status. * Performance improvements to on demand feature views, protobuf serialization and deserialization, and the Python feature server. @@ -22,7 +22,7 @@ Training datasets generated via `get_historical_features` can now be persisted i ### [Experimental] Data quality monitoring -Feast 0.18 includes the first milestone of our data quality monitoring work. Many users have requested ways to validate their training and serving data, as well as monitor for training-serving skew. Feast 0.18 allows users to validate their training data through an integration with [Great Expectations](https://greatexpectations.io/). Users can declare one of the previously generated training datasets as a reference for this validation by persisting it as a "saved dataset" (see previous section). More details about future milestones of data quality monitoring can be found [here](https://docs.feastsite.wpenginepowered.com/v/master/reference/data-quality). There's also a [tutorial on validating historical features](https://docs.feastsite.wpenginepowered.com/v/master/how-to-guides/validation/validating-historical-features) that demonstrates all new concepts in action. +Feast 0.18 includes the first milestone of our data quality monitoring work. Many users have requested ways to validate their training and serving data, as well as monitor for training-serving skew. Feast 0.18 allows users to validate their training data by declaring previously generated training datasets as a reference for validation, persisted as "saved datasets" (see previous section). This initial integration has since been superseded by Feast's native [Feature Quality Monitoring](../how-to-guides/feature-monitoring.md) system, which provides built-in metrics computation, drift detection, serving log monitoring, and a UI dashboard. ### Performance improvements diff --git a/docs/getting-started/concepts/dataset.md b/docs/getting-started/concepts/dataset.md index 061de54ebca..c86c13503ed 100644 --- a/docs/getting-started/concepts/dataset.md +++ b/docs/getting-started/concepts/dataset.md @@ -1,6 +1,6 @@ # \[Alpha] Saved dataset -Feast datasets allow for conveniently saving dataframes that include both features and entities to be subsequently used for data analysis and model training. Data Quality Monitoring was the original motivation for creating the dataset concept. Note that the Great Expectations-based validation that used saved datasets is now deprecated in favor of Feast's built-in [Feature Quality Monitoring](../../how-to-guides/feature-monitoring.md) system, which does not require saved datasets. +Feast datasets allow for conveniently saving dataframes that include both features and entities to be subsequently used for data analysis and model training. Data Quality Monitoring was the original motivation for creating the dataset concept. Dataset's metadata is stored in the Feast registry and raw data (features, entities, additional input keys and timestamp) is stored in the [offline store](../components/offline-store.md). diff --git a/docs/reference/codebase-structure.md b/docs/reference/codebase-structure.md index 6c5ad9a141e..4783773c270 100644 --- a/docs/reference/codebase-structure.md +++ b/docs/reference/codebase-structure.md @@ -28,7 +28,7 @@ The majority of Feast logic lives in these Python files: There are also several important submodules: * `infra/` contains all the infrastructure components, such as the provider, offline store, online store, batch materialization engine, and registry. -* `dqm/` covers data quality monitoring. The legacy Great Expectations profiler (`profilers/ge_profiler`) is deprecated; see [`monitoring/`](../../sdk/python/feast/monitoring/) for the current built-in monitoring system. +* `dqm/` covers data quality monitoring. See [`monitoring/`](../../sdk/python/feast/monitoring/) for the built-in monitoring system. * `diff/` covers the logic for determining how to apply infrastructure changes upon feature repo changes (e.g. the output of `feast plan` and `feast apply`). * `embedded_go/` covers the Go feature server. * `ui/` contains the embedded Web UI, to be launched on the `feast ui` command. diff --git a/docs/reference/dqm.md b/docs/reference/dqm.md index 51f98686dee..47090b5dd1c 100644 --- a/docs/reference/dqm.md +++ b/docs/reference/dqm.md @@ -1,89 +1,81 @@ # Data Quality Monitoring -{% hint style="warning" %} -**Deprecated:** The Great Expectations-based validation described on this page is deprecated and will be removed in a future release. It has been superseded by Feast's built-in [Feature Quality Monitoring](../how-to-guides/feature-monitoring.md) system, which provides richer metrics (histograms, percentiles, drift detection), works across batch data and serving logs, requires no external dependencies, and includes a built-in UI dashboard. +Feast's Data Quality Monitoring (DQM) system computes, stores, and serves statistical metrics for every registered feature. It gives you visibility into feature health — distributions, null rates, percentiles, histograms — across batch data and feature serving logs. -Please migrate to the new monitoring system. See the [Feature Quality Monitoring guide](../how-to-guides/feature-monitoring.md) for setup instructions. -{% endhint %} +Its goal is to address several complex data problems: -## Legacy: Great Expectations Integration - -The following documents the deprecated Great Expectations-based validation that was previously the only DQM option in Feast. This integration relied on `pip install 'feast[ge]'` and only supported validation during historical retrieval. - ---- +* **Data consistency** — new training datasets can differ significantly from previous datasets, potentially requiring changes in model architecture. +* **Upstream pipeline bugs** — bugs in upstream pipelines can cause invalid values to overwrite existing valid values in an online store. +* **Training/serving skew** — distribution shift between training and serving data can decrease model performance. ### Overview -The legacy validation process consists of the following steps: -1. User prepares reference dataset (only [saved datasets](../getting-started/concepts/dataset.md) from historical retrieval are supported). -2. User defines a profiler function that produces a profile using [Great Expectations](https://docs.greatexpectations.io). -3. Validation of the tested dataset is performed with the reference dataset and profiler provided as parameters. +Feast's DQM system works natively with your configured offline store — no additional infrastructure or external dependencies are required. The workflow is: -### Installation -```shell -pip install 'feast[ge]' -``` +1. **Register features** — run `feast apply` to register feature views. If `auto_baseline: true` is configured, baseline metrics are computed automatically. +2. **Schedule monitoring** — run `feast monitor run` on a schedule (daily recommended) to compute metrics across multiple time windows. +3. **Read metrics** — query metrics via the REST API or view them in the Feast UI. -### Dataset profile +### Configuration -This integration uses [Great Expectation's](https://greatexpectations.io/) [ExpectationSuite](https://legacy.docs.greatexpectations.io/en/latest/autoapi/great_expectations/core/expectation_suite/index.html#great_expectations.core.expectation_suite.ExpectationSuite) -as the dataset profile format. The user defines a profiler function that receives a dataset and returns an ExpectationSuite. +Enable DQM in your `feature_store.yaml`: + +```yaml +data_quality_monitoring: + auto_baseline: true +``` -```python -from great_expectations.dataset import Dataset -from great_expectations.core.expectation_suite import ExpectationSuite +### Computing Metrics -from feast.dqm.profilers.ge_profiler import ge_profiler +**Auto mode (recommended for production):** -@ge_profiler -def manual_profiler(dataset: Dataset) -> ExpectationSuite: - dataset.expect_column_max_to_be_between("column", 1, 2) - return dataset.get_expectation_suite() +```bash +feast monitor run ``` -### Validating Training Dataset +This detects the latest event timestamp in the source data and computes metrics for 5 time windows: daily, weekly, biweekly, monthly, and quarterly. -During retrieval of historical features, `validation_reference` can be passed as a parameter to methods `.to_df(validation_reference=...)` or `.to_arrow(validation_reference=...)` of RetrievalJob. -If validation is successful, the materialized dataset is returned. Otherwise, `feast.dqm.errors.ValidationFailed` exception is raised with details for expectations that didn't pass. +**Target a specific feature view:** -```python -from feast import FeatureStore +```bash +feast monitor run --feature-view driver_stats +``` -fs = FeatureStore(".") +**Explicit date range:** -job = fs.get_historical_features(...) -job.to_df( - validation_reference=fs - .get_saved_dataset("my_reference_dataset") - .as_reference(profiler=manual_profiler) -) +```bash +feast monitor run \ + --feature-view driver_stats \ + --start-date 2025-01-01 \ + --end-date 2025-01-07 \ + --granularity weekly ``` ---- +**Set a manual baseline:** -## Migration Guide +```bash +feast monitor run \ + --feature-view driver_stats \ + --start-date 2025-01-01 \ + --end-date 2025-03-31 \ + --granularity daily \ + --set-baseline +``` -The new [Feature Quality Monitoring](../how-to-guides/feature-monitoring.md) system replaces this integration with: +### Monitoring Feature Serving Logs -| Capability | GE-based (deprecated) | New DQM | -|---|---|---| -| Scope | Historical retrieval only | Batch data + serving logs | -| Dependencies | `feast[ge]` extra required | No extra dependencies | -| Metrics | User-defined expectations | Automatic: null rates, percentiles, histograms, drift | -| UI | None | Built-in monitoring dashboard | -| Automation | Manual profiler code | `feast monitor run` CLI + REST API | -| Backends | Limited | All offline store backends | +If your feature services have logging configured, you can compute metrics from the actual features served to models in production: -To migrate: +```bash +feast monitor run --source-type log +``` -1. Enable DQM in `feature_store.yaml`: - ```yaml - data_quality_monitoring: - auto_baseline: true - ``` +### Reading Metrics -2. Run `feast apply` to compute baseline metrics automatically. +Metrics are accessible via the REST API: -3. Schedule `feast monitor run` for ongoing monitoring. +``` +GET /monitoring/metrics/features?project=my_project&feature_view_name=driver_stats&granularity=daily +``` -4. Remove the `feast[ge]` dependency from your requirements. +See the [Feature Quality Monitoring guide](../how-to-guides/feature-monitoring.md) for full API reference, UI integration, and orchestrator examples. diff --git a/docs/reference/openlineage.md b/docs/reference/openlineage.md index 3acc6e7872b..bf9e18750ed 100644 --- a/docs/reference/openlineage.md +++ b/docs/reference/openlineage.md @@ -404,7 +404,7 @@ The consumer automatically links datasets across different producers when they r 2. **SymlinksDatasetFacet** — Producers can declare aliases. For example, Feast can declare that its internal `driver_hourly_stats` is a symlink to the Spark output at `s3://bucket/features/driver_hourly_stats/`. 3. **dataSource URI matching** — Datasets with matching `dataSource.uri` facets are linked even if their namespace or name differ. -Compatible producers include Airflow, Spark, dbt, Flink, Feast, Dagster, and Great Expectations. +Compatible producers include Airflow, Spark, dbt, Flink, Feast, and Dagster. ### RBAC for Lineage diff --git a/docs/roadmap.md b/docs/roadmap.md index 4eac7e68b3f..d92ffa38f24 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -90,7 +90,6 @@ The list below contains the functionality that contributors are planning to deve * [x] [Offline Feature Server (alpha)](https://docs.feast.dev/reference/feature-servers/offline-feature-server) * [x] [Registry server (alpha)](https://github.com/feast-dev/feast/blob/master/docs/reference/feature-servers/registry-server.md) * **Data Quality Management (See [RFC](https://docs.google.com/document/d/110F72d4NTv80p35wDSONxhhPBqWRwbZXG4f9mNEMd98/edit))** - * [x] ~~Data profiling and validation (Great Expectations)~~ (deprecated) * [x] [Feature Quality Monitoring](https://docs.feast.dev/how-to-guides/feature-monitoring) — built-in metrics, drift detection, serving log monitoring, and UI dashboard * **Feature Discovery and Governance** * [x] Python SDK for browsing feature registry diff --git a/docs/tutorials/validating-historical-features.md b/docs/tutorials/validating-historical-features.md deleted file mode 100644 index f2037f7c9c9..00000000000 --- a/docs/tutorials/validating-historical-features.md +++ /dev/null @@ -1,920 +0,0 @@ -# Validating historical features with Great Expectations - -{% hint style="warning" %} -**Deprecated:** This tutorial demonstrates the legacy Great Expectations-based validation which is deprecated. For new projects, use Feast's built-in [Feature Quality Monitoring](../how-to-guides/feature-monitoring.md) system which provides automatic metrics computation, drift detection, and a monitoring UI — with no external dependencies required. See also the [Monitoring Quickstart notebook](../../examples/monitoring/monitoring-quickstart.ipynb). -{% endhint %} - -In this tutorial, we will use the public dataset of Chicago taxi trips to present data validation capabilities of Feast. -- The original dataset is stored in BigQuery and consists of raw data for each taxi trip (one row per trip) since 2013. -- We will generate several training datasets (aka historical features in Feast) for different periods and evaluate expectations made on one dataset against another. - -Types of features we're ingesting and generating: -- Features that aggregate raw data with daily intervals (eg, trips per day, average fare or speed for a specific day, etc.). -- Features using SQL while pulling data from BigQuery (like total trips time or total miles travelled). -- Features calculated on the fly when requested using Feast's on-demand transformations - -Our plan: - -0. Prepare environment -1. Pull data from BigQuery (optional) -2. Declare & apply features and feature views in Feast -3. Generate reference dataset -4. Develop & test profiler function -5. Run validation on different dataset using reference dataset & profiler - - -> The original notebook and datasets for this tutorial can be found on [GitHub](https://github.com/feast-dev/dqm-tutorial). - -### 0. Setup - -Install Feast Python SDK and great expectations: - - -```python -!pip install 'feast[ge]' -``` - - -### 1. Dataset preparation (Optional) - -**You can skip this step if you don't have GCP account. Please use parquet files that are coming with this tutorial instead** - - -```python -!pip install google-cloud-bigquery -``` - - -```python -import pyarrow.parquet - -from google.cloud.bigquery import Client -``` - - -```python -bq_client = Client(project='kf-feast') -``` - -Running some basic aggregations while pulling data from BigQuery. Grouping by taxi_id and day: - - -```python -data_query = """SELECT - taxi_id, - TIMESTAMP_TRUNC(trip_start_timestamp, DAY) as day, - SUM(trip_miles) as total_miles_travelled, - SUM(trip_seconds) as total_trip_seconds, - SUM(fare) as total_earned, - COUNT(*) as trip_count -FROM `bigquery-public-data.chicago_taxi_trips.taxi_trips` -WHERE - trip_miles > 0 AND trip_seconds > 60 AND - trip_start_timestamp BETWEEN '2019-01-01' and '2020-12-31' AND - trip_total < 1000 -GROUP BY taxi_id, TIMESTAMP_TRUNC(trip_start_timestamp, DAY)""" -``` - - -```python -driver_stats_table = bq_client.query(data_query).to_arrow() - -# Storing resulting dataset into parquet file -pyarrow.parquet.write_table(driver_stats_table, "trips_stats.parquet") -``` - - -```python -def entities_query(year): - return f"""SELECT - distinct taxi_id -FROM `bigquery-public-data.chicago_taxi_trips.taxi_trips` -WHERE - trip_miles > 0 AND trip_seconds > 0 AND - trip_start_timestamp BETWEEN '{year}-01-01' and '{year}-12-31' -""" -``` - - -```python -entities_2019_table = bq_client.query(entities_query(2019)).to_arrow() - -# Storing entities (taxi ids) into parquet file -pyarrow.parquet.write_table(entities_2019_table, "entities.parquet") -``` - - -## 2. Declaring features - - -```python -import pyarrow.parquet -import pandas as pd - -from feast import FeatureView, Entity, FeatureStore, Field, BatchFeatureView -from feast.types import Float64, Int64 -from feast.value_type import ValueType -from feast.data_format import ParquetFormat -from feast.on_demand_feature_view import on_demand_feature_view -from feast.infra.offline_stores.file_source import FileSource -from feast.infra.offline_stores.file import SavedDatasetFileStorage -from datetime import timedelta - -``` - - -```python -batch_source = FileSource( - timestamp_field="day", - path="trips_stats.parquet", # using parquet file that we created on previous step - file_format=ParquetFormat() -) -``` - - -```python -taxi_entity = Entity(name='taxi', join_keys=['taxi_id']) -``` - - -```python -trips_stats_fv = BatchFeatureView( - name='trip_stats', - entities=[taxi_entity], - schema=[ - Field(name="total_miles_travelled", dtype=Float64), - Field(name="total_trip_seconds", dtype=Float64), - Field(name="total_earned", dtype=Float64), - Field(name="trip_count", dtype=Int64), - - ], - ttl=timedelta(seconds=86400), - source=batch_source, -) -``` - -*Read more about feature views in [Feast docs](https://docs.feast.dev/getting-started/concepts/feature-view)* - - -```python -@on_demand_feature_view( - sources=[ - trips_stats_fv, - ], - schema=[ - Field(name="avg_fare", dtype=Float64), - Field(name="avg_speed", dtype=Float64), - Field(name="avg_trip_seconds", dtype=Float64), - Field(name="earned_per_hour", dtype=Float64), - ] -) -def on_demand_stats(inp: pd.DataFrame) -> pd.DataFrame: - out = pd.DataFrame() - out["avg_fare"] = inp["total_earned"] / inp["trip_count"] - out["avg_speed"] = 3600 * inp["total_miles_travelled"] / inp["total_trip_seconds"] - out["avg_trip_seconds"] = inp["total_trip_seconds"] / inp["trip_count"] - out["earned_per_hour"] = 3600 * inp["total_earned"] / inp["total_trip_seconds"] - return out -``` - -*Read more about on demand feature views [here](../reference/beta-on-demand-feature-view.md)* - - -```python -store = FeatureStore(".") # using feature_store.yaml that stored in the same directory -``` - - -```python -store.apply([taxi_entity, trips_stats_fv, on_demand_stats]) # writing to the registry -``` - - -## 3. Generating training (reference) dataset - - -```python -taxi_ids = pyarrow.parquet.read_table("entities.parquet").to_pandas() -``` - -Generating range of timestamps with daily frequency: - - -```python -timestamps = pd.DataFrame() -timestamps["event_timestamp"] = pd.date_range("2019-06-01", "2019-07-01", freq='D') -``` - -Cross merge (aka relation multiplication) produces entity dataframe with each taxi_id repeated for each timestamp: - - -```python -entity_df = pd.merge(taxi_ids, timestamps, how='cross') -entity_df -``` - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
taxi_idevent_timestamp
091d5288487e87c5917b813ba6f75ab1c3a9749af906a2d...2019-06-01
191d5288487e87c5917b813ba6f75ab1c3a9749af906a2d...2019-06-02
291d5288487e87c5917b813ba6f75ab1c3a9749af906a2d...2019-06-03
391d5288487e87c5917b813ba6f75ab1c3a9749af906a2d...2019-06-04
491d5288487e87c5917b813ba6f75ab1c3a9749af906a2d...2019-06-05
.........
1569797ebf27414a0c7b128e7925e1da56d51a8b81484f7630cf...2019-06-27
1569807ebf27414a0c7b128e7925e1da56d51a8b81484f7630cf...2019-06-28
1569817ebf27414a0c7b128e7925e1da56d51a8b81484f7630cf...2019-06-29
1569827ebf27414a0c7b128e7925e1da56d51a8b81484f7630cf...2019-06-30
1569837ebf27414a0c7b128e7925e1da56d51a8b81484f7630cf...2019-07-01
-

156984 rows × 2 columns

-
- - - -Retrieving historical features for resulting entity dataframe and persisting output as a saved dataset: - - -```python -job = store.get_historical_features( - entity_df=entity_df, - features=[ - "trip_stats:total_miles_travelled", - "trip_stats:total_trip_seconds", - "trip_stats:total_earned", - "trip_stats:trip_count", - "on_demand_stats:avg_fare", - "on_demand_stats:avg_trip_seconds", - "on_demand_stats:avg_speed", - "on_demand_stats:earned_per_hour", - ] -) - -store.create_saved_dataset( - from_=job, - name='my_training_ds', - storage=SavedDatasetFileStorage(path='my_training_ds.parquet') -) -``` - -```python -, full_feature_names = False, tags = {}, _retrieval_job = , min_event_timestamp = 2019-06-01 00:00:00, max_event_timestamp = 2019-07-01 00:00:00)> -``` - - -## 4. Developing dataset profiler - -Dataset profiler is a function that accepts dataset and generates set of its characteristics. This charasteristics will be then used to evaluate (validate) next datasets. - -**Important: datasets are not compared to each other! -Feast use a reference dataset and a profiler function to generate a reference profile. -This profile will be then used during validation of the tested dataset.** - - -```python -import numpy as np - -from feast.dqm.profilers.ge_profiler import ge_profiler - -from great_expectations.core.expectation_suite import ExpectationSuite -from great_expectations.dataset import PandasDataset -``` - - -Loading saved dataset first and exploring the data: - - -```python -ds = store.get_saved_dataset('my_training_ds') -ds.to_df() -``` - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
total_earnedavg_trip_secondstaxi_idtotal_miles_travelledtrip_countearned_per_hourevent_timestamptotal_trip_secondsavg_fareavg_speed
068.252270.00000091d5288487e87c5917b813ba6f75ab1c3a9749af906a2d...24.702.054.1189432019-06-01 00:00:00+00:004540.034.12500019.585903
1221.00560.5000007a4a6162eaf27805aef407d25d5cb21fe779cd962922cb...54.1824.059.1436222019-06-01 00:00:00+00:0013452.09.20833314.499554
2160.501010.769231f4c9d05b215d7cbd08eca76252dae51cdb7aca9651d4ef...41.3013.043.9726032019-06-01 00:00:00+00:0013140.012.34615411.315068
3183.75697.550000c1f533318f8480a59173a9728ea0248c0d3eb187f4b897...37.3020.047.4159562019-06-01 00:00:00+00:0013951.09.1875009.625116
4217.751054.076923455b6b5cae6ca5a17cddd251485f2266d13d6a2c92f07c...69.6913.057.2064512019-06-01 00:00:00+00:0013703.016.75000018.308692
.................................
15697938.001980.0000000cccf0ec1f46d1e0beefcfdeaf5188d67e170cdff92618...14.901.069.0909092019-07-01 00:00:00+00:001980.038.00000027.090909
156980135.00551.250000beefd3462e3f5a8e854942a2796876f6db73ebbd25b435...28.4016.055.1020412019-07-01 00:00:00+00:008820.08.43750011.591837
156981NaNNaN9a3c52aa112f46cf0d129fafbd42051b0fb9b0ff8dcb0e...NaNNaNNaN2019-07-01 00:00:00+00:00NaNNaNNaN
15698263.00815.00000008308c31cd99f495dea73ca276d19a6258d7b4c9c88e43...19.964.069.5705522019-07-01 00:00:00+00:003260.015.75000022.041718
156983NaNNaN7ebf27414a0c7b128e7925e1da56d51a8b81484f7630cf...NaNNaNNaN2019-07-01 00:00:00+00:00NaNNaNNaN
-

156984 rows × 10 columns

-
- - - -Feast uses [Great Expectations](https://docs.greatexpectations.io/docs/) as a validation engine and [ExpectationSuite](https://legacy.docs.greatexpectations.io/en/latest/autoapi/great_expectations/core/expectation_suite/index.html#great_expectations.core.expectation_suite.ExpectationSuite) as a dataset's profile. Hence, we need to develop a function that will generate ExpectationSuite. This function will receive instance of [PandasDataset](https://legacy.docs.greatexpectations.io/en/latest/autoapi/great_expectations/dataset/index.html?highlight=pandasdataset#great_expectations.dataset.PandasDataset) (wrapper around pandas.DataFrame) so we can utilize both Pandas DataFrame API and some helper functions from PandasDataset during profiling. - - -```python -DELTA = 0.1 # controlling allowed window in fraction of the value on scale [0, 1] - -@ge_profiler -def stats_profiler(ds: PandasDataset) -> ExpectationSuite: - # simple checks on data consistency - ds.expect_column_values_to_be_between( - "avg_speed", - min_value=0, - max_value=60, - mostly=0.99 # allow some outliers - ) - - ds.expect_column_values_to_be_between( - "total_miles_travelled", - min_value=0, - max_value=500, - mostly=0.99 # allow some outliers - ) - - # expectation of means based on observed values - observed_mean = ds.trip_count.mean() - ds.expect_column_mean_to_be_between("trip_count", - min_value=observed_mean * (1 - DELTA), - max_value=observed_mean * (1 + DELTA)) - - observed_mean = ds.earned_per_hour.mean() - ds.expect_column_mean_to_be_between("earned_per_hour", - min_value=observed_mean * (1 - DELTA), - max_value=observed_mean * (1 + DELTA)) - - - # expectation of quantiles - qs = [0.5, 0.75, 0.9, 0.95] - observed_quantiles = ds.avg_fare.quantile(qs) - - ds.expect_column_quantile_values_to_be_between( - "avg_fare", - quantile_ranges={ - "quantiles": qs, - "value_ranges": [[None, max_value] for max_value in observed_quantiles] - }) - - return ds.get_expectation_suite() -``` - -Testing our profiler function: - - -```python -ds.get_profile(profiler=stats_profiler) -``` - 02/02/2022 02:43:47 PM INFO: 5 expectation(s) included in expectation_suite. result_format settings filtered. - - - - -**Verify that all expectations that we coded in our profiler are present here. Otherwise (if you can't find some expectations) it means that it failed to pass on the reference dataset (do it silently is default behavior of Great Expectations).** - -Now we can create validation reference from dataset and profiler function: - - -```python -validation_reference = ds.as_reference(name="validation_reference_dataset", profiler=stats_profiler) -``` - -and test it against our existing retrieval job - - -```python -_ = job.to_df(validation_reference=validation_reference) -``` - - 02/02/2022 02:43:52 PM INFO: 5 expectation(s) included in expectation_suite. result_format settings filtered. - 02/02/2022 02:43:53 PM INFO: Validating data_asset_name None with expectation_suite_name default - - -Validation successfully passed as no exceptions were raised. - - -### 5. Validating new historical retrieval - -Creating new timestamps for Dec 2020: - - -```python -from feast.dqm.errors import ValidationFailed -``` - - -```python -timestamps = pd.DataFrame() -timestamps["event_timestamp"] = pd.date_range("2020-12-01", "2020-12-07", freq='D') -``` - - -```python -entity_df = pd.merge(taxi_ids, timestamps, how='cross') -entity_df -``` - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
taxi_idevent_timestamp
091d5288487e87c5917b813ba6f75ab1c3a9749af906a2d...2020-12-01
191d5288487e87c5917b813ba6f75ab1c3a9749af906a2d...2020-12-02
291d5288487e87c5917b813ba6f75ab1c3a9749af906a2d...2020-12-03
391d5288487e87c5917b813ba6f75ab1c3a9749af906a2d...2020-12-04
491d5288487e87c5917b813ba6f75ab1c3a9749af906a2d...2020-12-05
.........
354437ebf27414a0c7b128e7925e1da56d51a8b81484f7630cf...2020-12-03
354447ebf27414a0c7b128e7925e1da56d51a8b81484f7630cf...2020-12-04
354457ebf27414a0c7b128e7925e1da56d51a8b81484f7630cf...2020-12-05
354467ebf27414a0c7b128e7925e1da56d51a8b81484f7630cf...2020-12-06
354477ebf27414a0c7b128e7925e1da56d51a8b81484f7630cf...2020-12-07
-

35448 rows × 2 columns

-
- - -```python -job = store.get_historical_features( - entity_df=entity_df, - features=[ - "trip_stats:total_miles_travelled", - "trip_stats:total_trip_seconds", - "trip_stats:total_earned", - "trip_stats:trip_count", - "on_demand_stats:avg_fare", - "on_demand_stats:avg_trip_seconds", - "on_demand_stats:avg_speed", - "on_demand_stats:earned_per_hour", - ] -) -``` - -Execute retrieval job with validation reference: - - -```python -try: - df = job.to_df(validation_reference=validation_reference) -except ValidationFailed as exc: - print(exc.validation_report) -``` - - 02/02/2022 02:43:58 PM INFO: 5 expectation(s) included in expectation_suite. result_format settings filtered. - 02/02/2022 02:43:59 PM INFO: Validating data_asset_name None with expectation_suite_name default - - [ - { - "expectation_config": { - "expectation_type": "expect_column_mean_to_be_between", - "kwargs": { - "column": "trip_count", - "min_value": 10.387244591346153, - "max_value": 12.695521167200855, - "result_format": "COMPLETE" - }, - "meta": {} - }, - "meta": {}, - "result": { - "observed_value": 6.692920555429092, - "element_count": 35448, - "missing_count": 31055, - "missing_percent": 87.6071992778154 - }, - "exception_info": { - "raised_exception": false, - "exception_message": null, - "exception_traceback": null - }, - "success": false - }, - { - "expectation_config": { - "expectation_type": "expect_column_mean_to_be_between", - "kwargs": { - "column": "earned_per_hour", - "min_value": 52.320624975640214, - "max_value": 63.94743052578249, - "result_format": "COMPLETE" - }, - "meta": {} - }, - "meta": {}, - "result": { - "observed_value": 68.99268345164135, - "element_count": 35448, - "missing_count": 31055, - "missing_percent": 87.6071992778154 - }, - "exception_info": { - "raised_exception": false, - "exception_message": null, - "exception_traceback": null - }, - "success": false - }, - { - "expectation_config": { - "expectation_type": "expect_column_quantile_values_to_be_between", - "kwargs": { - "column": "avg_fare", - "quantile_ranges": { - "quantiles": [ - 0.5, - 0.75, - 0.9, - 0.95 - ], - "value_ranges": [ - [ - null, - 16.4 - ], - [ - null, - 26.229166666666668 - ], - [ - null, - 36.4375 - ], - [ - null, - 42.0 - ] - ] - }, - "result_format": "COMPLETE" - }, - "meta": {} - }, - "meta": {}, - "result": { - "observed_value": { - "quantiles": [ - 0.5, - 0.75, - 0.9, - 0.95 - ], - "values": [ - 19.5, - 28.1, - 38.0, - 44.125 - ] - }, - "element_count": 35448, - "missing_count": 31055, - "missing_percent": 87.6071992778154, - "details": { - "success_details": [ - false, - false, - false, - false - ] - } - }, - "exception_info": { - "raised_exception": false, - "exception_message": null, - "exception_traceback": null - }, - "success": false - } - ] - - -Validation failed since several expectations didn't pass: -* Trip count (mean) decreased more than 10% (which is expected when comparing Dec 2020 vs June 2019) -* Average Fare increased - all quantiles are higher than expected -* Earn per hour (mean) increased more than 10% (most probably due to increased fare) - diff --git a/infra/website/docs/blog/feast-data-quality-monitoring.md b/infra/website/docs/blog/feast-data-quality-monitoring.md index 88e8550f83f..2c918c83a63 100644 --- a/infra/website/docs/blog/feast-data-quality-monitoring.md +++ b/infra/website/docs/blog/feast-data-quality-monitoring.md @@ -31,7 +31,7 @@ The biggest change is that monitoring is now a first-class Feast workflow: ## From validation to monitoring -Feast previously supported a Great Expectations-based DQM path for validating historical retrievals. That integration was useful, but it lived outside the normal feature store workflow: users had to install the `feast[ge]` extra, write profiler code, and run validation against saved datasets. +Feast previously supported an external-library-based validation path for historical retrievals. That integration was useful, but it lived outside the normal feature store workflow: users had to install extra dependencies, write profiler code, and run validation against saved datasets. That original integration proved the need for data quality inside Feast. It helped answer an important question: after generating a training dataset, does this dataset satisfy the expectations we care about? diff --git a/infra/website/docs/blog/feast-mlflow-kubeflow.md b/infra/website/docs/blog/feast-mlflow-kubeflow.md index 5199da61da5..d0b89ac7138 100644 --- a/infra/website/docs/blog/feast-mlflow-kubeflow.md +++ b/infra/website/docs/blog/feast-mlflow-kubeflow.md @@ -199,10 +199,6 @@ For cross-system lineage that extends beyond Feast into upstream data pipelines ### Data quality monitoring -:::note -The Great Expectations integration described in earlier versions of this post is now **deprecated**. Feast includes a built-in [Feature Quality Monitoring](/docs/how-to-guides/feature-monitoring) system that provides richer metrics, requires no external dependencies, and includes a monitoring UI. -::: - Feast's native data quality monitoring system automatically computes statistical metrics — null rates, distributions, percentiles, histograms — for every registered feature across both batch data and serving logs. It detects drift by comparing current metrics against baselines computed during `feast apply`. ```yaml