diff --git a/.DS_Store b/.DS_Store deleted file mode 100644 index 6cf2a1b..0000000 Binary files a/.DS_Store and /dev/null differ diff --git a/README.md b/README.md index fb3245c..4974546 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,7 @@ knowledge_graph_banner

- Getting set up - • - Tutorials + Getting set up

@@ -17,7 +15,7 @@ Key use cases include: * **Compare state standards**: Adapt content aligned to one state standard to other states, initially in math across Common Core State Standards and 15+ additional states * **Curriculum alignment:** Align your content or create additional materials aligned to curriculum (private-beta access only \- details below on how to join) -Knowledge Graph is distributed as CSV and JSON export files, making it accessible without specialized infrastructure. These files reflect a graph-based model, allowing developers to work with the data in relational databases or other environments. This structure enables rich querying and supports AI-enhanced educational applications. +Knowledge Graph is distributed as graph-native JSONL export files, making it accessible without specialized infrastructure. These files directly represent the underlying graph model, enabling developers to work with the data in graph databases, while remaining easy to ingest with standard data-processing tools. Developers can load the data into graph databases for relationship-centric querying or transform it for use in relational databases and data pipelines. This structure enables rich querying and supports AI-enhanced educational applications. For complete setup instructions and usage examples, see the [full docs](https://docs.learningcommons.org/knowledge-graph/). @@ -25,82 +23,49 @@ For complete setup instructions and usage examples, see the [full docs](https:// | Path | Description | | :---- | :---- | -| [import\_scripts/](./import_scripts/) | Helper scripts to import Knowledge Graph data into relational databases | -| [sample\_queries/](./sample_queries/) | Example SQL queries for exploring Knowledge Graph data | -| [tutorials/](./tutorials/) | Standalone example apps to demonstrate how Knowledge Graph data could be applied to solve different use cases | | [LICENSE](./LICENSE.md) | Open source license details | ## **Quick Start** -**v1.2.0 will be the final version where Knowledge Graph can be downloaded as CSV flat files. Starting from v1.3.0 onwards, Knowledge Graph will be accessible as graph-native JSON flat files and we’ll be starting to grant access to our REST API in early 2026. Any CSV and JSON flat files that were previously downloaded will be unaffected.** - -The knowledge graph data is available for download in both CSV and JSON formats. The graph data is exported with each file representing a specific entity type, and a relationships file capturing the connections between entities. - -**CSV files:** UTF-8 encoded with comma delimiters and quoted fields. All CSV files include header rows with column names. - -**JSON files:** Newline delimited JSON format with UTF-8 encoding. +The Knowledge graph data is available for download in newline delimited jsonl format with UTF-8 encoding. The graph data is exported with nodes.jsonl representing the nodes of the knowledge graph and the relationships.jsonl file capturing the connections between nodes. ## Files -* `StandardsFramework`: Educational standards frameworks -* `StandardsFrameworkItem`: Individual standards and learning objectives within frameworks -* `LearningComponent`: Granular, precise representations of individual skills or concepts -* `Relationships`: Connections and associations between all entity types +* `nodes.jsonl`: This file contains graph node records, defining each node by a unique identifier, labels, and a set of associated properties. +* `relationships.jsonl`: This file contains graph relationship records, describing how nodes are connected, including the relationship type, properties, and the source and target nodes. ## Download options -There are two options to download the files: direct s3 links, or using curl commands. +There are two options to download the files: direct links, or using curl commands. -### Direct S3 links +### Direct links Click links to download files directly. Files will download to your browser's default location (typically `~/Downloads`). -**CSV files:** -- [StandardsFramework.csv](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/StandardsFramework.csv?ref=github) -- [StandardsFrameworkItem.csv](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/StandardsFrameworkItem.csv?ref=github) -- [LearningComponent.csv](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/LearningComponent.csv?ref=github) -- [Relationships.csv](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/Relationships.csv?ref=github) - -**For SQL database imports:** Move the downloaded CSV files to `/tmp/kg-data/` to use the import scripts without modification: - -```bash -mkdir -p /tmp/kg-data -mv ~/Downloads/StandardsFramework.csv ~/Downloads/StandardsFrameworkItem.csv ~/Downloads/LearningComponent.csv ~/Downloads/Relationships.csv /tmp/kg-data/ -``` - -**JSON files:** -- [StandardsFramework.json](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/StandardsFramework.json?ref=github) -- [StandardsFrameworkItem.json](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/StandardsFrameworkItem.json?ref=github) -- [LearningComponent.json](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/LearningComponent.json?ref=github) -- [Relationships.json](https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/Relationships.json?ref=github) +- [nodes.jsonl](https://cdn.learningcommons.org/knowledge-graph/v1.3.0/exports/nodes.jsonl?ref=github) +- [relationships.jsonl](https://cdn.learningcommons.org/knowledge-graph/v1.3.0/exports/relationships.jsonl?ref=github) ### Using curl commands If you don't have `curl` installed, see [installation instructions](https://github.com/curl/curl). -**Recommended**: Download files to `/tmp/kg-data/` for compatibility with the SQL import scripts: - ```bash -# Create directory and download CSV files -mkdir -p /tmp/kg-data -cd /tmp/kg-data - -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/StandardsFramework.csv?ref=gh_curl" -o StandardsFramework.csv -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/StandardsFrameworkItem.csv?ref=gh_curl" -o StandardsFrameworkItem.csv -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/LearningComponent.csv?ref=gh_curl" -o LearningComponent.csv -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/csv/Relationships.csv?ref=gh_curl" -o Relationships.csv +curl -L "https://cdn.learningcommons.org/knowledge-graph/v1.3.0/exports/nodes.jsonl?ref=gh_curl" -o nodes.jsonl +curl -L "https://cdn.learningcommons.org/knowledge-graph/v1.3.0/exports/relationships.jsonl?ref=gh_curl" -o relationships.jsonl ``` + +## Querying with jq + +One option to query the JSONL files is using [jq](https://jqlang.github.io/jq/). Example to extract Common Core math standards: + ```bash -# Download JSON files -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/StandardsFramework.json?ref=gh_curl" -o StandardsFramework.json -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/StandardsFrameworkItem.json?ref=gh_curl" -o StandardsFrameworkItem.json -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/LearningComponent.json?ref=gh_curl" -o LearningComponent.json -curl -L "https://aidt-knowledge-graph-datasets-public-prod.s3.us-west-2.amazonaws.com/knowledge-graph/v1.2.0/json/Relationships.json?ref=gh_curl" -o Relationships.json +jq -c 'select((.labels | contains(["StandardsFrameworkItem"])) and .properties.jurisdiction == "Multi-State" and .properties.academicSubject == "Mathematics")' nodes.jsonl > common_core_math_standards.jsonl ``` -### **Next steps** - -This quick start walked you through how to download Knowledge Graph so you can start using it. You can explore next steps under [import\_scripts/](./import_scripts/), [sample\_queries/](./sample_queries/) and [tutorials/](./tutorials/). +This filters for nodes with: +- Label: `StandardsFrameworkItem` +- Jurisdiction: `Multi-State` (Common Core) +- Academic Subject: `Mathematics` ## **Support & Feedback** diff --git a/import_scripts/.DS_Store b/import_scripts/.DS_Store deleted file mode 100644 index f112dce..0000000 Binary files a/import_scripts/.DS_Store and /dev/null differ diff --git a/import_scripts/mysql/README.md b/import_scripts/mysql/README.md deleted file mode 100644 index f223b9a..0000000 --- a/import_scripts/mysql/README.md +++ /dev/null @@ -1,71 +0,0 @@ -**v1.2.0 will be the final version where Knowledge Graph can be downloaded as CSV flat files. Starting from v1.3.0 onwards, Knowledge Graph will be accessible as graph-native JSON flat files and we’ll be starting to grant access to our REST API in early 2026. Any CSV and JSON flat files that were previously downloaded will be unaffected.** - -# MySQL Import Guide - -This guide provides instructions for loading the Learning Commons Knowledge Graph dataset into a MySQL database. - -## Prerequisites - -- A running MySQL instance. If you don't have MySQL installed locally, you can use Docker to run a MySQL container. For detailed instructions on setting up and running MySQL with Docker, refer to the [official MySQL Docker documentation](https://hub.docker.com/_/mysql). -- Downloaded CSV data files (see the [main README](../../README.md) for download instructions) - -## Setup Steps - -### 1. Create Database and Tables - -First create a dedicated database if that's preferred: - -```bash -mysql -u -p -e "CREATE DATABASE ;" -``` - -Then connect to your MySQL database and create the necessary tables by running: - -```bash -mysql -u -p < create_tables.sql -``` - -Or execute the SQL file content directly in your MySQL client. See `create_tables.sql` for the complete table definitions. - -### 2. Load CSV Data - -After creating the tables, load the data: - -1. If you downloaded files to `/tmp/kg-data/` using the commands in the main README, the script will work without modification. Otherwise, update the file paths in `load_data.sql` to point to your downloaded CSV files. - -2. Enable local file loading on the server (if not already enabled): - -```sql -mysql -u -p -e "SET GLOBAL local_infile=1;" -``` - -3. Execute the load script with local file loading enabled: - -```bash -mysql -u -p --local-infile=1 < load_data.sql -``` - -**Note**: The `local_infile` setting must be enabled on both the server (`SET GLOBAL local_infile=1`) and client (`--local-infile=1`) for `LOAD DATA LOCAL INFILE` to work. - -## Query Examples - -Once the data is loaded, you can run queries to explore the tables: - -```sql --- Show all tables -SHOW TABLES; - --- Sample data queries -SELECT * FROM standards_framework LIMIT 10; -SELECT * FROM standards_framework_item LIMIT 10; -SELECT * FROM learning_component LIMIT 10; -SELECT * FROM relationships LIMIT 10; -``` - -## Docker MySQL Setup - -If using Docker MySQL, you can connect using: - -```bash -docker exec -it mysql -u -p -``` diff --git a/import_scripts/mysql/create_tables.sql b/import_scripts/mysql/create_tables.sql deleted file mode 100644 index a9d64e0..0000000 --- a/import_scripts/mysql/create_tables.sql +++ /dev/null @@ -1,75 +0,0 @@ --- MySQL version of the init.sql script -CREATE TABLE IF NOT EXISTS standards_framework_item ( - `identifier` TEXT, - `caseIdentifierURI` TEXT, - `caseIdentifierUUID` TEXT, - `statementCode` TEXT, - `description` TEXT, - `statementType` TEXT, - `normalizedStatementType` TEXT, - `jurisdiction` TEXT, - `academicSubject` TEXT, - `gradeLevel` TEXT, - `inLanguage` TEXT, - `dateCreated` DATE, - `dateModified` DATE, - `notes` TEXT, - `author` TEXT, - `provider` TEXT, - `license` TEXT, - `attributionStatement` TEXT -); - -CREATE TABLE IF NOT EXISTS standards_framework ( - `identifier` TEXT, - `caseIdentifierURI` TEXT, - `caseIdentifierUUID` TEXT, - `name` TEXT, - `description` TEXT, - `jurisdiction` TEXT, - `academicSubject` TEXT, - `inLanguage` TEXT, - `adoptionStatus` TEXT, - `dateCreated` DATE, - `dateModified` DATE, - `notes` TEXT, - `author` TEXT, - `provider` TEXT, - `license` TEXT, - `attributionStatement` TEXT -); - -CREATE TABLE IF NOT EXISTS learning_component ( - `identifier` TEXT, - `description` TEXT, - `academicSubject` TEXT, - `inLanguage` TEXT, - `dateCreated` DATE, - `dateModified` DATE, - `author` TEXT, - `provider` TEXT, - `license` TEXT, - `attributionStatement` TEXT -); - -CREATE TABLE IF NOT EXISTS relationships ( - `identifier` TEXT, - `relationshipType` TEXT, - `description` TEXT, - `sourceEntity` TEXT, - `sourceEntityKey` TEXT, - `sourceEntityValue` TEXT, - `targetEntity` TEXT, - `targetEntityKey` TEXT, - `targetEntityValue` TEXT, - `dateCreated` DATE, - `dateModified` DATE, - `author` TEXT, - `provider` TEXT, - `license` TEXT, - `attributionStatement` TEXT, - `jaccard` DOUBLE, - `ccssLCCount` INT, - `sharedLCCount` INT, - `stateLCCount` INT -); diff --git a/import_scripts/mysql/load_data.sql b/import_scripts/mysql/load_data.sql deleted file mode 100644 index abb8734..0000000 --- a/import_scripts/mysql/load_data.sql +++ /dev/null @@ -1,32 +0,0 @@ --- Load data from CSV files --- Default location: /tmp/kg-data/ --- If you downloaded the files to a different location, update the paths below --- Note: Use mysql --local-infile=1 when running this script - -LOAD DATA LOCAL INFILE '/tmp/kg-data/StandardsFramework.csv' -INTO TABLE standards_framework -FIELDS TERMINATED BY ',' -OPTIONALLY ENCLOSED BY '"' -LINES TERMINATED BY '\n' -IGNORE 1 ROWS; - -LOAD DATA LOCAL INFILE '/tmp/kg-data/StandardsFrameworkItem.csv' -INTO TABLE standards_framework_item -FIELDS TERMINATED BY ',' -OPTIONALLY ENCLOSED BY '"' -LINES TERMINATED BY '\n' -IGNORE 1 ROWS; - -LOAD DATA LOCAL INFILE '/tmp/kg-data/LearningComponent.csv' -INTO TABLE learning_component -FIELDS TERMINATED BY ',' -OPTIONALLY ENCLOSED BY '"' -LINES TERMINATED BY '\n' -IGNORE 1 ROWS; - -LOAD DATA LOCAL INFILE '/tmp/kg-data/Relationships.csv' -INTO TABLE relationships -FIELDS TERMINATED BY ',' -OPTIONALLY ENCLOSED BY '"' -LINES TERMINATED BY '\n' -IGNORE 1 ROWS; \ No newline at end of file diff --git a/import_scripts/postgresql/README.md b/import_scripts/postgresql/README.md deleted file mode 100644 index 21cfe5e..0000000 --- a/import_scripts/postgresql/README.md +++ /dev/null @@ -1,65 +0,0 @@ -**v1.2.0 will be the final version where Knowledge Graph can be downloaded as CSV flat files. Starting from v1.3.0 onwards, Knowledge Graph will be accessible as graph-native JSON flat files and we’ll be starting to grant access to our REST API in early 2026. Any CSV and JSON flat files that were previously downloaded will be unaffected.** - -# PostgreSQL Import Guide - -This guide provides instructions for loading the Learning Commons Knowledge Graph dataset into a PostgreSQL database. - -## Prerequisites - -- A running PostgreSQL instance. If you don't have PostgreSQL installed locally, you can use Docker to run a PostgreSQL container. For detailed instructions on setting up and running PostgreSQL with Docker, refer to the [official PostgreSQL Docker documentation](https://hub.docker.com/_/postgres). -- Downloaded CSV data files (see the [main README](../../README.md) for download instructions) - -## Setup Steps - -### 1. Create Database and Tables - -First create a dedicated database if that's preferred: - -```bash -psql -U -c "CREATE DATABASE ;" -``` - -Then connect to your PostgreSQL database and create the necessary tables by running: - -```bash -psql -U -d -f create_tables.sql -``` - -Or execute the SQL file content directly in your PostgreSQL client. See `create_tables.sql` for the complete table definitions. - -### 2. Load CSV Data - -After creating the tables, load the data: - -1. If you downloaded files to `/tmp/kg-data/` using the commands in the main README, the script will work without modification. Otherwise, update the file paths in `load_data.sql` to point to your downloaded CSV files. - -2. Execute the load script: - -```bash -psql -U -d -f load_data.sql -``` - -**Note**: Ensure PostgreSQL has read access to the file paths specified in the load script. - -## Query Examples - -Once the data is loaded, you can run queries to explore the tables: - -```sql --- List all tables -\dt - --- Sample data queries -SELECT * FROM standards_framework LIMIT 10; -SELECT * FROM standards_framework_item LIMIT 10; -SELECT * FROM learning_component LIMIT 10; -SELECT * FROM relationships LIMIT 10; -``` - -## Docker PostgreSQL Setup - -If using Docker PostgreSQL, you can connect using: - -```bash -docker exec -it psql -U -d -``` diff --git a/import_scripts/postgresql/create_tables.sql b/import_scripts/postgresql/create_tables.sql deleted file mode 100644 index 42927c8..0000000 --- a/import_scripts/postgresql/create_tables.sql +++ /dev/null @@ -1,74 +0,0 @@ -CREATE TABLE IF NOT EXISTS standards_framework_item ( - "identifier" TEXT, - "caseIdentifierURI" TEXT, - "caseIdentifierUUID" TEXT, - "statementCode" TEXT, - "description" TEXT, - "statementType" TEXT, - "normalizedStatementType" TEXT, - "jurisdiction" TEXT, - "academicSubject" TEXT, - "gradeLevel" JSON, - "inLanguage" TEXT, - "dateCreated" DATE, - "dateModified" DATE, - "notes" TEXT, - "author" TEXT, - "provider" TEXT, - "license" TEXT, - "attributionStatement" TEXT -); - -CREATE TABLE IF NOT EXISTS standards_framework ( - "identifier" TEXT, - "caseIdentifierURI" TEXT, - "caseIdentifierUUID" TEXT, - "name" TEXT, - "description" TEXT, - "jurisdiction" TEXT, - "academicSubject" TEXT, - "inLanguage" TEXT, - "adoptionStatus" TEXT, - "dateCreated" DATE, - "dateModified" DATE, - "notes" TEXT, - "author" TEXT, - "provider" TEXT, - "license" TEXT, - "attributionStatement" TEXT -); - -CREATE TABLE IF NOT EXISTS learning_component ( - "identifier" TEXT, - "description" TEXT, - "academicSubject" TEXT, - "inLanguage" TEXT, - "dateCreated" DATE, - "dateModified" DATE, - "author" TEXT, - "provider" TEXT, - "license" TEXT, - "attributionStatement" TEXT -); - -CREATE TABLE IF NOT EXISTS relationships ( - "identifier" TEXT, - "relationshipType" TEXT, - "description" TEXT, - "sourceEntity" TEXT, - "sourceEntityKey" TEXT, - "sourceEntityValue" TEXT, - "targetEntity" TEXT, - "targetEntityKey" TEXT, - "targetEntityValue" TEXT, - "dateCreated" DATE, - "dateModified" DATE, - "author" TEXT, - "provider" TEXT, - "license" TEXT, - "attributionStatement" TEXT, - "jaccard" DOUBLE PRECISION, - "ccssLCCount" INT, - "sharedLCCount" INT, - "stateLCCount" INT -); diff --git a/import_scripts/postgresql/load_data.sql b/import_scripts/postgresql/load_data.sql deleted file mode 100644 index c137a4e..0000000 --- a/import_scripts/postgresql/load_data.sql +++ /dev/null @@ -1,8 +0,0 @@ --- Load data from CSV files --- Default location: /tmp/kg-data/ --- If you downloaded the files to a different location, update the paths below - -\COPY standards_framework_item FROM '/tmp/kg-data/StandardsFrameworkItem.csv' DELIMITER ',' CSV HEADER; -\COPY standards_framework FROM '/tmp/kg-data/StandardsFramework.csv' DELIMITER ',' CSV HEADER; -\COPY learning_component FROM '/tmp/kg-data/LearningComponent.csv' DELIMITER ',' CSV HEADER; -\COPY relationships FROM '/tmp/kg-data/Relationships.csv' DELIMITER ',' CSV HEADER; \ No newline at end of file diff --git a/sample_queries/.DS_Store b/sample_queries/.DS_Store deleted file mode 100644 index 9909f34..0000000 Binary files a/sample_queries/.DS_Store and /dev/null differ diff --git a/sample_queries/mysql/.keep b/sample_queries/mysql/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/sample_queries/mysql/combined_queries/list_all_learning_components_in_a_grade.sql b/sample_queries/mysql/combined_queries/list_all_learning_components_in_a_grade.sql deleted file mode 100644 index ddd9140..0000000 --- a/sample_queries/mysql/combined_queries/list_all_learning_components_in_a_grade.sql +++ /dev/null @@ -1,20 +0,0 @@ -WITH grade6_standards AS ( - SELECT `caseIdentifierUUID` - FROM standards_framework_item - WHERE `gradeLevel` LIKE '%"6"%' -), -supporting_components AS ( - SELECT DISTINCT r.`sourceEntityValue` AS lc_id - FROM relationships r - JOIN grade6_standards g - ON g.`caseIdentifierUUID` = r.`targetEntityValue` - WHERE r.`relationshipType` = 'supports' - AND r.`sourceEntity` = 'LearningComponent' - AND r.`sourceEntityKey` = 'identifier' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' -) -SELECT lc.* -FROM learning_component lc -JOIN supporting_components c - ON lc.`identifier` = c.lc_id; \ No newline at end of file diff --git a/sample_queries/mysql/combined_queries/list_learning_components_supporting_a_standard.sql b/sample_queries/mysql/combined_queries/list_learning_components_supporting_a_standard.sql deleted file mode 100644 index f4bc13f..0000000 --- a/sample_queries/mysql/combined_queries/list_learning_components_supporting_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT lc.* -FROM relationships r -JOIN learning_component lc - ON lc.`identifier` = r.`sourceEntityValue` -WHERE r.`relationshipType` = 'supports' - AND r.`sourceEntity` = 'LearningComponent' - AND r.`sourceEntityKey` = 'identifier' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntityValue` = '6ba1c7ad-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/mysql/combined_queries/list_standards_supported_by_a_learning_component.sql b/sample_queries/mysql/combined_queries/list_standards_supported_by_a_learning_component.sql deleted file mode 100644 index 5d5ada2..0000000 --- a/sample_queries/mysql/combined_queries/list_standards_supported_by_a_learning_component.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT sfi.* -FROM relationships r -JOIN standards_framework_item sfi - ON sfi.`caseIdentifierUUID` = r.`targetEntityValue` -WHERE r.`relationshipType` = 'supports' - AND r.`sourceEntity` = 'LearningComponent' - AND r.`sourceEntityKey` = 'identifier' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' - AND r.`sourceEntityValue` = '019a93e7-1877-5e9b-b4b3-476f201fccc8'; \ No newline at end of file diff --git a/sample_queries/mysql/crosswalk_queries/find_best_state_match_for_ccssm_standard.sql b/sample_queries/mysql/crosswalk_queries/find_best_state_match_for_ccssm_standard.sql deleted file mode 100644 index 42a572e..0000000 --- a/sample_queries/mysql/crosswalk_queries/find_best_state_match_for_ccssm_standard.sql +++ /dev/null @@ -1,37 +0,0 @@ --- Find the best Texas state standard matches for a given CCSSM standard --- Returns crosswalks ordered by Jaccard score (highest similarity first) --- with metadata about both the CCSSM and matching Texas state standards - -SELECT - -- CCSSM Standard Information - ccss.`statementCode` AS ccss_standard_code, - ccss.`description` AS ccss_description, - ccss.`gradeLevel` AS ccss_grade_level, - ccss.`jurisdiction` AS ccss_jurisdiction, - - -- State Standard Information - state.`statementCode` AS state_standard_code, - state.`description` AS state_description, - state.`gradeLevel` AS state_grade_level, - state.`jurisdiction` AS state_jurisdiction, - - -- Crosswalk Metrics - r.`jaccard`, - r.`sharedLCCount`, - r.`stateLCCount`, - r.`ccssLCCount`, - - -- Entity Values for further joins if needed - r.`sourceEntityValue` AS state_uuid, - r.`targetEntityValue` AS ccss_uuid -FROM relationships r -JOIN standards_framework_item state - ON state.`caseIdentifierUUID` = r.`sourceEntityValue` -JOIN standards_framework_item ccss - ON ccss.`caseIdentifierUUID` = r.`targetEntityValue` -WHERE r.`relationshipType` = 'hasStandardAlignment' - AND ccss.`statementCode` = '6.EE.B.5' - AND ccss.`jurisdiction` = 'Multi-State' - AND state.`jurisdiction` = 'Texas' -ORDER BY r.`jaccard` DESC -LIMIT 10; diff --git a/sample_queries/mysql/crosswalk_queries/get_all_crosswalks.sql b/sample_queries/mysql/crosswalk_queries/get_all_crosswalks.sql deleted file mode 100644 index fe18fe8..0000000 --- a/sample_queries/mysql/crosswalk_queries/get_all_crosswalks.sql +++ /dev/null @@ -1,17 +0,0 @@ --- Get all crosswalks with state standard information --- Returns state → CCSSM standard alignments ordered by Jaccard score - -SELECT - state.`statementCode` AS state_standard_code, - state.`jurisdiction` AS state_jurisdiction, - r.`sourceEntityValue`, - r.`targetEntityValue`, - r.`jaccard`, - r.`stateLCCount`, - r.`ccssLCCount`, - r.`sharedLCCount` -FROM relationships r -JOIN standards_framework_item state - ON state.`caseIdentifierUUID` = r.`sourceEntityValue` -WHERE r.`relationshipType` = 'hasStandardAlignment' -ORDER BY r.`jaccard` DESC; diff --git a/sample_queries/mysql/crosswalk_queries/get_crosswalks_by_jaccard_threshold.sql b/sample_queries/mysql/crosswalk_queries/get_crosswalks_by_jaccard_threshold.sql deleted file mode 100644 index 14cf2c1..0000000 --- a/sample_queries/mysql/crosswalk_queries/get_crosswalks_by_jaccard_threshold.sql +++ /dev/null @@ -1,22 +0,0 @@ --- Get all Texas crosswalks that meet or exceed a specified Jaccard similarity threshold --- Ordered by CCSSM standard code, then Jaccard score - -SELECT - state_std.`jurisdiction` AS state_jurisdiction, - state_std.`statementCode` AS state_standard_code, - ccss_std.`statementCode` AS ccss_standard_code, - r.`jaccard`, - r.`sharedLCCount`, - r.`stateLCCount`, - r.`ccssLCCount` -FROM relationships r -JOIN standards_framework_item state_std - ON state_std.`caseIdentifierUUID` = r.`sourceEntityValue` -JOIN standards_framework_item ccss_std - ON ccss_std.`caseIdentifierUUID` = r.`targetEntityValue` -WHERE r.`relationshipType` = 'hasStandardAlignment' - AND r.`jaccard` >= 0.7 - AND state_std.`jurisdiction` = 'Texas' -ORDER BY - ccss_std.`statementCode`, - r.`jaccard` DESC; diff --git a/sample_queries/mysql/crosswalk_queries/get_crosswalks_for_state.sql b/sample_queries/mysql/crosswalk_queries/get_crosswalks_for_state.sql deleted file mode 100644 index 073ccc7..0000000 --- a/sample_queries/mysql/crosswalk_queries/get_crosswalks_for_state.sql +++ /dev/null @@ -1,35 +0,0 @@ --- Get all crosswalks for a specific state jurisdiction --- Returns comprehensive metadata for both state and CCSSM standards --- Ordered by state standard code and Jaccard score - -SELECT - -- State Standard Information - state_std.`jurisdiction` AS state_jurisdiction, - state_std.`statementCode` AS state_standard_code, - state_std.`gradeLevel` AS state_grade_level, - state_std.`description` AS state_description, - state_std.`academicSubject` AS state_academic_subject, - - -- CCSSM Standard Information - ccss_std.`statementCode` AS ccss_standard_code, - ccss_std.`gradeLevel` AS ccss_grade_level, - ccss_std.`description` AS ccss_description, - ccss_std.`academicSubject` AS ccss_academic_subject, - - -- Crosswalk Metrics - r.`jaccard`, - r.`sharedLCCount`, - r.`stateLCCount`, - r.`ccssLCCount` -FROM relationships r -JOIN standards_framework_item state_std - ON state_std.`caseIdentifierUUID` = r.`sourceEntityValue` -JOIN standards_framework_item ccss_std - ON ccss_std.`caseIdentifierUUID` = r.`targetEntityValue` -WHERE r.`relationshipType` = 'hasStandardAlignment' - AND state_std.`jurisdiction` = 'Texas' - AND state_std.`academicSubject` = 'Mathematics' -ORDER BY - state_std.`statementCode`, - ccss_std.`statementCode`, - r.`jaccard` DESC; diff --git a/sample_queries/mysql/crosswalk_queries/get_crosswalks_with_standards_metadata.sql b/sample_queries/mysql/crosswalk_queries/get_crosswalks_with_standards_metadata.sql deleted file mode 100644 index ae987c2..0000000 --- a/sample_queries/mysql/crosswalk_queries/get_crosswalks_with_standards_metadata.sql +++ /dev/null @@ -1,34 +0,0 @@ --- Find Texas state standard matches for a given CCSSM standard with full metadata --- Returns multiple matching Texas state standards ordered by similarity - -SELECT - -- CCSSM Standard Information - ccss_std.`statementCode` AS ccss_standard_code, - ccss_std.`jurisdiction` AS ccss_jurisdiction, - ccss_std.`gradeLevel` AS ccss_grade_level, - ccss_std.`description` AS ccss_description, - ccss_std.`academicSubject` AS ccss_academic_subject, - - -- State Standard Information - state_std.`statementCode` AS state_standard_code, - state_std.`jurisdiction` AS state_jurisdiction, - state_std.`gradeLevel` AS state_grade_level, - state_std.`description` AS state_description, - state_std.`academicSubject` AS state_academic_subject, - - -- Crosswalk Metrics - r.`jaccard`, - r.`sharedLCCount`, - r.`stateLCCount`, - r.`ccssLCCount` -FROM relationships r -JOIN standards_framework_item state_std - ON state_std.`caseIdentifierUUID` = r.`sourceEntityValue` -JOIN standards_framework_item ccss_std - ON ccss_std.`caseIdentifierUUID` = r.`targetEntityValue` -WHERE r.`relationshipType` = 'hasStandardAlignment' - AND ccss_std.`statementCode` = '6.EE.B.5' - AND ccss_std.`jurisdiction` = 'Multi-State' - AND state_std.`jurisdiction` = 'Texas' -ORDER BY r.`jaccard` DESC -LIMIT 10; diff --git a/sample_queries/mysql/crosswalk_queries/get_shared_learning_components_for_crosswalk.sql b/sample_queries/mysql/crosswalk_queries/get_shared_learning_components_for_crosswalk.sql deleted file mode 100644 index 85728c2..0000000 --- a/sample_queries/mysql/crosswalk_queries/get_shared_learning_components_for_crosswalk.sql +++ /dev/null @@ -1,55 +0,0 @@ --- Get the Learning Components that support both a state standard and a CCSSM standard --- Returns three categories: shared LCs (in both), state-only LCs, and CCSSM-only LCs --- This shows the pedagogical overlap and differences between crosswalked standards - -WITH state_lcs AS ( - SELECT lc.`identifier`, lc.`description` - FROM relationships r - JOIN standards_framework_item sfi - ON sfi.`caseIdentifierUUID` = r.`targetEntityValue` - JOIN learning_component lc - ON lc.`identifier` = r.`sourceEntityValue` - WHERE r.`relationshipType` = 'supports' - AND sfi.`statementCode` = '111.26.b.4.D' - AND sfi.`jurisdiction` = 'Texas' -), -ccss_lcs AS ( - SELECT lc.`identifier`, lc.`description` - FROM relationships r - JOIN standards_framework_item sfi - ON sfi.`caseIdentifierUUID` = r.`targetEntityValue` - JOIN learning_component lc - ON lc.`identifier` = r.`sourceEntityValue` - WHERE r.`relationshipType` = 'supports' - AND sfi.`statementCode` = '6.RP.A.2' - AND sfi.`jurisdiction` = 'Multi-State' -) -SELECT - 'shared' AS lc_type, - state_lcs.`identifier`, - state_lcs.`description` -FROM state_lcs -INNER JOIN ccss_lcs - ON state_lcs.`identifier` = ccss_lcs.`identifier` - -UNION ALL - -SELECT - 'state_only' AS lc_type, - state_lcs.`identifier`, - state_lcs.`description` -FROM state_lcs -LEFT JOIN ccss_lcs - ON state_lcs.`identifier` = ccss_lcs.`identifier` -WHERE ccss_lcs.`identifier` IS NULL - -UNION ALL - -SELECT - 'ccss_only' AS lc_type, - ccss_lcs.`identifier`, - ccss_lcs.`description` -FROM ccss_lcs -LEFT JOIN state_lcs - ON ccss_lcs.`identifier` = state_lcs.`identifier` -WHERE state_lcs.`identifier` IS NULL; diff --git a/sample_queries/mysql/learning_component_queries/get_a_learning_component_by_id.sql b/sample_queries/mysql/learning_component_queries/get_a_learning_component_by_id.sql deleted file mode 100644 index 21e5e49..0000000 --- a/sample_queries/mysql/learning_component_queries/get_a_learning_component_by_id.sql +++ /dev/null @@ -1,3 +0,0 @@ -SELECT * -FROM learning_component -WHERE `identifier` = '0046446a-0a9b-5ace-92a3-23d4bb158c68'; \ No newline at end of file diff --git a/sample_queries/mysql/learning_component_queries/list_all_learning_components.sql b/sample_queries/mysql/learning_component_queries/list_all_learning_components.sql deleted file mode 100644 index e7958bf..0000000 --- a/sample_queries/mysql/learning_component_queries/list_all_learning_components.sql +++ /dev/null @@ -1 +0,0 @@ -SELECT * FROM learning_component; \ No newline at end of file diff --git a/sample_queries/mysql/learning_progression_queries/list_all_standards_that_relates_to_a_standard.sql b/sample_queries/mysql/learning_progression_queries/list_all_standards_that_relates_to_a_standard.sql deleted file mode 100644 index aa20c65..0000000 --- a/sample_queries/mysql/learning_progression_queries/list_all_standards_that_relates_to_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT sfi.* -FROM relationships r -JOIN standards_framework_item sfi - ON sfi.`caseIdentifierUUID` = r.`sourceEntityValue` -WHERE r.`relationshipType` = 'relatesTo' - AND r.`sourceEntity` = 'StandardsFrameworkItem' - AND r.`sourceEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntityValue` = '6ba115f2-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/mysql/learning_progression_queries/list_standards_that_build_toward_a_standard.sql b/sample_queries/mysql/learning_progression_queries/list_standards_that_build_toward_a_standard.sql deleted file mode 100644 index 08aedfd..0000000 --- a/sample_queries/mysql/learning_progression_queries/list_standards_that_build_toward_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT sfi.* -FROM relationships r -JOIN standards_framework_item sfi - ON sfi.`caseIdentifierUUID` = r.`sourceEntityValue` -WHERE r.`relationshipType` = 'buildsTowards' - AND r.`sourceEntity` = 'StandardsFrameworkItem' - AND r.`sourceEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntityValue` = '6ba1b8b8-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/mysql/standard_framework_queries/find_standard_framework_by_subject_or_state.sql b/sample_queries/mysql/standard_framework_queries/find_standard_framework_by_subject_or_state.sql deleted file mode 100644 index 0fabfb7..0000000 --- a/sample_queries/mysql/standard_framework_queries/find_standard_framework_by_subject_or_state.sql +++ /dev/null @@ -1,4 +0,0 @@ -SELECT * -FROM standards_framework -WHERE `academicSubject` = 'Mathematics' - AND `jurisdiction` = 'California'; \ No newline at end of file diff --git a/sample_queries/mysql/standard_framework_queries/get_standard_framework_by_id.sql b/sample_queries/mysql/standard_framework_queries/get_standard_framework_by_id.sql deleted file mode 100644 index 8879bd5..0000000 --- a/sample_queries/mysql/standard_framework_queries/get_standard_framework_by_id.sql +++ /dev/null @@ -1,2 +0,0 @@ -SELECT * -FROM standards_framework WHERE `caseIdentifierUUID` = 'c64961be-d7cb-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/mysql/standard_framework_queries/list_all_standard_frameworks.sql b/sample_queries/mysql/standard_framework_queries/list_all_standard_frameworks.sql deleted file mode 100644 index c579f3e..0000000 --- a/sample_queries/mysql/standard_framework_queries/list_all_standard_frameworks.sql +++ /dev/null @@ -1 +0,0 @@ -SELECT * FROM standards_framework; \ No newline at end of file diff --git a/sample_queries/mysql/standard_queries/get_all_standard_items_in_a_standard_framework.sql b/sample_queries/mysql/standard_queries/get_all_standard_items_in_a_standard_framework.sql deleted file mode 100644 index d37e3fd..0000000 --- a/sample_queries/mysql/standard_queries/get_all_standard_items_in_a_standard_framework.sql +++ /dev/null @@ -1,33 +0,0 @@ --- WARNING: This recursive query may hit performance issues or database recursion limits --- on large/deep hierarchies. If execution fails, either increase your database's recursion --- limit settings or use an iterative approach in application code. - -WITH RECURSIVE all_descendants AS ( - -- Base case: direct children of the StandardsFramework - SELECT sfi.`caseIdentifierUUID` - FROM relationships r - JOIN standards_framework_item sfi - ON sfi.`caseIdentifierUUID` = r.`targetEntityValue` - WHERE r.`sourceEntity` = 'StandardsFramework' - AND r.`sourceEntityKey` = 'caseIdentifierUUID' - AND r.`sourceEntityValue` = 'c64961be-d7cb-11e8-824f-0242ac160002' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' - - UNION ALL - - -- Recursive case: children of already found items - SELECT sfi.`caseIdentifierUUID` - FROM relationships r - JOIN standards_framework_item sfi - ON sfi.`caseIdentifierUUID` = r.`targetEntityValue` - JOIN all_descendants ad - ON ad.`caseIdentifierUUID` = r.`sourceEntityValue` - WHERE r.`sourceEntity` = 'StandardsFrameworkItem' - AND r.`sourceEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' - AND r.`relationshipType` = 'hasChild' -) -SELECT `caseIdentifierUUID` -FROM all_descendants; \ No newline at end of file diff --git a/sample_queries/mysql/standard_queries/get_parent_standard_for_a_standard.sql b/sample_queries/mysql/standard_queries/get_parent_standard_for_a_standard.sql deleted file mode 100644 index 3395a17..0000000 --- a/sample_queries/mysql/standard_queries/get_parent_standard_for_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT parent.* -FROM relationships r -JOIN standards_framework_item parent - ON parent.`caseIdentifierUUID` = r.`sourceEntityValue` -WHERE r.`relationshipType` = 'hasChild' - AND r.`sourceEntity` = 'StandardsFrameworkItem' - AND r.`sourceEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID' - AND r.`targetEntityValue` = '57165d0c-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/mysql/standard_queries/get_standard_by_standard_code.sql b/sample_queries/mysql/standard_queries/get_standard_by_standard_code.sql deleted file mode 100644 index 0b0c888..0000000 --- a/sample_queries/mysql/standard_queries/get_standard_by_standard_code.sql +++ /dev/null @@ -1,3 +0,0 @@ -SELECT * -FROM standards_framework_item -WHERE `statementCode` = '7.G.A.1'; \ No newline at end of file diff --git a/sample_queries/mysql/standard_queries/get_standard_by_standard_id.sql b/sample_queries/mysql/standard_queries/get_standard_by_standard_id.sql deleted file mode 100644 index e7be152..0000000 --- a/sample_queries/mysql/standard_queries/get_standard_by_standard_id.sql +++ /dev/null @@ -1,3 +0,0 @@ -SELECT * -FROM standards_framework_item -WHERE `caseIdentifierUUID` = 'c4401baa-b0e5-496c-92c3-352fda95e5ae'; \ No newline at end of file diff --git a/sample_queries/mysql/standard_queries/get_standards_for_a_grade.sql b/sample_queries/mysql/standard_queries/get_standards_for_a_grade.sql deleted file mode 100644 index 4098ac0..0000000 --- a/sample_queries/mysql/standard_queries/get_standards_for_a_grade.sql +++ /dev/null @@ -1,4 +0,0 @@ -SELECT * -FROM standards_framework_item -WHERE `gradeLevel` LIKE '%"6"%' -AND `jurisdiction` = 'Multi-State'; \ No newline at end of file diff --git a/sample_queries/mysql/standard_queries/list_child_standards_for_a_standard.sql b/sample_queries/mysql/standard_queries/list_child_standards_for_a_standard.sql deleted file mode 100644 index 19a22d6..0000000 --- a/sample_queries/mysql/standard_queries/list_child_standards_for_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT child.* -FROM relationships r -JOIN standards_framework_item child - ON child.`caseIdentifierUUID` = r.`targetEntityValue` -WHERE r.`relationshipType` = 'hasChild' - AND r.`sourceEntity` = 'StandardsFrameworkItem' - AND r.`sourceEntityKey` = 'caseIdentifierUUID' - AND r.`sourceEntityValue` = '57165a88-d7cc-11e8-824f-0242ac160002' - AND r.`targetEntity` = 'StandardsFrameworkItem' - AND r.`targetEntityKey` = 'caseIdentifierUUID'; \ No newline at end of file diff --git a/sample_queries/postgresql/.DS_Store b/sample_queries/postgresql/.DS_Store deleted file mode 100644 index 18fd051..0000000 Binary files a/sample_queries/postgresql/.DS_Store and /dev/null differ diff --git a/sample_queries/postgresql/.keep b/sample_queries/postgresql/.keep deleted file mode 100644 index e69de29..0000000 diff --git a/sample_queries/postgresql/combined_queries/list_all_learning_components_in_a_grade.sql b/sample_queries/postgresql/combined_queries/list_all_learning_components_in_a_grade.sql deleted file mode 100644 index 1dcf44f..0000000 --- a/sample_queries/postgresql/combined_queries/list_all_learning_components_in_a_grade.sql +++ /dev/null @@ -1,23 +0,0 @@ -WITH grade6_standards AS ( - SELECT "caseIdentifierUUID" - FROM standards_framework_item - WHERE EXISTS ( - SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem - WHERE elem = '6' - ) -), -supporting_components AS ( - SELECT DISTINCT r."sourceEntityValue" AS lc_id - FROM relationships r - JOIN grade6_standards g - ON g."caseIdentifierUUID" = r."targetEntityValue" - WHERE r."relationshipType" = 'supports' - AND r."sourceEntity" = 'LearningComponent' - AND r."sourceEntityKey" = 'identifier' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' -) -SELECT lc.* -FROM learning_component lc -JOIN supporting_components c - ON lc."identifier" = c.lc_id; \ No newline at end of file diff --git a/sample_queries/postgresql/combined_queries/list_learning_components_supporting_a_standard.sql b/sample_queries/postgresql/combined_queries/list_learning_components_supporting_a_standard.sql deleted file mode 100644 index c987126..0000000 --- a/sample_queries/postgresql/combined_queries/list_learning_components_supporting_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT lc.* -FROM relationships r -JOIN learning_component lc - ON lc."identifier" = r."sourceEntityValue" -WHERE r."relationshipType" = 'supports' - AND r."sourceEntity" = 'LearningComponent' - AND r."sourceEntityKey" = 'identifier' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' - AND r."targetEntityValue" = '6ba1c7ad-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/postgresql/combined_queries/list_standards_supported_by_a_learning_component.sql b/sample_queries/postgresql/combined_queries/list_standards_supported_by_a_learning_component.sql deleted file mode 100644 index 4e59611..0000000 --- a/sample_queries/postgresql/combined_queries/list_standards_supported_by_a_learning_component.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT sfi.* -FROM relationships r -JOIN standards_framework_item sfi - ON sfi."caseIdentifierUUID" = r."targetEntityValue" -WHERE r."relationshipType" = 'supports' - AND r."sourceEntity" = 'LearningComponent' - AND r."sourceEntityKey" = 'identifier' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' - AND r."sourceEntityValue" = '019a93e7-1877-5e9b-b4b3-476f201fccc8'; \ No newline at end of file diff --git a/sample_queries/postgresql/crosswalk_queries/find_best_state_match_for_ccssm_standard.sql b/sample_queries/postgresql/crosswalk_queries/find_best_state_match_for_ccssm_standard.sql deleted file mode 100644 index acaffbc..0000000 --- a/sample_queries/postgresql/crosswalk_queries/find_best_state_match_for_ccssm_standard.sql +++ /dev/null @@ -1,37 +0,0 @@ --- Find the best Texas state standard matches for a given CCSSM standard --- Returns crosswalks ordered by Jaccard score (highest similarity first) --- with metadata about both the CCSSM and matching Texas state standards - -SELECT - -- CCSSM Standard Information - ccss."statementCode" AS ccss_standard_code, - ccss."description" AS ccss_description, - ccss."gradeLevel" AS ccss_grade_level, - ccss."jurisdiction" AS ccss_jurisdiction, - - -- State Standard Information - state."statementCode" AS state_standard_code, - state."description" AS state_description, - state."gradeLevel" AS state_grade_level, - state."jurisdiction" AS state_jurisdiction, - - -- Crosswalk Metrics - r."jaccard", - r."sharedLCCount", - r."stateLCCount", - r."ccssLCCount", - - -- Entity Values for further joins if needed - r."sourceEntityValue" AS state_uuid, - r."targetEntityValue" AS ccss_uuid -FROM relationships r -JOIN standards_framework_item state - ON state."caseIdentifierUUID" = r."sourceEntityValue" -JOIN standards_framework_item ccss - ON ccss."caseIdentifierUUID" = r."targetEntityValue" -WHERE r."relationshipType" = 'hasStandardAlignment' - AND ccss."statementCode" = '6.EE.B.5' - AND ccss."jurisdiction" = 'Multi-State' - AND state."jurisdiction" = 'Texas' -ORDER BY r."jaccard" DESC -LIMIT 10; diff --git a/sample_queries/postgresql/crosswalk_queries/get_all_crosswalks.sql b/sample_queries/postgresql/crosswalk_queries/get_all_crosswalks.sql deleted file mode 100644 index 2836e76..0000000 --- a/sample_queries/postgresql/crosswalk_queries/get_all_crosswalks.sql +++ /dev/null @@ -1,17 +0,0 @@ --- Get all crosswalks with state standard information --- Returns state → CCSSM standard alignments ordered by Jaccard score - -SELECT - state."statementCode" AS state_standard_code, - state."jurisdiction" AS state_jurisdiction, - r."sourceEntityValue", - r."targetEntityValue", - r."jaccard", - r."stateLCCount", - r."ccssLCCount", - r."sharedLCCount" -FROM relationships r -JOIN standards_framework_item state - ON state."caseIdentifierUUID" = r."sourceEntityValue" -WHERE r."relationshipType" = 'hasStandardAlignment' -ORDER BY r."jaccard" DESC; diff --git a/sample_queries/postgresql/crosswalk_queries/get_crosswalks_by_jaccard_threshold.sql b/sample_queries/postgresql/crosswalk_queries/get_crosswalks_by_jaccard_threshold.sql deleted file mode 100644 index dfd761f..0000000 --- a/sample_queries/postgresql/crosswalk_queries/get_crosswalks_by_jaccard_threshold.sql +++ /dev/null @@ -1,22 +0,0 @@ --- Get all Texas crosswalks that meet or exceed a specified Jaccard similarity threshold --- Ordered by CCSSM standard code, then Jaccard score - -SELECT - state_std."jurisdiction" AS state_jurisdiction, - state_std."statementCode" AS state_standard_code, - ccss_std."statementCode" AS ccss_standard_code, - r."jaccard", - r."sharedLCCount", - r."stateLCCount", - r."ccssLCCount" -FROM relationships r -JOIN standards_framework_item state_std - ON state_std."caseIdentifierUUID" = r."sourceEntityValue" -JOIN standards_framework_item ccss_std - ON ccss_std."caseIdentifierUUID" = r."targetEntityValue" -WHERE r."relationshipType" = 'hasStandardAlignment' - AND r."jaccard" >= 0.7 - AND state_std."jurisdiction" = 'Texas' -ORDER BY - ccss_std."statementCode", - r."jaccard" DESC; diff --git a/sample_queries/postgresql/crosswalk_queries/get_crosswalks_for_state.sql b/sample_queries/postgresql/crosswalk_queries/get_crosswalks_for_state.sql deleted file mode 100644 index 849d6f6..0000000 --- a/sample_queries/postgresql/crosswalk_queries/get_crosswalks_for_state.sql +++ /dev/null @@ -1,35 +0,0 @@ --- Get all crosswalks for a specific state jurisdiction --- Returns comprehensive metadata for both state and CCSSM standards --- Ordered by state standard code and Jaccard score - -SELECT - -- State Standard Information - state_std."jurisdiction" AS state_jurisdiction, - state_std."statementCode" AS state_standard_code, - state_std."gradeLevel" AS state_grade_level, - state_std."description" AS state_description, - state_std."academicSubject" AS state_academic_subject, - - -- CCSSM Standard Information - ccss_std."statementCode" AS ccss_standard_code, - ccss_std."gradeLevel" AS ccss_grade_level, - ccss_std."description" AS ccss_description, - ccss_std."academicSubject" AS ccss_academic_subject, - - -- Crosswalk Metrics - r."jaccard", - r."sharedLCCount", - r."stateLCCount", - r."ccssLCCount" -FROM relationships r -JOIN standards_framework_item state_std - ON state_std."caseIdentifierUUID" = r."sourceEntityValue" -JOIN standards_framework_item ccss_std - ON ccss_std."caseIdentifierUUID" = r."targetEntityValue" -WHERE r."relationshipType" = 'hasStandardAlignment' - AND state_std."jurisdiction" = 'Texas' - AND state_std."academicSubject" = 'Mathematics' -ORDER BY - state_std."statementCode", - ccss_std."statementCode", - r."jaccard" DESC; diff --git a/sample_queries/postgresql/crosswalk_queries/get_crosswalks_with_standards_metadata.sql b/sample_queries/postgresql/crosswalk_queries/get_crosswalks_with_standards_metadata.sql deleted file mode 100644 index 34b1781..0000000 --- a/sample_queries/postgresql/crosswalk_queries/get_crosswalks_with_standards_metadata.sql +++ /dev/null @@ -1,34 +0,0 @@ --- Find Texas state standard matches for a given CCSSM standard with full metadata --- Returns multiple matching Texas state standards ordered by similarity - -SELECT - -- CCSSM Standard Information - ccss_std."statementCode" AS ccss_standard_code, - ccss_std."jurisdiction" AS ccss_jurisdiction, - ccss_std."gradeLevel" AS ccss_grade_level, - ccss_std."description" AS ccss_description, - ccss_std."academicSubject" AS ccss_academic_subject, - - -- State Standard Information - state_std."statementCode" AS state_standard_code, - state_std."jurisdiction" AS state_jurisdiction, - state_std."gradeLevel" AS state_grade_level, - state_std."description" AS state_description, - state_std."academicSubject" AS state_academic_subject, - - -- Crosswalk Metrics - r."jaccard", - r."sharedLCCount", - r."stateLCCount", - r."ccssLCCount" -FROM relationships r -JOIN standards_framework_item state_std - ON state_std."caseIdentifierUUID" = r."sourceEntityValue" -JOIN standards_framework_item ccss_std - ON ccss_std."caseIdentifierUUID" = r."targetEntityValue" -WHERE r."relationshipType" = 'hasStandardAlignment' - AND ccss_std."statementCode" = '6.EE.B.5' - AND ccss_std."jurisdiction" = 'Multi-State' - AND state_std."jurisdiction" = 'Texas' -ORDER BY r."jaccard" DESC -LIMIT 10; diff --git a/sample_queries/postgresql/crosswalk_queries/get_shared_learning_components_for_crosswalk.sql b/sample_queries/postgresql/crosswalk_queries/get_shared_learning_components_for_crosswalk.sql deleted file mode 100644 index cd51ead..0000000 --- a/sample_queries/postgresql/crosswalk_queries/get_shared_learning_components_for_crosswalk.sql +++ /dev/null @@ -1,55 +0,0 @@ --- Get the Learning Components that support both a state standard and a CCSSM standard --- Returns three categories: shared LCs (in both), state-only LCs, and CCSSM-only LCs --- This shows the pedagogical overlap and differences between crosswalked standards - -WITH state_lcs AS ( - SELECT lc."identifier", lc."description" - FROM relationships r - JOIN standards_framework_item sfi - ON sfi."caseIdentifierUUID" = r."targetEntityValue" - JOIN learning_component lc - ON lc."identifier" = r."sourceEntityValue" - WHERE r."relationshipType" = 'supports' - AND sfi."statementCode" = '111.26.b.4.D' - AND sfi."jurisdiction" = 'Texas' -), -ccss_lcs AS ( - SELECT lc."identifier", lc."description" - FROM relationships r - JOIN standards_framework_item sfi - ON sfi."caseIdentifierUUID" = r."targetEntityValue" - JOIN learning_component lc - ON lc."identifier" = r."sourceEntityValue" - WHERE r."relationshipType" = 'supports' - AND sfi."statementCode" = '6.RP.A.2' - AND sfi."jurisdiction" = 'Multi-State' -) -SELECT - 'shared' AS lc_type, - state_lcs."identifier", - state_lcs."description" -FROM state_lcs -INNER JOIN ccss_lcs - ON state_lcs."identifier" = ccss_lcs."identifier" - -UNION ALL - -SELECT - 'state_only' AS lc_type, - state_lcs."identifier", - state_lcs."description" -FROM state_lcs -LEFT JOIN ccss_lcs - ON state_lcs."identifier" = ccss_lcs."identifier" -WHERE ccss_lcs."identifier" IS NULL - -UNION ALL - -SELECT - 'ccss_only' AS lc_type, - ccss_lcs."identifier", - ccss_lcs."description" -FROM ccss_lcs -LEFT JOIN state_lcs - ON ccss_lcs."identifier" = state_lcs."identifier" -WHERE state_lcs."identifier" IS NULL; diff --git a/sample_queries/postgresql/learning_component_queries/get_a_learning_component_by_id.sql b/sample_queries/postgresql/learning_component_queries/get_a_learning_component_by_id.sql deleted file mode 100644 index fc93841..0000000 --- a/sample_queries/postgresql/learning_component_queries/get_a_learning_component_by_id.sql +++ /dev/null @@ -1,3 +0,0 @@ -SELECT * -FROM learning_component -WHERE "identifier" = '0046446a-0a9b-5ace-92a3-23d4bb158c68'; \ No newline at end of file diff --git a/sample_queries/postgresql/learning_component_queries/list_all_learning_components.sql b/sample_queries/postgresql/learning_component_queries/list_all_learning_components.sql deleted file mode 100644 index e7958bf..0000000 --- a/sample_queries/postgresql/learning_component_queries/list_all_learning_components.sql +++ /dev/null @@ -1 +0,0 @@ -SELECT * FROM learning_component; \ No newline at end of file diff --git a/sample_queries/postgresql/learning_progression_queries/list_all_standards_that_relates_to_a_standard.sql b/sample_queries/postgresql/learning_progression_queries/list_all_standards_that_relates_to_a_standard.sql deleted file mode 100644 index 3b15040..0000000 --- a/sample_queries/postgresql/learning_progression_queries/list_all_standards_that_relates_to_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT sfi.* -FROM relationships r -JOIN standards_framework_item sfi - ON sfi."caseIdentifierUUID" = r."sourceEntityValue" -WHERE r."relationshipType" = 'relatesTo' - AND r."sourceEntity" = 'StandardsFrameworkItem' - AND r."sourceEntityKey" = 'caseIdentifierUUID' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' - AND r."targetEntityValue" = '6ba115f2-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/postgresql/learning_progression_queries/list_standards_that_build_toward_a_standard.sql b/sample_queries/postgresql/learning_progression_queries/list_standards_that_build_toward_a_standard.sql deleted file mode 100644 index 9051387..0000000 --- a/sample_queries/postgresql/learning_progression_queries/list_standards_that_build_toward_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT sfi.* -FROM relationships r -JOIN standards_framework_item sfi - ON sfi."caseIdentifierUUID" = r."sourceEntityValue" -WHERE r."relationshipType" = 'buildsTowards' - AND r."sourceEntity" = 'StandardsFrameworkItem' - AND r."sourceEntityKey" = 'caseIdentifierUUID' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' - AND r."targetEntityValue" = '6ba1b8b8-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_framework_queries/find_standard_framework_by_subject_or_state.sql b/sample_queries/postgresql/standard_framework_queries/find_standard_framework_by_subject_or_state.sql deleted file mode 100644 index b61b9a4..0000000 --- a/sample_queries/postgresql/standard_framework_queries/find_standard_framework_by_subject_or_state.sql +++ /dev/null @@ -1,4 +0,0 @@ -SELECT * -FROM standards_framework -WHERE "academicSubject" = 'Mathematics' - AND "jurisdiction" = 'California'; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_framework_queries/get_standard_framework_by_id.sql b/sample_queries/postgresql/standard_framework_queries/get_standard_framework_by_id.sql deleted file mode 100644 index 20b2622..0000000 --- a/sample_queries/postgresql/standard_framework_queries/get_standard_framework_by_id.sql +++ /dev/null @@ -1,2 +0,0 @@ -SELECT * -FROM standards_framework WHERE "caseIdentifierUUID" = 'c64961be-d7cb-11e8-824f-0242ac160002' \ No newline at end of file diff --git a/sample_queries/postgresql/standard_framework_queries/list_all_standard_frameworks.sql b/sample_queries/postgresql/standard_framework_queries/list_all_standard_frameworks.sql deleted file mode 100644 index c579f3e..0000000 --- a/sample_queries/postgresql/standard_framework_queries/list_all_standard_frameworks.sql +++ /dev/null @@ -1 +0,0 @@ -SELECT * FROM standards_framework; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_queries/get_all_standard_items_in_a_standard_framework.sql b/sample_queries/postgresql/standard_queries/get_all_standard_items_in_a_standard_framework.sql deleted file mode 100644 index c84095f..0000000 --- a/sample_queries/postgresql/standard_queries/get_all_standard_items_in_a_standard_framework.sql +++ /dev/null @@ -1,33 +0,0 @@ --- WARNING: This recursive query may hit performance issues or database recursion limits --- on large/deep hierarchies. If execution fails, either increase your database's recursion --- limit settings or use an iterative approach in application code. - -WITH RECURSIVE all_descendants AS ( - -- Base case: direct children of the StandardsFramework - SELECT sfi."caseIdentifierUUID" - FROM relationships r - JOIN standards_framework_item sfi - ON sfi."caseIdentifierUUID" = r."targetEntityValue" - WHERE r."sourceEntity" = 'StandardsFramework' - AND r."sourceEntityKey" = 'caseIdentifierUUID' - AND r."sourceEntityValue" = 'c64961be-d7cb-11e8-824f-0242ac160002' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' - - UNION ALL - - -- Recursive case: children of already found items - SELECT sfi."caseIdentifierUUID" - FROM relationships r - JOIN standards_framework_item sfi - ON sfi."caseIdentifierUUID" = r."targetEntityValue" - JOIN all_descendants ad - ON ad."caseIdentifierUUID" = r."sourceEntityValue" - WHERE r."sourceEntity" = 'StandardsFrameworkItem' - AND r."sourceEntityKey" = 'caseIdentifierUUID' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' - AND r."relationshipType" = 'hasChild' -) -SELECT "caseIdentifierUUID" -FROM all_descendants; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_queries/get_parent_standard_for_a_standard.sql b/sample_queries/postgresql/standard_queries/get_parent_standard_for_a_standard.sql deleted file mode 100644 index a57f840..0000000 --- a/sample_queries/postgresql/standard_queries/get_parent_standard_for_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT parent.* -FROM relationships r -JOIN standards_framework_item parent - ON parent."caseIdentifierUUID" = r."sourceEntityValue" -WHERE r."relationshipType" = 'hasChild' - AND r."sourceEntity" = 'StandardsFrameworkItem' - AND r."sourceEntityKey" = 'caseIdentifierUUID' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID' - AND r."targetEntityValue" = '57165d0c-d7cc-11e8-824f-0242ac160002'; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_queries/get_standard_by_standard_code.sql b/sample_queries/postgresql/standard_queries/get_standard_by_standard_code.sql deleted file mode 100644 index ba23ce9..0000000 --- a/sample_queries/postgresql/standard_queries/get_standard_by_standard_code.sql +++ /dev/null @@ -1,3 +0,0 @@ -SELECT * -FROM standards_framework_item -WHERE "statementCode" = '7.G.A.1'; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_queries/get_standard_by_standard_id.sql b/sample_queries/postgresql/standard_queries/get_standard_by_standard_id.sql deleted file mode 100644 index ca1e418..0000000 --- a/sample_queries/postgresql/standard_queries/get_standard_by_standard_id.sql +++ /dev/null @@ -1,3 +0,0 @@ -SELECT * -FROM standards_framework_item -WHERE "caseIdentifierUUID" = 'c4401baa-b0e5-496c-92c3-352fda95e5ae'; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_queries/get_standards_for_a_grade.sql b/sample_queries/postgresql/standard_queries/get_standards_for_a_grade.sql deleted file mode 100644 index 6ce951f..0000000 --- a/sample_queries/postgresql/standard_queries/get_standards_for_a_grade.sql +++ /dev/null @@ -1,7 +0,0 @@ -SELECT * -FROM standards_framework_item -WHERE EXISTS ( - SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem - WHERE elem = '6' -) -AND "jurisdiction" = 'Multi-State'; \ No newline at end of file diff --git a/sample_queries/postgresql/standard_queries/list_child_standards_for_a_standard.sql b/sample_queries/postgresql/standard_queries/list_child_standards_for_a_standard.sql deleted file mode 100644 index d58cffe..0000000 --- a/sample_queries/postgresql/standard_queries/list_child_standards_for_a_standard.sql +++ /dev/null @@ -1,10 +0,0 @@ -SELECT child.* -FROM relationships r -JOIN standards_framework_item child - ON child."caseIdentifierUUID" = r."targetEntityValue" -WHERE r."relationshipType" = 'hasChild' - AND r."sourceEntity" = 'StandardsFrameworkItem' - AND r."sourceEntityKey" = 'caseIdentifierUUID' - AND r."sourceEntityValue" = '57165a88-d7cc-11e8-824f-0242ac160002' - AND r."targetEntity" = 'StandardsFrameworkItem' - AND r."targetEntityKey" = 'caseIdentifierUUID'; \ No newline at end of file diff --git a/tutorials/.gitignore b/tutorials/.gitignore deleted file mode 100644 index 5800e14..0000000 --- a/tutorials/.gitignore +++ /dev/null @@ -1,17 +0,0 @@ -**/.DS_Store -**/.env -**/node_modules - -# Python -**/__pycache__/ -**/*.py[cod] -**/*$py.class -**/.pytest_cache/ -**/.coverage -**/venv/ -**/env/ -**/.venv/ -**/.env/ - -# Data files (embeddings) -**/*_embeddings.json \ No newline at end of file diff --git a/tutorials/compare_standards/js/.env.example b/tutorials/compare_standards/js/.env.example deleted file mode 100644 index f2d3f1f..0000000 --- a/tutorials/compare_standards/js/.env.example +++ /dev/null @@ -1,2 +0,0 @@ -# Knowledge Graph data path - update with your actual path to CSV files -KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files \ No newline at end of file diff --git a/tutorials/compare_standards/js/README.md b/tutorials/compare_standards/js/README.md deleted file mode 100644 index e7a4d11..0000000 --- a/tutorials/compare_standards/js/README.md +++ /dev/null @@ -1,38 +0,0 @@ -# Compare Standards - -Demonstrates how to compare educational standards across different frameworks (Common Core vs Texas) - -Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowledge-graph/v1-1-0/getting-started/tutorials/comparing-standards-across-states) - -## Prerequisites - -- Node.js (v14 or higher) -- Knowledge Graph CSV dataset files: - - `StandardsFrameworkItem.csv` - - `LearningComponent.csv` - - `Relationships.csv` - -## Dependencies - -- **arquero**: Data manipulation and analysis library -- **csv-parse**: CSV file parsing -- **dotenv**: Environment variable management - -## Quick Start - -1. **Clone and Install**: - ```bash - git clone git@github.com:learning-commons-org/knowledge-graph.git - cd tutorials/compare_standards/js - npm install - ``` - -2. **Set Environment Variables** (create `.env` file): - ```bash - KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files - ``` - -3. **Run Tutorial**: - ```bash - npm start - ``` diff --git a/tutorials/compare_standards/js/compare-standards.js b/tutorials/compare_standards/js/compare-standards.js deleted file mode 100644 index e18cd57..0000000 --- a/tutorials/compare_standards/js/compare-standards.js +++ /dev/null @@ -1,463 +0,0 @@ -/* ================================ - CONFIGURATION & SETUP - ================================ */ - -// Dependencies -const fs = require('fs'); -const path = require('path'); -const { parse } = require('csv-parse/sync'); -require('dotenv').config(); - -// Domain Constants -// Pick a CCSSM standard to find its best state standard matches -const TARGET_CCSSM_STANDARD_CODE = '6.EE.B.5'; // Common Core 6th grade math standard on solving equations and inequalities -const TARGET_CCSSM_JURISDICTION = 'Multi-State'; - -// Environment setup -const dataDir = process.env.KG_DATA_PATH; -if (!dataDir) { - console.error('❌ KG_DATA_PATH environment variable is not set.'); - process.exit(1); -} - - -/* ================================ - HELPER FUNCTIONS - ================================ */ - -function loadCSV(filename) { - try { - const content = fs.readFileSync(path.join(dataDir, filename), 'utf8'); - return parse(content, { columns: true, skip_empty_lines: true }); - } catch (error) { - console.error(`❌ Error loading CSV file ${filename}: ${error.message}`); - throw error; - } -} - - -/* ================================ - STEP 1: LOAD THE CROSSWALK DATA - ================================ */ - -function loadCrosswalkData(aq) { - /** - * Load crosswalk data from relationships.csv - * - * Purpose: Crosswalk data lives in the relationships.csv file. Standards that have - * crosswalk data include four crosswalk-specific columns: jaccard, stateLCCount, - * ccssLCCount, and sharedLCCount. - * - * Each row shows one state → CCSSM crosswalk relationship. - */ - - // Load CSV files - const relationshipsData = aq.from(loadCSV('Relationships.csv')); - const standardsFrameworkItemsData = aq.from(loadCSV('StandardsFrameworkItem.csv')); - const learningComponentsData = aq.from(loadCSV('LearningComponent.csv')); - - console.log('✅ Data loaded from KG CSV files'); - console.log(` Total Relationships: ${relationshipsData.numRows()}`); - console.log(` Standards Framework Items: ${standardsFrameworkItemsData.numRows()}`); - console.log(` Learning Components: ${learningComponentsData.numRows()}`); - - // Filter for crosswalk relationships (hasStandardAlignment) - const crosswalkData = relationshipsData - .filter(d => d.relationshipType === 'hasStandardAlignment'); - - console.log(`\n✅ Crosswalk data filtered:`); - console.log(` Total crosswalk relationships (state → CCSSM): ${crosswalkData.numRows()}`); - - // Show preview of crosswalk data - if (crosswalkData.numRows() > 0) { - console.log(`\n📊 Preview of crosswalk data (first 3 rows):`); - const preview = crosswalkData - .select('sourceEntityValue', 'targetEntityValue', 'jaccard', - 'stateLCCount', 'ccssLCCount', 'sharedLCCount') - .slice(0, 3) - .objects(); - - preview.forEach((row, idx) => { - console.log(` ${idx + 1}. Source: ${row.sourceEntityValue} → Target: ${row.targetEntityValue}`); - console.log(` Jaccard: ${row.jaccard}, State LCs: ${row.stateLCCount}, CCSS LCs: ${row.ccssLCCount}, Shared: ${row.sharedLCCount}`); - }); - } - - return { - crosswalkData, - standardsFrameworkItemsData, - learningComponentsData, - relationshipsData - }; -} - - -/* ================================ - STEP 2: FIND THE BEST-MATCHING STATE STANDARDS - ================================ */ - -function findBestStateMatches(ccssmStandardCode, jurisdiction, data, aq) { - /** - * Find the best state standard matches for a CCSSM standard - * - * Purpose: To find the best state standard matches for a CCSSM standard, filter rows by the - * CCSSM standard ID and sort by the Jaccard score. This identifies the state - * standards that contain the most similar skills and concept targets for student - * mastery (not necessarily the most similar semantically). - */ - - const { crosswalkData, standardsFrameworkItemsData } = data; - - // First, find the CCSSM standard by its statement code and jurisdiction - const ccssmStandard = standardsFrameworkItemsData - .params({ code: ccssmStandardCode, juris: jurisdiction }) - .filter(d => d.statementCode === code && d.jurisdiction === juris) - .object(); - - if (!ccssmStandard || !ccssmStandard.statementCode) { - console.log(`❌ CCSSM standard not found: ${ccssmStandardCode}`); - return null; - } - - const ccssmStandardUuid = ccssmStandard.caseIdentifierUUID; // Use 'caseIdentifierUUID' for crosswalk matching - - console.log(`✅ Found CCSSM standard: ${ccssmStandardCode}`); - console.log(` Case UUID: ${ccssmStandardUuid}`); - console.log(` Description: ${ccssmStandard.description}`); - console.log(` Jurisdiction: ${ccssmStandard.jurisdiction}`); - - // Filter crosswalk data for this CCSSM standard (it's the target in relationships) - // and filter for Texas matches only - const matches = crosswalkData - .params({ ccssmId: ccssmStandardUuid }) - .filter(d => d.targetEntityValue === ccssmId); - - if (matches.numRows() === 0) { - console.log(`\n❌ No state standard matches found for ${ccssmStandardCode}`); - return null; - } - - // Join with standards data to get jurisdiction and filter for Texas - const matchesWithJurisdiction = matches - .join( - standardsFrameworkItemsData.select('caseIdentifierUUID', 'jurisdiction'), - ['sourceEntityValue', 'caseIdentifierUUID'] - ); - - // Filter for Texas only - const texasMatches = matchesWithJurisdiction - .params({ state: 'Texas' }) - .filter(d => d.jurisdiction === state); - - if (texasMatches.numRows() === 0) { - console.log(`\n❌ No Texas standard matches found for ${ccssmStandardCode}`); - return null; - } - - // Drop the temporary columns added for filtering to avoid conflicts in later joins - const texasMatchesClean = texasMatches.select(aq.not('caseIdentifierUUID', 'jurisdiction')); - - // Sort by Jaccard score (highest first) - const sortedMatches = texasMatchesClean.orderby(aq.desc('jaccard')); - - console.log(`\n✅ Found ${sortedMatches.numRows()} Texas standard matches for ${ccssmStandardCode}`); - console.log(`\n📊 Top Texas match (highest Jaccard score):`); - - const topMatch = sortedMatches.object(); - console.log(` State Standard UUID: ${topMatch.sourceEntityValue}`); - console.log(` Jaccard Score: ${parseFloat(topMatch.jaccard).toFixed(4)}`); - console.log(` Shared LC Count: ${topMatch.sharedLCCount}`); - console.log(` State LC Count: ${topMatch.stateLCCount}`); - console.log(` CCSS LC Count: ${topMatch.ccssLCCount}`); - - return sortedMatches; -} - - -/* ================================ - STEP 3: INTERPRET THE RELATIONSHIP METRICS - ================================ */ - -function interpretRelationshipMetrics(matches) { - /** - * Interpret the relationship metrics for crosswalk matches - * - * Purpose: Each crosswalk relationship carries additional context about the degree - * of overlap: - * - sharedLCCount shows how many deconstructed skills are shared - * - stateLCCount and ccssLCCount show how many total skills support each standard - * - Together with the Jaccard score, these counts help interpret the strength and - * balance of the overlap - */ - - if (!matches || matches.numRows() === 0) { - return; - } - - console.log(`\n📊 INTERPRETATION OF TOP MATCHES:\n`); - - // Show top 5 matches with interpretation - const topMatches = matches.slice(0, 5).objects(); - - topMatches.forEach((match, idx) => { - const jaccard = parseFloat(match.jaccard); - const stateLc = parseFloat(match.stateLCCount); - const ccssLc = parseFloat(match.ccssLCCount); - const sharedLc = parseFloat(match.sharedLCCount); - - console.log(`Match #${idx + 1}:`); - console.log(` Jaccard Score: ${jaccard.toFixed(4)}`); - console.log(` State LC Count: ${stateLc}`); - console.log(` CCSS LC Count: ${ccssLc}`); - console.log(` Shared LC Count: ${sharedLc}`); - - // Interpret the metrics - let interpretation; - if (jaccard >= 0.9) { - interpretation = "Very strong overlap; standards share nearly all skills"; - } else if (jaccard >= 0.7) { - interpretation = "Strong overlap; substantial shared skills"; - } else if (jaccard >= 0.5) { - interpretation = "Moderate overlap; many shared skills"; - } else if (jaccard >= 0.3) { - interpretation = "Partial overlap; some shared skills"; - } else { - interpretation = "Weak overlap; few shared skills"; - } - - // Check scope balance - let scopeNote; - if (Math.abs(stateLc - ccssLc) <= 2) { - scopeNote = "Both standards have similar scope"; - } else if (stateLc > ccssLc) { - scopeNote = "State standard covers more content"; - } else { - scopeNote = "CCSS standard covers more content"; - } - - console.log(` Interpretation: ${interpretation}`); - console.log(` Scope: ${scopeNote}`); - console.log(); - }); -} - - -/* ================================ - STEP 4: JOIN CROSSWALKS WITH STANDARDS METADATA - ================================ */ - -function enrichCrosswalksWithMetadata(matches, data, aq) { - /** - * Join crosswalk data with standards metadata - * - * Purpose: Enrich the crosswalk data by joining it with StandardsFrameworkItems.csv, - * which contains metadata such as grade level and description. This provides a clear - * view of which state standards most closely align to their CCSSM counterparts, along - * with the strength of each connection. - */ - - if (!matches || matches.numRows() === 0) { - return null; - } - - const { standardsFrameworkItemsData } = data; - - // Rename columns to avoid conflicts when merging CCSS and state metadata - // We'll merge the same standards dataset twice (once for CCSS, once for state) - - // Join with CCSS standard metadata (target) - const enriched = matches - .join( - standardsFrameworkItemsData.select('caseIdentifierUUID', 'statementCode', 'description', - 'gradeLevel', 'academicSubject', 'jurisdiction') - .rename({ - caseIdentifierUUID: 'ccss_uuid', - statementCode: 'statementCode_ccss', - description: 'description_ccss', - gradeLevel: 'gradeLevel_ccss', - academicSubject: 'academicSubject_ccss', - jurisdiction: 'jurisdiction_ccss' - }), - ['targetEntityValue', 'ccss_uuid'] - ) - // Join with state standard metadata (source) - .join( - standardsFrameworkItemsData.select('caseIdentifierUUID', 'statementCode', 'description', - 'gradeLevel', 'academicSubject', 'jurisdiction') - .rename({ - caseIdentifierUUID: 'state_uuid', - statementCode: 'statementCode_state', - description: 'description_state', - gradeLevel: 'gradeLevel_state', - academicSubject: 'academicSubject_state' - }), - ['sourceEntityValue', 'state_uuid'] - ); - - console.log(`\n✅ Enriched crosswalk data with standards metadata\n`); - console.log(`📊 DETAILED COMPARISON (Top 3 matches):\n`); - - const top3 = enriched.slice(0, 3).objects(); - - top3.forEach((row, idx) => { - console.log(`Match #${idx + 1} (Jaccard: ${parseFloat(row.jaccard).toFixed(4)}):`); - console.log(` CCSS STANDARD:`); - console.log(` Code: ${row.statementCode_ccss}`); - console.log(` Jurisdiction: ${row.jurisdiction_ccss}`); - console.log(` Grade Level: ${row.gradeLevel_ccss}`); - console.log(` Description: ${row.description_ccss}`); - console.log(` `); - console.log(` STATE STANDARD:`); - console.log(` Code: ${row.statementCode_state}`); - console.log(` Jurisdiction: ${row.jurisdiction}`); - console.log(` Grade Level: ${row.gradeLevel_state}`); - console.log(` Description: ${row.description_state}`); - console.log(` `); - console.log(` ALIGNMENT METRICS:`); - console.log(` Shared LCs: ${row.sharedLCCount} / State LCs: ${row.stateLCCount} / CCSS LCs: ${row.ccssLCCount}`); - console.log(); - }); - - return enriched; -} - - -/* ================================ - STEP 5: JOIN CROSSWALKS TO LEARNING COMPONENTS - ================================ */ - -function showSharedLearningComponents(stateStandardCode, ccssStandardCode, stateJurisdiction, data, aq) { - /** - * Join crosswalks to Learning Components to show shared skills - * - * Purpose: Now that you have crosswalk pairs (state → CCSSM), you can see the - * actual skills behind each match by joining to the Learning Components dataset. - * We'll use the 'supports' relationships to fetch the LCs that support each standard - * and then intersect them to list the shared LCs (the evidence behind the crosswalk). - */ - - const { standardsFrameworkItemsData, relationshipsData, learningComponentsData } = data; - - // Find the standard identifiers - // Note: For LC relationships, we need to use caseIdentifierUUID, not identifier - const stateStandard = standardsFrameworkItemsData - .params({ code: stateStandardCode, juris: stateJurisdiction }) - .filter(d => d.statementCode === code && d.jurisdiction === juris) - .object(); - - const ccssStandard = standardsFrameworkItemsData - .params({ code: ccssStandardCode }) - .filter(d => d.statementCode === code && d.jurisdiction === 'Multi-State') - .object(); - - if (!stateStandard || !ccssStandard) { - console.log('❌ Could not find one or both standards'); - return; - } - - const stateUuid = stateStandard.caseIdentifierUUID; - const ccssUuid = ccssStandard.caseIdentifierUUID; - - // Get LCs that support the state standard - // LC relationships use caseIdentifierUUID for targetEntityValue - const stateLcIds = relationshipsData - .params({ uuid: stateUuid }) - .filter(d => d.relationshipType === 'supports' && d.targetEntityValue === uuid) - .array('sourceEntityValue'); - - const stateLcs = learningComponentsData - .filter(aq.escape(d => stateLcIds.includes(d.identifier))) - .select('identifier', 'description') - .dedupe('identifier'); - - // Get LCs that support the CCSS standard - const ccssLcIds = relationshipsData - .params({ uuid: ccssUuid }) - .filter(d => d.relationshipType === 'supports' && d.targetEntityValue === uuid) - .array('sourceEntityValue'); - - const ccssLcs = learningComponentsData - .filter(aq.escape(d => ccssLcIds.includes(d.identifier))) - .select('identifier', 'description') - .dedupe('identifier'); - - // Find shared LCs (intersection) using join - const sharedLcs = stateLcs - .semijoin(ccssLcs, 'identifier'); - - // Find state-only LCs (in state but not in CCSS) - const stateOnlyLcs = stateLcs - .antijoin(ccssLcs, 'identifier'); - - // Find CCSS-only LCs (in CCSS but not in state) - const ccssOnlyLcs = ccssLcs - .antijoin(stateLcs, 'identifier'); - - console.log(`\n✅ LEARNING COMPONENTS ANALYSIS:\n`); - console.log(`State Standard: ${stateStandardCode}`); - console.log(`CCSS Standard: ${ccssStandardCode}`); - console.log(); - - console.log(`📊 SHARED LEARNING COMPONENTS (${sharedLcs.numRows()}):`); - console.log('These are the concrete pedagogical overlaps between the two standards:\n'); - sharedLcs.objects().forEach((lc, idx) => { - console.log(` ✅ ${idx + 1}. ${lc.description}`); - }); - console.log(); - - console.log(`📊 STATE-ONLY LEARNING COMPONENTS (${stateOnlyLcs.numRows()}):`); - stateOnlyLcs.objects().forEach((lc, idx) => { - console.log(` ➖ ${idx + 1}. ${lc.description}`); - }); - console.log(); - - console.log(`📊 CCSS-ONLY LEARNING COMPONENTS (${ccssOnlyLcs.numRows()}):`); - ccssOnlyLcs.objects().forEach((lc, idx) => { - console.log(` ➕ ${idx + 1}. ${lc.description}`); - }); - console.log(); -} - - -/* ================================ - MAIN EXECUTION - ================================ */ - -async function main() { - const aq = await import('arquero'); - - console.log('\n=== USING CROSSWALKS TO COMPARE STATE STANDARDS TO COMMON CORE ===\n'); - - console.log('🔄 Step 1: Load the crosswalk data...'); - const data = loadCrosswalkData(aq); - - console.log('\n' + '='.repeat(70)); - console.log('🔄 Step 2: Find the best-matching state standards for a CCSSM standard...'); - const matches = findBestStateMatches(TARGET_CCSSM_STANDARD_CODE, TARGET_CCSSM_JURISDICTION, data, aq); - - if (matches && matches.numRows() > 0) { - console.log('\n' + '='.repeat(70)); - console.log('🔄 Step 3: Interpret the relationship metrics...'); - interpretRelationshipMetrics(matches); - - console.log('='.repeat(70)); - console.log('🔄 Step 4: Join crosswalks with standards metadata...'); - const enriched = enrichCrosswalksWithMetadata(matches, data, aq); - - if (enriched && enriched.numRows() > 0) { - console.log('='.repeat(70)); - console.log('🔄 Step 5: Join crosswalks to Learning Components...'); - // Use the top match for detailed LC analysis (already filtered for Texas) - const topMatch = enriched.object(); - showSharedLearningComponents( - topMatch.statementCode_state, - topMatch.statementCode_ccss, - topMatch.jurisdiction, - data, - aq - ); - } - } -} - -main().catch(console.error); diff --git a/tutorials/compare_standards/js/package-lock.json b/tutorials/compare_standards/js/package-lock.json deleted file mode 100644 index 2ae242c..0000000 --- a/tutorials/compare_standards/js/package-lock.json +++ /dev/null @@ -1,64 +0,0 @@ -{ - "name": "compare-standards", - "version": "1.0.0", - "lockfileVersion": 3, - "requires": true, - "packages": { - "": { - "name": "compare-standards", - "version": "1.0.0", - "license": "ISC", - "dependencies": { - "arquero": "^8.0.3", - "csv-parse": "^6.1.0", - "dotenv": "^16.3.1" - } - }, - "node_modules/@uwdata/flechette": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/@uwdata/flechette/-/flechette-2.1.0.tgz", - "integrity": "sha512-CsVgFIc94Rb5UhKvAK6E2rCFV+H6WkDzBEjYZvl2lfcI9vqLRafdngd9o5cq2lsA2offxCmJXdS1DE7ACPn33w==", - "license": "BSD-3-Clause" - }, - "node_modules/acorn": { - "version": "8.15.0", - "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.15.0.tgz", - "integrity": "sha512-NZyJarBfL7nWwIq+FDL6Zp/yHEhePMNnnJ0y3qfieCrmNvYct8uvtiV41UvlSe6apAfk0fY1FbWx+NwfmpvtTg==", - "license": "MIT", - "bin": { - "acorn": "bin/acorn" - }, - "engines": { - "node": ">=0.4.0" - } - }, - "node_modules/arquero": { - "version": "8.0.3", - "resolved": "https://registry.npmjs.org/arquero/-/arquero-8.0.3.tgz", - "integrity": "sha512-7YQwe/GPVBUiahaPwEwgvu6VHyuhX0Ut61JZlIJYsAobOH5unLBwTmh43BobhX/N4dW1sb8WyKlQ8GjFq2whzQ==", - "license": "BSD-3-Clause", - "dependencies": { - "@uwdata/flechette": "^2.0.0", - "acorn": "^8.14.1" - } - }, - "node_modules/csv-parse": { - "version": "6.1.0", - "resolved": "https://registry.npmjs.org/csv-parse/-/csv-parse-6.1.0.tgz", - "integrity": "sha512-CEE+jwpgLn+MmtCpVcPtiCZpVtB6Z2OKPTr34pycYYoL7sxdOkXDdQ4lRiw6ioC0q6BLqhc6cKweCVvral8yhw==", - "license": "MIT" - }, - "node_modules/dotenv": { - "version": "16.6.1", - "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.6.1.tgz", - "integrity": "sha512-uBq4egWHTcTt33a72vpSG0z3HnPuIl6NqYcTrKEg2azoEyl2hpW0zqlxysq2pK9HlDIHyHyakeYaYnSAwd8bow==", - "license": "BSD-2-Clause", - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://dotenvx.com" - } - } - } -} diff --git a/tutorials/compare_standards/js/package.json b/tutorials/compare_standards/js/package.json deleted file mode 100644 index 64d6b6d..0000000 --- a/tutorials/compare_standards/js/package.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "name": "compare-standards", - "version": "1.0.0", - "description": "Compare educational standards across different states and jurisdictions using Knowledge Graph datasets", - "main": "compare-standards.js", - "scripts": { - "start": "node compare-standards.js", - "test": "echo \"Error: no test specified\" && exit 1" - }, - "dependencies": { - "arquero": "^8.0.3", - "csv-parse": "^6.1.0", - "dotenv": "^16.3.1" - }, - "keywords": [ - "education", - "standards", - "knowledge-graph", - "comparison", - "analysis" - ], - "author": "", - "license": "ISC" -} \ No newline at end of file diff --git a/tutorials/compare_standards/python/.env.example b/tutorials/compare_standards/python/.env.example deleted file mode 100644 index f2d3f1f..0000000 --- a/tutorials/compare_standards/python/.env.example +++ /dev/null @@ -1,2 +0,0 @@ -# Knowledge Graph data path - update with your actual path to CSV files -KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files \ No newline at end of file diff --git a/tutorials/compare_standards/python/README.md b/tutorials/compare_standards/python/README.md deleted file mode 100644 index 99e5dba..0000000 --- a/tutorials/compare_standards/python/README.md +++ /dev/null @@ -1,39 +0,0 @@ -# Compare Standards - -Demonstrates how to use crosswalk data to compare state standards to Common Core State Standards (CCSSM) using the Knowledge Graph dataset. - -Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowledge-graph/v1-2-0/getting-started/tutorials/comparing-standards-across-states) - -## Prerequisites - -- Python 3.8 or higher -- Knowledge Graph CSV dataset files: - - `StandardsFrameworkItem.csv` - - `LearningComponent.csv` - - `Relationships.csv` - -## Dependencies - -- **pandas**: Data manipulation and analysis library -- **python-dotenv**: Environment variable management - -## Quick Start - -1. **Clone and Set Up Virtual Environment**: - ```bash - git clone git@github.com:learning-commons-org/knowledge-graph.git - cd tutorials/compare_standards/python - python -m venv venv - source venv/bin/activate - pip install -r requirements.txt - ``` - -2. **Set Environment Variables** (create `.env` file): - ```bash - KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files - ``` - -3. **Run Tutorial**: - ```bash - python compare_standards.py - ``` diff --git a/tutorials/compare_standards/python/compare_standards.py b/tutorials/compare_standards/python/compare_standards.py deleted file mode 100644 index 33a6020..0000000 --- a/tutorials/compare_standards/python/compare_standards.py +++ /dev/null @@ -1,502 +0,0 @@ -#!/usr/bin/env python3 -""" -Using crosswalks to compare state standards to Common Core - -This tutorial demonstrates how to use the crosswalk data in Knowledge Graph to compare -standards between a state framework and the Common Core State Standards (CCSSM). These -crosswalks help determine which CCSSM standard is most similar to a given state standard -and understand the similarities and differences between them. - -Crosswalks are evidence-based relationships between state standards and CCSSM standards, -derived from overlapping sets of Learning Components (LCs). Each crosswalk includes -similarity metrics, such as the Jaccard score and relative LC counts, to help interpret -how closely two standards align. -""" - -""" -================================ -CONFIGURATION & SETUP -================================ -""" - -# Dependencies -import os -import sys -import pandas as pd -from pathlib import Path -from dotenv import load_dotenv - -# Load environment variables -load_dotenv() - -# Domain Constants -# Pick a CCSSM standard to find its best state standard matches -TARGET_CCSSM_STANDARD_CODE = '6.EE.B.5' # Common Core 6th grade math standard on solving equations and inequalities -TARGET_CCSSM_JURISDICTION = 'Multi-State' - -# Environment Setup -data_dir = os.getenv('KG_DATA_PATH') -if not data_dir: - print('❌ KG_DATA_PATH environment variable is not set.') - sys.exit(1) - -data_path = Path(data_dir) - - -""" -================================ -HELPER FUNCTIONS -================================ -""" - -def load_csv(filename): - """ - Load and parse CSV file from data directory - - Args: - filename (str): Name of the CSV file to load - - Returns: - pd.DataFrame: Loaded CSV data as DataFrame - """ - try: - file_path = data_path / filename - return pd.read_csv(file_path, low_memory=False) - except Exception as error: - print(f'❌ Error loading CSV file {filename}: {str(error)}') - raise error - - -""" -================================ -STEP 1: LOAD THE CROSSWALK DATA -================================ -""" - -def load_crosswalk_data(): - """ - Load crosswalk data from relationships.csv - - Purpose: Crosswalk data lives in the relationships.csv file. Standards that have - crosswalk data include four crosswalk-specific columns: jaccard, stateLCCount, - ccssLCCount, and sharedLCCount. - - Each row shows one state → CCSSM crosswalk relationship. - - Returns: - dict: Dictionary containing crosswalk data and related datasets - """ - # Load CSV files - relationships_data = load_csv('Relationships.csv') - standards_framework_items_data = load_csv('StandardsFrameworkItem.csv') - learning_components_data = load_csv('LearningComponent.csv') - - print('✅ Data loaded from KG CSV files') - print(f' Total Relationships: {len(relationships_data)}') - print(f' Standards Framework Items: {len(standards_framework_items_data)}') - print(f' Learning Components: {len(learning_components_data)}') - - # Filter for crosswalk relationships (hasStandardAlignment) - crosswalk_data = relationships_data[ - relationships_data['relationshipType'] == 'hasStandardAlignment' - ].copy() - - print(f'\n✅ Crosswalk data filtered:') - print(f' Total crosswalk relationships (state → CCSSM): {len(crosswalk_data)}') - - # Show preview of crosswalk data - if len(crosswalk_data) > 0: - print(f'\n📊 Preview of crosswalk data (first 3 rows):') - preview_cols = ['sourceEntityValue', 'targetEntityValue', 'jaccard', - 'stateLCCount', 'ccssLCCount', 'sharedLCCount'] - available_cols = [col for col in preview_cols if col in crosswalk_data.columns] - print(crosswalk_data[available_cols].head(3).to_string(index=False)) - - return { - 'crosswalk_data': crosswalk_data, - 'standards_framework_items_data': standards_framework_items_data, - 'learning_components_data': learning_components_data, - 'relationships_data': relationships_data - } - - -""" -================================ -STEP 2: FIND THE BEST-MATCHING STATE STANDARDS -================================ -""" - -def find_best_state_matches(ccssm_standard_code, jurisdiction, data): - """ - Find the best state standard matches for a CCSSM standard - - Purpose: To find the best state standard matches for a CCSSM standard, filter rows by the - CCSSM standard ID and sort by the Jaccard score. This identifies the state - standards that contain the most similar skills and concept targets for student - mastery (not necessarily the most similar semantically). - - Args: - ccssm_standard_code (str): The statement code of the CCSSM standard - jurisdiction (str): The jurisdiction of the CCSSM standard (typically 'Multi-State') - data (dict): Dictionary containing the loaded datasets - - Returns: - pd.DataFrame: Crosswalk matches sorted by Jaccard score (highest first) - """ - crosswalk_data = data['crosswalk_data'] - standards_data = data['standards_framework_items_data'] - - # First, find the CCSSM standard by its statement code and jurisdiction - ccssm_standard = standards_data[ - (standards_data['statementCode'] == ccssm_standard_code) & - (standards_data['jurisdiction'] == jurisdiction) - ] - - if len(ccssm_standard) == 0: - print(f'❌ CCSSM standard not found: {ccssm_standard_code}') - return None - - ccssm_standard = ccssm_standard.iloc[0] - ccssm_standard_uuid = ccssm_standard['caseIdentifierUUID'] # Use 'caseIdentifierUUID' for crosswalk matching - - print(f'✅ Found CCSSM standard: {ccssm_standard_code}') - print(f' Case UUID: {ccssm_standard_uuid}') - print(f' Description: {ccssm_standard["description"]}') - print(f' Jurisdiction: {ccssm_standard["jurisdiction"]}') - - # Filter crosswalk data for this CCSSM standard (it's the target in relationships) - # and filter for Texas matches only - matches = crosswalk_data[ - crosswalk_data['targetEntityValue'] == ccssm_standard_uuid - ].copy() - - if len(matches) == 0: - print(f'\n❌ No state standard matches found for {ccssm_standard_code}') - return None - - # Join with standards data to get jurisdiction and filter for Texas - matches = matches.merge( - standards_data[['caseIdentifierUUID', 'jurisdiction']], - left_on='sourceEntityValue', - right_on='caseIdentifierUUID', - how='left', - suffixes=('', '_temp') - ) - - # Filter for Texas only - texas_matches = matches[matches['jurisdiction'] == 'Texas'].copy() - - if len(texas_matches) == 0: - print(f'\n❌ No Texas standard matches found for {ccssm_standard_code}') - return None - - # Drop the temporary columns added for filtering - texas_matches = texas_matches.drop(columns=['caseIdentifierUUID', 'jurisdiction']) - - # Sort by Jaccard score (highest first) - texas_matches = texas_matches.sort_values('jaccard', ascending=False) - - print(f'\n✅ Found {len(texas_matches)} Texas standard matches for {ccssm_standard_code}') - print(f'\n📊 Top Texas match (highest Jaccard score):') - - top_match = texas_matches.iloc[0] - print(f' State Standard UUID: {top_match["sourceEntityValue"]}') - print(f' Jaccard Score: {top_match["jaccard"]:.4f}') - print(f' Shared LC Count: {top_match["sharedLCCount"]}') - print(f' State LC Count: {top_match["stateLCCount"]}') - print(f' CCSS LC Count: {top_match["ccssLCCount"]}') - - return texas_matches - - -""" -================================ -STEP 3: INTERPRET THE RELATIONSHIP METRICS -================================ -""" - -def interpret_relationship_metrics(matches): - """ - Interpret the relationship metrics for crosswalk matches - - Purpose: Each crosswalk relationship carries additional context about the degree - of overlap: - - sharedLCCount shows how many deconstructed skills are shared - - stateLCCount and ccssLCCount show how many total skills support each standard - - Together with the Jaccard score, these counts help interpret the strength and - balance of the overlap - - Args: - matches (pd.DataFrame): Crosswalk matches from Step 2 - """ - if matches is None or len(matches) == 0: - return - - print(f'\n📊 INTERPRETATION OF TOP MATCHES:\n') - - # Show top 5 matches with interpretation - for idx, (_, match) in enumerate(matches.head(5).iterrows(), 1): - jaccard = match['jaccard'] - state_lc = match['stateLCCount'] - ccss_lc = match['ccssLCCount'] - shared_lc = match['sharedLCCount'] - - print(f'Match #{idx}:') - print(f' Jaccard Score: {jaccard:.4f}') - print(f' State LC Count: {state_lc}') - print(f' CCSS LC Count: {ccss_lc}') - print(f' Shared LC Count: {shared_lc}') - - # Interpret the metrics - if jaccard >= 0.9: - interpretation = "Very strong overlap; standards share nearly all skills" - elif jaccard >= 0.7: - interpretation = "Strong overlap; substantial shared skills" - elif jaccard >= 0.5: - interpretation = "Moderate overlap; many shared skills" - elif jaccard >= 0.3: - interpretation = "Partial overlap; some shared skills" - else: - interpretation = "Weak overlap; few shared skills" - - # Check scope balance - if abs(state_lc - ccss_lc) <= 2: - scope_note = "Both standards have similar scope" - elif state_lc > ccss_lc: - scope_note = "State standard covers more content" - else: - scope_note = "CCSS standard covers more content" - - print(f' Interpretation: {interpretation}') - print(f' Scope: {scope_note}') - print() - - -""" -================================ -STEP 4: JOIN CROSSWALKS WITH STANDARDS METADATA -================================ -""" - -def enrich_crosswalks_with_metadata(matches, data): - """ - Join crosswalk data with standards metadata - - Purpose: Enrich the crosswalk data by joining it with StandardsFrameworkItems.csv, - which contains metadata such as grade level and description. This provides a clear - view of which state standards most closely align to their CCSSM counterparts, along - with the strength of each connection. - - Args: - matches (pd.DataFrame): Crosswalk matches from Step 2 - data (dict): Dictionary containing the loaded datasets - - Returns: - pd.DataFrame: Enriched crosswalk data with metadata - """ - if matches is None or len(matches) == 0: - return None - - standards_data = data['standards_framework_items_data'] - - # Rename columns to avoid conflicts when merging CCSS and state metadata - # We'll merge the same standards dataset twice (once for CCSS, once for state) - - # Join with CCSS standard metadata (target) - ccss_standards = standards_data[['caseIdentifierUUID', 'statementCode', 'description', - 'gradeLevel', 'academicSubject', 'jurisdiction']].copy() - ccss_standards.columns = ['ccss_uuid', 'statementCode_ccss', 'description_ccss', - 'gradeLevel_ccss', 'academicSubject_ccss', 'jurisdiction_ccss'] - - enriched = matches.merge( - ccss_standards, - left_on='targetEntityValue', - right_on='ccss_uuid', - how='left' - ) - - # Join with state standard metadata (source) - state_standards = standards_data[['caseIdentifierUUID', 'statementCode', 'description', - 'gradeLevel', 'academicSubject', 'jurisdiction']].copy() - state_standards.columns = ['state_uuid', 'statementCode_state', 'description_state', - 'gradeLevel_state', 'academicSubject_state', 'jurisdiction'] - - enriched = enriched.merge( - state_standards, - left_on='sourceEntityValue', - right_on='state_uuid', - how='left' - ) - - print(f'\n✅ Enriched crosswalk data with standards metadata\n') - print(f'📊 DETAILED COMPARISON (Top 3 matches):\n') - - for idx, (_, row) in enumerate(enriched.head(3).iterrows(), 1): - print(f'Match #{idx} (Jaccard: {row["jaccard"]:.4f}):') - print(f' CCSS STANDARD:') - print(f' Code: {row["statementCode_ccss"]}') - print(f' Jurisdiction: {row["jurisdiction_ccss"]}') - print(f' Grade Level: {row["gradeLevel_ccss"]}') - print(f' Description: {row["description_ccss"]}') - print(f' ') - print(f' STATE STANDARD:') - print(f' Code: {row["statementCode_state"]}') - print(f' Jurisdiction: {row["jurisdiction"]}') - print(f' Grade Level: {row["gradeLevel_state"]}') - print(f' Description: {row["description_state"]}') - print(f' ') - print(f' ALIGNMENT METRICS:') - print(f' Shared LCs: {row["sharedLCCount"]} / State LCs: {row["stateLCCount"]} / CCSS LCs: {row["ccssLCCount"]}') - print() - - return enriched - - -""" -================================ -STEP 5: JOIN CROSSWALKS TO LEARNING COMPONENTS -================================ -""" - -def show_shared_learning_components(state_standard_code, ccss_standard_code, state_jurisdiction, data): - """ - Join crosswalks to Learning Components to show shared skills - - Purpose: Now that you have crosswalk pairs (state → CCSSM), you can see the - actual skills behind each match by joining to the Learning Components dataset. - We'll use the 'supports' relationships to fetch the LCs that support each standard - and then intersect them to list the shared LCs (the evidence behind the crosswalk). - - Args: - state_standard_code (str): State standard code - ccss_standard_code (str): CCSS standard code - state_jurisdiction (str): State jurisdiction (to ensure correct standard match) - data (dict): Dictionary containing the loaded datasets - """ - standards_data = data['standards_framework_items_data'] - relationships_data = data['relationships_data'] - learning_components_data = data['learning_components_data'] - - # Find the standard identifiers - # Note: For LC relationships, we need to use caseIdentifierUUID, not identifier - state_standard = standards_data[ - (standards_data['statementCode'] == state_standard_code) & - (standards_data['jurisdiction'] == state_jurisdiction) - ] - ccss_standard = standards_data[ - (standards_data['statementCode'] == ccss_standard_code) & - (standards_data['jurisdiction'] == 'Multi-State') - ] - - if len(state_standard) == 0 or len(ccss_standard) == 0: - print('❌ Could not find one or both standards') - return - - state_uuid = state_standard.iloc[0]['caseIdentifierUUID'] - ccss_uuid = ccss_standard.iloc[0]['caseIdentifierUUID'] - - # Get LCs that support the state standard - # LC relationships use caseIdentifierUUID for targetEntityValue - state_lc_relationships = relationships_data[ - (relationships_data['relationshipType'] == 'supports') & - (relationships_data['targetEntityValue'] == state_uuid) - ] - - state_lc_ids = state_lc_relationships['sourceEntityValue'].unique() - state_lcs = learning_components_data[ - learning_components_data['identifier'].isin(state_lc_ids) - ][['identifier', 'description']].drop_duplicates() - - # Get LCs that support the CCSS standard - ccss_lc_relationships = relationships_data[ - (relationships_data['relationshipType'] == 'supports') & - (relationships_data['targetEntityValue'] == ccss_uuid) - ] - - ccss_lc_ids = ccss_lc_relationships['sourceEntityValue'].unique() - ccss_lcs = learning_components_data[ - learning_components_data['identifier'].isin(ccss_lc_ids) - ][['identifier', 'description']].drop_duplicates() - - # Find shared LCs (intersection) using merge - shared_lcs = state_lcs.merge( - ccss_lcs[['identifier']], - on='identifier', - how='inner' - ) - - # Find state-only LCs (in state but not in CCSS) - state_only_lcs = state_lcs[ - ~state_lcs['identifier'].isin(ccss_lcs['identifier']) - ] - - # Find CCSS-only LCs (in CCSS but not in state) - ccss_only_lcs = ccss_lcs[ - ~ccss_lcs['identifier'].isin(state_lcs['identifier']) - ] - - print(f'\n✅ LEARNING COMPONENTS ANALYSIS:\n') - print(f'CCSS Standard: {ccss_standard_code}') - print(f'State Standard: {state_standard_code}') - print() - - print(f'📊 SHARED LEARNING COMPONENTS ({len(shared_lcs)}):') - print('These are the concrete pedagogical overlaps between the two standards:\n') - for idx, (_, lc) in enumerate(shared_lcs.iterrows(), 1): - print(f' ✅ {idx}. {lc["description"]}') - print() - - print(f'📊 STATE-ONLY LEARNING COMPONENTS ({len(state_only_lcs)}):') - for idx, (_, lc) in enumerate(state_only_lcs.iterrows(), 1): - print(f' ➖ {idx}. {lc["description"]}') - print() - - print(f'📊 CCSS-ONLY LEARNING COMPONENTS ({len(ccss_only_lcs)}):') - for idx, (_, lc) in enumerate(ccss_only_lcs.iterrows(), 1): - print(f' ➕ {idx}. {lc["description"]}') - print() - - -""" -================================ -MAIN EXECUTION -================================ -""" - -def main(): - """ - Main execution function - orchestrates all tutorial steps - """ - print('\n=== USING CROSSWALKS TO COMPARE STATE STANDARDS TO COMMON CORE ===\n') - - print('🔄 Step 1: Load the crosswalk data...') - data = load_crosswalk_data() - - print('\n' + '='*70) - print('🔄 Step 2: Find the best-matching state standards for a CCSSM standard...') - matches = find_best_state_matches(TARGET_CCSSM_STANDARD_CODE, TARGET_CCSSM_JURISDICTION, data) - - if matches is not None and len(matches) > 0: - print('\n' + '='*70) - print('🔄 Step 3: Interpret the relationship metrics...') - interpret_relationship_metrics(matches) - - print('='*70) - print('🔄 Step 4: Join crosswalks with standards metadata...') - enriched = enrich_crosswalks_with_metadata(matches, data) - - if enriched is not None and len(enriched) > 0: - print('='*70) - print('🔄 Step 5: Join crosswalks to Learning Components...') - # Use the top match for detailed LC analysis (already filtered for Texas) - top_match = enriched.iloc[0] - show_shared_learning_components( - top_match['statementCode_state'], - top_match['statementCode_ccss'], - top_match['jurisdiction'], - data - ) - - -if __name__ == '__main__': - main() diff --git a/tutorials/compare_standards/python/requirements.txt b/tutorials/compare_standards/python/requirements.txt deleted file mode 100644 index 36953cb..0000000 --- a/tutorials/compare_standards/python/requirements.txt +++ /dev/null @@ -1,9 +0,0 @@ -# Core data manipulation and analysis -pandas>=2.0.0 - -# Environment variable management -python-dotenv>=1.0.0 - -# Optional: Enhanced data processing (uncomment if needed) -# numpy>=1.24.0 -# scipy>=1.10.0 \ No newline at end of file diff --git a/tutorials/generate_prereq_practice/js/.env.example b/tutorials/generate_prereq_practice/js/.env.example deleted file mode 100644 index 0483037..0000000 --- a/tutorials/generate_prereq_practice/js/.env.example +++ /dev/null @@ -1,5 +0,0 @@ -# Knowledge Graph data path - update with your actual path to CSV files -KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files - -# OpenAI API key for generating practice questions -OPENAI_API_KEY=your_openai_api_key_here \ No newline at end of file diff --git a/tutorials/generate_prereq_practice/js/README.md b/tutorials/generate_prereq_practice/js/README.md deleted file mode 100644 index 40bd1ef..0000000 --- a/tutorials/generate_prereq_practice/js/README.md +++ /dev/null @@ -1,46 +0,0 @@ -# Generate Prerequisite Practice - -Demonstrates how to generate prerequisite-based practice questions using Knowledge Graph data covering: -- **Data Loading**: Reading CSV files with educational frameworks and relationships -- **Prerequisite Analysis**: Finding standards that build towards a target standard -- **Relationship Mapping**: Connecting learning components to prerequisite standards -- **Practice Generation**: Creating structured practice questions based on prerequisite knowledge - -**Features**: Common Core Math analysis, prerequisite relationship traversal, learning component mapping - -Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowledge-graph/v1-1-0/getting-started/tutorials/generating-prerequisite-practice-questions) - -## Prerequisites - -- Node.js (v14 or higher) -- Knowledge Graph CSV dataset files: - - `StandardsFrameworkItem.csv` - - `LearningComponent.csv` - - `Relationships.csv` - -## Dependencies - -- **openai**: OpenAI API for generating practice questions -- **csv-parse**: CSV file parsing -- **dotenv**: Environment variable management -- **arquero**: Data manipulation - -## Quick Start - -1. **Clone and Install**: - ```bash - git clone git@github.com:learning-commons-org/knowledge-graph.git - cd tutorials/generate_prereq_practice/js - npm install - ``` - -2. **Set Environment Variables** (create `.env` file): - ```bash - KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files - OPENAI_API_KEY=your_openai_api_key_here - ``` - -3. **Run Tutorial**: - ```bash - npm start - ``` diff --git a/tutorials/generate_prereq_practice/js/generate-prereq-practice.js b/tutorials/generate_prereq_practice/js/generate-prereq-practice.js deleted file mode 100644 index 45f5a2f..0000000 --- a/tutorials/generate_prereq_practice/js/generate-prereq-practice.js +++ /dev/null @@ -1,332 +0,0 @@ -/* ================================ - CONFIGURATION & SETUP - ================================ */ - -// Dependencies -const fs = require('fs'); -const path = require('path'); -const { parse } = require('csv-parse/sync'); -const OpenAI = require('openai'); -require('dotenv').config(); - -// Constants -const GENERATE_PRACTICE = true; -// Filter criteria for mathematics standards -const JURISDICTION = 'Multi-State'; -const ACADEMIC_SUBJECT = 'Mathematics'; -const TARGET_CODE = '6.NS.B.4'; -// OpenAI configuration -const OPENAI_MODEL = 'gpt-4'; -const OPENAI_TEMPERATURE = 0.7; - -// Environment setup -const dataDir = process.env.KG_DATA_PATH; -if (!dataDir) { - console.error('❌ KG_DATA_PATH environment variable is not set.'); - process.exit(1); -} - -const openai = new OpenAI({ - apiKey: process.env.OPENAI_API_KEY, -}); - -/* ================================ - HELPER FUNCTIONS - ================================ */ - -function loadCSV(filename) { - try { - const content = fs.readFileSync(path.join(dataDir, filename), 'utf8'); - return parse(content, { columns: true, skip_empty_lines: true }); - } catch (error) { - console.error(`❌ Error loading CSV file ${filename}: ${error.message}`); - throw error; - } -} - - -/* ================================ - STEP 1: LOAD DATA - ================================ */ -function loadData(aq) { - /* Load CSV data files and build filtered dataset - */ - - const standardsFrameworkItems = aq.from(loadCSV('StandardsFrameworkItem.csv')); - const learningComponents = aq.from(loadCSV('LearningComponent.csv')); - const relationships = aq.from(loadCSV('Relationships.csv')); - - console.log('✅ Files loaded from KG CSV files'); - console.log({ - standardsFrameworkItems: standardsFrameworkItems.numRows(), - learningComponents: learningComponents.numRows(), - relationships: relationships.numRows() - }); - - // Filter for relevant StandardsFrameworkItems by jurisdiction and subject - const relevantStandards = standardsFrameworkItems - .params({ jurisdiction: JURISDICTION, academicSubject: ACADEMIC_SUBJECT }) - .filter(d => d.jurisdiction === jurisdiction && d.academicSubject === academicSubject); - - // Get array of relevant identifiers for filtering - const relevantStandardIds = relevantStandards.array('caseIdentifierUUID'); - const relevantStandardSet = new Set(relevantStandardIds); - - // Filter relationships for buildsTowards and supports relationships - const relevantRelationships = relationships - .filter(aq.escape(d => - (d.relationshipType === 'buildsTowards' && - relevantStandardSet.has(d.sourceEntityValue) && - relevantStandardSet.has(d.targetEntityValue)) || - (d.relationshipType === 'supports' && - relevantStandardSet.has(d.targetEntityValue)) - )); - - // Get learning component IDs from supports relationships - const supportRelationships = relevantRelationships - .filter(d => d.relationshipType === 'supports'); - const linkedLearningComponentIds = supportRelationships.array('sourceEntityValue'); - const linkedLearningComponentSet = new Set(linkedLearningComponentIds); - - // Filter learning components by identifier - const relevantLearningComponents = learningComponents - .filter(aq.escape(d => linkedLearningComponentSet.has(d.identifier))); - - console.log('✅ Retrieved scoped graph:'); - console.log({ - standardsFrameworkItems: relevantStandards.numRows(), - learningComponents: relevantLearningComponents.numRows(), - relationships: relevantRelationships.numRows() - }); - - return { - relevantStandards, - relevantRelationships, - relevantLearningComponents - }; -} - -/* ================================ - STEP 2: QUERY PREREQUISITE DATA - ================================ */ -function getStandardAndPrerequisites(relevantStandards, relevantRelationships) { - const targetStandardTable = relevantStandards - .params({ targetCode: TARGET_CODE }) - .filter(d => d.statementCode === targetCode); - - if (targetStandardTable.numRows() === 0) { - console.error(`❌ No StandardsFrameworkItem found for statementCode = "${TARGET_CODE}"`); - return null; - } - - const targetStandard = targetStandardTable.object(); - console.log(`✅ Found StandardsFrameworkItem for ${TARGET_CODE}:`) - console.log({ - caseIdentifierUUID: targetStandard.caseIdentifierUUID, - statementCode: targetStandard.statementCode, - description: targetStandard.description - }); - - const prerequisiteLinks = relevantRelationships - .params({ targetIdentifier: targetStandard.caseIdentifierUUID }) - .filter(d => d.relationshipType === 'buildsTowards' && - d.targetEntityValue === targetIdentifier); - - const prerequisiteStandards = prerequisiteLinks - .join(relevantStandards, ['sourceEntityValue', 'caseIdentifierUUID']) - .select('sourceEntityValue', 'statementCode', 'description_2') - .rename({ sourceEntityValue: 'caseIdentifierUUID', description_2: 'standardDescription' }); - - console.log(`✅ Found ${prerequisiteStandards.numRows()} prerequisite(s) for ${targetStandard.statementCode}:`); - console.log(prerequisiteStandards.objects()); - - return { targetStandard, prerequisiteStandards }; -} - -function getLearningComponentsForPrerequisites(prerequisiteStandards, relevantRelationships, relevantLearningComponents) { - const prerequisiteLearningComponents = prerequisiteStandards - .join(relevantRelationships, ['caseIdentifierUUID', 'targetEntityValue']) - .params({ supportsType: 'supports' }) - .filter(d => d.relationshipType === supportsType) - .join(relevantLearningComponents, ['sourceEntityValue', 'identifier']) - .select('caseIdentifierUUID', 'statementCode', 'standardDescription', 'description_2') - .rename({ description_2: 'learningComponentDescription' }); - - console.log(`✅ Found ${prerequisiteLearningComponents.numRows()} supporting learning components for prerequisites:`); - console.log(prerequisiteLearningComponents.objects()); - - return prerequisiteLearningComponents; -} - -function queryPrerequisiteData(aq, relevantStandards, relevantRelationships, relevantLearningComponents) { - /* Analyze prerequisite relationships for the target standard - * This step identifies prerequisites and supporting learning components - * - * SQL: WITH target AS ( - * SELECT "caseIdentifierUUID" - * FROM standards_framework_item - * WHERE "statementCode" = '6.NS.B.4' - * ), - * prerequisite_standards AS ( - * SELECT - * sfi."caseIdentifierUUID", - * sfi."statementCode", - * sfi."description" - * FROM standards_framework_item sfi - * JOIN relationships r - * ON sfi."caseIdentifierUUID" = r."sourceEntityValue" - * JOIN target t - * ON r."targetEntityValue" = t."caseIdentifierUUID" - * WHERE r."relationshipType" = 'buildsTowards' - * ) - * SELECT - * ps."caseIdentifierUUID", - * ps."statementCode", - * ps."description", - * lc."description" - * FROM prerequisite_standards ps - * JOIN relationships r - * ON ps."caseIdentifierUUID" = r."targetEntityValue" - * JOIN learning_component lc - * ON r."sourceEntityValue" = lc."identifier" - * WHERE r."relationshipType" = 'supports'; - * - * Cypher: MATCH (target:StandardsFrameworkItem {statementCode: '6.NS.B.4'}) - * MATCH (prereq:StandardsFrameworkItem)-[:buildsTowards]->(target) - * MATCH (lc:LearningComponent)-[:supports]->(prereq) - * RETURN prereq.caseIdentifierUUID, prereq.statementCode, prereq.description, - * lc.description - */ - - const standardAndPrereqData = getStandardAndPrerequisites(relevantStandards, relevantRelationships); - if (!standardAndPrereqData) { - return null; - } - - const { targetStandard, prerequisiteStandards } = standardAndPrereqData; - const prerequisiteLearningComponents = getLearningComponentsForPrerequisites(prerequisiteStandards, relevantRelationships, relevantLearningComponents); - - return { targetStandard, prerequisiteLearningComponents }; -} - -/* ================================ - STEP 3: GENERATE PRACTICE - ================================ */ -function packageContextData(targetStandard, prerequisiteLearningComponents) { - /* Package the standards and learning components data for text generation - * This creates a structured context that can be used for generating practice questions - */ - - // Convert dataframe to context format for LLM - const allRows = prerequisiteLearningComponents.objects(); - const standardsMap = new Map(); - - // Group learning components by standard for context - for (const row of allRows) { - if (!standardsMap.has(row.caseIdentifierUUID)) { - standardsMap.set(row.caseIdentifierUUID, { - statementCode: row.statementCode, - description: row.standardDescription || '(no statement)', - supportingLearningComponents: [] - }); - } - - standardsMap.get(row.caseIdentifierUUID).supportingLearningComponents.push({ - description: row.learningComponentDescription || '(no description)' - }); - } - - const fullStandardsContext = { - targetStandard: { - statementCode: targetStandard.statementCode, - description: targetStandard.description || '(no statement)' - }, - prereqStandards: Array.from(standardsMap.values()) - }; - - console.log(`✅ Packaged full standards context for text generation`); - console.log(JSON.stringify(fullStandardsContext, null, 2)); - return fullStandardsContext; -} - -function generatePracticeData(fullStandardsContext) { - /* Generate practice questions using OpenAI API - * This creates educational content based on prerequisite data - */ - return async function generatePractice() { - console.log(`🔄 Generating practice questions for ${fullStandardsContext.targetStandard.statementCode}...`); - - try { - // Build prompt inline - let prerequisiteText = ''; - for (const prereq of fullStandardsContext.prereqStandards) { - prerequisiteText += `- ${prereq.statementCode}: ${prereq.description}\n`; - prerequisiteText += ' Supporting Learning Components:\n'; - for (const lc of prereq.supportingLearningComponents) { - prerequisiteText += ` • ${lc.description}\n`; - } - } - - const prompt = `You are a math tutor helping middle school students. Based on the following information, generate 3 practice questions for the target standard. Questions should help reinforce the key concept and build on prerequisite knowledge. - -Target Standard: -- ${fullStandardsContext.targetStandard.statementCode}: ${fullStandardsContext.targetStandard.description} - -Prerequisite Standards & Supporting Learning Components: -${prerequisiteText}`; - - const response = await openai.chat.completions.create({ - model: OPENAI_MODEL, - messages: [ - { role: 'system', content: 'You are an expert middle school math tutor.' }, - { role: 'user', content: prompt } - ], - temperature: OPENAI_TEMPERATURE - }); - - const practiceQuestions = response.choices[0].message.content.trim(); - - console.log(`✅ Generated practice questions:\n`); - console.log(practiceQuestions); - - return { - aiGenerated: practiceQuestions, - targetStandard: fullStandardsContext.targetStandard.statementCode, - prerequisiteCount: fullStandardsContext.prereqStandards.length - }; - } catch (err) { - console.error('❌ Error generating practice questions:', err.message); - throw err; - } - }; -} - - -async function main() { - console.log('\n=== GENERATE PREREQUISITE PRACTICE TUTORIAL ===\n'); - - console.log('🔄 Step 1: Loading data...'); - const aq = await import('arquero'); - const { relevantStandards, relevantRelationships, relevantLearningComponents } = loadData(aq); - - console.log('\n🔄 Step 2: Querying prerequisite data...'); - const prerequisiteData = queryPrerequisiteData(aq, relevantStandards, relevantRelationships, relevantLearningComponents); - - if (!prerequisiteData) { - console.error('❌ Failed to find prerequisite data'); - return; - } - - const { targetStandard, prerequisiteLearningComponents } = prerequisiteData; - - console.log('\n🔄 Step 3: Generating practice...'); - const fullStandardsContext = packageContextData(targetStandard, prerequisiteLearningComponents); - const generatePractice = generatePracticeData(fullStandardsContext); - if (GENERATE_PRACTICE) { - await generatePractice(); - } else { - console.log('🚫 Practice question generation disabled'); - } -} - -main().catch(console.error); \ No newline at end of file diff --git a/tutorials/generate_prereq_practice/js/package-lock.json b/tutorials/generate_prereq_practice/js/package-lock.json deleted file mode 100644 index 21bfb88..0000000 --- a/tutorials/generate_prereq_practice/js/package-lock.json +++ /dev/null @@ -1,515 +0,0 @@ -{ - "name": "generate-prereq-practice", - "version": "1.0.0", - "lockfileVersion": 3, - "requires": true, - "packages": { - "": { - "name": "generate-prereq-practice", - "version": "1.0.0", - "license": "ISC", - "dependencies": { - "arquero": "^8.0.3", - "csv-parse": "^6.1.0", - "dotenv": "^16.3.1", - "openai": "^4.0.0" - } - }, - "node_modules/@types/node": { - "version": "24.3.0", - "resolved": "https://registry.npmjs.org/@types/node/-/node-24.3.0.tgz", - "integrity": "sha512-aPTXCrfwnDLj4VvXrm+UUCQjNEvJgNA8s5F1cvwQU+3KNltTOkBm1j30uNLyqqPNe7gE3KFzImYoZEfLhp4Yow==", - "license": "MIT", - "dependencies": { - "undici-types": "~7.10.0" - } - }, - "node_modules/@types/node-fetch": { - "version": "2.6.13", - "resolved": "https://registry.npmjs.org/@types/node-fetch/-/node-fetch-2.6.13.tgz", - "integrity": "sha512-QGpRVpzSaUs30JBSGPjOg4Uveu384erbHBoT1zeONvyCfwQxIkUshLAOqN/k9EjGviPRmWTTe6aH2qySWKTVSw==", - "license": "MIT", - "dependencies": { - "@types/node": "*", - "form-data": "^4.0.4" - } - }, - "node_modules/@uwdata/flechette": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/@uwdata/flechette/-/flechette-2.1.0.tgz", - "integrity": "sha512-CsVgFIc94Rb5UhKvAK6E2rCFV+H6WkDzBEjYZvl2lfcI9vqLRafdngd9o5cq2lsA2offxCmJXdS1DE7ACPn33w==", - "license": "BSD-3-Clause" - }, - "node_modules/abort-controller": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/abort-controller/-/abort-controller-3.0.0.tgz", - "integrity": "sha512-h8lQ8tacZYnR3vNQTgibj+tODHI5/+l06Au2Pcriv/Gmet0eaj4TwWH41sO9wnHDiQsEj19q0drzdWdeAHtweg==", - "license": "MIT", - "dependencies": { - "event-target-shim": "^5.0.0" - }, - "engines": { - "node": ">=6.5" - } - }, - "node_modules/acorn": { - "version": "8.15.0", - "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.15.0.tgz", - "integrity": "sha512-NZyJarBfL7nWwIq+FDL6Zp/yHEhePMNnnJ0y3qfieCrmNvYct8uvtiV41UvlSe6apAfk0fY1FbWx+NwfmpvtTg==", - "license": "MIT", - "bin": { - "acorn": "bin/acorn" - }, - "engines": { - "node": ">=0.4.0" - } - }, - "node_modules/agentkeepalive": { - "version": "4.6.0", - "resolved": "https://registry.npmjs.org/agentkeepalive/-/agentkeepalive-4.6.0.tgz", - "integrity": "sha512-kja8j7PjmncONqaTsB8fQ+wE2mSU2DJ9D4XKoJ5PFWIdRMa6SLSN1ff4mOr4jCbfRSsxR4keIiySJU0N9T5hIQ==", - "license": "MIT", - "dependencies": { - "humanize-ms": "^1.2.1" - }, - "engines": { - "node": ">= 8.0.0" - } - }, - "node_modules/arquero": { - "version": "8.0.3", - "resolved": "https://registry.npmjs.org/arquero/-/arquero-8.0.3.tgz", - "integrity": "sha512-7YQwe/GPVBUiahaPwEwgvu6VHyuhX0Ut61JZlIJYsAobOH5unLBwTmh43BobhX/N4dW1sb8WyKlQ8GjFq2whzQ==", - "license": "BSD-3-Clause", - "dependencies": { - "@uwdata/flechette": "^2.0.0", - "acorn": "^8.14.1" - } - }, - "node_modules/asynckit": { - "version": "0.4.0", - "resolved": "https://registry.npmjs.org/asynckit/-/asynckit-0.4.0.tgz", - "integrity": "sha512-Oei9OH4tRh0YqU3GxhX79dM/mwVgvbZJaSNaRk+bshkj0S5cfHcgYakreBjrHwatXKbz+IoIdYLxrKim2MjW0Q==", - "license": "MIT" - }, - "node_modules/call-bind-apply-helpers": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/call-bind-apply-helpers/-/call-bind-apply-helpers-1.0.2.tgz", - "integrity": "sha512-Sp1ablJ0ivDkSzjcaJdxEunN5/XvksFJ2sMBFfq6x0ryhQV/2b/KwFe21cMpmHtPOSij8K99/wSfoEuTObmuMQ==", - "license": "MIT", - "dependencies": { - "es-errors": "^1.3.0", - "function-bind": "^1.1.2" - }, - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/combined-stream": { - "version": "1.0.8", - "resolved": "https://registry.npmjs.org/combined-stream/-/combined-stream-1.0.8.tgz", - "integrity": "sha512-FQN4MRfuJeHf7cBbBMJFXhKSDq+2kAArBlmRBvcvFE5BB1HZKXtSFASDhdlz9zOYwxh8lDdnvmMOe/+5cdoEdg==", - "license": "MIT", - "dependencies": { - "delayed-stream": "~1.0.0" - }, - "engines": { - "node": ">= 0.8" - } - }, - "node_modules/csv-parse": { - "version": "6.1.0", - "resolved": "https://registry.npmjs.org/csv-parse/-/csv-parse-6.1.0.tgz", - "integrity": "sha512-CEE+jwpgLn+MmtCpVcPtiCZpVtB6Z2OKPTr34pycYYoL7sxdOkXDdQ4lRiw6ioC0q6BLqhc6cKweCVvral8yhw==", - "license": "MIT" - }, - "node_modules/delayed-stream": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/delayed-stream/-/delayed-stream-1.0.0.tgz", - "integrity": "sha512-ZySD7Nf91aLB0RxL4KGrKHBXl7Eds1DAmEdcoVawXnLD7SDhpNgtuII2aAkg7a7QS41jxPSZ17p4VdGnMHk3MQ==", - "license": "MIT", - "engines": { - "node": ">=0.4.0" - } - }, - "node_modules/dotenv": { - "version": "16.6.1", - "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.6.1.tgz", - "integrity": "sha512-uBq4egWHTcTt33a72vpSG0z3HnPuIl6NqYcTrKEg2azoEyl2hpW0zqlxysq2pK9HlDIHyHyakeYaYnSAwd8bow==", - "license": "BSD-2-Clause", - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://dotenvx.com" - } - }, - "node_modules/dunder-proto": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz", - "integrity": "sha512-KIN/nDJBQRcXw0MLVhZE9iQHmG68qAVIBg9CqmUYjmQIhgij9U5MFvrqkUL5FbtyyzZuOeOt0zdeRe4UY7ct+A==", - "license": "MIT", - "dependencies": { - "call-bind-apply-helpers": "^1.0.1", - "es-errors": "^1.3.0", - "gopd": "^1.2.0" - }, - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/es-define-property": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/es-define-property/-/es-define-property-1.0.1.tgz", - "integrity": "sha512-e3nRfgfUZ4rNGL232gUgX06QNyyez04KdjFrF+LTRoOXmrOgFKDg4BCdsjW8EnT69eqdYGmRpJwiPVYNrCaW3g==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/es-errors": { - "version": "1.3.0", - "resolved": "https://registry.npmjs.org/es-errors/-/es-errors-1.3.0.tgz", - "integrity": "sha512-Zf5H2Kxt2xjTvbJvP2ZWLEICxA6j+hAmMzIlypy4xcBg1vKVnx89Wy0GbS+kf5cwCVFFzdCFh2XSCFNULS6csw==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/es-object-atoms": { - "version": "1.1.1", - "resolved": "https://registry.npmjs.org/es-object-atoms/-/es-object-atoms-1.1.1.tgz", - "integrity": "sha512-FGgH2h8zKNim9ljj7dankFPcICIK9Cp5bm+c2gQSYePhpaG5+esrLODihIorn+Pe6FGJzWhXQotPv73jTaldXA==", - "license": "MIT", - "dependencies": { - "es-errors": "^1.3.0" - }, - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/es-set-tostringtag": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/es-set-tostringtag/-/es-set-tostringtag-2.1.0.tgz", - "integrity": "sha512-j6vWzfrGVfyXxge+O0x5sh6cvxAog0a/4Rdd2K36zCMV5eJ+/+tOAngRO8cODMNWbVRdVlmGZQL2YS3yR8bIUA==", - "license": "MIT", - "dependencies": { - "es-errors": "^1.3.0", - "get-intrinsic": "^1.2.6", - "has-tostringtag": "^1.0.2", - "hasown": "^2.0.2" - }, - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/event-target-shim": { - "version": "5.0.1", - "resolved": "https://registry.npmjs.org/event-target-shim/-/event-target-shim-5.0.1.tgz", - "integrity": "sha512-i/2XbnSz/uxRCU6+NdVJgKWDTM427+MqYbkQzD321DuCQJUqOuJKIA0IM2+W2xtYHdKOmZ4dR6fExsd4SXL+WQ==", - "license": "MIT", - "engines": { - "node": ">=6" - } - }, - "node_modules/form-data": { - "version": "4.0.4", - "resolved": "https://registry.npmjs.org/form-data/-/form-data-4.0.4.tgz", - "integrity": "sha512-KrGhL9Q4zjj0kiUt5OO4Mr/A/jlI2jDYs5eHBpYHPcBEVSiipAvn2Ko2HnPe20rmcuuvMHNdZFp+4IlGTMF0Ow==", - "license": "MIT", - "dependencies": { - "asynckit": "^0.4.0", - "combined-stream": "^1.0.8", - "es-set-tostringtag": "^2.1.0", - "hasown": "^2.0.2", - "mime-types": "^2.1.12" - }, - "engines": { - "node": ">= 6" - } - }, - "node_modules/form-data-encoder": { - "version": "1.7.2", - "resolved": "https://registry.npmjs.org/form-data-encoder/-/form-data-encoder-1.7.2.tgz", - "integrity": "sha512-qfqtYan3rxrnCk1VYaA4H+Ms9xdpPqvLZa6xmMgFvhO32x7/3J/ExcTd6qpxM0vH2GdMI+poehyBZvqfMTto8A==", - "license": "MIT" - }, - "node_modules/formdata-node": { - "version": "4.4.1", - "resolved": "https://registry.npmjs.org/formdata-node/-/formdata-node-4.4.1.tgz", - "integrity": "sha512-0iirZp3uVDjVGt9p49aTaqjk84TrglENEDuqfdlZQ1roC9CWlPk6Avf8EEnZNcAqPonwkG35x4n3ww/1THYAeQ==", - "license": "MIT", - "dependencies": { - "node-domexception": "1.0.0", - "web-streams-polyfill": "4.0.0-beta.3" - }, - "engines": { - "node": ">= 12.20" - } - }, - "node_modules/function-bind": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/function-bind/-/function-bind-1.1.2.tgz", - "integrity": "sha512-7XHNxH7qX9xG5mIwxkhumTox/MIRNcOgDrxWsMt2pAr23WHp6MrRlN7FBSFpCpr+oVO0F744iUgR82nJMfG2SA==", - "license": "MIT", - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/get-intrinsic": { - "version": "1.3.0", - "resolved": "https://registry.npmjs.org/get-intrinsic/-/get-intrinsic-1.3.0.tgz", - "integrity": "sha512-9fSjSaos/fRIVIp+xSJlE6lfwhES7LNtKaCBIamHsjr2na1BiABJPo0mOjjz8GJDURarmCPGqaiVg5mfjb98CQ==", - "license": "MIT", - "dependencies": { - "call-bind-apply-helpers": "^1.0.2", - "es-define-property": "^1.0.1", - "es-errors": "^1.3.0", - "es-object-atoms": "^1.1.1", - "function-bind": "^1.1.2", - "get-proto": "^1.0.1", - "gopd": "^1.2.0", - "has-symbols": "^1.1.0", - "hasown": "^2.0.2", - "math-intrinsics": "^1.1.0" - }, - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/get-proto": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/get-proto/-/get-proto-1.0.1.tgz", - "integrity": "sha512-sTSfBjoXBp89JvIKIefqw7U2CCebsc74kiY6awiGogKtoSGbgjYE/G/+l9sF3MWFPNc9IcoOC4ODfKHfxFmp0g==", - "license": "MIT", - "dependencies": { - "dunder-proto": "^1.0.1", - "es-object-atoms": "^1.0.0" - }, - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/gopd": { - "version": "1.2.0", - "resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz", - "integrity": "sha512-ZUKRh6/kUFoAiTAtTYPZJ3hw9wNxx+BIBOijnlG9PnrJsCcSjs1wyyD6vJpaYtgnzDrKYRSqf3OO6Rfa93xsRg==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/has-symbols": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/has-symbols/-/has-symbols-1.1.0.tgz", - "integrity": "sha512-1cDNdwJ2Jaohmb3sg4OmKaMBwuC48sYni5HUw2DvsC8LjGTLK9h+eb1X6RyuOHe4hT0ULCW68iomhjUoKUqlPQ==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/has-tostringtag": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/has-tostringtag/-/has-tostringtag-1.0.2.tgz", - "integrity": "sha512-NqADB8VjPFLM2V0VvHUewwwsw0ZWBaIdgo+ieHtK3hasLz4qeCRjYcqfB6AQrBggRKppKF8L52/VqdVsO47Dlw==", - "license": "MIT", - "dependencies": { - "has-symbols": "^1.0.3" - }, - "engines": { - "node": ">= 0.4" - }, - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/hasown": { - "version": "2.0.2", - "resolved": "https://registry.npmjs.org/hasown/-/hasown-2.0.2.tgz", - "integrity": "sha512-0hJU9SCPvmMzIBdZFqNPXWa6dqh7WdH0cII9y+CyS8rG3nL48Bclra9HmKhVVUHyPWNH5Y7xDwAB7bfgSjkUMQ==", - "license": "MIT", - "dependencies": { - "function-bind": "^1.1.2" - }, - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/humanize-ms": { - "version": "1.2.1", - "resolved": "https://registry.npmjs.org/humanize-ms/-/humanize-ms-1.2.1.tgz", - "integrity": "sha512-Fl70vYtsAFb/C06PTS9dZBo7ihau+Tu/DNCk/OyHhea07S+aeMWpFFkUaXRa8fI+ScZbEI8dfSxwY7gxZ9SAVQ==", - "license": "MIT", - "dependencies": { - "ms": "^2.0.0" - } - }, - "node_modules/math-intrinsics": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/math-intrinsics/-/math-intrinsics-1.1.0.tgz", - "integrity": "sha512-/IXtbwEk5HTPyEwyKX6hGkYXxM9nbj64B+ilVJnC/R6B0pH5G4V3b0pVbL7DBj4tkhBAppbQUlf6F6Xl9LHu1g==", - "license": "MIT", - "engines": { - "node": ">= 0.4" - } - }, - "node_modules/mime-db": { - "version": "1.52.0", - "resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.52.0.tgz", - "integrity": "sha512-sPU4uV7dYlvtWJxwwxHD0PuihVNiE7TyAbQ5SWxDCB9mUYvOgroQOwYQQOKPJ8CIbE+1ETVlOoK1UC2nU3gYvg==", - "license": "MIT", - "engines": { - "node": ">= 0.6" - } - }, - "node_modules/mime-types": { - "version": "2.1.35", - "resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.35.tgz", - "integrity": "sha512-ZDY+bPm5zTTF+YpCrAU9nK0UgICYPT0QtT1NZWFv4s++TNkcgVaT0g6+4R2uI4MjQjzysHB1zxuWL50hzaeXiw==", - "license": "MIT", - "dependencies": { - "mime-db": "1.52.0" - }, - "engines": { - "node": ">= 0.6" - } - }, - "node_modules/ms": { - "version": "2.1.3", - "resolved": "https://registry.npmjs.org/ms/-/ms-2.1.3.tgz", - "integrity": "sha512-6FlzubTLZG3J2a/NVCAleEhjzq5oxgHyaCU9yYXvcLsvoVaHJq/s5xXI6/XXP6tz7R9xAOtHnSO/tXtF3WRTlA==", - "license": "MIT" - }, - "node_modules/node-domexception": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/node-domexception/-/node-domexception-1.0.0.tgz", - "integrity": "sha512-/jKZoMpw0F8GRwl4/eLROPA3cfcXtLApP0QzLmUT/HuPCZWyB7IY9ZrMeKw2O/nFIqPQB3PVM9aYm0F312AXDQ==", - "deprecated": "Use your platform's native DOMException instead", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/jimmywarting" - }, - { - "type": "github", - "url": "https://paypal.me/jimmywarting" - } - ], - "license": "MIT", - "engines": { - "node": ">=10.5.0" - } - }, - "node_modules/node-fetch": { - "version": "2.7.0", - "resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.7.0.tgz", - "integrity": "sha512-c4FRfUm/dbcWZ7U+1Wq0AwCyFL+3nt2bEw05wfxSz+DWpWsitgmSgYmy2dQdWyKC1694ELPqMs/YzUSNozLt8A==", - "license": "MIT", - "dependencies": { - "whatwg-url": "^5.0.0" - }, - "engines": { - "node": "4.x || >=6.0.0" - }, - "peerDependencies": { - "encoding": "^0.1.0" - }, - "peerDependenciesMeta": { - "encoding": { - "optional": true - } - } - }, - "node_modules/openai": { - "version": "4.104.0", - "resolved": "https://registry.npmjs.org/openai/-/openai-4.104.0.tgz", - "integrity": "sha512-p99EFNsA/yX6UhVO93f5kJsDRLAg+CTA2RBqdHK4RtK8u5IJw32Hyb2dTGKbnnFmnuoBv5r7Z2CURI9sGZpSuA==", - "license": "Apache-2.0", - "dependencies": { - "@types/node": "^18.11.18", - "@types/node-fetch": "^2.6.4", - "abort-controller": "^3.0.0", - "agentkeepalive": "^4.2.1", - "form-data-encoder": "1.7.2", - "formdata-node": "^4.3.2", - "node-fetch": "^2.6.7" - }, - "bin": { - "openai": "bin/cli" - }, - "peerDependencies": { - "ws": "^8.18.0", - "zod": "^3.23.8" - }, - "peerDependenciesMeta": { - "ws": { - "optional": true - }, - "zod": { - "optional": true - } - } - }, - "node_modules/openai/node_modules/@types/node": { - "version": "18.19.123", - "resolved": "https://registry.npmjs.org/@types/node/-/node-18.19.123.tgz", - "integrity": "sha512-K7DIaHnh0mzVxreCR9qwgNxp3MH9dltPNIEddW9MYUlcKAzm+3grKNSTe2vCJHI1FaLpvpL5JGJrz1UZDKYvDg==", - "license": "MIT", - "dependencies": { - "undici-types": "~5.26.4" - } - }, - "node_modules/openai/node_modules/undici-types": { - "version": "5.26.5", - "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-5.26.5.tgz", - "integrity": "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA==", - "license": "MIT" - }, - "node_modules/tr46": { - "version": "0.0.3", - "resolved": "https://registry.npmjs.org/tr46/-/tr46-0.0.3.tgz", - "integrity": "sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw==", - "license": "MIT" - }, - "node_modules/undici-types": { - "version": "7.10.0", - "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.10.0.tgz", - "integrity": "sha512-t5Fy/nfn+14LuOc2KNYg75vZqClpAiqscVvMygNnlsHBFpSXdJaYtXMcdNLpl/Qvc3P2cB3s6lOV51nqsFq4ag==", - "license": "MIT" - }, - "node_modules/web-streams-polyfill": { - "version": "4.0.0-beta.3", - "resolved": "https://registry.npmjs.org/web-streams-polyfill/-/web-streams-polyfill-4.0.0-beta.3.tgz", - "integrity": "sha512-QW95TCTaHmsYfHDybGMwO5IJIM93I/6vTRk+daHTWFPhwh+C8Cg7j7XyKrwrj8Ib6vYXe0ocYNrmzY4xAAN6ug==", - "license": "MIT", - "engines": { - "node": ">= 14" - } - }, - "node_modules/webidl-conversions": { - "version": "3.0.1", - "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-3.0.1.tgz", - "integrity": "sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ==", - "license": "BSD-2-Clause" - }, - "node_modules/whatwg-url": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-5.0.0.tgz", - "integrity": "sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw==", - "license": "MIT", - "dependencies": { - "tr46": "~0.0.3", - "webidl-conversions": "^3.0.0" - } - } - } -} diff --git a/tutorials/generate_prereq_practice/js/package.json b/tutorials/generate_prereq_practice/js/package.json deleted file mode 100644 index b731fd5..0000000 --- a/tutorials/generate_prereq_practice/js/package.json +++ /dev/null @@ -1,25 +0,0 @@ -{ - "name": "generate-prereq-practice", - "version": "1.0.0", - "description": "Generate prerequisite practice questions using Knowledge Graph datasets", - "main": "generate-prereq-practice.js", - "scripts": { - "start": "node generate-prereq-practice.js", - "test": "echo \"Error: no test specified\" && exit 1" - }, - "dependencies": { - "openai": "^4.0.0", - "arquero": "^8.0.3", - "csv-parse": "^6.1.0", - "dotenv": "^16.3.1" - }, - "keywords": [ - "education", - "prerequisites", - "knowledge-graph", - "practice-questions", - "ai" - ], - "author": "", - "license": "ISC" -} \ No newline at end of file diff --git a/tutorials/generate_prereq_practice/python/.env.example b/tutorials/generate_prereq_practice/python/.env.example deleted file mode 100644 index 0483037..0000000 --- a/tutorials/generate_prereq_practice/python/.env.example +++ /dev/null @@ -1,5 +0,0 @@ -# Knowledge Graph data path - update with your actual path to CSV files -KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files - -# OpenAI API key for generating practice questions -OPENAI_API_KEY=your_openai_api_key_here \ No newline at end of file diff --git a/tutorials/generate_prereq_practice/python/README.md b/tutorials/generate_prereq_practice/python/README.md deleted file mode 100644 index 5db696e..0000000 --- a/tutorials/generate_prereq_practice/python/README.md +++ /dev/null @@ -1,47 +0,0 @@ -# Generate Prerequisite Practice - -Demonstrates how to generate prerequisite-based practice questions using Knowledge Graph data covering: -- **Data Loading**: Reading CSV files with educational frameworks and relationships -- **Prerequisite Analysis**: Finding standards that build towards a target standard -- **Relationship Mapping**: Connecting learning components to prerequisite standards -- **Practice Generation**: Creating structured practice questions based on prerequisite knowledge - -**Features**: Common Core Math analysis, prerequisite relationship traversal, learning component mapping - -Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowledge-graph/v1-1-0/getting-started/tutorials/generating-prerequisite-practice-questions) - -## Prerequisites - -- Python 3.8 or higher -- Knowledge Graph CSV dataset files: - - `StandardsFrameworkItem.csv` - - `LearningComponent.csv` - - `Relationships.csv` - -## Dependencies - -- **openai**: OpenAI API for generating practice questions -- **pandas**: Data manipulation and analysis library -- **python-dotenv**: Environment variable management - -## Quick Start - -1. **Clone and Set Up Virtual Environment**: - ```bash - git clone git@github.com:learning-commons-org/knowledge-graph.git - cd tutorials/generate_prereq_practice/python - python -m venv venv - source venv/bin/activate - pip install -r requirements.txt - ``` - -2. **Set Environment Variables** (create `.env` file): - ```bash - KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files - OPENAI_API_KEY=your_openai_api_key_here - ``` - -3. **Run Tutorial**: - ```bash - python generate_prereq_practice.py - ``` diff --git a/tutorials/generate_prereq_practice/python/generate_prereq_practice.py b/tutorials/generate_prereq_practice/python/generate_prereq_practice.py deleted file mode 100644 index b1d4d49..0000000 --- a/tutorials/generate_prereq_practice/python/generate_prereq_practice.py +++ /dev/null @@ -1,373 +0,0 @@ -#!/usr/bin/env python3 -""" -================================ -CONFIGURATION & SETUP -================================ -""" - -# Dependencies -import os -import sys -import json -import pandas as pd -from pathlib import Path -from dotenv import load_dotenv -from openai import OpenAI - -# Load environment variables -load_dotenv() - -# Domain Constants -GENERATE_PRACTICE = True -# Filter criteria for mathematics standards -JURISDICTION = 'Multi-State' -ACADEMIC_SUBJECT = 'Mathematics' -TARGET_CODE = '6.NS.B.4' -# OpenAI configuration -OPENAI_MODEL = 'gpt-4' -OPENAI_TEMPERATURE = 0.7 - -# Environment Setup -data_dir = os.getenv('KG_DATA_PATH') -if not data_dir: - print('❌ KG_DATA_PATH environment variable is not set.') - sys.exit(1) - -data_path = Path(data_dir) - -openai_client = OpenAI( - api_key=os.getenv('OPENAI_API_KEY') -) - - -""" -================================ -HELPER FUNCTIONS -================================ -""" - -def load_csv(filename): - """Load and parse CSV file from data directory""" - try: - file_path = data_path / filename - return pd.read_csv(file_path) - except Exception as error: - print(f'❌ Error loading CSV file {filename}: {str(error)}') - raise error - - -""" -================================ -STEP 1: LOAD DATA -================================ -""" - -def load_data(): - """Load CSV data files and build filtered dataset""" - - standards_framework_items = load_csv('StandardsFrameworkItem.csv') - learning_components = load_csv('LearningComponent.csv') - relationships = load_csv('Relationships.csv') - - print('✅ Files loaded from KG CSV files') - print(f' Standards Framework Items: {len(standards_framework_items)}') - print(f' Learning Components: {len(learning_components)}') - print(f' Relationships: {len(relationships)}') - - # Filter for relevant StandardsFrameworkItems by jurisdiction and subject - relevant_standards = standards_framework_items[ - (standards_framework_items['jurisdiction'] == JURISDICTION) & - (standards_framework_items['academicSubject'] == ACADEMIC_SUBJECT) - ] - - # Get set of relevant identifiers for filtering - relevant_standard_ids = set(relevant_standards['caseIdentifierUUID']) - - # Filter relationships for buildsTowards and supports relationships - relevant_relationships = relationships[ - ((relationships['relationshipType'] == 'buildsTowards') & - (relationships['sourceEntityValue'].isin(relevant_standard_ids)) & - (relationships['targetEntityValue'].isin(relevant_standard_ids))) | - ((relationships['relationshipType'] == 'supports') & - (relationships['targetEntityValue'].isin(relevant_standard_ids))) - ] - - # Get learning component IDs from supports relationships - support_relationships = relevant_relationships[ - relevant_relationships['relationshipType'] == 'supports' - ] - linked_learning_component_ids = set(support_relationships['sourceEntityValue']) - - # Filter learning components by identifier - relevant_learning_components = learning_components[ - learning_components['identifier'].isin(linked_learning_component_ids) - ] - - print('✅ Retrieved scoped graph:') - print(f' Standards Framework Items: {len(relevant_standards)}') - print(f' Learning Components: {len(relevant_learning_components)}') - print(f' Relationships: {len(relevant_relationships)}') - - return { - 'relevant_standards': relevant_standards, - 'relevant_relationships': relevant_relationships, - 'relevant_learning_components': relevant_learning_components - } - - -""" -================================ -STEP 2: QUERY PREREQUISITE DATA -================================ -""" - -def get_standard_and_prerequisites(relevant_standards, relevant_relationships): - target_standard_df = relevant_standards[ - relevant_standards['statementCode'] == TARGET_CODE - ] - - if len(target_standard_df) == 0: - print(f'❌ No StandardsFrameworkItem found for statementCode = "{TARGET_CODE}"') - return None - - target_standard = target_standard_df.iloc[0] - print(f'✅ Found StandardsFrameworkItem for "{TARGET_CODE}":') - print(f' UUID: {target_standard["caseIdentifierUUID"]}') - print(f' Statement Code: {target_standard["statementCode"]}') - print(f' Description: {target_standard["description"]}') - - prerequisite_links = relevant_relationships[ - (relevant_relationships['relationshipType'] == 'buildsTowards') & - (relevant_relationships['targetEntityValue'] == target_standard['caseIdentifierUUID']) - ] - - prerequisite_standards = prerequisite_links.merge( - relevant_standards, - left_on='sourceEntityValue', - right_on='caseIdentifierUUID', - suffixes=('_rel', '_std') - )[['sourceEntityValue', 'statementCode', 'description_std']].rename(columns={ - 'sourceEntityValue': 'caseIdentifierUUID', - 'description_std': 'standardDescription' - }) - - print(f'✅ Found {len(prerequisite_standards)} prerequisite(s) for {target_standard["statementCode"]}:') - for i, (_, std) in enumerate(prerequisite_standards.iterrows(), 1): - print(f' {i}. {std["statementCode"]}: {std["standardDescription"][:100]}...') - - return {'target_standard': target_standard, 'prerequisite_standards': prerequisite_standards} - - -def get_learning_components_for_prerequisites(prerequisite_standards, relevant_relationships, relevant_learning_components): - prerequisite_learning_components = prerequisite_standards.merge( - relevant_relationships, - left_on='caseIdentifierUUID', - right_on='targetEntityValue', - suffixes=('_std', '_rel') - ) - - prerequisite_learning_components = prerequisite_learning_components[ - prerequisite_learning_components['relationshipType'] == 'supports' - ].merge( - relevant_learning_components, - left_on='sourceEntityValue', - right_on='identifier', - suffixes=('_rel', '_lc') - ) - - # Select the correct columns - after merge with suffixes, learning component description becomes description_lc - prerequisite_learning_components = prerequisite_learning_components[ - ['caseIdentifierUUID', 'statementCode', 'standardDescription', 'description_lc'] - ].rename(columns={ - 'description_lc': 'learningComponentDescription' - }) - - print(f'✅ Found {len(prerequisite_learning_components)} supporting learning components for prerequisites:') - for i, (_, lc) in enumerate(prerequisite_learning_components.iterrows(), 1): - print(f' {i}. {lc["learningComponentDescription"][:100]}...') - - return prerequisite_learning_components - - -def query_prerequisite_data(relevant_standards, relevant_relationships, relevant_learning_components): - """ - Analyze prerequisite relationships for the target standard - This step identifies prerequisites and supporting learning components - - SQL: WITH target AS ( - SELECT "caseIdentifierUUID" - FROM standards_framework_item - WHERE "statementCode" = '6.NS.B.4' - ), - prerequisite_standards AS ( - SELECT - sfi."caseIdentifierUUID", - sfi."statementCode", - sfi."description" - FROM standards_framework_item sfi - JOIN relationships r - ON sfi."caseIdentifierUUID" = r."sourceEntityValue" - JOIN target t - ON r."targetEntityValue" = t."caseIdentifierUUID" - WHERE r."relationshipType" = 'buildsTowards' - ) - SELECT - ps."caseIdentifierUUID", - ps."statementCode", - ps."description", - lc."description" - FROM prerequisite_standards ps - JOIN relationships r - ON ps."caseIdentifierUUID" = r."targetEntityValue" - JOIN learning_component lc - ON r."sourceEntityValue" = lc."identifier" - WHERE r."relationshipType" = 'supports'; - - Cypher: MATCH (target:StandardsFrameworkItem {statementCode: '6.NS.B.4'}) - MATCH (prereq:StandardsFrameworkItem)-[:buildsTowards]->(target) - MATCH (lc:LearningComponent)-[:supports]->(prereq) - RETURN prereq.caseIdentifierUUID, prereq.statementCode, prereq.description, - lc.description - """ - - standard_and_prereq_data = get_standard_and_prerequisites(relevant_standards, relevant_relationships) - if not standard_and_prereq_data: - return None - - target_standard = standard_and_prereq_data['target_standard'] - prerequisite_standards = standard_and_prereq_data['prerequisite_standards'] - prerequisite_learning_components = get_learning_components_for_prerequisites(prerequisite_standards, relevant_relationships, relevant_learning_components) - - return {'target_standard': target_standard, 'prerequisite_learning_components': prerequisite_learning_components} - - -""" -================================ -STEP 3: GENERATE PRACTICE -================================ -""" - -def package_context_data(target_standard, prerequisite_learning_components): - """ - Package the standards and learning components data for text generation - This creates a structured context that can be used for generating practice questions - """ - - # Convert dataframe to context format for LLM - all_rows = prerequisite_learning_components.to_dict('records') - standards_map = {} - - # Group learning components by standard for context - for row in all_rows: - case_id = row['caseIdentifierUUID'] - if case_id not in standards_map: - standards_map[case_id] = { - 'statementCode': row['statementCode'], - 'description': row['standardDescription'] or '(no statement)', - 'supportingLearningComponents': [] - } - - standards_map[case_id]['supportingLearningComponents'].append({ - 'description': row['learningComponentDescription'] or '(no description)' - }) - - full_standards_context = { - 'targetStandard': { - 'statementCode': target_standard['statementCode'], - 'description': target_standard['description'] or '(no statement)' - }, - 'prereqStandards': list(standards_map.values()) - } - - print('✅ Packaged full standards context for text generation') - print(json.dumps(full_standards_context, indent=2)) - return full_standards_context - - -def generate_practice_data(full_standards_context): - """Generate practice questions using OpenAI API - This creates educational content based on prerequisite data - """ - def generate_practice(): - print(f'🔄 Generating practice questions for {full_standards_context["targetStandard"]["statementCode"]}...') - - try: - # Build prompt inline - prerequisite_text = '' - for prereq in full_standards_context['prereqStandards']: - prerequisite_text += f'- {prereq["statementCode"]}: {prereq["description"]}\n' - prerequisite_text += ' Supporting Learning Components:\n' - for lc in prereq['supportingLearningComponents']: - prerequisite_text += f' • {lc["description"]}\n' - - prompt = f"""You are a math tutor helping middle school students. Based on the following information, generate 3 practice questions for the target standard. Questions should help reinforce the key concept and build on prerequisite knowledge. - -Target Standard: -- {full_standards_context["targetStandard"]["statementCode"]}: {full_standards_context["targetStandard"]["description"]} - -Prerequisite Standards & Supporting Learning Components: -{prerequisite_text}""" - - response = openai_client.chat.completions.create( - model=OPENAI_MODEL, - messages=[ - {'role': 'system', 'content': 'You are an expert middle school math tutor.'}, - {'role': 'user', 'content': prompt} - ], - temperature=OPENAI_TEMPERATURE - ) - - practice_questions = response.choices[0].message.content.strip() - - print('✅ Generated practice questions:\n') - print(practice_questions) - - return { - 'aiGenerated': practice_questions, - 'targetStandard': full_standards_context['targetStandard']['statementCode'], - 'prerequisiteCount': len(full_standards_context['prereqStandards']) - } - except Exception as err: - print(f'❌ Error generating practice questions: {str(err)}') - raise err - - return generate_practice - - -""" -================================ -MAIN EXECUTION -================================ -""" - -def main(): - """Main execution function - orchestrates all tutorial steps""" - print('\n=== GENERATE PREREQUISITE PRACTICE TUTORIAL ===\n') - - print('🔄 Step 1: Loading data...') - data = load_data() - relevant_standards = data['relevant_standards'] - relevant_relationships = data['relevant_relationships'] - relevant_learning_components = data['relevant_learning_components'] - - print('\n🔄 Step 2: Querying prerequisite data...') - prerequisite_data = query_prerequisite_data(relevant_standards, relevant_relationships, relevant_learning_components) - - if not prerequisite_data: - print('❌ Failed to find prerequisite data') - return - - target_standard = prerequisite_data['target_standard'] - prerequisite_learning_components = prerequisite_data['prerequisite_learning_components'] - - print('\n🔄 Step 3: Generating practice...') - full_standards_context = package_context_data(target_standard, prerequisite_learning_components) - generate_practice = generate_practice_data(full_standards_context) - if GENERATE_PRACTICE: - generate_practice() - else: - print('🚫 Practice question generation disabled') - - -if __name__ == '__main__': - main() \ No newline at end of file diff --git a/tutorials/generate_prereq_practice/python/requirements.txt b/tutorials/generate_prereq_practice/python/requirements.txt deleted file mode 100644 index 875454d..0000000 --- a/tutorials/generate_prereq_practice/python/requirements.txt +++ /dev/null @@ -1,8 +0,0 @@ -# Core data manipulation and analysis -pandas>=2.0.0 - -# Environment variable management -python-dotenv>=1.0.0 - -# OpenAI API for generating practice questions -openai>=1.0.0 \ No newline at end of file diff --git a/tutorials/working_with_standards/js/.env.example b/tutorials/working_with_standards/js/.env.example deleted file mode 100644 index f2d3f1f..0000000 --- a/tutorials/working_with_standards/js/.env.example +++ /dev/null @@ -1,2 +0,0 @@ -# Knowledge Graph data path - update with your actual path to CSV files -KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files \ No newline at end of file diff --git a/tutorials/working_with_standards/js/README.md b/tutorials/working_with_standards/js/README.md deleted file mode 100644 index 2b2bfce..0000000 --- a/tutorials/working_with_standards/js/README.md +++ /dev/null @@ -1,42 +0,0 @@ -Demonstrates how to work with state educational standards data covering: -- **Data Loading**: Reading CSV files with standards frameworks, items, and relationships -- **Hierarchical Queries**: Traversing parent-child relationships in educational standards -- **AI Embeddings**: Generating vector embeddings for semantic search -- **Vector Search**: Performing similarity-based searches on educational standards - -Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowledge-graph/v1-1-0/getting-started/tutorials/working-with-state-standards) - -## Prerequisites - -- Node.js (v14 or higher) -- Knowledge Graph CSV dataset files: - - `StandardsFramework.csv` - - `StandardsFrameworkItem.csv` - - `Relationships.csv` - -## Dependencies - -- **@xenova/transformers**: Local AI embeddings (no API required) -- **arquero**: Data manipulation and analysis library -- **csv-parse**: CSV file parsing -- **fast-cosine-similarity**: Vector similarity calculations -- **dotenv**: Environment variable management - -## Quick Start - -1. **Clone and Install**: - ```bash - git clone git@github.com:learning-commons-org/knowledge-graph.git - cd tutorials/working_with_standards/js - npm install - ``` - -2. **Set Environment Variables** (create `.env` file): - ```bash - KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files - ``` - -3. **Run Tutorial**: - ```bash - npm start - ``` diff --git a/tutorials/working_with_standards/js/package-lock.json b/tutorials/working_with_standards/js/package-lock.json deleted file mode 100644 index ca416cb..0000000 --- a/tutorials/working_with_standards/js/package-lock.json +++ /dev/null @@ -1,912 +0,0 @@ -{ - "name": "work-with-state-standards", - "version": "1.0.0", - "lockfileVersion": 3, - "requires": true, - "packages": { - "": { - "name": "work-with-state-standards", - "version": "1.0.0", - "license": "ISC", - "dependencies": { - "@xenova/transformers": "^2.6.0", - "arquero": "^8.0.3", - "csv-parse": "^6.1.0", - "dotenv": "^16.3.1", - "fast-cosine-similarity": "^1.2.2" - } - }, - "node_modules/@huggingface/jinja": { - "version": "0.2.2", - "resolved": "https://registry.npmjs.org/@huggingface/jinja/-/jinja-0.2.2.tgz", - "integrity": "sha512-/KPde26khDUIPkTGU82jdtTW9UAuvUTumCAbFs/7giR0SxsvZC4hru51PBvpijH6BVkHcROcvZM/lpy5h1jRRA==", - "license": "MIT", - "engines": { - "node": ">=18" - } - }, - "node_modules/@protobufjs/aspromise": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/@protobufjs/aspromise/-/aspromise-1.1.2.tgz", - "integrity": "sha512-j+gKExEuLmKwvz3OgROXtrJ2UG2x8Ch2YZUxahh+s1F2HZ+wAceUNLkvy6zKCPVRkU++ZWQrdxsUeQXmcg4uoQ==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/base64": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/@protobufjs/base64/-/base64-1.1.2.tgz", - "integrity": "sha512-AZkcAA5vnN/v4PDqKyMR5lx7hZttPDgClv83E//FMNhR2TMcLUhfRUBHCmSl0oi9zMgDDqRUJkSxO3wm85+XLg==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/codegen": { - "version": "2.0.4", - "resolved": "https://registry.npmjs.org/@protobufjs/codegen/-/codegen-2.0.4.tgz", - "integrity": "sha512-YyFaikqM5sH0ziFZCN3xDC7zeGaB/d0IUb9CATugHWbd1FRFwWwt4ld4OYMPWu5a3Xe01mGAULCdqhMlPl29Jg==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/eventemitter": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@protobufjs/eventemitter/-/eventemitter-1.1.0.tgz", - "integrity": "sha512-j9ednRT81vYJ9OfVuXG6ERSTdEL1xVsNgqpkxMsbIabzSo3goCjDIveeGv5d03om39ML71RdmrGNjG5SReBP/Q==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/fetch": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@protobufjs/fetch/-/fetch-1.1.0.tgz", - "integrity": "sha512-lljVXpqXebpsijW71PZaCYeIcE5on1w5DlQy5WH6GLbFryLUrBD4932W/E2BSpfRJWseIL4v/KPgBFxDOIdKpQ==", - "license": "BSD-3-Clause", - "dependencies": { - "@protobufjs/aspromise": "^1.1.1", - "@protobufjs/inquire": "^1.1.0" - } - }, - "node_modules/@protobufjs/float": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/@protobufjs/float/-/float-1.0.2.tgz", - "integrity": "sha512-Ddb+kVXlXst9d+R9PfTIxh1EdNkgoRe5tOX6t01f1lYWOvJnSPDBlG241QLzcyPdoNTsblLUdujGSE4RzrTZGQ==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/inquire": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@protobufjs/inquire/-/inquire-1.1.0.tgz", - "integrity": "sha512-kdSefcPdruJiFMVSbn801t4vFK7KB/5gd2fYvrxhuJYg8ILrmn9SKSX2tZdV6V+ksulWqS7aXjBcRXl3wHoD9Q==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/path": { - "version": "1.1.2", - "resolved": "https://registry.npmjs.org/@protobufjs/path/-/path-1.1.2.tgz", - "integrity": "sha512-6JOcJ5Tm08dOHAbdR3GrvP+yUUfkjG5ePsHYczMFLq3ZmMkAD98cDgcT2iA1lJ9NVwFd4tH/iSSoe44YWkltEA==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/pool": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@protobufjs/pool/-/pool-1.1.0.tgz", - "integrity": "sha512-0kELaGSIDBKvcgS4zkjz1PeddatrjYcmMWOlAuAPwAeccUrPHdUqo/J6LiymHHEiJT5NrF1UVwxY14f+fy4WQw==", - "license": "BSD-3-Clause" - }, - "node_modules/@protobufjs/utf8": { - "version": "1.1.0", - "resolved": "https://registry.npmjs.org/@protobufjs/utf8/-/utf8-1.1.0.tgz", - "integrity": "sha512-Vvn3zZrhQZkkBE8LSuW3em98c0FwgO4nxzv6OdSxPKJIEKY2bGbHn+mhGIPerzI4twdxaP8/0+06HBpwf345Lw==", - "license": "BSD-3-Clause" - }, - "node_modules/@types/long": { - "version": "4.0.2", - "resolved": "https://registry.npmjs.org/@types/long/-/long-4.0.2.tgz", - "integrity": "sha512-MqTGEo5bj5t157U6fA/BiDynNkn0YknVdh48CMPkTSpFTVmvao5UQmm7uEF6xBEo7qIMAlY/JSleYaE6VOdpaA==", - "license": "MIT" - }, - "node_modules/@types/node": { - "version": "18.19.123", - "resolved": "https://registry.npmjs.org/@types/node/-/node-18.19.123.tgz", - "integrity": "sha512-K7DIaHnh0mzVxreCR9qwgNxp3MH9dltPNIEddW9MYUlcKAzm+3grKNSTe2vCJHI1FaLpvpL5JGJrz1UZDKYvDg==", - "license": "MIT", - "dependencies": { - "undici-types": "~5.26.4" - } - }, - "node_modules/@uwdata/flechette": { - "version": "2.1.0", - "resolved": "https://registry.npmjs.org/@uwdata/flechette/-/flechette-2.1.0.tgz", - "integrity": "sha512-CsVgFIc94Rb5UhKvAK6E2rCFV+H6WkDzBEjYZvl2lfcI9vqLRafdngd9o5cq2lsA2offxCmJXdS1DE7ACPn33w==", - "license": "BSD-3-Clause" - }, - "node_modules/@xenova/transformers": { - "version": "2.17.2", - "resolved": "https://registry.npmjs.org/@xenova/transformers/-/transformers-2.17.2.tgz", - "integrity": "sha512-lZmHqzrVIkSvZdKZEx7IYY51TK0WDrC8eR0c5IMnBsO8di8are1zzw8BlLhyO2TklZKLN5UffNGs1IJwT6oOqQ==", - "license": "Apache-2.0", - "dependencies": { - "@huggingface/jinja": "^0.2.2", - "onnxruntime-web": "1.14.0", - "sharp": "^0.32.0" - }, - "optionalDependencies": { - "onnxruntime-node": "1.14.0" - } - }, - "node_modules/acorn": { - "version": "8.15.0", - "resolved": "https://registry.npmjs.org/acorn/-/acorn-8.15.0.tgz", - "integrity": "sha512-NZyJarBfL7nWwIq+FDL6Zp/yHEhePMNnnJ0y3qfieCrmNvYct8uvtiV41UvlSe6apAfk0fY1FbWx+NwfmpvtTg==", - "license": "MIT", - "bin": { - "acorn": "bin/acorn" - }, - "engines": { - "node": ">=0.4.0" - } - }, - "node_modules/arquero": { - "version": "8.0.3", - "resolved": "https://registry.npmjs.org/arquero/-/arquero-8.0.3.tgz", - "integrity": "sha512-7YQwe/GPVBUiahaPwEwgvu6VHyuhX0Ut61JZlIJYsAobOH5unLBwTmh43BobhX/N4dW1sb8WyKlQ8GjFq2whzQ==", - "license": "BSD-3-Clause", - "dependencies": { - "@uwdata/flechette": "^2.0.0", - "acorn": "^8.14.1" - } - }, - "node_modules/b4a": { - "version": "1.6.7", - "resolved": "https://registry.npmjs.org/b4a/-/b4a-1.6.7.tgz", - "integrity": "sha512-OnAYlL5b7LEkALw87fUVafQw5rVR9RjwGd4KUwNQ6DrrNmaVaUCgLipfVlzrPQ4tWOR9P0IXGNOx50jYCCdSJg==", - "license": "Apache-2.0" - }, - "node_modules/bare-events": { - "version": "2.6.1", - "resolved": "https://registry.npmjs.org/bare-events/-/bare-events-2.6.1.tgz", - "integrity": "sha512-AuTJkq9XmE6Vk0FJVNq5QxETrSA/vKHarWVBG5l/JbdCL1prJemiyJqUS0jrlXO0MftuPq4m3YVYhoNc5+aE/g==", - "license": "Apache-2.0", - "optional": true - }, - "node_modules/bare-fs": { - "version": "4.2.0", - "resolved": "https://registry.npmjs.org/bare-fs/-/bare-fs-4.2.0.tgz", - "integrity": "sha512-oRfrw7gwwBVAWx9S5zPMo2iiOjxyiZE12DmblmMQREgcogbNO0AFaZ+QBxxkEXiPspcpvO/Qtqn8LabUx4uYXg==", - "license": "Apache-2.0", - "optional": true, - "dependencies": { - "bare-events": "^2.5.4", - "bare-path": "^3.0.0", - "bare-stream": "^2.6.4" - }, - "engines": { - "bare": ">=1.16.0" - }, - "peerDependencies": { - "bare-buffer": "*" - }, - "peerDependenciesMeta": { - "bare-buffer": { - "optional": true - } - } - }, - "node_modules/bare-os": { - "version": "3.6.1", - "resolved": "https://registry.npmjs.org/bare-os/-/bare-os-3.6.1.tgz", - "integrity": "sha512-uaIjxokhFidJP+bmmvKSgiMzj2sV5GPHaZVAIktcxcpCyBFFWO+YlikVAdhmUo2vYFvFhOXIAlldqV29L8126g==", - "license": "Apache-2.0", - "optional": true, - "engines": { - "bare": ">=1.14.0" - } - }, - "node_modules/bare-path": { - "version": "3.0.0", - "resolved": "https://registry.npmjs.org/bare-path/-/bare-path-3.0.0.tgz", - "integrity": "sha512-tyfW2cQcB5NN8Saijrhqn0Zh7AnFNsnczRcuWODH0eYAXBsJ5gVxAUuNr7tsHSC6IZ77cA0SitzT+s47kot8Mw==", - "license": "Apache-2.0", - "optional": true, - "dependencies": { - "bare-os": "^3.0.1" - } - }, - "node_modules/bare-stream": { - "version": "2.7.0", - "resolved": "https://registry.npmjs.org/bare-stream/-/bare-stream-2.7.0.tgz", - "integrity": "sha512-oyXQNicV1y8nc2aKffH+BUHFRXmx6VrPzlnaEvMhram0nPBrKcEdcyBg5r08D0i8VxngHFAiVyn1QKXpSG0B8A==", - "license": "Apache-2.0", - "optional": true, - "dependencies": { - "streamx": "^2.21.0" - }, - "peerDependencies": { - "bare-buffer": "*", - "bare-events": "*" - }, - "peerDependenciesMeta": { - "bare-buffer": { - "optional": true - }, - "bare-events": { - "optional": true - } - } - }, - "node_modules/base64-js": { - "version": "1.5.1", - "resolved": "https://registry.npmjs.org/base64-js/-/base64-js-1.5.1.tgz", - "integrity": "sha512-AKpaYlHn8t4SVbOHCy+b5+KKgvR4vrsD8vbvrbiQJps7fKDTkjkDry6ji0rUJjC0kzbNePLwzxq8iypo41qeWA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT" - }, - "node_modules/bl": { - "version": "4.1.0", - "resolved": "https://registry.npmjs.org/bl/-/bl-4.1.0.tgz", - "integrity": "sha512-1W07cM9gS6DcLperZfFSj+bWLtaPGSOHWhPiGzXmvVJbRLdG82sH/Kn8EtW1VqWVA54AKf2h5k5BbnIbwF3h6w==", - "license": "MIT", - "dependencies": { - "buffer": "^5.5.0", - "inherits": "^2.0.4", - "readable-stream": "^3.4.0" - } - }, - "node_modules/buffer": { - "version": "5.7.1", - "resolved": "https://registry.npmjs.org/buffer/-/buffer-5.7.1.tgz", - "integrity": "sha512-EHcyIPBQ4BSGlvjB16k5KgAJ27CIsHY/2JBmCRReo48y9rQ3MaUzWX3KVlBa4U7MyX02HdVj0K7C3WaB3ju7FQ==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "dependencies": { - "base64-js": "^1.3.1", - "ieee754": "^1.1.13" - } - }, - "node_modules/chownr": { - "version": "1.1.4", - "resolved": "https://registry.npmjs.org/chownr/-/chownr-1.1.4.tgz", - "integrity": "sha512-jJ0bqzaylmJtVnNgzTeSOs8DPavpbYgEr/b0YL8/2GO3xJEhInFmhKMUnEJQjZumK7KXGFhUy89PrsJWlakBVg==", - "license": "ISC" - }, - "node_modules/color": { - "version": "4.2.3", - "resolved": "https://registry.npmjs.org/color/-/color-4.2.3.tgz", - "integrity": "sha512-1rXeuUUiGGrykh+CeBdu5Ie7OJwinCgQY0bc7GCRxy5xVHy+moaqkpL/jqQq0MtQOeYcrqEz4abc5f0KtU7W4A==", - "license": "MIT", - "dependencies": { - "color-convert": "^2.0.1", - "color-string": "^1.9.0" - }, - "engines": { - "node": ">=12.5.0" - } - }, - "node_modules/color-convert": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/color-convert/-/color-convert-2.0.1.tgz", - "integrity": "sha512-RRECPsj7iu/xb5oKYcsFHSppFNnsj/52OVTRKb4zP5onXwVF3zVmmToNcOfGC+CRDpfK/U584fMg38ZHCaElKQ==", - "license": "MIT", - "dependencies": { - "color-name": "~1.1.4" - }, - "engines": { - "node": ">=7.0.0" - } - }, - "node_modules/color-name": { - "version": "1.1.4", - "resolved": "https://registry.npmjs.org/color-name/-/color-name-1.1.4.tgz", - "integrity": "sha512-dOy+3AuW3a2wNbZHIuMZpTcgjGuLU/uBL/ubcZF9OXbDo8ff4O8yVp5Bf0efS8uEoYo5q4Fx7dY9OgQGXgAsQA==", - "license": "MIT" - }, - "node_modules/color-string": { - "version": "1.9.1", - "resolved": "https://registry.npmjs.org/color-string/-/color-string-1.9.1.tgz", - "integrity": "sha512-shrVawQFojnZv6xM40anx4CkoDP+fZsw/ZerEMsW/pyzsRbElpsL/DBVW7q3ExxwusdNXI3lXpuhEZkzs8p5Eg==", - "license": "MIT", - "dependencies": { - "color-name": "^1.0.0", - "simple-swizzle": "^0.2.2" - } - }, - "node_modules/csv-parse": { - "version": "6.1.0", - "resolved": "https://registry.npmjs.org/csv-parse/-/csv-parse-6.1.0.tgz", - "integrity": "sha512-CEE+jwpgLn+MmtCpVcPtiCZpVtB6Z2OKPTr34pycYYoL7sxdOkXDdQ4lRiw6ioC0q6BLqhc6cKweCVvral8yhw==", - "license": "MIT" - }, - "node_modules/decompress-response": { - "version": "6.0.0", - "resolved": "https://registry.npmjs.org/decompress-response/-/decompress-response-6.0.0.tgz", - "integrity": "sha512-aW35yZM6Bb/4oJlZncMH2LCoZtJXTRxES17vE3hoRiowU2kWHaJKFkSBDnDR+cm9J+9QhXmREyIfv0pji9ejCQ==", - "license": "MIT", - "dependencies": { - "mimic-response": "^3.1.0" - }, - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/deep-extend": { - "version": "0.6.0", - "resolved": "https://registry.npmjs.org/deep-extend/-/deep-extend-0.6.0.tgz", - "integrity": "sha512-LOHxIOaPYdHlJRtCQfDIVZtfw/ufM8+rVj649RIHzcm/vGwQRXFt6OPqIFWsm2XEMrNIEtWR64sY1LEKD2vAOA==", - "license": "MIT", - "engines": { - "node": ">=4.0.0" - } - }, - "node_modules/detect-libc": { - "version": "2.0.4", - "resolved": "https://registry.npmjs.org/detect-libc/-/detect-libc-2.0.4.tgz", - "integrity": "sha512-3UDv+G9CsCKO1WKMGw9fwq/SWJYbI0c5Y7LU1AXYoDdbhE2AHQ6N6Nb34sG8Fj7T5APy8qXDCKuuIHd1BR0tVA==", - "license": "Apache-2.0", - "engines": { - "node": ">=8" - } - }, - "node_modules/dotenv": { - "version": "16.6.1", - "resolved": "https://registry.npmjs.org/dotenv/-/dotenv-16.6.1.tgz", - "integrity": "sha512-uBq4egWHTcTt33a72vpSG0z3HnPuIl6NqYcTrKEg2azoEyl2hpW0zqlxysq2pK9HlDIHyHyakeYaYnSAwd8bow==", - "license": "BSD-2-Clause", - "engines": { - "node": ">=12" - }, - "funding": { - "url": "https://dotenvx.com" - } - }, - "node_modules/end-of-stream": { - "version": "1.4.5", - "resolved": "https://registry.npmjs.org/end-of-stream/-/end-of-stream-1.4.5.tgz", - "integrity": "sha512-ooEGc6HP26xXq/N+GCGOT0JKCLDGrq2bQUZrQ7gyrJiZANJ/8YDTxTpQBXGMn+WbIQXNVpyWymm7KYVICQnyOg==", - "license": "MIT", - "dependencies": { - "once": "^1.4.0" - } - }, - "node_modules/expand-template": { - "version": "2.0.3", - "resolved": "https://registry.npmjs.org/expand-template/-/expand-template-2.0.3.tgz", - "integrity": "sha512-XYfuKMvj4O35f/pOXLObndIRvyQ+/+6AhODh+OKWj9S9498pHHn/IMszH+gt0fBCRWMNfk1ZSp5x3AifmnI2vg==", - "license": "(MIT OR WTFPL)", - "engines": { - "node": ">=6" - } - }, - "node_modules/fast-cosine-similarity": { - "version": "1.2.2", - "resolved": "https://registry.npmjs.org/fast-cosine-similarity/-/fast-cosine-similarity-1.2.2.tgz", - "integrity": "sha512-VAAswiN4WOjvkbqSTXfatw+zKclDN9h2R3Si1pFoZSLd1jHjXizaOr1LLip7bI69IRcbyhOPQ3CjL5WfvHx2gg==", - "license": "MIT" - }, - "node_modules/fast-fifo": { - "version": "1.3.2", - "resolved": "https://registry.npmjs.org/fast-fifo/-/fast-fifo-1.3.2.tgz", - "integrity": "sha512-/d9sfos4yxzpwkDkuN7k2SqFKtYNmCTzgfEpz82x34IM9/zc8KGxQoXg1liNC/izpRM/MBdt44Nmx41ZWqk+FQ==", - "license": "MIT" - }, - "node_modules/flatbuffers": { - "version": "1.12.0", - "resolved": "https://registry.npmjs.org/flatbuffers/-/flatbuffers-1.12.0.tgz", - "integrity": "sha512-c7CZADjRcl6j0PlvFy0ZqXQ67qSEZfrVPynmnL+2zPc+NtMvrF8Y0QceMo7QqnSPc7+uWjUIAbvCQ5WIKlMVdQ==", - "license": "SEE LICENSE IN LICENSE.txt" - }, - "node_modules/fs-constants": { - "version": "1.0.0", - "resolved": "https://registry.npmjs.org/fs-constants/-/fs-constants-1.0.0.tgz", - "integrity": "sha512-y6OAwoSIf7FyjMIv94u+b5rdheZEjzR63GTyZJm5qh4Bi+2YgwLCcI/fPFZkL5PSixOt6ZNKm+w+Hfp/Bciwow==", - "license": "MIT" - }, - "node_modules/github-from-package": { - "version": "0.0.0", - "resolved": "https://registry.npmjs.org/github-from-package/-/github-from-package-0.0.0.tgz", - "integrity": "sha512-SyHy3T1v2NUXn29OsWdxmK6RwHD+vkj3v8en8AOBZ1wBQ/hCAQ5bAQTD02kW4W9tUp/3Qh6J8r9EvntiyCmOOw==", - "license": "MIT" - }, - "node_modules/guid-typescript": { - "version": "1.0.9", - "resolved": "https://registry.npmjs.org/guid-typescript/-/guid-typescript-1.0.9.tgz", - "integrity": "sha512-Y8T4vYhEfwJOTbouREvG+3XDsjr8E3kIr7uf+JZ0BYloFsttiHU0WfvANVsR7TxNUJa/WpCnw/Ino/p+DeBhBQ==", - "license": "ISC" - }, - "node_modules/ieee754": { - "version": "1.2.1", - "resolved": "https://registry.npmjs.org/ieee754/-/ieee754-1.2.1.tgz", - "integrity": "sha512-dcyqhDvX1C46lXZcVqCpK+FtMRQVdIMN6/Df5js2zouUsqG7I6sFxitIC+7KYK29KdXOLHdu9zL4sFnoVQnqaA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "BSD-3-Clause" - }, - "node_modules/inherits": { - "version": "2.0.4", - "resolved": "https://registry.npmjs.org/inherits/-/inherits-2.0.4.tgz", - "integrity": "sha512-k/vGaX4/Yla3WzyMCvTQOXYeIHvqOKtnqBduzTHpzpQZzAskKMhZ2K+EnBiSM9zGSoIFeMpXKxa4dYeZIQqewQ==", - "license": "ISC" - }, - "node_modules/ini": { - "version": "1.3.8", - "resolved": "https://registry.npmjs.org/ini/-/ini-1.3.8.tgz", - "integrity": "sha512-JV/yugV2uzW5iMRSiZAyDtQd+nxtUnjeLt0acNdw98kKLrvuRVyB80tsREOE7yvGVgalhZ6RNXCmEHkUKBKxew==", - "license": "ISC" - }, - "node_modules/is-arrayish": { - "version": "0.3.2", - "resolved": "https://registry.npmjs.org/is-arrayish/-/is-arrayish-0.3.2.tgz", - "integrity": "sha512-eVRqCvVlZbuw3GrM63ovNSNAeA1K16kaR/LRY/92w0zxQ5/1YzwblUX652i4Xs9RwAGjW9d9y6X88t8OaAJfWQ==", - "license": "MIT" - }, - "node_modules/long": { - "version": "4.0.0", - "resolved": "https://registry.npmjs.org/long/-/long-4.0.0.tgz", - "integrity": "sha512-XsP+KhQif4bjX1kbuSiySJFNAehNxgLb6hPRGJ9QsUr8ajHkuXGdrHmFUTUUXhDwVX2R5bY4JNZEwbUiMhV+MA==", - "license": "Apache-2.0" - }, - "node_modules/mimic-response": { - "version": "3.1.0", - "resolved": "https://registry.npmjs.org/mimic-response/-/mimic-response-3.1.0.tgz", - "integrity": "sha512-z0yWI+4FDrrweS8Zmt4Ej5HdJmky15+L2e6Wgn3+iK5fWzb6T3fhNFq2+MeTRb064c6Wr4N/wv0DzQTjNzHNGQ==", - "license": "MIT", - "engines": { - "node": ">=10" - }, - "funding": { - "url": "https://github.com/sponsors/sindresorhus" - } - }, - "node_modules/minimist": { - "version": "1.2.8", - "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.8.tgz", - "integrity": "sha512-2yyAR8qBkN3YuheJanUpWC5U3bb5osDywNB8RzDVlDwDHbocAJveqqj1u8+SVD7jkWT4yvsHCpWqqWqAxb0zCA==", - "license": "MIT", - "funding": { - "url": "https://github.com/sponsors/ljharb" - } - }, - "node_modules/mkdirp-classic": { - "version": "0.5.3", - "resolved": "https://registry.npmjs.org/mkdirp-classic/-/mkdirp-classic-0.5.3.tgz", - "integrity": "sha512-gKLcREMhtuZRwRAfqP3RFW+TK4JqApVBtOIftVgjuABpAtpxhPGaDcfvbhNvD0B8iD1oUr/txX35NjcaY6Ns/A==", - "license": "MIT" - }, - "node_modules/napi-build-utils": { - "version": "2.0.0", - "resolved": "https://registry.npmjs.org/napi-build-utils/-/napi-build-utils-2.0.0.tgz", - "integrity": "sha512-GEbrYkbfF7MoNaoh2iGG84Mnf/WZfB0GdGEsM8wz7Expx/LlWf5U8t9nvJKXSp3qr5IsEbK04cBGhol/KwOsWA==", - "license": "MIT" - }, - "node_modules/node-abi": { - "version": "3.75.0", - "resolved": "https://registry.npmjs.org/node-abi/-/node-abi-3.75.0.tgz", - "integrity": "sha512-OhYaY5sDsIka7H7AtijtI9jwGYLyl29eQn/W623DiN/MIv5sUqc4g7BIDThX+gb7di9f6xK02nkp8sdfFWZLTg==", - "license": "MIT", - "dependencies": { - "semver": "^7.3.5" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/node-addon-api": { - "version": "6.1.0", - "resolved": "https://registry.npmjs.org/node-addon-api/-/node-addon-api-6.1.0.tgz", - "integrity": "sha512-+eawOlIgy680F0kBzPUNFhMZGtJ1YmqM6l4+Crf4IkImjYrO/mqPwRMh352g23uIaQKFItcQ64I7KMaJxHgAVA==", - "license": "MIT" - }, - "node_modules/once": { - "version": "1.4.0", - "resolved": "https://registry.npmjs.org/once/-/once-1.4.0.tgz", - "integrity": "sha512-lNaJgI+2Q5URQBkccEKHTQOPaXdUxnZZElQTZY0MFUAuaEqe1E+Nyvgdz/aIyNi6Z9MzO5dv1H8n58/GELp3+w==", - "license": "ISC", - "dependencies": { - "wrappy": "1" - } - }, - "node_modules/onnx-proto": { - "version": "4.0.4", - "resolved": "https://registry.npmjs.org/onnx-proto/-/onnx-proto-4.0.4.tgz", - "integrity": "sha512-aldMOB3HRoo6q/phyB6QRQxSt895HNNw82BNyZ2CMh4bjeKv7g/c+VpAFtJuEMVfYLMbRx61hbuqnKceLeDcDA==", - "license": "MIT", - "dependencies": { - "protobufjs": "^6.8.8" - } - }, - "node_modules/onnxruntime-common": { - "version": "1.14.0", - "resolved": "https://registry.npmjs.org/onnxruntime-common/-/onnxruntime-common-1.14.0.tgz", - "integrity": "sha512-3LJpegM2iMNRX2wUmtYfeX/ytfOzNwAWKSq1HbRrKc9+uqG/FsEA0bbKZl1btQeZaXhC26l44NWpNUeXPII7Ew==", - "license": "MIT" - }, - "node_modules/onnxruntime-node": { - "version": "1.14.0", - "resolved": "https://registry.npmjs.org/onnxruntime-node/-/onnxruntime-node-1.14.0.tgz", - "integrity": "sha512-5ba7TWomIV/9b6NH/1x/8QEeowsb+jBEvFzU6z0T4mNsFwdPqXeFUM7uxC6QeSRkEbWu3qEB0VMjrvzN/0S9+w==", - "license": "MIT", - "optional": true, - "os": [ - "win32", - "darwin", - "linux" - ], - "dependencies": { - "onnxruntime-common": "~1.14.0" - } - }, - "node_modules/onnxruntime-web": { - "version": "1.14.0", - "resolved": "https://registry.npmjs.org/onnxruntime-web/-/onnxruntime-web-1.14.0.tgz", - "integrity": "sha512-Kcqf43UMfW8mCydVGcX9OMXI2VN17c0p6XvR7IPSZzBf/6lteBzXHvcEVWDPmCKuGombl997HgLqj91F11DzXw==", - "license": "MIT", - "dependencies": { - "flatbuffers": "^1.12.0", - "guid-typescript": "^1.0.9", - "long": "^4.0.0", - "onnx-proto": "^4.0.4", - "onnxruntime-common": "~1.14.0", - "platform": "^1.3.6" - } - }, - "node_modules/platform": { - "version": "1.3.6", - "resolved": "https://registry.npmjs.org/platform/-/platform-1.3.6.tgz", - "integrity": "sha512-fnWVljUchTro6RiCFvCXBbNhJc2NijN7oIQxbwsyL0buWJPG85v81ehlHI9fXrJsMNgTofEoWIQeClKpgxFLrg==", - "license": "MIT" - }, - "node_modules/prebuild-install": { - "version": "7.1.3", - "resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz", - "integrity": "sha512-8Mf2cbV7x1cXPUILADGI3wuhfqWvtiLA1iclTDbFRZkgRQS0NqsPZphna9V+HyTEadheuPmjaJMsbzKQFOzLug==", - "license": "MIT", - "dependencies": { - "detect-libc": "^2.0.0", - "expand-template": "^2.0.3", - "github-from-package": "0.0.0", - "minimist": "^1.2.3", - "mkdirp-classic": "^0.5.3", - "napi-build-utils": "^2.0.0", - "node-abi": "^3.3.0", - "pump": "^3.0.0", - "rc": "^1.2.7", - "simple-get": "^4.0.0", - "tar-fs": "^2.0.0", - "tunnel-agent": "^0.6.0" - }, - "bin": { - "prebuild-install": "bin.js" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/prebuild-install/node_modules/tar-fs": { - "version": "2.1.4", - "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-2.1.4.tgz", - "integrity": "sha512-mDAjwmZdh7LTT6pNleZ05Yt65HC3E+NiQzl672vQG38jIrehtJk/J3mNwIg+vShQPcLF/LV7CMnDW6vjj6sfYQ==", - "license": "MIT", - "dependencies": { - "chownr": "^1.1.1", - "mkdirp-classic": "^0.5.2", - "pump": "^3.0.0", - "tar-stream": "^2.1.4" - } - }, - "node_modules/prebuild-install/node_modules/tar-stream": { - "version": "2.2.0", - "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-2.2.0.tgz", - "integrity": "sha512-ujeqbceABgwMZxEJnk2HDY2DlnUZ+9oEcb1KzTVfYHio0UE6dG71n60d8D2I4qNvleWrrXpmjpt7vZeF1LnMZQ==", - "license": "MIT", - "dependencies": { - "bl": "^4.0.3", - "end-of-stream": "^1.4.1", - "fs-constants": "^1.0.0", - "inherits": "^2.0.3", - "readable-stream": "^3.1.1" - }, - "engines": { - "node": ">=6" - } - }, - "node_modules/protobufjs": { - "version": "6.11.4", - "resolved": "https://registry.npmjs.org/protobufjs/-/protobufjs-6.11.4.tgz", - "integrity": "sha512-5kQWPaJHi1WoCpjTGszzQ32PG2F4+wRY6BmAT4Vfw56Q2FZ4YZzK20xUYQH4YkfehY1e6QSICrJquM6xXZNcrw==", - "hasInstallScript": true, - "license": "BSD-3-Clause", - "dependencies": { - "@protobufjs/aspromise": "^1.1.2", - "@protobufjs/base64": "^1.1.2", - "@protobufjs/codegen": "^2.0.4", - "@protobufjs/eventemitter": "^1.1.0", - "@protobufjs/fetch": "^1.1.0", - "@protobufjs/float": "^1.0.2", - "@protobufjs/inquire": "^1.1.0", - "@protobufjs/path": "^1.1.2", - "@protobufjs/pool": "^1.1.0", - "@protobufjs/utf8": "^1.1.0", - "@types/long": "^4.0.1", - "@types/node": ">=13.7.0", - "long": "^4.0.0" - }, - "bin": { - "pbjs": "bin/pbjs", - "pbts": "bin/pbts" - } - }, - "node_modules/pump": { - "version": "3.0.3", - "resolved": "https://registry.npmjs.org/pump/-/pump-3.0.3.tgz", - "integrity": "sha512-todwxLMY7/heScKmntwQG8CXVkWUOdYxIvY2s0VWAAMh/nd8SoYiRaKjlr7+iCs984f2P8zvrfWcDDYVb73NfA==", - "license": "MIT", - "dependencies": { - "end-of-stream": "^1.1.0", - "once": "^1.3.1" - } - }, - "node_modules/rc": { - "version": "1.2.8", - "resolved": "https://registry.npmjs.org/rc/-/rc-1.2.8.tgz", - "integrity": "sha512-y3bGgqKj3QBdxLbLkomlohkvsA8gdAiUQlSBJnBhfn+BPxg4bc62d8TcBW15wavDfgexCgccckhcZvywyQYPOw==", - "license": "(BSD-2-Clause OR MIT OR Apache-2.0)", - "dependencies": { - "deep-extend": "^0.6.0", - "ini": "~1.3.0", - "minimist": "^1.2.0", - "strip-json-comments": "~2.0.1" - }, - "bin": { - "rc": "cli.js" - } - }, - "node_modules/readable-stream": { - "version": "3.6.2", - "resolved": "https://registry.npmjs.org/readable-stream/-/readable-stream-3.6.2.tgz", - "integrity": "sha512-9u/sniCrY3D5WdsERHzHE4G2YCXqoG5FTHUiCC4SIbr6XcLZBY05ya9EKjYek9O5xOAwjGq+1JdGBAS7Q9ScoA==", - "license": "MIT", - "dependencies": { - "inherits": "^2.0.3", - "string_decoder": "^1.1.1", - "util-deprecate": "^1.0.1" - }, - "engines": { - "node": ">= 6" - } - }, - "node_modules/safe-buffer": { - "version": "5.2.1", - "resolved": "https://registry.npmjs.org/safe-buffer/-/safe-buffer-5.2.1.tgz", - "integrity": "sha512-rp3So07KcdmmKbGvgaNxQSJr7bGVSVk5S9Eq1F+ppbRo70+YeaDxkw5Dd8NPN+GD6bjnYm2VuPuCXmpuYvmCXQ==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT" - }, - "node_modules/semver": { - "version": "7.7.2", - "resolved": "https://registry.npmjs.org/semver/-/semver-7.7.2.tgz", - "integrity": "sha512-RF0Fw+rO5AMf9MAyaRXI4AV0Ulj5lMHqVxxdSgiVbixSCXoEmmX/jk0CuJw4+3SqroYO9VoUh+HcuJivvtJemA==", - "license": "ISC", - "bin": { - "semver": "bin/semver.js" - }, - "engines": { - "node": ">=10" - } - }, - "node_modules/sharp": { - "version": "0.32.6", - "resolved": "https://registry.npmjs.org/sharp/-/sharp-0.32.6.tgz", - "integrity": "sha512-KyLTWwgcR9Oe4d9HwCwNM2l7+J0dUQwn/yf7S0EnTtb0eVS4RxO0eUSvxPtzT4F3SY+C4K6fqdv/DO27sJ/v/w==", - "hasInstallScript": true, - "license": "Apache-2.0", - "dependencies": { - "color": "^4.2.3", - "detect-libc": "^2.0.2", - "node-addon-api": "^6.1.0", - "prebuild-install": "^7.1.1", - "semver": "^7.5.4", - "simple-get": "^4.0.1", - "tar-fs": "^3.0.4", - "tunnel-agent": "^0.6.0" - }, - "engines": { - "node": ">=14.15.0" - }, - "funding": { - "url": "https://opencollective.com/libvips" - } - }, - "node_modules/simple-concat": { - "version": "1.0.1", - "resolved": "https://registry.npmjs.org/simple-concat/-/simple-concat-1.0.1.tgz", - "integrity": "sha512-cSFtAPtRhljv69IK0hTVZQ+OfE9nePi/rtJmw5UjHeVyVroEqJXP1sFztKUy1qU+xvz3u/sfYJLa947b7nAN2Q==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT" - }, - "node_modules/simple-get": { - "version": "4.0.1", - "resolved": "https://registry.npmjs.org/simple-get/-/simple-get-4.0.1.tgz", - "integrity": "sha512-brv7p5WgH0jmQJr1ZDDfKDOSeWWg+OVypG99A/5vYGPqJ6pxiaHLy8nxtFjBA7oMa01ebA9gfh1uMCFqOuXxvA==", - "funding": [ - { - "type": "github", - "url": "https://github.com/sponsors/feross" - }, - { - "type": "patreon", - "url": "https://www.patreon.com/feross" - }, - { - "type": "consulting", - "url": "https://feross.org/support" - } - ], - "license": "MIT", - "dependencies": { - "decompress-response": "^6.0.0", - "once": "^1.3.1", - "simple-concat": "^1.0.0" - } - }, - "node_modules/simple-swizzle": { - "version": "0.2.2", - "resolved": "https://registry.npmjs.org/simple-swizzle/-/simple-swizzle-0.2.2.tgz", - "integrity": "sha512-JA//kQgZtbuY83m+xT+tXJkmJncGMTFT+C+g2h2R9uxkYIrE2yy9sgmcLhCnw57/WSD+Eh3J97FPEDFnbXnDUg==", - "license": "MIT", - "dependencies": { - "is-arrayish": "^0.3.1" - } - }, - "node_modules/streamx": { - "version": "2.22.1", - "resolved": "https://registry.npmjs.org/streamx/-/streamx-2.22.1.tgz", - "integrity": "sha512-znKXEBxfatz2GBNK02kRnCXjV+AA4kjZIUxeWSr3UGirZMJfTE9uiwKHobnbgxWyL/JWro8tTq+vOqAK1/qbSA==", - "license": "MIT", - "dependencies": { - "fast-fifo": "^1.3.2", - "text-decoder": "^1.1.0" - }, - "optionalDependencies": { - "bare-events": "^2.2.0" - } - }, - "node_modules/string_decoder": { - "version": "1.3.0", - "resolved": "https://registry.npmjs.org/string_decoder/-/string_decoder-1.3.0.tgz", - "integrity": "sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA==", - "license": "MIT", - "dependencies": { - "safe-buffer": "~5.2.0" - } - }, - "node_modules/strip-json-comments": { - "version": "2.0.1", - "resolved": "https://registry.npmjs.org/strip-json-comments/-/strip-json-comments-2.0.1.tgz", - "integrity": "sha512-4gB8na07fecVVkOI6Rs4e7T6NOTki5EmL7TUduTs6bu3EdnSycntVJ4re8kgZA+wx9IueI2Y11bfbgwtzuE0KQ==", - "license": "MIT", - "engines": { - "node": ">=0.10.0" - } - }, - "node_modules/tar-fs": { - "version": "3.1.1", - "resolved": "https://registry.npmjs.org/tar-fs/-/tar-fs-3.1.1.tgz", - "integrity": "sha512-LZA0oaPOc2fVo82Txf3gw+AkEd38szODlptMYejQUhndHMLQ9M059uXR+AfS7DNo0NpINvSqDsvyaCrBVkptWg==", - "license": "MIT", - "dependencies": { - "pump": "^3.0.0", - "tar-stream": "^3.1.5" - }, - "optionalDependencies": { - "bare-fs": "^4.0.1", - "bare-path": "^3.0.0" - } - }, - "node_modules/tar-stream": { - "version": "3.1.7", - "resolved": "https://registry.npmjs.org/tar-stream/-/tar-stream-3.1.7.tgz", - "integrity": "sha512-qJj60CXt7IU1Ffyc3NJMjh6EkuCFej46zUqJ4J7pqYlThyd9bO0XBTmcOIhSzZJVWfsLks0+nle/j538YAW9RQ==", - "license": "MIT", - "dependencies": { - "b4a": "^1.6.4", - "fast-fifo": "^1.2.0", - "streamx": "^2.15.0" - } - }, - "node_modules/text-decoder": { - "version": "1.2.3", - "resolved": "https://registry.npmjs.org/text-decoder/-/text-decoder-1.2.3.tgz", - "integrity": "sha512-3/o9z3X0X0fTupwsYvR03pJ/DjWuqqrfwBgTQzdWDiQSm9KitAyz/9WqsT2JQW7KV2m+bC2ol/zqpW37NHxLaA==", - "license": "Apache-2.0", - "dependencies": { - "b4a": "^1.6.4" - } - }, - "node_modules/tunnel-agent": { - "version": "0.6.0", - "resolved": "https://registry.npmjs.org/tunnel-agent/-/tunnel-agent-0.6.0.tgz", - "integrity": "sha512-McnNiV1l8RYeY8tBgEpuodCC1mLUdbSN+CYBL7kJsJNInOP8UjDDEwdk6Mw60vdLLrr5NHKZhMAOSrR2NZuQ+w==", - "license": "Apache-2.0", - "dependencies": { - "safe-buffer": "^5.0.1" - }, - "engines": { - "node": "*" - } - }, - "node_modules/undici-types": { - "version": "5.26.5", - "resolved": "https://registry.npmjs.org/undici-types/-/undici-types-5.26.5.tgz", - "integrity": "sha512-JlCMO+ehdEIKqlFxk6IfVoAUVmgz7cU7zD/h9XZ0qzeosSHmUJVOzSQvvYSYWXkFXC+IfLKSIffhv0sVZup6pA==", - "license": "MIT" - }, - "node_modules/util-deprecate": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/util-deprecate/-/util-deprecate-1.0.2.tgz", - "integrity": "sha512-EPD5q1uXyFxJpCrLnCc1nHnq3gOa6DZBocAIiI2TaSCA7VCJ1UJDMagCzIkXNsUYfD1daK//LTEQ8xiIbrHtcw==", - "license": "MIT" - }, - "node_modules/wrappy": { - "version": "1.0.2", - "resolved": "https://registry.npmjs.org/wrappy/-/wrappy-1.0.2.tgz", - "integrity": "sha512-l4Sp/DRseor9wL6EvV2+TuQn63dMkPjZ/sp9XkghTEbV9KlPS1xUsZ3u7/IQO4wxtcFB4bgpQPRcR3QCvezPcQ==", - "license": "ISC" - } - } -} diff --git a/tutorials/working_with_standards/js/package.json b/tutorials/working_with_standards/js/package.json deleted file mode 100644 index ecb022e..0000000 --- a/tutorials/working_with_standards/js/package.json +++ /dev/null @@ -1,25 +0,0 @@ -{ - "name": "work-with-state-standards", - "version": "1.0.0", - "description": "Working with state standards data using Knowledge Graph datasets", - "main": "work-with-state-standards.js", - "scripts": { - "start": "node work-with-state-standards.js", - "test": "echo \"Error: no test specified\" && exit 1" - }, - "dependencies": { - "@xenova/transformers": "^2.6.0", - "arquero": "^8.0.3", - "csv-parse": "^6.1.0", - "dotenv": "^16.3.1", - "fast-cosine-similarity": "^1.2.2" - }, - "keywords": [ - "education", - "standards", - "knowledge-graph", - "embeddings" - ], - "author": "", - "license": "ISC" -} diff --git a/tutorials/working_with_standards/js/work-with-state-standards.js b/tutorials/working_with_standards/js/work-with-state-standards.js deleted file mode 100644 index 62eb772..0000000 --- a/tutorials/working_with_standards/js/work-with-state-standards.js +++ /dev/null @@ -1,332 +0,0 @@ -/* ================================ - CONFIGURATION & SETUP - ================================ */ - -// Dependencies -const fs = require('fs'); -const path = require('path'); -const { parse } = require('csv-parse/sync'); -const { cosineSimilarity } = require('fast-cosine-similarity'); -require('dotenv').config(); - -// Constants -const GENERATE_EMBEDDINGS = true; -const MIDDLE_SCHOOL_GRADES = ['6', '7', '8']; -// For this tutorial, we use 'all-MiniLM-L6-v2' which provides good quality embeddings -// for short text. You can substitute any compatible embedding model. -const EMBEDDING_MODEL = 'Xenova/all-MiniLM-L6-v2'; - -// Environment setup -const dataDir = process.env.KG_DATA_PATH; -if (!dataDir) { - console.error('❌ KG_DATA_PATH environment variable is not set.'); - process.exit(1); -} -const EMBEDDING_FILE_PATH = path.join(dataDir, 'california_math_embeddings.json'); - -// Initialize embedding pipeline (will be loaded on first use) -let embedder = null; -let pipeline = null; - -/* ================================ - HELPER FUNCTIONS - ================================ */ - -function loadCSV(filename) { - try { - const content = fs.readFileSync(path.join(dataDir, filename), 'utf8'); - return parse(content, { columns: true, skip_empty_lines: true }); - } catch (error) { - console.error(`❌ Error loading CSV file ${filename}: ${error.message}`); - throw error; - } -} - -function findFrameworkItem(caseIdentifierUUID, standardsFrameworkItemsData) { - return standardsFrameworkItemsData.find(item => item.caseIdentifierUUID === caseIdentifierUUID); -} - - -/* ================================ - STEP 1: LOAD DATA - ================================ */ -function loadData(aq) { - /* Load CSV data files needed for the tutorial - */ - const standardsFrameworksData = loadCSV('StandardsFramework.csv'); - const standardsFrameworkItemsData = loadCSV('StandardsFrameworkItem.csv'); - - console.log('✅ Files loaded from KG CSV files'); - console.log({ - standardsFrameworks: standardsFrameworksData.length, - standardsFrameworkItems: standardsFrameworkItemsData.length - }); - - return { standardsFrameworksData, standardsFrameworkItemsData }; -} - -/* ================================ - STEP 2: QUERY FOR STANDARDS DATA - ================================ */ - -function getMathStandardsFrameworks(aq, standardsFrameworksData) { - /* Get snapshot of mathematics standards frameworks - * - * SQL: SELECT "name", "academicSubject", "jurisdiction", "identifier" - * FROM standards_framework - * WHERE "academicSubject" = 'Mathematics'; - * - * Cypher: MATCH (sf:StandardsFramework {academicSubject: 'Mathematics'}) - * RETURN sf.name, sf.academicSubject, sf.jurisdiction, sf.identifier - */ - const mathFrameworks = aq.from(standardsFrameworksData) - .filter(d => d.academicSubject === 'Mathematics') - .select('caseIdentifierUUID', 'name', 'jurisdiction', 'academicSubject'); - - console.log(`✅ Retrieved ${mathFrameworks.numRows()} state standard frameworks for math (dataframe):`); - console.log('Sample of first 5 frameworks:'); - console.log(mathFrameworks.slice(0,5).objects()); - - /* Get California math framework metadata - * - * SQL: SELECT * - * FROM standards_framework - * WHERE "jurisdiction" = 'California' - * AND "academicSubject" = 'Mathematics'; - * Cypher: MATCH (sf:StandardsFramework {jurisdiction: 'California', academicSubject: 'Mathematics'}) RETURN sf - */ - const californiaFrameworkTable = aq.from(standardsFrameworksData) - .filter(d => d.jurisdiction === 'California' && d.academicSubject === 'Mathematics') - .select('caseIdentifierUUID', 'name', 'jurisdiction', 'academicSubject'); - - const californiaFramework = californiaFrameworkTable.object(); - - console.log(`✅ Retrieved ${californiaFramework ? 1 : 0} California math standards framework:`); - if (californiaFramework) { - console.log(californiaFramework); - } - - return { mathFrameworks, californiaFramework }; -} - -function getMiddleSchoolStandardsGroupings(aq, standardsFrameworkItemsData) { - /* Filter for middle school standard groupings from California framework - * - * SQL: SELECT * - * FROM standards_framework_item - * WHERE "jurisdiction" = 'California' - * AND "academicSubject" = 'Mathematics' - * AND "normalizedStatementType" = 'Standard Grouping' - * AND ( - * EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '6') - * OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '7') - * OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '8') - * ); - * - * Cypher: MATCH (sfi:StandardsFrameworkItem) - * WHERE sfi.jurisdiction = 'California' - * AND sfi.academicSubject = 'Mathematics' - * AND sfi.normalizedStatementType = 'Standard Grouping' - * AND any(g IN sfi.gradeLevel WHERE g IN ['6','7','8']) - * RETURN sfi; - */ - const groupings = aq.from(standardsFrameworkItemsData) - .filter(aq.escape(item => item.jurisdiction === 'California' && - item.academicSubject === 'Mathematics' && - item.normalizedStatementType === 'Standard Grouping' && - (JSON.parse(item.gradeLevel || '[]')).some(level => MIDDLE_SCHOOL_GRADES.includes(level)))) - .select('caseIdentifierUUID', 'statementCode', 'description', 'normalizedStatementType', 'statementType', 'gradeLevel'); - - console.log(`✅ Retrieved ${groupings.numRows()} standard groupings for middle school math in California (dataframe):`); - console.log('Sample of first 5 standard groupings:'); - console.log(groupings.slice(0,5).objects()); - - return groupings; -} - -function getMiddleSchoolStandards(aq, standardsFrameworkItemsData) { - /* Get all standards for California middle school mathematics - * - * SQL: SELECT * - * FROM standards_framework_item - * WHERE "jurisdiction" = 'California' - * AND "academicSubject" = 'Mathematics' - * AND "normalizedStatementType" = 'Standard' - * AND ( - * EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '6') - * OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '7') - * OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '8') - * ); - * - * Cypher: MATCH (sfi:StandardsFrameworkItem) - * WHERE sfi.jurisdiction = 'California' - * AND sfi.academicSubject = 'Mathematics' - * AND sfi.normalizedStatementType = 'Standard' - * AND any(g IN sfi.gradeLevel WHERE g IN ['6','7','8']) - * RETURN sfi; - */ - const standards = aq.from(standardsFrameworkItemsData) - .filter(aq.escape(item => item.jurisdiction === 'California' && - item.academicSubject === 'Mathematics' && - item.normalizedStatementType === 'Standard' && - (JSON.parse(item.gradeLevel || '[]')).some(level => MIDDLE_SCHOOL_GRADES.includes(level)))) - .select('caseIdentifierUUID', 'statementCode', 'description', 'normalizedStatementType', 'gradeLevel'); - - console.log(`✅ Retrieved ${standards.numRows()} standards for California middle school mathematics (dataframe):`); - console.log('Sample of first 5 standards:'); - console.log(standards.slice(100, 105).objects()); - - return standards; -} - -function queryForStandardsData(aq, standardsFrameworksData, standardsFrameworkItemsData) { - const { mathFrameworks, californiaFramework } = getMathStandardsFrameworks(aq, standardsFrameworksData); - const groupings = getMiddleSchoolStandardsGroupings(aq, standardsFrameworkItemsData); - const standards = getMiddleSchoolStandards(aq, standardsFrameworkItemsData); - - return { californiaFramework, groupings, standards }; -} - -/* ================================ - STEP 3: EMBED STANDARD DATA - ================================ */ -function embedStandardData(aq, standardsFrameworkItemsData) { - // Generate embeddings for California middle school mathematics standards - const embeddingStandards = aq.from(standardsFrameworkItemsData) - .filter(aq.escape(item => item.jurisdiction === 'California' && - item.academicSubject === 'Mathematics' && - item.normalizedStatementType === 'Standard' && - (JSON.parse(item.gradeLevel || '[]')).some(level => MIDDLE_SCHOOL_GRADES.includes(level)) && - !!item.description)) - .objects(); - - /* Generate and save embeddings for each standard - * This creates vector representations of standard descriptions for semantic search - */ - return async function generateEmbeddings() { - const results = []; - console.log(`🔄 Generating embeddings for ${embeddingStandards.length} standards...`); - - // Initialize embedder if not already done - if (!embedder) { - console.log('📥 Loading embedding model (first time only)...'); - if (!pipeline) { - const { pipeline: pipelineImport } = await import('@xenova/transformers'); - pipeline = pipelineImport; - } - embedder = await pipeline('feature-extraction', EMBEDDING_MODEL); - console.log('✅ Embedding model loaded'); - } - - for (const standard of embeddingStandards) { - const code = standard.statementCode || '(no code)'; - - try { - const output = await embedder(standard.description, { pooling: 'mean', normalize: true }); - const embedding = Array.from(output.data); - - results.push({ - caseIdentifierUUID: standard.caseIdentifierUUID, - statementCode: standard.statementCode, - embedding: embedding - }); - - console.log(`✅ ${code}`); - } catch (err) { - console.error(`❌ ${code}: ${err.message}`); - throw err; - } - } - - // Save embeddings to file - fs.writeFileSync(EMBEDDING_FILE_PATH, JSON.stringify(results, null, 2)); - console.log(`✅ Saved ${results.length} embeddings to ${EMBEDDING_FILE_PATH}`); - }; -} - -/* ================================ - STEP 4: VECTOR SEARCH STANDARD DATA - ================================ */ -function vectorSearchStandardData(standardsFrameworkItemsData) { - /* Perform vector search using cosine similarity - */ - return async function vectorSearch(query, topK = 5) { - // Initialize embedder if not already done - if (!embedder) { - console.log('📥 Loading embedding model...'); - if (!pipeline) { - const { pipeline: pipelineImport } = await import('@xenova/transformers'); - pipeline = pipelineImport; - } - embedder = await pipeline('feature-extraction', EMBEDDING_MODEL); - console.log('✅ Embedding model loaded'); - } - - let queryEmbedding; - try { - const output = await embedder(query, { pooling: 'mean', normalize: true }); - queryEmbedding = Array.from(output.data); - } catch (error) { - console.error(`❌ Error generating embedding for query "${query}": ${error.message}`); - return; - } - - let storedEmbeddings; - try { - storedEmbeddings = JSON.parse(fs.readFileSync(EMBEDDING_FILE_PATH, 'utf8')); - } catch (error) { - console.error(`❌ Error loading embeddings from ${EMBEDDING_FILE_PATH}: ${error.message}`); - console.error('💡 Make sure to run the embedding generation step first (Step 3)'); - return; - } - - const topResults = storedEmbeddings - .map(item => ({ - caseIdentifierUUID: item.caseIdentifierUUID, - score: cosineSimilarity(queryEmbedding, item.embedding) - })) - .sort((a, b) => b.score - a.score) - .slice(0, topK); - - console.log(`\nTop ${topK} results for "${query}":\n`); - - - - topResults.forEach((result, i) => { - const frameworkItem = findFrameworkItem(result.caseIdentifierUUID, standardsFrameworkItemsData); - const statementCode = frameworkItem?.statementCode || '(no code)'; - const description = frameworkItem?.description || '(no statement)'; - - topResults[i].statementCode = statementCode; - topResults[i].description = description; - }); - - console.log(topResults); - }; -} - -async function main() { - const aq = await import('arquero'); - - console.log('\n=== WORKING WITH STATE STANDARDS TUTORIAL ===\n'); - - console.log('🔄 Step 1: Loading data...'); - const { standardsFrameworksData, standardsFrameworkItemsData } = loadData(aq); - - console.log('\n🔄 Step 2: Querying for standards data...'); - queryForStandardsData(aq, standardsFrameworksData, standardsFrameworkItemsData); - - console.log('\n🔄 Step 3: Embedding standard data...'); - const generateEmbeddings = embedStandardData(aq, standardsFrameworkItemsData); - if (GENERATE_EMBEDDINGS) { - await generateEmbeddings(); - } else { - console.log('🚫 Embedding generation disabled'); - } - - console.log('\n🔄 Step 4: Vector searching standard data...'); - const vectorSearch = vectorSearchStandardData(standardsFrameworkItemsData); - await vectorSearch('linear equations'); -} - -main().catch(console.error); \ No newline at end of file diff --git a/tutorials/working_with_standards/python/.env.example b/tutorials/working_with_standards/python/.env.example deleted file mode 100644 index f2d3f1f..0000000 --- a/tutorials/working_with_standards/python/.env.example +++ /dev/null @@ -1,2 +0,0 @@ -# Knowledge Graph data path - update with your actual path to CSV files -KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files \ No newline at end of file diff --git a/tutorials/working_with_standards/python/README.md b/tutorials/working_with_standards/python/README.md deleted file mode 100644 index ad4d586..0000000 --- a/tutorials/working_with_standards/python/README.md +++ /dev/null @@ -1,44 +0,0 @@ -Demonstrates how to work with state educational standards data covering: -- **Data Loading**: Reading CSV files with standards frameworks, items, and relationships -- **Hierarchical Queries**: Traversing parent-child relationships in educational standards -- **AI Embeddings**: Generating vector embeddings for semantic search -- **Vector Search**: Performing similarity-based searches on educational standards - -Follow the step-by-step tutorial [here](https://docs.learningcommons.org/knowledge-graph/v1-1-0/getting-started/tutorials/working-with-state-standards) - -## Prerequisites - -- Python 3.8 or higher -- Knowledge Graph CSV dataset files: - - `StandardsFramework.csv` - - `StandardsFrameworkItem.csv` - - `Relationships.csv` - -## Dependencies - -- **transformers**: Hugging Face transformers library for embeddings -- **pandas**: Data manipulation and analysis library -- **torch**: PyTorch for deep learning models -- **scikit-learn**: Machine learning library for cosine similarity -- **python-dotenv**: Environment variable management - -## Quick Start - -1. **Clone and Set Up Virtual Environment**: - ```bash - git clone git@github.com:learning-commons-org/knowledge-graph.git - cd tutorials/working_with_standards/python - python -m venv venv - source venv/bin/activate - pip install -r requirements.txt - ``` - -2. **Set Environment Variables** (create `.env` file): - ```bash - KG_DATA_PATH=/path/to/your/knowledge-graph/csv/files - ``` - -3. **Run Tutorial**: - ```bash - python work_with_state_standards.py - ``` diff --git a/tutorials/working_with_standards/python/requirements.txt b/tutorials/working_with_standards/python/requirements.txt deleted file mode 100644 index 7d90ef3..0000000 --- a/tutorials/working_with_standards/python/requirements.txt +++ /dev/null @@ -1,13 +0,0 @@ -# Core data manipulation and analysis -pandas>=2.0.0 - -# Environment variable management -python-dotenv>=1.0.0 - -# Machine learning and embeddings -transformers>=4.21.0 -torch>=1.12.0 -scikit-learn>=1.1.0 - -# Optional: For GPU acceleration (uncomment if needed) -# torch-audio>=0.12.0 \ No newline at end of file diff --git a/tutorials/working_with_standards/python/work_with_state_standards.py b/tutorials/working_with_standards/python/work_with_state_standards.py deleted file mode 100644 index 0ed7c62..0000000 --- a/tutorials/working_with_standards/python/work_with_state_standards.py +++ /dev/null @@ -1,434 +0,0 @@ -#!/usr/bin/env python3 -""" -================================ -CONFIGURATION & SETUP -================================ -""" - -# Dependencies -import os -import sys -import json -import pandas as pd -from pathlib import Path -from dotenv import load_dotenv -from transformers import AutoTokenizer, AutoModel -import torch -from sklearn.metrics.pairwise import cosine_similarity - -# Load environment variables -load_dotenv() - -# Domain Constants -GENERATE_EMBEDDINGS = True -MIDDLE_SCHOOL_GRADES = ['6', '7', '8'] -# For this tutorial, we use 'all-MiniLM-L6-v2' which provides good quality embeddings -# for short text. You can substitute any compatible embedding model. -EMBEDDING_MODEL = 'sentence-transformers/all-MiniLM-L6-v2' - -# Environment Setup -data_dir = os.getenv('KG_DATA_PATH') -if not data_dir: - print('❌ KG_DATA_PATH environment variable is not set.') - sys.exit(1) - -data_path = Path(data_dir) -EMBEDDING_FILE_PATH = data_path / 'california_math_embeddings.json' - -# Initialize embedding components (will be loaded on first use) -tokenizer = None -model = None - - -""" -================================ -HELPER FUNCTIONS -================================ -""" - -def load_csv(filename): - """Load and parse CSV file from data directory""" - try: - file_path = data_path / filename - return pd.read_csv(file_path) - except Exception as error: - print(f'❌ Error loading CSV file {filename}: {str(error)}') - raise error - - -def find_framework_item(case_identifier_uuid, standards_framework_items_data): - """Find a framework item by UUID""" - matching_items = standards_framework_items_data[ - standards_framework_items_data['caseIdentifierUUID'] == case_identifier_uuid - ] - if len(matching_items) > 0: - return matching_items.iloc[0] - return None - - -""" -================================ -STEP 1: LOAD DATA -================================ -""" - -def load_data(): - """Load CSV data files needed for the tutorial""" - standards_frameworks_data = load_csv('StandardsFramework.csv') - standards_framework_items_data = load_csv('StandardsFrameworkItem.csv') - - print('✅ Files loaded from KG CSV files') - print(f' Standards Frameworks: {len(standards_frameworks_data)}') - print(f' Standards Framework Items: {len(standards_framework_items_data)}') - - return {'standards_frameworks_data': standards_frameworks_data, - 'standards_framework_items_data': standards_framework_items_data} - - -""" -================================ -STEP 2: QUERY FOR STANDARDS DATA -================================ -""" - -def get_math_standards_frameworks(standards_frameworks_data): - """ - Get snapshot of mathematics standards frameworks - - SQL Equivalent: - SELECT "name", "academicSubject", "jurisdiction", "identifier" - FROM standards_framework - WHERE "academicSubject" = 'Mathematics'; - - Cypher Equivalent: - MATCH (sf:StandardsFramework {academicSubject: 'Mathematics'}) - RETURN sf.name, sf.academicSubject, sf.jurisdiction, sf.identifier - """ - math_frameworks = standards_frameworks_data[ - standards_frameworks_data['academicSubject'] == 'Mathematics' - ][['caseIdentifierUUID', 'name', 'jurisdiction', 'academicSubject']] - - print(f'✅ Retrieved {len(math_frameworks)} state standard frameworks for math (dataframe):') - print('Sample of first 5 frameworks:') - for i, (_, framework) in enumerate(math_frameworks.head(5).iterrows()): - print(f' {i+1}. {framework["jurisdiction"]}: {framework["name"]}') - - # Get California math framework metadata - # - # SQL: SELECT * - # FROM standards_framework - # WHERE "jurisdiction" = 'California' - # AND "academicSubject" = 'Mathematics'; - # Cypher: MATCH (sf:StandardsFramework {jurisdiction: 'California', academicSubject: 'Mathematics'}) RETURN sf - california_framework_df = standards_frameworks_data[ - (standards_frameworks_data['jurisdiction'] == 'California') & - (standards_frameworks_data['academicSubject'] == 'Mathematics') - ][['caseIdentifierUUID', 'name', 'jurisdiction', 'academicSubject']] - - california_framework = california_framework_df.iloc[0] if len(california_framework_df) > 0 else None - - print(f'✅ Retrieved {1 if california_framework is not None else 0} California math standards framework:') - if california_framework is not None: - print(f' UUID: {california_framework["caseIdentifierUUID"]}') - print(f' Name: {california_framework["name"]}') - print(f' Jurisdiction: {california_framework["jurisdiction"]}') - print(f' Academic Subject: {california_framework["academicSubject"]}') - - return {'math_frameworks': math_frameworks, 'california_framework': california_framework} - - -def get_middle_school_standards_groupings(standards_framework_items_data): - """ - Filter for middle school standard groupings from California framework - - SQL Equivalent: - SELECT * - FROM standards_framework_item - WHERE "jurisdiction" = 'California' - AND "academicSubject" = 'Mathematics' - AND "normalizedStatementType" = 'Standard Grouping' - AND ( - EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '6') - OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '7') - OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '8') - ); - - Cypher Equivalent: - MATCH (sfi:StandardsFrameworkItem) - WHERE sfi.jurisdiction = 'California' - AND sfi.academicSubject = 'Mathematics' - AND sfi.normalizedStatementType = 'Standard Grouping' - AND any(g IN sfi.gradeLevel WHERE g IN ['6','7','8']) - RETURN sfi; - """ - def has_middle_school_grade(grade_level_str): - try: - if pd.isna(grade_level_str): - return False - grade_levels = json.loads(grade_level_str) - return any(level in MIDDLE_SCHOOL_GRADES for level in grade_levels) - except: - return False - - ca_math_items = standards_framework_items_data[ - (standards_framework_items_data['jurisdiction'] == 'California') & - (standards_framework_items_data['academicSubject'] == 'Mathematics') & - (standards_framework_items_data['normalizedStatementType'] == 'Standard Grouping') - ] - - groupings = ca_math_items[ - ca_math_items['gradeLevel'].apply(has_middle_school_grade) - ][['caseIdentifierUUID', 'statementCode', 'description', 'normalizedStatementType', 'statementType', 'gradeLevel']] - - print(f'✅ Retrieved {len(groupings)} standard groupings for middle school math in California (dataframe):') - print('Sample of first 5 standard groupings:') - for i, (_, grouping) in enumerate(groupings.head(5).iterrows()): - print(f' {i+1}. {grouping["statementCode"]}: {grouping["description"][:100]}...') - - return groupings - - -def get_middle_school_standards(standards_framework_items_data): - """ - Get all standards for California middle school mathematics - - SQL Equivalent: - SELECT * - FROM standards_framework_item - WHERE "jurisdiction" = 'California' - AND "academicSubject" = 'Mathematics' - AND "normalizedStatementType" = 'Standard' - AND ( - EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '6') - OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '7') - OR EXISTS (SELECT 1 FROM json_array_elements_text("gradeLevel") AS elem WHERE elem = '8') - ); - - Cypher Equivalent: - MATCH (sfi:StandardsFrameworkItem) - WHERE sfi.jurisdiction = 'California' - AND sfi.academicSubject = 'Mathematics' - AND sfi.normalizedStatementType = 'Standard' - AND any(g IN sfi.gradeLevel WHERE g IN ['6','7','8']) - RETURN sfi; - """ - def has_middle_school_grade(grade_level_str): - try: - if pd.isna(grade_level_str): - return False - grade_levels = json.loads(grade_level_str) - return any(level in MIDDLE_SCHOOL_GRADES for level in grade_levels) - except: - return False - - ca_math_items = standards_framework_items_data[ - (standards_framework_items_data['jurisdiction'] == 'California') & - (standards_framework_items_data['academicSubject'] == 'Mathematics') & - (standards_framework_items_data['normalizedStatementType'] == 'Standard') - ] - - standards = ca_math_items[ - ca_math_items['gradeLevel'].apply(has_middle_school_grade) - ][['caseIdentifierUUID', 'statementCode', 'description', 'normalizedStatementType', 'gradeLevel']] - - print(f'✅ Retrieved {len(standards)} standards for California middle school mathematics (dataframe):') - print('Sample of first 5 standards:') - # Show slice from 100-105 like JS version - start_idx = min(100, len(standards)) - end_idx = min(105, len(standards)) - sample_standards = standards.iloc[start_idx:end_idx] - - for i, (_, standard) in enumerate(sample_standards.iterrows(), start_idx + 1): - print(f' {i}. {standard["statementCode"]}: {standard["description"][:100] if pd.notna(standard["description"]) else "No description"}...') - - return standards - - -def query_for_standards_data(standards_frameworks_data, standards_framework_items_data): - """Query and organize standards data""" - math_data = get_math_standards_frameworks(standards_frameworks_data) - groupings = get_middle_school_standards_groupings(standards_framework_items_data) - standards = get_middle_school_standards(standards_framework_items_data) - - return { - 'california_framework': math_data['california_framework'], - 'groupings': groupings, - 'standards': standards - } - - -""" -================================ -STEP 3: EMBED STANDARD DATA -================================ -""" - -def embed_standard_data(standards_framework_items_data): - """Generate embeddings for California middle school mathematics standards""" - - def has_middle_school_grade(grade_level_str): - try: - if pd.isna(grade_level_str): - return False - grade_levels = json.loads(grade_level_str) - return any(level in MIDDLE_SCHOOL_GRADES for level in grade_levels) - except: - return False - - # Filter for embedding standards - ca_math_standards = standards_framework_items_data[ - (standards_framework_items_data['jurisdiction'] == 'California') & - (standards_framework_items_data['academicSubject'] == 'Mathematics') & - (standards_framework_items_data['normalizedStatementType'] == 'Standard') & - (standards_framework_items_data['gradeLevel'].apply(has_middle_school_grade)) & - (standards_framework_items_data['description'].notna()) - ] - - embedding_standards = ca_math_standards.to_dict('records') - - def generate_embeddings(): - """Generate and save embeddings for each standard""" - global tokenizer, model - - results = [] - print(f'🔄 Generating embeddings for {len(embedding_standards)} standards...') - - # Initialize embedding model if not already done - if tokenizer is None or model is None: - print('📥 Loading embedding model (first time only)...') - tokenizer = AutoTokenizer.from_pretrained(EMBEDDING_MODEL) - model = AutoModel.from_pretrained(EMBEDDING_MODEL) - print('✅ Embedding model loaded') - - for standard in embedding_standards: - code = standard.get('statementCode') or '(no code)' - - try: - # Generate embedding - inputs = tokenizer(standard['description'], return_tensors='pt', padding=True, truncation=True) - with torch.no_grad(): - outputs = model(**inputs) - # Mean pooling - embedding = outputs.last_hidden_state.mean(dim=1).squeeze().numpy() - - results.append({ - 'caseIdentifierUUID': standard['caseIdentifierUUID'], - 'statementCode': standard['statementCode'], - 'embedding': embedding.tolist() - }) - - print(f'✅ {code}') - except Exception as err: - print(f'❌ {code}: {str(err)}') - raise err - - # Save embeddings to file - with open(EMBEDDING_FILE_PATH, 'w') as f: - json.dump(results, f, indent=2) - print(f'✅ Saved {len(results)} embeddings to {EMBEDDING_FILE_PATH}') - - return generate_embeddings - - -""" -================================ -STEP 4: VECTOR SEARCH STANDARD DATA -================================ -""" - -def vector_search_standard_data(standards_framework_items_data): - """Perform vector search using cosine similarity""" - - def vector_search(query, top_k=5): - global tokenizer, model - - # Initialize embedding model if not already done - if tokenizer is None or model is None: - print('📥 Loading embedding model...') - tokenizer = AutoTokenizer.from_pretrained(EMBEDDING_MODEL) - model = AutoModel.from_pretrained(EMBEDDING_MODEL) - print('✅ Embedding model loaded') - - # Generate query embedding - try: - inputs = tokenizer(query, return_tensors='pt', padding=True, truncation=True) - with torch.no_grad(): - outputs = model(**inputs) - query_embedding = outputs.last_hidden_state.mean(dim=1).squeeze().numpy() - except Exception as error: - print(f'❌ Error generating embedding for query "{query}": {str(error)}') - return - - # Load stored embeddings - try: - with open(EMBEDDING_FILE_PATH, 'r') as f: - stored_embeddings = json.load(f) - except Exception as error: - print(f'❌ Error loading embeddings from {EMBEDDING_FILE_PATH}: {str(error)}') - print('💡 Make sure to run the embedding generation step first (Step 3)') - return - - # Calculate similarities and get top results - similarities = [] - for item in stored_embeddings: - similarity = cosine_similarity([query_embedding], [item['embedding']])[0][0] - similarities.append({ - 'caseIdentifierUUID': item['caseIdentifierUUID'], - 'score': float(similarity) - }) - - top_results = sorted(similarities, key=lambda x: x['score'], reverse=True)[:top_k] - - print(f'\nTop {top_k} results for "{query}":\n') - - # Add framework item details to results - for i, result in enumerate(top_results): - framework_item = find_framework_item(result['caseIdentifierUUID'], standards_framework_items_data) - - statement_code = framework_item['statementCode'] if framework_item is not None and pd.notna(framework_item['statementCode']) else '(no code)' - description = framework_item['description'] if framework_item is not None and pd.notna(framework_item['description']) else '(no statement)' - - top_results[i]['statementCode'] = statement_code - top_results[i]['description'] = description - - # Print results in readable format - for i, result in enumerate(top_results, 1): - print(f' {i}. {result["statementCode"]} (Score: {result["score"]:.4f})') - print(f' {result["description"][:200]}...' if len(result["description"]) > 200 else f' {result["description"]}') - print() - - return vector_search - - -""" -================================ -MAIN EXECUTION -================================ -""" - -def main(): - """Main execution function - orchestrates all tutorial steps""" - print('\n=== WORKING WITH STATE STANDARDS TUTORIAL ===\n') - - print('🔄 Step 1: Loading data...') - data = load_data() - standards_frameworks_data = data['standards_frameworks_data'] - standards_framework_items_data = data['standards_framework_items_data'] - - print('\n🔄 Step 2: Querying for standards data...') - query_for_standards_data(standards_frameworks_data, standards_framework_items_data) - - print('\n🔄 Step 3: Embedding standard data...') - generate_embeddings = embed_standard_data(standards_framework_items_data) - if GENERATE_EMBEDDINGS: - generate_embeddings() - else: - print('🚫 Embedding generation disabled') - - print('\n🔄 Step 4: Vector searching standard data...') - vector_search = vector_search_standard_data(standards_framework_items_data) - vector_search('linear equations') - - -if __name__ == '__main__': - main() \ No newline at end of file