Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,12 @@
# DLSync Changelog

This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [3.2.0] - 2026-02-27
### Added
- Added PLAN command for deployment preview (dry-run without database modifications)
- Added support for TAGS object type
- Added support for ROW_ACCESS_POLICIES object type
## [3.1.0] - 2026-02-10
### Added
- Added support for AGENTS object type
Expand Down
117 changes: 117 additions & 0 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,117 @@
# Contributing to DLSync

Thank you for your interest in contributing to DLSync! We welcome contributions from the community.

## Prerequisites

- Java 11 or higher
- Gradle 7.0 or higher
- Git
- Snowflake account (for testing changes)

## Quick Start

### 1. Fork the Repository

1. Click **Fork** on the [DLSync GitHub repository](https://github.com/Snowflake-Labs/dlsync)
2. This creates a copy under your GitHub account

### 2. Clone Your Fork

```bash
git clone https://github.com/YOUR_USERNAME/dlsync.git
cd dlsync
```

### 3. Add Upstream Remote

```bash
git remote add upstream https://github.com/Snowflake-Labs/dlsync.git
```

---

## Making Changes

### 1. Create a Feature Branch

```bash
# Update your local main branch
git fetch upstream
git checkout main
git rebase upstream/main

# Create a new branch for your changes
git checkout -b feature/your-feature-name
```

### 2. Make Your Changes

- Edit the necessary files
- Keep changes focused and concise
- Follow the existing code style

### 3. Test Your Changes

```bash
# Run tests
./gradlew test

# Build the project
./gradlew build
```

### 4. Update Documentation

- Update `README.md` if user-facing changes
- add changelog to `CHANGELOG.md`

### 5. Commit Your Changes

```bash
git add .
git commit -m "Description of your changes"
```

---

## Creating a Pull Request

### 1. Push Your Branch

```bash
git push origin feature/your-feature-name
```

### 2. Create Pull Request on GitHub

1. Go to your forked repository on GitHub: `https://github.com/YOUR_USERNAME/dlsync`
2. You should see a notification: "Your branch is ahead of Snowflake-Labs:main"
3. Click the **Compare & pull request** button (or go to **Pull Requests** tab → **New Pull Request**)
4. Verify the pull request details:
- **Base repository**: `Snowflake-Labs/dlsync`
- **Base branch**: `main`
- **Head repository**: `YOUR_USERNAME/dlsync`
- **Compare branch**: `feature/your-feature-name`
5. Add a title and description:
```
Title: [FEATURE|FIX|DOCS] Brief description of changes

Description:
- What changes are included
- Why this change is needed
- Any related issues (e.g., Closes #123)
```
6. Click **Create Pull Request**

### 4. Review Process

- A maintainer will review your PR
- CI/CD checks must pass
- Address any feedback or changes requested
- Once approved, your PR will be merged!

---

Thank you for contributing! 🙏

73 changes: 64 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ DLSync also understands interdependencies between different scripts, thus applyi
1. [How to use this tool](#how-to-use-this-tool)
1. [Deploy](#deploy)
1. [Test](#test)
1. [Plan](#plan)
1. [Rollback](#rollback)
1. [Verify](#verify)
1. [Create script](#create-script)
Expand All @@ -41,6 +42,7 @@ DLSync also understands interdependencies between different scripts, thus applyi
1. [dl_sync_change_sync](#dl_sync_change_sync)
1. [dl_sync_script_event](#dl_sync_script_event)
1. [Example scripts](#example-scripts)
1. [Contributing](#contributing)

## Key Features
- Hybrid Change Management: It combines declarative and migration based change management to manage database changes
Expand Down Expand Up @@ -113,7 +115,7 @@ Where
- Other account-level objects (users, resource monitors, integrations, etc.)
- **database_name_*:** is the database name of your project,
- **schema_name_*:** are schemas inside the database,
- **object_type:** is type of the object only 1 of the following (VIEWS, FUNCTIONS, PROCEDURES, FILE_FORMATS, TABLES, SEQUENCES, STAGES, STREAMS, TASKS, STREAMLITS, PIPES, ALERTS, DYNAMIC_TABLES, MASKING_POLICIES, NOTEBOOKS, CORTEX_SEARCH_SERVICES, SEMANTIC_VIEWS, AGENTS),
- **object_type:** is type of the object only 1 of the following (VIEWS, FUNCTIONS, PROCEDURES, FILE_FORMATS, TABLES, SEQUENCES, STAGES, STREAMS, TASKS, STREAMLITS, PIPES, ALERTS, DYNAMIC_TABLES, MASKING_POLICIES, ROW_ACCESS_POLICIES, TAGS, NOTEBOOKS, CORTEX_SEARCH_SERVICES, SEMANTIC_VIEWS, AGENTS),
- **object_name_*.sql:** are individual database object scripts.
- **config.yml:** is a configuration file used to configure DLSync behavior.
- **parameter-[profile-*].properties:** is parameter to value map file. This is going to be used by corresponding individual instances of your database.
Expand All @@ -131,7 +133,7 @@ For example, if you have a view named `SAMPLE_VIEW` in schema `MY_SCHEMA` in dat

The structure and content of the scripts differ based on the type of script. This tool categorizes scripts into 2 types: Declarative scripts and Migration scripts.
#### 1. Declarative Script
This type of script is used for object types of VIEWS, FUNCTIONS, PROCEDURES, FILE_FORMATS, PIPES, MASKING_POLICIES, NOTEBOOKS, CORTEX_SEARCH_SERVICES, SEMANTIC_VIEWS, AGENTS, STREAMLITS, RESOURCE_MONITORS, NETWORK_POLICIES, SESSION_POLICIES, PASSWORD_POLICIES, AUTHENTICATION_POLICIES, API_INTEGRATIONS, NOTIFICATION_INTEGRATIONS, SECURITY_INTEGRATIONS, STORAGE_INTEGRATIONS, and WAREHOUSES.
This type of script is used for object types of VIEWS, FUNCTIONS, PROCEDURES, FILE_FORMATS, PIPES, MASKING_POLICIES, ROW_ACCESS_POLICIES, TAGS, NOTEBOOKS, CORTEX_SEARCH_SERVICES, SEMANTIC_VIEWS, AGENTS, STREAMLITS, RESOURCE_MONITORS, NETWORK_POLICIES, SESSION_POLICIES, PASSWORD_POLICIES, AUTHENTICATION_POLICIES, API_INTEGRATIONS, NOTIFICATION_INTEGRATIONS, SECURITY_INTEGRATIONS, STORAGE_INTEGRATIONS, and WAREHOUSES.
In this type of script, you define the current state (desired state) of the object.
When a change is made to the script, DLSync replaces the current object with the updated definition.
These types of scripts must always have a `create or replace` statement. Every time you make a change to the script, DLSync will replace the object with the new definition.
Expand Down Expand Up @@ -230,9 +232,9 @@ Writing unit tests follows a 3-step process:
- Add the query to refer to the database object with a select statement.
For example, if you have a view named `SAMPLE_VIEW` with the following content:
```
create or replace view ${MY_DB}.{MY_SCHEMA}.SAMPLE_VIEW as
select tb1.id, tb1.my_column || '->' || tb2.my_column as my_new_column from ${MY_DB}.{MY_SECOND_SCHEMA}.MY_TABLE_1 tb_1
join ${MY_DB}.{MY_SECOND_SCHEMA}.MY_TABLE_2 tb2
create or replace view ${MY_DB}.${MY_SCHEMA}.SAMPLE_VIEW as
select tb1.id, tb1.my_column || '->' || tb2.my_column as my_new_column from ${MY_DB}.${MY_SECOND_SCHEMA}.MY_TABLE_1 tb_1
join ${MY_DB}.${MY_SECOND_SCHEMA}.MY_TABLE_2 tb2
on tb1.id = tb2.id;
```

Expand All @@ -250,7 +252,7 @@ MY_TABLE_2 as (
expected_data as (
select '1' as id, 'old_value1->new_value1' as my_new_column
)
select * from ${MY_DB}.{MY_SCHEMA}.SAMPLE_VIEW;
select * from ${MY_DB}.${MY_SCHEMA}.SAMPLE_VIEW;
```
Then dlsync will generate a query with the following content to validate the test:
```
Expand All @@ -267,8 +269,8 @@ expected_data as (
select '1' as id, 'old_value1->new_value1' as my_new_column
),
ACTUAL_DATA as (
select tb1.id, tb1.my_column || '->' || tb2.my_column as my_new_column from ${MY_DB}.{MY_SECOND_SCHEMA}.MY_TABLE_1 tb_1
join ${MY_DB}.{MY_SECOND_SCHEMA}.MY_TABLE_2 tb2
select tb1.id, tb1.my_column || '->' || tb2.my_column as my_new_column from ${MY_DB}.${MY_SECOND_SCHEMA}.MY_TABLE_1 tb_1
join ${MY_DB}.${MY_SECOND_SCHEMA}.MY_TABLE_2 tb2
on tb1.id = tb2.id
),
assertion as (
Expand Down Expand Up @@ -408,6 +410,53 @@ The test module can be triggered using the following command:
dlsync test -s path/to/db_scripts -p dev
```

#### Plan
This module is used to preview changes before deploying them to the database. It shows what changes would be made without actually executing them, allowing you to review and understand the impact before committing to the deployment.
The plan command performs a dry-run analysis that:
- Identifies which scripts have changed since the last deployment
- Shows the dependency order in which changes would be applied
- Displays the complete content of all scripts that would be executed
- Provides metadata about each script (type, version, author, hash)
- **Does NOT modify the database in any way** (read-only operation)

The plan module can be triggered using the following command:
```
dlsync plan -s path/to/db_scripts -p dev
```

**Output Example:**
```
========== DEPLOYMENT PLAN ==========
Total changes to deploy: 2
========== DEPLOYMENT ORDER ==========
1 of 2: MY_DB.MY_SCHEMA.SAMPLE_TABLE
Type: Migration
Object: MY_DB.MY_SCHEMA.SAMPLE_TABLE (Version: 1)
Author: john.doe@company.com
Hash: a1b2c3d4e5f6g7h8
Content:
CREATE TABLE MY_DB.MY_SCHEMA.SAMPLE_TABLE(id VARCHAR, my_column VARCHAR);
========== END CONTENT ==========
2 of 2: MY_DB.MY_SCHEMA.SAMPLE_VIEW
Type: Declarative
Object: MY_DB.MY_SCHEMA.SAMPLE_VIEW
Object Type: VIEW
Hash: b2c3d4e5f6g7h8i9
Content:
CREATE VIEW MY_DB.MY_SCHEMA.SAMPLE_VIEW AS SELECT * FROM MY_DB.MY_SCHEMA.SAMPLE_TABLE;
========== END CONTENT ==========
========== END PLAN ==========
```

**Use Cases:**
- Review changes before deployment in development/staging environments
- Audit what will be deployed to production
- Integrate with CI/CD pipelines as a pre-deployment validation step
- Generate deployment plans for approval workflows
- Verify script dependencies are correct before execution

**Note:** The plan command is read-only and safe to run multiple times without side effects. However, it will create the necessary DLSync metadata tables if they don't already exist in the target schema.

#### Rollback
This module is used to rollback changes to the previous deployment. It will rollback the changes to the database objects based on the script files. This should be triggered after you have rolled back the git repository of the script files.
The rollback works first by identifying the changes between the current deployment and the previous deployment. For declarative scripts (views, udf, stored procedures and file formats) it will replace them with the current script(i.e previous version as you have already made git rollback).
Expand Down Expand Up @@ -450,7 +499,7 @@ The role must have privileges to create and manage DLSync tracking tables in the
> **Note:** the database and schema specified in the connection will be used to store the DLSync tracking tables.

### Database-Level Objects
For managing database-level objects (VIEWS, FUNCTIONS, PROCEDURES, TABLES, SEQUENCES, STAGES, STREAMS, TASKS, ALERTS, DYNAMIC_TABLES, FILE_FORMATS, PIPES, MASKING_POLICIES, SESSION_POLICIES, PASSWORD_POLICIES, AUTHENTICATION_POLICIES, NOTEBOOKS), the role must have:
For managing database-level objects (VIEWS, FUNCTIONS, PROCEDURES, TABLES, SEQUENCES, STAGES, STREAMS, TASKS, ALERTS, DYNAMIC_TABLES, FILE_FORMATS, PIPES, MASKING_POLICIES, ROW_ACCESS_POLICIES, TAGS, SESSION_POLICIES, PASSWORD_POLICIES, AUTHENTICATION_POLICIES, NOTEBOOKS), the role must have:
- **USAGE** on the target schema
- **CREATE** privileges for the specific object types being deployed (CREATE VIEW, CREATE FUNCTION, CREATE TABLE, etc.)
- **ALTER** privileges on existing objects in the schema
Expand Down Expand Up @@ -517,3 +566,9 @@ created_ts: the timestamp when was this change added
```
## Example scripts
To explore the tool, you can use the example scripts provided in the `example_scripts` directory.

## Contributing

We welcome contributions from the community! Whether you're fixing bugs, adding features, or improving documentation, your help is appreciated.

Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on how to contribute to this project.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
CREATE OR REPLACE ROW ACCESS POLICY ${EXAMPLE_DB}.${MAIN_SCHEMA}.CUSTOMER_ACCESS
AS (customer_region VARCHAR)
RETURNS BOOLEAN ->
CASE
WHEN CURRENT_ROLE() IN ('ADMIN', 'DATA_ADMIN') THEN TRUE
WHEN CURRENT_ROLE() = 'REGIONAL_ANALYST' AND customer_region = CURRENT_USER()
THEN TRUE
ELSE FALSE
END;
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
create or replace TAG ${EXAMPLE_DB}.${MAIN_SCHEMA}.COLUMN_CLASSIFICATION allowed_values 'PUBLIC', 'INTERNAL', 'RESTRICTED', 'SENSITIVE';
65 changes: 63 additions & 2 deletions src/main/java/com/snowflake/dlsync/ChangeManager.java
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,69 @@ public void deploy(boolean onlyHashes) throws SQLException, IOException, NoSuchA
}
}

public void plan() throws SQLException, IOException, NoSuchAlgorithmException {
log.info("Started PLAN command for deployment");

try {
scriptRepo.loadScriptHash();

// Get all changed scripts (same logic as deploy)
List<Script> changedScripts = scriptSource.getAllScripts()
.stream()
.filter(script -> !config.isScriptExcluded(script))
.filter(script -> scriptRepo.isScriptChanged(script))
.collect(Collectors.toList());
dependencyGraph.addNodes(changedScripts);
List<Script> sequencedScripts = dependencyGraph.topologicalSort();

// Show deployment summary
System.out.println("========== DEPLOYMENT PLAN ==========");
System.out.println("Total changes to deploy: " + changedScripts.size());
System.out.println("========== DEPLOYMENT ORDER ==========");

if (changedScripts.isEmpty()) {
System.out.println("PLAN: No changes to deploy");
System.out.println("========== END PLAN ==========");
return;
}

int index = 1;
for (Script script : sequencedScripts) {
parameterInjector.injectParameters(script);
validateScript(script);
System.out.println(index + " of " + sequencedScripts.size() + ": " + script);

if (script instanceof MigrationScript) {
MigrationScript migration = (MigrationScript) script;
System.out.println(" Type: Migration");
System.out.println(" Object: " + migration.getFullObjectName() + " (Version: " + migration.getVersion() + ")");
System.out.println(" Author: " + (migration.getAuthor() != null ? migration.getAuthor() : "N/A"));
System.out.println(" Hash: " + script.getHash());
} else {
System.out.println(" Type: Declarative");
System.out.println(" Object: " + script.getFullObjectName());
System.out.println(" Object Type: " + script.getObjectType());
System.out.println(" Hash: " + script.getHash());
}

// Show full SQL content without truncation
System.out.println(" Content:");
System.out.println(script.getContent());
System.out.println("========== END CONTENT ==========");

index++;
}

System.out.println("========== END PLAN ==========");

} catch (Exception e) {
System.err.println("ERROR: Plan command failed: " + e.getMessage());
log.error("Plan command failed: {}", e.getMessage());
System.out.println("========== END PLAN ==========");
throw e;
}
}

public void rollback() throws SQLException, IOException {
log.info("Starting ROLLBACK scripts.");
startSync(ChangeType.ROLLBACK);
Expand Down Expand Up @@ -282,7 +345,5 @@ public void endSyncError(ChangeType changeType, String message) throws SQLExcept
public void endSyncSuccess(ChangeType changeType, Long changeCount) throws SQLException {
scriptRepo.updateChangeSync(changeType, Status.SUCCESS, "Successfully completed " + changeType.toString() , changeCount);
}


}

6 changes: 5 additions & 1 deletion src/main/java/com/snowflake/dlsync/Main.java
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,10 @@ public static void main(String[] args) throws SQLException {
changeManager.test();
log.info("DLsync successfully tested.");
break;
case PLAN:
changeManager.plan();
log.info("DLSync plan completed successfully.");
break;
default:
log.error("Change type not specified as an argument.");
}
Expand Down Expand Up @@ -103,7 +107,7 @@ public static CommandLine buildCommandOptions(String[] args) throws ParseExcepti
CommandLine commandLine = new DefaultParser().parse(options, argsWithoutCommand);
return commandLine;
} catch (ParseException e) {
new HelpFormatter().printHelp("dlsync [deploy|rollback|verify|create-script|create-lineage] [options]", "options:", options, "");
new HelpFormatter().printHelp("dlsync [deploy|rollback|verify|create-script|create-lineage|test|plan] [options]", "options:", options, "");
throw e;
}

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
package com.snowflake.dlsync.models;

public enum ChangeType {
DEPLOY, VERIFY, ROLLBACK, CREATE_SCRIPT, CREATE_LINEAGE, TEST
DEPLOY, VERIFY, ROLLBACK, CREATE_SCRIPT, CREATE_LINEAGE, TEST, PLAN
};
Loading