MTSWebServices · annamikhailovapsk · Mar 20, 2026 · Mar 20, 2026
diff --git a/mddocs/Makefile b/mddocs/Makefile
@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = .
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
diff --git a/mddocs/_static/architecture.png b/mddocs/_static/architecture.png
diff --git a/mddocs/_static/custom.css b/mddocs/_static/custom.css
@@ -0,0 +1,3 @@
+.logo {
+  width: 200px !important;
+}
diff --git a/mddocs/_static/openapi.json b/mddocs/_static/openapi.json
@@ -0,0 +1,9 @@
+{
+  "openapi": "3.1.0",
+  "version": "unknown",
+  "info": {
+    "title": "Generated in CI",
+    "version": "unknown"
+  },
+  "paths": {}
+}
diff --git a/mddocs/_static/redoc.html b/mddocs/_static/redoc.html
@@ -0,0 +1,28 @@
+
+<!DOCTYPE html>
+<html>
+  <head>
+    <title>SyncMaster - ReDoc</title>
+    <!-- needed for adaptive design -->
+    <meta charset="utf-8"/>
+    <meta name="viewport" content="width=device-width, initial-scale=1">
+
+    <link rel="shortcut icon" href="../../_static/icon.svg">
+    <!--
+    ReDoc doesn't change outer page styles
+    -->
+    <style>
+      body {
+        margin: 0;
+        padding: 0;
+      }
+    </style>
+  </head>
+  <body>
+    <noscript>
+        ReDoc requires Javascript to function. Please enable it to browse the documentation.
+    </noscript>
+    <redoc spec-url="../../_static/openapi.json"></redoc>
+    <script src="https://cdn.jsdelivr.net/npm/redoc@next/bundles/redoc.standalone.js"></script>
+  </body>
+</html>
diff --git a/mddocs/_static/swagger.html b/mddocs/_static/swagger.html
@@ -0,0 +1,26 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+  <meta charset="utf-8" />
+  <meta name="viewport" content="width=device-width, initial-scale=1" />
+  <meta
+    name="description"
+    content="SwaggerUI"
+  />
+  <title>SwaggerUI</title>
+  <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/swagger-ui-dist@5/swagger-ui.css" />
+  <link rel="shortcut icon" href="../../_static/icon.svg">
+</head>
+<body>
+  <div id="swagger-ui"></div>
+  <script src="https://cdn.jsdelivr.net/npm/swagger-ui-dist@5/swagger-ui-bundle.js" crossorigin></script>
+  <script>
+    window.onload = () => {
+      window.ui = SwaggerUIBundle({
+        url: '../../_static/openapi.json',
+        dom_id: '#swagger-ui',
+      });
+    };
+  </script>
+</body>
+</html>
diff --git a/mddocs/changelog.md b/mddocs/changelog.md
@@ -0,0 +1,13 @@
+# Changelog { #changelog }
+
+- [0.2.5 [UNRELEASED] (2025-09-30)][DRAFT]
+- [0.2.4 (2025-07-09)][0.2.4]
+- [0.2.3 (2025-04-11)][0.2.3]
+- [0.2.2 (2025-04-11)][0.2.2]
+- [0.2.1 (2025-04-07)][0.2.1]
+- [0.2.0 (2025-04-04)][0.2.0]
+- [0.1.5 (2024-04-22)][0.1.5]
+- [0.1.4 (2024-04-15)][0.1.4]
+- [0.1.3 (2024-04-15)][0.1.3]
+- [0.1.2 (2024-04-15)][0.1.2]
+- [0.1.1 (2024-03-29)][0.1.1]
diff --git a/mddocs/changelog/0.1.1.md b/mddocs/changelog/0.1.1.md
@@ -0,0 +1,3 @@
+# 0.1.1 (2024-03-29) { #0.1.1 }
+
+SyncMaster is now open source!
diff --git a/mddocs/changelog/0.1.2.md b/mddocs/changelog/0.1.2.md
@@ -0,0 +1,12 @@
+# 0.1.2 (2024-04-15) { #0.1.2 }
+
+## Features
+
+- Rename `syncmaster.backend.main` module to `syncmaster.backend`. It now also accepts the same args as `uvicorn`. ([#35](https://github.com/MobileTeleSystems/syncmaster/issues/35))
+- Add `syncmaster.db.migrations` module to run `alembic` with proper config. ([#35](https://github.com/MobileTeleSystems/syncmaster/issues/35))
+- Change backend and worker images to use entrypoint. ([#35](https://github.com/MobileTeleSystems/syncmaster/issues/35))
+
+## Improvements
+
+- Run database migrations in the entrypoint of backend image, before starting the backend server. ([#35](https://github.com/MobileTeleSystems/syncmaster/issues/35))
+- Add healthchecks to example `docker-compose.yml`. ([#35](https://github.com/MobileTeleSystems/syncmaster/issues/35))
diff --git a/mddocs/changelog/0.1.3.md b/mddocs/changelog/0.1.3.md
@@ -0,0 +1,5 @@
+# 0.1.3 (2024-04-15) { #0.1.3 }
+
+## Bug Fixes
+
+Fix backend image entrypoint.
diff --git a/mddocs/changelog/0.1.4.md b/mddocs/changelog/0.1.4.md
@@ -0,0 +1,6 @@
+# 0.1.4 (2024-04-15) { #0.1.4 }
+
+## Bug Fixes
+
+- Fix missing backend factory for uvicorn.
+- Fix missing `kinit` executable in worker image.
diff --git a/mddocs/changelog/0.1.5.md b/mddocs/changelog/0.1.5.md
@@ -0,0 +1,17 @@
+# 0.1.5 (2024-04-22) { #0.1.5 }
+
+## Breaking Changes
+
+- Pass current `Run` to `CREATE_SPARK_SESSION_FUNCTION`. This allows using run/transfer/group information for Spark session options,
+  like `appName` or custom ones. ([#38](https://github.com/MobileTeleSystems/syncmaster/issues/38))
+
+## Improvements
+
+- Reduce backend image size ([#44](https://github.com/MobileTeleSystems/syncmaster/issues/44))
+
+## Bug Fixes
+
+- Fix 500 error in case of `PATCH v1/connections/:id` request with passed `auth_data.password` field value ([#39](https://github.com/MobileTeleSystems/syncmaster/issues/39))
+- Do not use `asyncio.gather` with SQLAlchemy requests ([#40](https://github.com/MobileTeleSystems/syncmaster/issues/40))
+- Fix 500 error while creating HDFS connection ([#41](https://github.com/MobileTeleSystems/syncmaster/issues/41))
+- Fix missing `options` field from Transfer params with `hdfs` and `s3` type
diff --git a/mddocs/changelog/0.2.0.md b/mddocs/changelog/0.2.0.md
@@ -0,0 +1,69 @@
+# 0.2.0 (2025-04-04) { #0.2.0 }
+
+## TL;DR
+
+- Completely new UI.
+- Add support for FileSystem connections.
+- Add support for simple transformations within transfer - filter files, filter rows, change columns.
+- Add support for incremental read strategy.
+- Add support for running transfers by schedule.
+- Add support for changing SparkSession resource limits.
+
+Huge thanks to [Dmitry Pedchenko](https://github.com/dmitry-pedchenko), [Maxim Lixakov](https://github.com/maxim-lixakov), [Ilyas Gasanov](https://github.com/IlyasDevelopment), [Kirill Chernikov](https://github.com/Zabilsya).
+
+## Breaking Changes
+
+- Implement a single error handling format to improve consistency ([#95](https://github.com/MobileTeleSystems/syncmaster/issues/95))
+- Change response format for `GET /v1/groups` - add **current user role** for each group ([#97](https://github.com/MobileTeleSystems/syncmaster/issues/97))
+- Change response format for `GET /v1/groups/:id` - add **current user role** for group ([#109](https://github.com/MobileTeleSystems/syncmaster/issues/109))
+- Now migrations are executed in a dedicated one-off container, instead of being run as a part of `backend` container. ([#163](https://github.com/MobileTeleSystems/syncmaster/issues/163))
+- Delete **transfers**, **queues** and **groups** records instead of marking them as deleted ([#168](https://github.com/MobileTeleSystems/syncmaster/issues/168))
+- Move the `type` field from nested `connection_data` field to the root level of the `Connection` response. ([#169](https://github.com/MobileTeleSystems/syncmaster/issues/169))
+- Decouple `auth_data.type` from the connection type (e.g. `postgres`) and link it to the authentication type (e.g. `basic`). ([#169](https://github.com/MobileTeleSystems/syncmaster/issues/169))
+- Add deletion of **connections** records instead of marking them as deleted ([#170](https://github.com/MobileTeleSystems/syncmaster/issues/170))
+- Use PUT instead of PATCH for `Connection` and `Transfer` models ([#215](https://github.com/MobileTeleSystems/syncmaster/issues/215))
+- Return new connection and transfer object in `POST /v1/transfer/copy` and `POST /v1/connection/copy` endpoints.
+- Change response status from 200 to 204 for all `DELETE` endpoints.
+
+## Features
+
+- Add hive to known types ([#67](https://github.com/MobileTeleSystems/syncmaster/issues/67))
+- Allow search for **groups**, **users**, **connections**, **transfers**, **queues** ([#92](https://github.com/MobileTeleSystems/syncmaster/issues/92), [#94](https://github.com/MobileTeleSystems/syncmaster/issues/94), [#99](https://github.com/MobileTeleSystems/syncmaster/issues/99), [#100](https://github.com/MobileTeleSystems/syncmaster/issues/100), [#101](https://github.com/MobileTeleSystems/syncmaster/issues/101), [#103](https://github.com/MobileTeleSystems/syncmaster/issues/103))
+- Add filters for **connections**, **transfers** and **runs** ([#94](https://github.com/MobileTeleSystems/syncmaster/issues/94), [#102](https://github.com/MobileTeleSystems/syncmaster/issues/102), [#106](https://github.com/MobileTeleSystems/syncmaster/issues/106))
+- Implement a scheduler to run celery tasks on a schedule. This can be done by setting `Transfer.is_scheduled=True` and `Transfer.schedule="..."` (cron-like expression).
+  The Run model now has a `type` field with options `MANUAL` and `SCHEDULED`. ([#114](https://github.com/MobileTeleSystems/syncmaster/issues/114))
+- Add GET `v1/monitoring/metrics` endpoint to provide basic HTTP server metrics in Prometheus format ([#121](https://github.com/MobileTeleSystems/syncmaster/issues/121))
+- Implemented `KeycloakAuthProvider` for Single Sign-On (SSO) authentication.([#123](https://github.com/MobileTeleSystems/syncmaster/issues/123))
+- Implemented `DummyAuthProvider` for development and testing environments. ([#123](https://github.com/MobileTeleSystems/syncmaster/issues/123))
+- Add API schemas for new DB sources - Clickhouse, MSSQL, MySQL ([#124](https://github.com/MobileTeleSystems/syncmaster/issues/124), [#125](https://github.com/MobileTeleSystems/syncmaster/issues/125), [#126](https://github.com/MobileTeleSystems/syncmaster/issues/126), [#160](https://github.com/MobileTeleSystems/syncmaster/issues/160))
+- Add logic for handling FTP, FTPS, SFTP, Samba, WebDAV transfers ([#189](https://github.com/MobileTeleSystems/syncmaster/issues/189), [#191](https://github.com/MobileTeleSystems/syncmaster/issues/191), [#192](https://github.com/MobileTeleSystems/syncmaster/issues/192), [#194](https://github.com/MobileTeleSystems/syncmaster/issues/194))
+- Add API schemas for file sources - SFTP, FTP, FTPS, WebDAV, Samba ([#187](https://github.com/MobileTeleSystems/syncmaster/issues/187))
+- Add API schemas for file formats - Excel, XML, ORC, Parquet ([#140](https://github.com/MobileTeleSystems/syncmaster/issues/140), [#142](https://github.com/MobileTeleSystems/syncmaster/issues/142), [#143](https://github.com/MobileTeleSystems/syncmaster/issues/143), [#144](https://github.com/MobileTeleSystems/syncmaster/issues/144))
+- Add compression options to file formats CSV, JSON, JSONLine, Excel, ORC, Parquet, XML ([#159](https://github.com/MobileTeleSystems/syncmaster/issues/159), [#161](https://github.com/MobileTeleSystems/syncmaster/issues/161))
+- Add transformations for **Transfers** with dataframe row filtering ([#184](https://github.com/MobileTeleSystems/syncmaster/issues/184))
+- Add transformations for **Transfers** with dataframe column filtering ([#186](https://github.com/MobileTeleSystems/syncmaster/issues/186))
+- Add transformations for **Transfers** with file filtering ([#198](https://github.com/MobileTeleSystems/syncmaster/issues/198))
+- Add `increment_by` field to `strategy_params` ([#202](https://github.com/MobileTeleSystems/syncmaster/issues/202))
+- Implement increment strategy for transfers with file sources ([#209](https://github.com/MobileTeleSystems/syncmaster/issues/209))
+- Implement increment strategy for transfers with database sources ([#211](https://github.com/MobileTeleSystems/syncmaster/issues/211))
+- Add `resources` field to `Transfer`. ([#214](https://github.com/MobileTeleSystems/syncmaster/issues/214))
+- Add `file_name_template` field to `target_params` ([#196](https://github.com/MobileTeleSystems/syncmaster/issues/196), [#201](https://github.com/MobileTeleSystems/syncmaster/issues/201))
+
+## Improvements
+
+- Updated `User` model to include `email`, `first_name`, `middle_name`, and `last_name` fields, all optional. ([#123](https://github.com/MobileTeleSystems/syncmaster/issues/123))
+- Read env variable `SYNCMASTER__ENTRYPOINT__SUPERUSERS` to promote users to `SUPERUSER` role during server startup. ([#137](https://github.com/MobileTeleSystems/syncmaster/issues/137))
+- Enabled dynamic selection of authentication provider via environment variable `SYNCMASTER__AUTH__PROVIDER`. ([#123](https://github.com/MobileTeleSystems/syncmaster/issues/123))
+- Enable parallel reading from JDBC sources. ([#219](https://github.com/MobileTeleSystems/syncmaster/issues/219))
+- Reset HWM when changing strategy from `incremental` to `full`. ([#217](https://github.com/MobileTeleSystems/syncmaster/issues/217))
+- Grant read-only permissions for the previous group owner when ownership is transferred ([#135](https://github.com/MobileTeleSystems/syncmaster/issues/135))
+
+## Bug Fixes
+
+- Use Hadoop AWS `magic` committer only if transfer *target* is S3. ([#46](https://github.com/MobileTeleSystems/syncmaster/issues/46))
+- Check that `service_name` and `sid` are mutually exclusive when editing Oracle connection. ([#52](https://github.com/MobileTeleSystems/syncmaster/issues/52))
+- Queue name is unique within a group, new field `slug` is globally-unique. ([#54](https://github.com/MobileTeleSystems/syncmaster/issues/54), [#119](https://github.com/MobileTeleSystems/syncmaster/issues/119))
+- Prohibit updating connection type it if there is a transfer associated with this connection. ([#55](https://github.com/MobileTeleSystems/syncmaster/issues/55))
+- Fix error when `is_scheduled` field value was ignored. ([#57](https://github.com/MobileTeleSystems/syncmaster/issues/57))
+- Group without any users assigned was missing in groups list. ([#62](https://github.com/MobileTeleSystems/syncmaster/issues/62))
+- Dump connection credentials while starting a transfer. ([#63](https://github.com/MobileTeleSystems/syncmaster/issues/63))
diff --git a/mddocs/changelog/0.2.1.md b/mddocs/changelog/0.2.1.md
@@ -0,0 +1,11 @@
+# 0.2.1 (2025-04-07) { #0.2.1 }
+
+## Improvements
+
+- Change docker image user from `root` to `syncmaster`, to improve security.
+- Move server healthcheck to Docker image.
+- SBOM file is generated on release.
+
+## Bug fixes
+
+- Fix missing Swagger docs in prod image.
diff --git a/mddocs/changelog/0.2.2.md b/mddocs/changelog/0.2.2.md
@@ -0,0 +1,19 @@
+# 0.2.2 (2025-04-11) { #0.2.2 }
+
+## Breaking
+
+- Use `PUT /v1/qroups/:id` instead of `PATCH /v1/qroups/:id`.
+- Use `PUT /v1/qroups/:id/users/:id` instead of `PATCH /v1/qroups/:id/users/:id`.
+- Use `PUT /v1/queues/:id` instead of `PATCH /v1/queues/:id`.
+- Now allowed names length should be in 3..128 symbols range, not 1..inf.
+
+## Improvements
+
+- Now queue name can include any ASCII printable characters.
+- Queue slug is always lowercase. Spaces, hyphens and underscores are replaced with `-` symbol.
+
+## Bug fixes
+
+- Call `kinit` before starting Spark session connecting to `Hive` cluster. ([#225](https://github.com/MobileTeleSystems/syncmaster/issues/225))
+- Fix `HDFS` connection was trying to use anonymous auth instead of user/password. ([#225](https://github.com/MobileTeleSystems/syncmaster/issues/225))
+- Fix updating queue ignored name and didn’t reset description.
diff --git a/mddocs/changelog/0.2.3.md b/mddocs/changelog/0.2.3.md
@@ -0,0 +1,9 @@
+# 0.2.3 (2025-04-11) { #0.2.3 }
+
+## Bug fixes
+
+- Fix Worker not updating Run `status` and `ended_at` fields after executing a very long ETL process.
+
+## Improvements
+
+- Change Celery log level from DEBUG to INFO.
diff --git a/mddocs/changelog/0.2.4.md b/mddocs/changelog/0.2.4.md
@@ -0,0 +1,5 @@
+# 0.2.4 (2025-07-09) { #0.2.4 }
+
+## Improvements
+
+Include all required jars from Maven to worker image. This increases image size, but drastically reduces time of Spark session startup.
diff --git a/mddocs/changelog/0.2.5.md b/mddocs/changelog/0.2.5.md
@@ -0,0 +1,17 @@
+# 0.2.5 (2025-10-10) { #0.2.5 }
+
+## Features
+
+- Implement Keycloak login page on frontend. ([#128](https://github.com/MTSWebServices/syncmaster-ui/pull/128))
+- Implement ``GET /v1/auth/logout`` endpoint for ``KeycloakAuthProvider``. ([#275](https://github.com/MobileTeleSystems/syncmaster/issues/255))
+
+## Improvements
+
+- Improved full-text search for technical fields such as hostnames, table names, and directory paths. ([#255](https://github.com/MobileTeleSystems/syncmaster/issues/255))
+- Replace 307 redirect to Keycloak auth page with 401 response, due to browser restrictions for redirect + CORS + localhost. ([#274](https://github.com/MobileTeleSystems/syncmaster/issues/274))
+
+## Bug Fixes
+
+- Replace sync methods of Keycloak client with async ones.  ([#177](https://github.com/MobileTeleSystems/syncmaster/issues/177))
+
+  Previously interaction with Keycloak could block asyncio event loop.
diff --git a/mddocs/changelog/0.3.0.md b/mddocs/changelog/0.3.0.md
@@ -0,0 +1,101 @@
+# 0.3.0 (2025-12-18) { #0.3.0 }
+
+Release of Data.SyncMaster 0.3.0 brings up support for Iceberg, Spark-on-K8s and Spark-on-Yarn.
+
+!!! note
+    Currently Spark-on-K8s and Spark-on-Yarn do not support FTP, FTPS, SFTP, Samba and WebDAV.
+
+## Breaking Changes
+
+- Worker container command should be changed from `--queues 123-myqueue` to `worker --queues 123-myqueue` ([#295](https://github.com/MobileTeleSystems/syncmaster/issues/295)).
+
+- Application should be configured via `config.yml` file ([#289](https://github.com/MobileTeleSystems/syncmaster/issues/289)).
+
+  It's still possible to use environment variables instead. But it is not recommended for security reasons, as docker/k8s envs can be read by other users.
+
+  Other notable changes:
+
+- Environment variable `SYNCMASTER__ENTRYPOINT__SUPERUSERS` is renamed to `SYNCMASTER__SUPERUSERS`.
+- Logging format is configured explicitly via `config.yml` instead of having few predefined configuration files.
+
+- Moved `server.session` middleware settings to `auth` block ([#304](https://github.com/MobileTeleSystems/syncmaster/issues/304)).
+  Also rename some fields in `auth.keycloak` settings block.
+
+Before vs after
+
+Before:
+
+```yaml
+auth:
+  provider: ...
+  keycloak:
+    server_url: ...
+     redirect_url: ...
+
+server:
+  session:
+    enabled: true
+    secret_key: ...
+```
+
+Now:
+
+```yaml
+auth:
+  provider:
+  keycloak:
+    api_url: ...
+    ui_callback_url: ...
+  cookie:
+    secret_key: ...
+```
+
+## Features
+
+- Added Iceberg support (([#282](https://github.com/MobileTeleSystems/syncmaster/issues/282)), ([#284](https://github.com/MobileTeleSystems/syncmaster/issues/284)), ([#294](https://github.com/MobileTeleSystems/syncmaster/issues/294)), ([#297](https://github.com/MobileTeleSystems/syncmaster/issues/297))).
+
+  Iceberg connection currently supports only Iceberg REST Catalog with S3 warehouse.
+
+- Allow using SyncMaster worker image as `spark.kubernetes.container.image`. ([#295](https://github.com/MobileTeleSystems/syncmaster/issues/295))
+
+- Allow passing default Spark session config via worker settings ([#291](https://github.com/MobileTeleSystems/syncmaster/issues/291)):
+
+### Example config.yml
+
+```yaml
+config.yml:
+worker:
+  spark_session_default_config:
+  spark.master: local
+  spark.driver.host: 127.0.0.1
+  spark.driver.bindAddress: 0.0.0.0
+  spark.sql.pyspark.jvmStacktrace.enabled: true
+  spark.ui.enabled: false
+```
+
+- Added OAuth2GatewayProvider ([#283](https://github.com/MobileTeleSystems/syncmaster/issues/283)).
+
+  This allows using Data.SyncMaster under OAuth2 Gateway. Implementation is similar to DummyAuthProvider.
+
+- Allow disabling `SessionMiddleware`, as it only required by `KeycloakAuthProvider`.
+
+- Add hooks support to worker classes (TransferController, Handler) ([#279](https://github.com/MobileTeleSystems/syncmaster/issues/279)).
+
+- Pass transfer name and group name to Handlers ([#308](https://github.com/MobileTeleSystems/syncmaster/issues/308)).
+
+## Improvements
+
+- Make S3 connection `region` a mandatory option, to prevent possible errors.
+- Hide `database_name` from Clickhouse and MySQL connection pages.
+- Frontend: add placeholders to connection params, like host, port and so on.
+- Sync frontend and backend checks for some field patterns, e.g. table name should be in format `schema.table`.
+- Improve OpenAPI schema fields description.
+
+## Bug Fixes
+
+Fix some file format options were ignored by SyncMaster worker:
+
+- XML: `root_tag`, `row_tag`
+- Excel `start_cell`, `include_header`
+- CSV `include_header`, `line_sep`
+- JSON, JSONLine: `line_sep`
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,3 @@
		# 0.1.1 (2024-03-29) { #0.1.1 }

		SyncMaster is now open source!