Skip to content

Commit a102a29

Browse files
fupelaqufupelaqu
authored
Merge pull request #30 from SOFTNETWORK-APP/feature/sqlDMLAndDDL
add support for : - DDL statements - CREATE, ALTER, SHOW, DESCRIBE, DROP, TRUNCATE TABLE - CREATE, ALTER, SHOW, DESCRIBE, DROP PIPELINE - DML statements - INSERT INTO ... VALUES - INSERT INTO ... AS SELECT ... [ON CONFLICT ...] - UPDATE ... SET ... [WHERE] - DELETE FROM ... [WHERE] - COPY INTO ... FROM ... [FILE_FORMAT = ...] [ON CONFLICT ...]
2 parents 3a9a6ab + 38ae45c commit a102a29

164 files changed

Lines changed: 24762 additions & 2560 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 126 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -30,21 +30,23 @@ SoftClient4ES provides a trait-based interface (`ElasticClientApi`) that aggrega
3030

3131
#### **Core APIs**
3232

33-
| API | Description | Documentation |
34-
|------------------------------------------------------------|-------------------------------------------------------------------------------------|-------------------------------------------------|
35-
| **[RefreshApi](documentation/client/refresh.md)** | Control index refresh for real-time search | [📖 Docs](documentation/client/refresh.md) |
36-
| **[IndicesApi](documentation/client/indices.md)** | Create, update, and manage indices with settings and mappings | [📖 Docs](documentation/client/indices.md) |
37-
| **[SettingsApi](documentation/client/settings.md)** | Dynamic index settings management | [📖 Docs](documentation/client/settings.md) |
38-
| **[AliasApi](documentation/client/aliases.md)** | Manage index aliases for zero-downtime deployments | [📖 Docs](documentation/client/aliases.md) |
39-
| **[MappingApi](documentation/client/mappings.md)** | Smart mapping management with automatic migration and rollback | [📖 Docs](documentation/client/mappings.md) |
40-
| **[IndexApi](documentation/client/index.md)** | Index documents | [📖 Docs](documentation/client/index.md) |
41-
| **[UpdateApi](documentation/client/update.md)** | Partial document updates with script support | [📖 Docs](documentation/client/update.md) |
42-
| **[DeleteApi](documentation/client/delete.md)** | Delete documents by ID or query | [📖 Docs](documentation/client/delete.md) |
43-
| **[BulkApi](documentation/client/bulk.md)** | High-performance bulk operations with Akka Streams | [📖 Docs](documentation/client/bulk.md) |
44-
| **[GetApi](documentation/client/get.md)** | Get documents by ID | [📖 Docs](documentation/client/get.md) |
45-
| **[SearchApi](documentation/client/search.md)** | Advanced search with SQL and aggregations support | [📖 Docs](documentation/client/search.md) |
46-
| **[ScrollApi](documentation/client/scroll.md)** | Stream large datasets with automatic strategy detection (PIT, search_after, scroll) | [📖 Docs](documentation/client/scroll.md) |
33+
| API | Description | Documentation |
34+
|--------------------------------------------------------|-------------------------------------------------------------------------------------|-------------------------------------------------|
35+
| **[RefreshApi](documentation/client/refresh.md)** | Control index refresh for real-time search | [📖 Docs](documentation/client/refresh.md) |
36+
| **[IndicesApi](documentation/client/indices.md)** | Create, update, and manage indices with settings and mappings | [📖 Docs](documentation/client/indices.md) |
37+
| **[SettingsApi](documentation/client/settings.md)** | Dynamic index settings management | [📖 Docs](documentation/client/settings.md) |
38+
| **[AliasApi](documentation/client/aliases.md)** | Manage index aliases for zero-downtime deployments | [📖 Docs](documentation/client/aliases.md) |
39+
| **[MappingApi](documentation/client/mappings.md)** | Smart mapping management with automatic migration and rollback | [📖 Docs](documentation/client/mappings.md) |
40+
| **[IndexApi](documentation/client/index.md)** | Index documents | [📖 Docs](documentation/client/index.md) |
41+
| **[UpdateApi](documentation/client/update.md)** | Partial document updates with script support | [📖 Docs](documentation/client/update.md) |
42+
| **[DeleteApi](documentation/client/delete.md)** | Delete documents by ID or query | [📖 Docs](documentation/client/delete.md) |
43+
| **[BulkApi](documentation/client/bulk.md)** | High-performance bulk operations with Akka Streams | [📖 Docs](documentation/client/bulk.md) |
44+
| **[GetApi](documentation/client/get.md)** | Get documents by ID | [📖 Docs](documentation/client/get.md) |
45+
| **[SearchApi](documentation/client/search.md)** | Advanced search with SQL and aggregations support | [📖 Docs](documentation/client/search.md) |
46+
| **[ScrollApi](documentation/client/scroll.md)** | Stream large datasets with automatic strategy detection (PIT, search_after, scroll) | [📖 Docs](documentation/client/scroll.md) |
4747
| **[AggregationApi](documentation/client/aggregations.md)** | Type-safe way to execute aggregations using SQL queries | [📖 Docs](documentation/client/aggregations.md) |
48+
| **[TemplateApi](documentation/client/templates.md)** | Templates management | [📖 Docs](documentation/client/templates.md) |
49+
| **[GatewayApi](documentation/client/gateway.md)** | Unified SQL interface for DQL, DML, and DDL statements | [📖 Docs](documentation/client/gateway.md) |
4850

4951
#### **Client Implementations**
5052

@@ -181,7 +183,105 @@ result match {
181183

182184
### **3. SQL compatible**
183185

184-
### **3.1 SQL to Elasticsearch Query DSL**
186+
### **3.1 SQL Gateway — Unified SQL Interface for Elasticsearch**
187+
188+
SoftClient4ES includes a high‑level SQL Gateway that allows you to execute **DQL, DML, DDL, and Pipeline statements** directly against Elasticsearch using standard SQL syntax.
189+
190+
The Gateway exposes a single entry point:
191+
192+
```scala
193+
gateway.run(sql: String): Future[ElasticResult[QueryResult]]
194+
```
195+
196+
It automatically:
197+
198+
- normalizes SQL (removes comments, trims whitespace)
199+
- parses SQL into AST nodes
200+
- routes statements to the appropriate executor
201+
- returns a typed `QueryResult` (`QueryRows`, `TableResult`, `PipelineResult`, `DmlResult`, `DdlResult`, `SQLResult`)
202+
203+
#### **Architecture Diagram — SQL Gateway**
204+
205+
```text
206+
+----------------------+
207+
| GatewayApi |
208+
| run(sql: String) |
209+
+----------+-----------+
210+
|
211+
v
212+
+----------------------+
213+
| Parser |
214+
| SQL → Statement |
215+
+----------+-----------+
216+
|
217+
------------------------------------------------
218+
| | |
219+
v v v
220+
+---------------+ +---------------+ +----------------+
221+
| DqlStatement | | DmlStatement | | DdlStatement |
222+
+-------+-------+ +-------+-------+ +--------+-------+
223+
| | |
224+
v v v
225+
+---------------+ +---------------+ +----------------------+
226+
| DqlExecutor | | DmlExecutor | | DdlRouterExecutor |
227+
+-------+-------+ +-------+-------+ +----------+-----------+
228+
/ \
229+
/ \
230+
v v
231+
+----------------+ +------------------+
232+
| TableExecutor | | PipelineExecutor |
233+
+--------+-------+ +--------+---------+
234+
| |
235+
v v
236+
+-----------+ +-----------+
237+
|ElasticSearch| |ElasticSearch|
238+
+-----------+ +-----------+
239+
240+
+-----------------------------------------------+
241+
| QueryResult |
242+
+-----------------------------------------------+
243+
| QueryRows | QueryStream | QueryStructured |
244+
| DmlResult | DdlResult | TableResult |
245+
| PipelineResult | SQLResult (SHOW CREATE) |
246+
+-----------------------------------------------+
247+
```
248+
249+
---
250+
251+
#### **Supported SQL Categories**
252+
253+
| Category | Examples |
254+
|---------------------|--------------------------------------------------------------------------------------------------------------------|
255+
| **DQL** | `SELECT`, `JOIN UNNEST`, `GROUP BY`, `HAVING`, window functions |
256+
| **DML** | `INSERT`, `UPDATE`, `DELETE`, `COPY INTO` |
257+
| **DDL (Tables)** | `CREATE TABLE`, `ALTER TABLE`, `DROP TABLE`, `TRUNCATE TABLE`, `DESCRIBE TABLE`, `SHOW TABLE`, `SHOW CREATE TABLE` |
258+
| **DDL (Pipelines)** | `CREATE PIPELINE`, `ALTER PIPELINE`, `DROP PIPELINE`, `DESCRIBE PIPELINE`, `SHOW PIPELINE`, `SHOW CREATE PIPELINE` |
259+
260+
#### **Example**
261+
262+
```scala
263+
gateway.run("""
264+
CREATE TABLE users (
265+
id INT,
266+
name TEXT,
267+
age INT,
268+
PRIMARY KEY (id)
269+
);
270+
INSERT INTO users VALUES (1, 'Alice', 30);
271+
SELECT * FROM users;
272+
""")
273+
```
274+
275+
#### **Documentation**
276+
277+
- 📘 **Gateway API** — `documentation/client/gateway.md`
278+
- 📘 **DQL** — `documentation/sql/dql_statements.md`
279+
- 📘 **DML** — `documentation/sql/dml_statements.md`
280+
- 📘 **DDL** — `documentation/sql/ddl_statements.md`
281+
282+
---
283+
284+
### **3.2 SQL to Elasticsearch Query DSL**
185285

186286
SoftClient4ES includes a powerful SQL parser that translates standard SQL `SELECT` queries into native Elasticsearch queries.
187287

@@ -210,6 +310,9 @@ SoftClient4ES includes a powerful SQL parser that translates standard SQL `SELEC
210310
- ✅ Date / Time functions (`YEAR`, `QUARTER`, `MONTH`, `WEEK`, `DAY`, `HOUR`, `MINUTE`, `SECOND`, `MILLISECOND`, `MICROSECOND`, `NANOSECOND`, `EPOCHDAY`, `OFFSET_SECONDS`, `LAST_DAY`, `WEEKDAY`, `YEARDAY`, `INTERVAL`, `CURRENT_DATE`, `CURDATE`, `TODAY`, `NOW`, `CURRENT_TIME`, `CURTIME`, `CURRENT_DATETIME`, `CURRENT_TIMESTAMP`, `DATE_ADD`, `DATEADD`, `DATE_SUB`, `DATESUB`, `DATETIME_ADD`, `DATETIMEADD`, `DATETIME_SUB`, `DATETIMESUB`, `DATE_DIFF`, `DATEDIFF`, `DATE_FORMAT`, `DATE_PARSE`, `DATETIME_FORMAT`, `DATETIME_PARSE`, `DATE_TRUNC`, `EXTRACT`)
211311
- ✅ Geospatial functions (`POINT`, `ST_DISTANCE`)
212312
- ✅ Aggregate functions (`COUNT`, `SUM`, `AVG`, `MIN`, `MAX`, `DISTINCT`, `FIRST_VALUE`, `LAST_VALUE`, `ARRAY_AGG`)
313+
- ✅ [Window functions](#32-window-functions-support) with `OVER` clause
314+
- ✅ [DML Support](#34-dml-support) (`INSERT`, `UPDATE`, `DELETE`)
315+
- ✅ [DDL Support](#35-ddl-support) (`CREATE TABLE`, `ALTER TABLE`, `DROP TABLE`, `TRUNCATE TABLE`, `CREATE PIPELINE`, `ALTER PIPELINE`, `DROP PIPELINE`)
213316

214317
**Example:**
215318

@@ -1188,6 +1291,8 @@ client.searchAsUnchecked[Product](SQLQuery(dynamicQuery))
11881291
client.scrollAsUnchecked[Product](dynamicQuery)
11891292
```
11901293

1294+
---
1295+
11911296
📖 **[Full SQL Validation Documentation](documentation/sql/validation.md)**
11921297

11931298
📖 **[Full SQL Documentation](documentation/sql/README.md)**
@@ -1517,18 +1622,18 @@ ThisBuild / resolvers ++= Seq(
15171622

15181623
// For Elasticsearch 6
15191624
// Using Jest client
1520-
libraryDependencies += "app.softnetwork.elastic" %% s"softclient4es6-jest-client" % 0.14.2
1625+
libraryDependencies += "app.softnetwork.elastic" %% s"softclient4es6-jest-client" % 0.15.0
15211626
// Or using Rest High Level client
1522-
libraryDependencies += "app.softnetwork.elastic" %% s"softclient4es6-rest-client" % 0.14.2
1627+
libraryDependencies += "app.softnetwork.elastic" %% s"softclient4es6-rest-client" % 0.15.0
15231628

15241629
// For Elasticsearch 7
1525-
libraryDependencies += "app.softnetwork.elastic" %% s"softclient4es7-rest-client" % 0.14.2
1630+
libraryDependencies += "app.softnetwork.elastic" %% s"softclient4es7-rest-client" % 0.15.0
15261631

15271632
// For Elasticsearch 8
1528-
libraryDependencies += "app.softnetwork.elastic" %% s"softclient4es8-java-client" % 0.14.2
1633+
libraryDependencies += "app.softnetwork.elastic" %% s"softclient4es8-java-client" % 0.15.0
15291634

15301635
// For Elasticsearch 9
1531-
libraryDependencies += "app.softnetwork.elastic" %% s"softclient4es9-java-client" % 0.14.2
1636+
libraryDependencies += "app.softnetwork.elastic" %% s"softclient4es9-java-client" % 0.15.0
15321637
```
15331638

15341639
### **Quick Example**
@@ -1567,12 +1672,10 @@ client.createIndex("users", mapping) match {
15671672

15681673
### **Short-term**
15691674

1570-
- [ ] Support for `INSERT`, `UPDATE`, `DELETE` SQL operations
1571-
- [ ] Support for `CREATE TABLE`, `ALTER TABLE` SQL operations
1675+
- [ ] Full **JDBC connector for Elasticsearch**
15721676

15731677
### **Long-term**
15741678

1575-
- [ ] Full **JDBC connector for Elasticsearch**
15761679
- [ ] Advanced monitoring and metrics
15771680

15781681
---

‎bridge/src/main/scala/app/softnetwork/elastic/sql/bridge/ElasticAggregation.scala‎

Lines changed: 9 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,6 @@ import app.softnetwork.elastic.sql.query.{
2222
Asc,
2323
BucketIncludesExcludes,
2424
BucketNode,
25-
BucketTree,
2625
Criteria,
2726
Desc,
2827
Field,
@@ -101,7 +100,7 @@ object ElasticAggregation {
101100
having: Option[Criteria],
102101
bucketsDirection: Map[String, SortOrder],
103102
allAggregations: Map[String, SQLAggregation]
104-
): ElasticAggregation = {
103+
)(implicit timestamp: Long): ElasticAggregation = {
105104
import sqlAgg._
106105
val sourceField = identifier.path
107106

@@ -156,14 +155,14 @@ object ElasticAggregation {
156155
if (transformFuncs.nonEmpty) {
157156
val context = PainlessContext()
158157
val scriptSrc = identifier.painless(Some(context))
159-
val script = Script(s"$context$scriptSrc").lang("painless")
158+
val script = now(Script(s"$context$scriptSrc").lang("painless"))
160159
buildScript(aggName, script)
161160
} else {
162161
aggType match {
163162
case th: WindowFunction if th.shouldBeScripted =>
164163
val context = PainlessContext()
165164
val scriptSrc = th.identifier.painless(Some(context))
166-
val script = Script(s"$context$scriptSrc").lang("painless")
165+
val script = now(Script(s"$context$scriptSrc").lang("painless"))
167166
buildScript(aggName, script)
168167
case _ => buildField(aggName, sourceField)
169168
}
@@ -231,7 +230,7 @@ object ElasticAggregation {
231230
.filter(_.isScriptField)
232231
.groupBy(_.sourceField)
233232
.map(_._2.head)
234-
.map(f => f.sourceField -> Script(f.painless(None)).lang("painless"))
233+
.map(f => f.sourceField -> now(Script(f.painless(None)).lang("painless")))
235234
.toMap,
236235
size = limit,
237236
sorts = th.orderBy
@@ -269,14 +268,14 @@ object ElasticAggregation {
269268
val painless = script.identifier.painless(None)
270269
bucketScriptAggregation(
271270
aggName,
272-
Script(s"$painless").lang("painless"),
271+
now(Script(s"$painless").lang("painless")),
273272
params.toMap
274273
)
275274
case _ =>
276275
throw new IllegalArgumentException(s"Unsupported aggregation type: $aggType")
277276
}
278277

279-
val nestedElement = identifier.nestedElement
278+
val nestedElement = sqlAgg.nestedElement
280279

281280
val nestedElements: Seq[NestedElement] =
282281
nestedElement.map(n => NestedElements.buildNestedTrees(Seq(n))).getOrElse(Nil)
@@ -349,11 +348,7 @@ object ElasticAggregation {
349348
having: Option[Criteria],
350349
nested: Option[NestedElement],
351350
allElasticAggregations: Seq[ElasticAggregation]
352-
): Seq[Aggregation] = {
353-
val trees = BucketTree(buckets.flatMap(_.headOption))
354-
println(
355-
s"[DEBUG] buildBuckets called with buckets: \n$trees"
356-
)
351+
)(implicit timestamp: Long): Seq[Aggregation] = {
357352
for (tree <- buckets) yield {
358353
val treeNodes =
359354
tree.sortBy(_.level).reverse.foldLeft(Seq.empty[NodeAggregation]) { (current, node) =>
@@ -371,7 +366,7 @@ object ElasticAggregation {
371366
if (!bucket.isBucketScript && bucket.shouldBeScripted) {
372367
val context = PainlessContext()
373368
val painless = bucket.painless(Some(context))
374-
Some(Script(s"$context$painless").lang("painless"))
369+
Some(now(Script(s"$context$painless").lang("painless")))
375370
} else {
376371
None
377372
}
@@ -520,7 +515,7 @@ object ElasticAggregation {
520515
val bucketSelector =
521516
bucketSelectorAggregation(
522517
"having_filter",
523-
Script(script),
518+
now(Script(script)),
524519
extractMetricsPathForBucket(
525520
criteria,
526521
nested,

‎bridge/src/main/scala/app/softnetwork/elastic/sql/bridge/ElasticBridge.scala‎

Lines changed: 5 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ case class ElasticBridge(filter: ElasticFilter) {
4545
def query(
4646
innerHitsNames: Set[String] = Set.empty,
4747
currentQuery: Option[ElasticBoolQuery]
48-
): Query = {
48+
)(implicit timestamp: Long): Query = {
4949
filter match {
5050
case boolQuery: ElasticBoolQuery =>
5151
import boolQuery._
@@ -77,13 +77,10 @@ case class ElasticBridge(filter: ElasticFilter) {
7777
case _ =>
7878
}
7979
if (n.sources.nonEmpty) {
80-
inner = inner.fetchSource(
81-
FetchSourceContext(
82-
fetchSource = true,
83-
includes = n.sources.map { source =>
84-
(n.path.split('.').toSeq ++ Seq(source)).mkString(".")
85-
}.toArray
86-
)
80+
inner = inner.docValueFields(
81+
n.sources.map { source =>
82+
(n.path.split('.').toSeq ++ Seq(source)).mkString(".")
83+
}
8784
)
8885
}
8986
inner

‎bridge/src/main/scala/app/softnetwork/elastic/sql/bridge/ElasticCriteria.scala‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,9 @@ import com.sksamuel.elastic4s.requests.searches.queries.Query
2121

2222
case class ElasticCriteria(criteria: Criteria) {
2323

24-
def asQuery(group: Boolean = true, innerHitsNames: Set[String] = Set.empty): Query = {
24+
def asQuery(group: Boolean = true, innerHitsNames: Set[String] = Set.empty)(implicit
25+
timestamp: Long
26+
): Query = {
2527
val query = criteria.boolQuery.copy(group = group)
2628
query
2729
.filter(criteria.asFilter(Option(query)))

0 commit comments

Comments
 (0)