Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .lycheeignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# example opensearch/dashboards ports
http://localhost:5601
http://localhost:9200

# sites that block scraping
https://hg.openjdk.org/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/
https://www.linkedin.com/in/*/

# we have many pull requests links due to changelogs, skip these as they're unlikely to break and it saves API usage
https://github.com/opendistro-for-elasticsearch/sql/pull/*
https://github.com/opensearch-project/sql/pull/*

2 changes: 0 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,6 @@ The following projects are related to the SQL plugin, but stored in the differen
| [![untriaged open][untriaged-badge]][untriaged-link] |
| [![nolabel open][nolabel-badge]][nolabel-link] |

[dco-badge]: https://github.com/opensearch-project/sql/actions/workflows/dco.yml/badge.svg
[dco-badge-link]: https://github.com/opensearch-project/sql/actions/workflows/dco.yml
[link-check-badge]: https://github.com/opensearch-project/sql/actions/workflows/link-checker.yml/badge.svg
[link-check-link]: https://github.com/opensearch-project/sql/actions/workflows/link-checker.yml
[bwc-tests-badge]: https://img.shields.io/badge/BWC%20tests-in%20progress-yellow
Expand Down
4 changes: 2 additions & 2 deletions async-query-core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Following is the list of extension points where the consumer of the library need

- Data store interface
- [AsyncQueryJobMetadataStorageService](src/main/java/org/opensearch/sql/spark/asyncquery/AsyncQueryJobMetadataStorageService.java)
- [SessionStorageService](java/org/opensearch/sql/spark/execution/statestore/SessionStorageService.java)
- [SessionStorageService](src/main/java/org/opensearch/sql/spark/execution/statestore/SessionStorageService.java)
- [StatementStorageService](src/main/java/org/opensearch/sql/spark/execution/statestore/StatementStorageService.java)
- [FlintIndexMetadataService](src/main/java/org/opensearch/sql/spark/flint/FlintIndexMetadataService.java)
- [FlintIndexStateModelService](src/main/java/org/opensearch/sql/spark/flint/FlintIndexStateModelService.java)
Expand All @@ -39,4 +39,4 @@ To update the grammar files, update `build.gradle` file (in `downloadG4Files` ta
```
./gradlew async-query-core:downloadG4Files
```
This will overwrite the files under `src/main/antlr`.
This will overwrite the files under `src/main/antlr`.
6 changes: 3 additions & 3 deletions benchmarks/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# OpenSearch SQL/PPL Microbenchmark Suite

This directory contains the microbenchmark suite of OpenSearch SQL/PPL. It relies on [JMH](http://openjdk.java.net/projects/code-tools/jmh/).
This directory contains the microbenchmark suite of OpenSearch SQL/PPL. It relies on [JMH](https://openjdk.java.net/projects/code-tools/jmh/).

## Purpose

Expand All @@ -25,6 +25,6 @@ Run specific benchmarks using the `-Pjmh.includes` parameter:

## Adding Microbenchmarks

Before adding a new microbenchmark, make yourself familiar with the JMH API. You can check our existing microbenchmarks and also the [JMH samples](http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/).
Before adding a new microbenchmark, make yourself familiar with the JMH API. You can check our existing microbenchmarks and also the [JMH samples](https://hg.openjdk.org/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/).

In contrast to tests, the actual name of the benchmark class is not relevant to JMH. However, stick to the naming convention and end the class name of a benchmark with `Benchmark`. To have JMH execute a benchmark, annotate the respective methods with `@Benchmark`.
In contrast to tests, the actual name of the benchmark class is not relevant to JMH. However, stick to the naming convention and end the class name of a benchmark with `Benchmark`. To have JMH execute a benchmark, annotate the respective methods with `@Benchmark`.
2 changes: 1 addition & 1 deletion docs/attributions.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Apart from the problems we identified earlier, we made significant improvement i

1. *Integration Test*: We migrated all integrate tests to standard OpenSearch IT framework which spins up in-memory cluster for testing. Now all test cases treat plugin code as blackbox and verify functionality from externally.
2. *New JDBC Driver*: We developed our own JDBC driver without any dependency on Elasticsearch proprietary code.
[sql-jdbc](https://github.com/opensearch-project/sql/tree/main/sql-jdbc)
[sql-jdbc](https://github.com/opensearch-project/sql-jdbc)
3. *Better Hash JOIN*: Block Hash Join implementation with circuit break mechanism protect your OpenSearch memory. Performance testing showed our implementation is 1.5 ~ 2x better than old hash join in terms of throughput and latency and much lower error rate under heavy pressure.
4. *Query Planner*: Logical and physical planner was added to support JOIN query in efficient and extendible way.
5. *PartiQL Compatibility*: we are partially compatible with PartiQL specification which allows for query involved in nested JSON documents.
Expand Down
2 changes: 1 addition & 1 deletion docs/dev/intro-v3-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ PPL is specifically designed to simplify tasks in observability and security ana
The current PPL engine (shared with SQL v2 engine) is built with custom components, including a parser, analyzer, optimizer, and relies heavily on OpenSearch DSL capabilities to execute query plans. By aligning its syntax and concepts with familiar languages like Splunk SPL and SQL, we aim to streamline migration for users from these backgrounds, allowing them to adopt PPL with minimal effort. The lack of comprehensive ability is a critical blocker for Splunk-to-OpenSearch migrations. We added ~20 new commands in PPL-on-Spark, but there are still dozens of command gaps to be filled. Not to mention that there are still a large number of functions to be implemented.

### 2.2 Lack of Unified PPL Experience
The PPL language is currently inconsistent across [PPL-on-OpenSearch](https://github.com/opensearch-project/sql/blob/main/ppl/src/main/antlr/OpenSearchPPLParser.g4) and [PPL-on-Spark](https://github.com/opensearch-project/opensearch-spark/blob/main/ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4). There are a lot of new commands added in PPL-on-Spark, such as `join`, `lookup` and `subsearch` are not yet supported in PPL-on-OpenSearch. As more and more new commands and functions are implemented in PPL-on-Spark, this gap will continue to widen.
The PPL language is currently inconsistent across [PPL-on-OpenSearch](https://github.com/opensearch-project/sql/blob/main/ppl/src/main/antlr/OpenSearchPPLParser.g4) and [PPL-on-Spark](https://github.com/opensearch-project/sql/blob/main/language-grammar/src/main/antlr4/OpenSearchPPLParser.g4). There are a lot of new commands added in PPL-on-Spark, such as `join`, `lookup` and `subsearch` are not yet supported in PPL-on-OpenSearch. As more and more new commands and functions are implemented in PPL-on-Spark, this gap will continue to widen.

### 2.3 Lack of mature query optimizer
Although the v2 engine framework comes with an optimizer class, it only has a few pushdown optimization rules and lacks of mature optimization rules and cost-based optimizer like those found in traditional databases. Query performance and scalability are core to PPL's design, enabling it to efficiently handle high-performance queries and scale to support large datasets and complex queries.
Expand Down
2 changes: 1 addition & 1 deletion docs/dev/intro-v3-engine.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,5 +89,5 @@ If you're interested in the new query engine, please find more details in [V3 Ar
The following items are on our roadmap with high priority:
- Resolve the [V3 limitation](#33-limitations).
- Advancing pushdown optimization and benchmarking
- Unified the PPL syntax between [PPL-on-OpenSearch](https://github.com/opensearch-project/sql/blob/main/ppl/src/main/antlr/OpenSearchPPLParser.g4) and [PPL-on-Spark](https://github.com/opensearch-project/opensearch-spark/blob/main/ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4)
- Unified the PPL syntax between [PPL-on-OpenSearch](https://github.com/opensearch-project/sql/blob/main/ppl/src/main/antlr/OpenSearchPPLParser.g4) and [PPL-on-Spark](https://github.com/opensearch-project/sql/blob/main/language-grammar/src/main/antlr4/OpenSearchPPLParser.g4)
- Support more DSL aggregation
4 changes: 2 additions & 2 deletions docs/dev/opensearch-relevancy-search.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,7 +119,7 @@ Besides, one of the query engine architecture tenets is to keep the job of every

#### Option B: Create a new query plan node dedicated for the search features.

The diagram below is simplified with only the logical plan and physical plan sections and leaves out others. Please check out [OpenSearch SQL Engine Architecture](https://github.com/opensearch-project/sql/blob/main/docs/dev/Architecture.md) for the complete architecture of the query engine.
The diagram below is simplified with only the logical plan and physical plan sections and leaves out others. Please check out [OpenSearch SQL Engine Architecture](https://github.com/opensearch-project/sql/blob/main/docs/dev/intro-architecture.md) for the complete architecture of the query engine.

![relevance-Page-2 (4)](https://user-images.githubusercontent.com/33583073/129938534-28fa4845-4246-4707-9519-e68c9e86d174.png)

Expand Down Expand Up @@ -291,4 +291,4 @@ All the code changes should be test driven. The pull requests should include uni

### A2. Search flow in search engine

![relevance](https://user-images.githubusercontent.com/33583073/129938659-5b49f43d-a83f-47d5-be5b-937b1c96e5bc.png)
![relevance](https://user-images.githubusercontent.com/33583073/129938659-5b49f43d-a83f-47d5-be5b-937b1c96e5bc.png)
2 changes: 1 addition & 1 deletion docs/dev/testing-doctest.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ Doctest is relying on the console/command line to run code examples in documenta
* https://github.com/crate/crate/blob/master/docs/general/dql/selects.rst


Similar to CrateDB using it’s CLI “crash”, we can make use of our own [SQL-CLI](https://github.com/opensearch-project/sql/tree/main/sql-cli)
Similar to CrateDB using it’s CLI “crash”, we can make use of our own [SQL-CLI](https://github.com/opensearch-project/sql-cli)

To support PPL, we need to add PPL support to SQL-CLI. Since PPL and SQL expose similar http endpoint for query and share similar response format. The update won’t be much of work.

Expand Down
2 changes: 1 addition & 1 deletion release-notes/opensearch-sql.release-notes-2.10.0.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,4 +40,4 @@ Compatible with OpenSearch and OpenSearch Dashboards Version 2.10.0
* [Backport 2.x] [spotless] Removes Checkstyle in favor of spotless by @MitchellGale in https://github.com/opensearch-project/sql/pull/2018
* [Backport 2.x] [Spotless] Entire project running spotless by @MitchellGale in https://github.com/opensearch-project/sql/pull/2016
---
**Full Changelog**: https://github.com/opensearch-project/sql/compare/2.3.0.0...v.2.10.0.0
**Full Changelog**: https://github.com/opensearch-project/sql/compare/2.3.0.0...2.10.0.0
2 changes: 1 addition & 1 deletion release-notes/opensearch-sql.release-notes-2.11.0.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,4 +52,4 @@ Compatible with OpenSearch and OpenSearch Dashboards Version 2.11.0
* bump okio to 3.4.0 by @joshuali925 in https://github.com/opensearch-project/sql/pull/2047

---
**Full Changelog**: https://github.com/opensearch-project/sql/compare/2.3.0.0...v.2.11.0.0
**Full Changelog**: https://github.com/opensearch-project/sql/compare/2.3.0.0...2.11.0.0
Loading