diff --git a/.lycheeignore b/.lycheeignore new file mode 100644 index 00000000000..335800e6047 --- /dev/null +++ b/.lycheeignore @@ -0,0 +1,12 @@ +# example opensearch/dashboards ports +http://localhost:5601 +http://localhost:9200 + +# sites that block scraping +https://hg.openjdk.org/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/ +https://www.linkedin.com/in/*/ + +# we have many pull requests links due to changelogs, skip these as they're unlikely to break and it saves API usage +https://github.com/opendistro-for-elasticsearch/sql/pull/* +https://github.com/opensearch-project/sql/pull/* + diff --git a/README.md b/README.md index 4a7e1e5ec9e..4059fd68ac2 100644 --- a/README.md +++ b/README.md @@ -55,8 +55,6 @@ The following projects are related to the SQL plugin, but stored in the differen | [![untriaged open][untriaged-badge]][untriaged-link] | | [![nolabel open][nolabel-badge]][nolabel-link] | -[dco-badge]: https://github.com/opensearch-project/sql/actions/workflows/dco.yml/badge.svg -[dco-badge-link]: https://github.com/opensearch-project/sql/actions/workflows/dco.yml [link-check-badge]: https://github.com/opensearch-project/sql/actions/workflows/link-checker.yml/badge.svg [link-check-link]: https://github.com/opensearch-project/sql/actions/workflows/link-checker.yml [bwc-tests-badge]: https://img.shields.io/badge/BWC%20tests-in%20progress-yellow diff --git a/async-query-core/README.md b/async-query-core/README.md index 08301c024d7..f6d08ad7103 100644 --- a/async-query-core/README.md +++ b/async-query-core/README.md @@ -16,7 +16,7 @@ Following is the list of extension points where the consumer of the library need - Data store interface - [AsyncQueryJobMetadataStorageService](src/main/java/org/opensearch/sql/spark/asyncquery/AsyncQueryJobMetadataStorageService.java) - - [SessionStorageService](java/org/opensearch/sql/spark/execution/statestore/SessionStorageService.java) + - [SessionStorageService](src/main/java/org/opensearch/sql/spark/execution/statestore/SessionStorageService.java) - [StatementStorageService](src/main/java/org/opensearch/sql/spark/execution/statestore/StatementStorageService.java) - [FlintIndexMetadataService](src/main/java/org/opensearch/sql/spark/flint/FlintIndexMetadataService.java) - [FlintIndexStateModelService](src/main/java/org/opensearch/sql/spark/flint/FlintIndexStateModelService.java) @@ -39,4 +39,4 @@ To update the grammar files, update `build.gradle` file (in `downloadG4Files` ta ``` ./gradlew async-query-core:downloadG4Files ``` -This will overwrite the files under `src/main/antlr`. \ No newline at end of file +This will overwrite the files under `src/main/antlr`. diff --git a/benchmarks/README.md b/benchmarks/README.md index 6a720c7201c..7467e37f2f0 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -1,6 +1,6 @@ # OpenSearch SQL/PPL Microbenchmark Suite -This directory contains the microbenchmark suite of OpenSearch SQL/PPL. It relies on [JMH](http://openjdk.java.net/projects/code-tools/jmh/). +This directory contains the microbenchmark suite of OpenSearch SQL/PPL. It relies on [JMH](https://openjdk.java.net/projects/code-tools/jmh/). ## Purpose @@ -25,6 +25,6 @@ Run specific benchmarks using the `-Pjmh.includes` parameter: ## Adding Microbenchmarks -Before adding a new microbenchmark, make yourself familiar with the JMH API. You can check our existing microbenchmarks and also the [JMH samples](http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/). +Before adding a new microbenchmark, make yourself familiar with the JMH API. You can check our existing microbenchmarks and also the [JMH samples](https://hg.openjdk.org/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/). -In contrast to tests, the actual name of the benchmark class is not relevant to JMH. However, stick to the naming convention and end the class name of a benchmark with `Benchmark`. To have JMH execute a benchmark, annotate the respective methods with `@Benchmark`. \ No newline at end of file +In contrast to tests, the actual name of the benchmark class is not relevant to JMH. However, stick to the naming convention and end the class name of a benchmark with `Benchmark`. To have JMH execute a benchmark, annotate the respective methods with `@Benchmark`. diff --git a/docs/attributions.md b/docs/attributions.md index cc16c35bae0..b5eb089181b 100644 --- a/docs/attributions.md +++ b/docs/attributions.md @@ -22,7 +22,7 @@ Apart from the problems we identified earlier, we made significant improvement i 1. *Integration Test*: We migrated all integrate tests to standard OpenSearch IT framework which spins up in-memory cluster for testing. Now all test cases treat plugin code as blackbox and verify functionality from externally. 2. *New JDBC Driver*: We developed our own JDBC driver without any dependency on Elasticsearch proprietary code. - [sql-jdbc](https://github.com/opensearch-project/sql/tree/main/sql-jdbc) + [sql-jdbc](https://github.com/opensearch-project/sql-jdbc) 3. *Better Hash JOIN*: Block Hash Join implementation with circuit break mechanism protect your OpenSearch memory. Performance testing showed our implementation is 1.5 ~ 2x better than old hash join in terms of throughput and latency and much lower error rate under heavy pressure. 4. *Query Planner*: Logical and physical planner was added to support JOIN query in efficient and extendible way. 5. *PartiQL Compatibility*: we are partially compatible with PartiQL specification which allows for query involved in nested JSON documents. diff --git a/docs/dev/intro-v3-architecture.md b/docs/dev/intro-v3-architecture.md index fc5bf3237d1..4f51fd1016b 100644 --- a/docs/dev/intro-v3-architecture.md +++ b/docs/dev/intro-v3-architecture.md @@ -24,7 +24,7 @@ PPL is specifically designed to simplify tasks in observability and security ana The current PPL engine (shared with SQL v2 engine) is built with custom components, including a parser, analyzer, optimizer, and relies heavily on OpenSearch DSL capabilities to execute query plans. By aligning its syntax and concepts with familiar languages like Splunk SPL and SQL, we aim to streamline migration for users from these backgrounds, allowing them to adopt PPL with minimal effort. The lack of comprehensive ability is a critical blocker for Splunk-to-OpenSearch migrations. We added ~20 new commands in PPL-on-Spark, but there are still dozens of command gaps to be filled. Not to mention that there are still a large number of functions to be implemented. ### 2.2 Lack of Unified PPL Experience -The PPL language is currently inconsistent across [PPL-on-OpenSearch](https://github.com/opensearch-project/sql/blob/main/ppl/src/main/antlr/OpenSearchPPLParser.g4) and [PPL-on-Spark](https://github.com/opensearch-project/opensearch-spark/blob/main/ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4). There are a lot of new commands added in PPL-on-Spark, such as `join`, `lookup` and `subsearch` are not yet supported in PPL-on-OpenSearch. As more and more new commands and functions are implemented in PPL-on-Spark, this gap will continue to widen. +The PPL language is currently inconsistent across [PPL-on-OpenSearch](https://github.com/opensearch-project/sql/blob/main/ppl/src/main/antlr/OpenSearchPPLParser.g4) and [PPL-on-Spark](https://github.com/opensearch-project/sql/blob/main/language-grammar/src/main/antlr4/OpenSearchPPLParser.g4). There are a lot of new commands added in PPL-on-Spark, such as `join`, `lookup` and `subsearch` are not yet supported in PPL-on-OpenSearch. As more and more new commands and functions are implemented in PPL-on-Spark, this gap will continue to widen. ### 2.3 Lack of mature query optimizer Although the v2 engine framework comes with an optimizer class, it only has a few pushdown optimization rules and lacks of mature optimization rules and cost-based optimizer like those found in traditional databases. Query performance and scalability are core to PPL's design, enabling it to efficiently handle high-performance queries and scale to support large datasets and complex queries. diff --git a/docs/dev/intro-v3-engine.md b/docs/dev/intro-v3-engine.md index fd73cd5c8e1..2d11f30aeaa 100644 --- a/docs/dev/intro-v3-engine.md +++ b/docs/dev/intro-v3-engine.md @@ -89,5 +89,5 @@ If you're interested in the new query engine, please find more details in [V3 Ar The following items are on our roadmap with high priority: - Resolve the [V3 limitation](#33-limitations). - Advancing pushdown optimization and benchmarking -- Unified the PPL syntax between [PPL-on-OpenSearch](https://github.com/opensearch-project/sql/blob/main/ppl/src/main/antlr/OpenSearchPPLParser.g4) and [PPL-on-Spark](https://github.com/opensearch-project/opensearch-spark/blob/main/ppl-spark-integration/src/main/antlr4/OpenSearchPPLParser.g4) +- Unified the PPL syntax between [PPL-on-OpenSearch](https://github.com/opensearch-project/sql/blob/main/ppl/src/main/antlr/OpenSearchPPLParser.g4) and [PPL-on-Spark](https://github.com/opensearch-project/sql/blob/main/language-grammar/src/main/antlr4/OpenSearchPPLParser.g4) - Support more DSL aggregation diff --git a/docs/dev/opensearch-relevancy-search.md b/docs/dev/opensearch-relevancy-search.md index 9b25cc757dd..94a63e9ed41 100644 --- a/docs/dev/opensearch-relevancy-search.md +++ b/docs/dev/opensearch-relevancy-search.md @@ -119,7 +119,7 @@ Besides, one of the query engine architecture tenets is to keep the job of every #### Option B: Create a new query plan node dedicated for the search features. -The diagram below is simplified with only the logical plan and physical plan sections and leaves out others. Please check out [OpenSearch SQL Engine Architecture](https://github.com/opensearch-project/sql/blob/main/docs/dev/Architecture.md) for the complete architecture of the query engine. +The diagram below is simplified with only the logical plan and physical plan sections and leaves out others. Please check out [OpenSearch SQL Engine Architecture](https://github.com/opensearch-project/sql/blob/main/docs/dev/intro-architecture.md) for the complete architecture of the query engine. ![relevance-Page-2 (4)](https://user-images.githubusercontent.com/33583073/129938534-28fa4845-4246-4707-9519-e68c9e86d174.png) @@ -291,4 +291,4 @@ All the code changes should be test driven. The pull requests should include uni ### A2. Search flow in search engine -![relevance](https://user-images.githubusercontent.com/33583073/129938659-5b49f43d-a83f-47d5-be5b-937b1c96e5bc.png) \ No newline at end of file +![relevance](https://user-images.githubusercontent.com/33583073/129938659-5b49f43d-a83f-47d5-be5b-937b1c96e5bc.png) diff --git a/docs/dev/testing-doctest.md b/docs/dev/testing-doctest.md index e8b88f19b0f..86fee86bd64 100644 --- a/docs/dev/testing-doctest.md +++ b/docs/dev/testing-doctest.md @@ -192,7 +192,7 @@ Doctest is relying on the console/command line to run code examples in documenta * https://github.com/crate/crate/blob/master/docs/general/dql/selects.rst -Similar to CrateDB using it’s CLI “crash”, we can make use of our own [SQL-CLI](https://github.com/opensearch-project/sql/tree/main/sql-cli) +Similar to CrateDB using it’s CLI “crash”, we can make use of our own [SQL-CLI](https://github.com/opensearch-project/sql-cli) To support PPL, we need to add PPL support to SQL-CLI. Since PPL and SQL expose similar http endpoint for query and share similar response format. The update won’t be much of work. diff --git a/release-notes/opensearch-sql.release-notes-2.10.0.0.md b/release-notes/opensearch-sql.release-notes-2.10.0.0.md index f1a730ce46b..67ce2c09275 100644 --- a/release-notes/opensearch-sql.release-notes-2.10.0.0.md +++ b/release-notes/opensearch-sql.release-notes-2.10.0.0.md @@ -40,4 +40,4 @@ Compatible with OpenSearch and OpenSearch Dashboards Version 2.10.0 * [Backport 2.x] [spotless] Removes Checkstyle in favor of spotless by @MitchellGale in https://github.com/opensearch-project/sql/pull/2018 * [Backport 2.x] [Spotless] Entire project running spotless by @MitchellGale in https://github.com/opensearch-project/sql/pull/2016 --- -**Full Changelog**: https://github.com/opensearch-project/sql/compare/2.3.0.0...v.2.10.0.0 \ No newline at end of file +**Full Changelog**: https://github.com/opensearch-project/sql/compare/2.3.0.0...2.10.0.0 diff --git a/release-notes/opensearch-sql.release-notes-2.11.0.0.md b/release-notes/opensearch-sql.release-notes-2.11.0.0.md index a560d5c8dd7..a517b86bc48 100644 --- a/release-notes/opensearch-sql.release-notes-2.11.0.0.md +++ b/release-notes/opensearch-sql.release-notes-2.11.0.0.md @@ -52,4 +52,4 @@ Compatible with OpenSearch and OpenSearch Dashboards Version 2.11.0 * bump okio to 3.4.0 by @joshuali925 in https://github.com/opensearch-project/sql/pull/2047 --- -**Full Changelog**: https://github.com/opensearch-project/sql/compare/2.3.0.0...v.2.11.0.0 \ No newline at end of file +**Full Changelog**: https://github.com/opensearch-project/sql/compare/2.3.0.0...2.11.0.0