Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .copyrightconfig
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,4 @@ startyear: 2024
# - Dotfiles already skipped automatically
# Enable by removing the leading '# ' from the next line and editing values.
# filesexcluded: third_party/*, docs/generated/*.md, assets/*.png, scripts/temp_*.py, vendor/lib.js
filesexcluded: .github/*, README.md, CONTRIBUTING.md, Jenkinsfile, gradle/*, *.yml, *.gradle, gradlew, gradlew.bat, *.json, *.xml, *.txt, **/test/resources/**, docs/**, flux-cli/hadoop/*, CODEOWNERS, *.properties, *.md, *.xqy, *version, *.lock
filesexcluded: .github/*, README.md, CONTRIBUTING.md, Jenkinsfile, gradle/*, *.yml, *.gradle, gradlew, gradlew.bat, *.json, *.xml, *.txt, **/test/resources/**, docs/**, flux-cli/hadoop/*, CODEOWNERS, *.properties, *.md, *.xqy, *version, *.lock, *.html
6 changes: 3 additions & 3 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,15 +198,15 @@ are all synonyms):

./gradlew shadowJar

This will produce an assembly jar at `./flux-cli/build/libs/marklogic-flux-2.0-SNAPSHOT-all.jar`.
This will produce an assembly jar at `./flux-cli/build/libs/marklogic-flux-2.2-SNAPSHOT-all.jar`.

You can now run any CLI command via spark-submit. This is an example of previewing an import of files - change the value
of `--path`, as an absolute path is needed, and of course change the value of `--master` to match that of your Spark
cluster:

```
$SPARK_HOME/bin/spark-submit --class com.marklogic.flux.spark.Submit \
--master spark://NYWHYC3G0W:7077 flux-cli/build/libs/marklogic-flux-2.0-SNAPSHOT-all.jar \
--master spark://NYWHYC3G0W:7077 flux-cli/build/libs/marklogic-flux-2.2-SNAPSHOT-all.jar \
import-files --path /Users/rudin/workspace/flux/flux-cli/src/test/resources/mixed-files \
--connection-string "admin:admin@localhost:8000" \
--preview 5 --preview-drop content
Expand All @@ -223,7 +223,7 @@ to something you can access):
$SPARK_HOME/bin/spark-submit --class com.marklogic.flux.spark.Submit \
--packages org.apache.hadoop:hadoop-aws:3.3.4,org.apache.hadoop:hadoop-client:3.3.4 \
--master spark://NYWHYC3G0W:7077 \
flux-cli/build/libs/marklogic-flux-1.2-SNAPSHOT-all.jar \
flux-cli/build/libs/marklogic-flux-2.2-SNAPSHOT-all.jar \
import-files --path "s3a://changeme/" \
--connection-string "admin:admin@localhost:8000" \
--s3-add-credentials \
Expand Down
561 changes: 240 additions & 321 deletions NOTICE.txt

Large diffs are not rendered by default.

11 changes: 10 additions & 1 deletion docs/assets/javadoc/allclasses-index.html
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html lang="en">
<head>
<!-- Generated by javadoc (17) -->
<title>All Classes and Interfaces (flux-cli 2.0.0 API)</title>
<title>All Classes and Interfaces (flux-cli 2.1.0 API)</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="description" content="class index">
Expand Down Expand Up @@ -34,6 +34,7 @@
<li><a href="com/marklogic/flux/api/package-summary.html">Package</a></li>
<li>Class</li>
<li><a href="overview-tree.html">Tree</a></li>
<li><a href="deprecated-list.html">Deprecated</a></li>
<li><a href="index-all.html">Index</a></li>
<li><a href="help-doc.html#all-classes">Help</a></li>
</ul>
Expand Down Expand Up @@ -282,6 +283,14 @@ <h1 title="All Classes and Interfaces" class="title">All Classes and Interfaces<
<div class="col-last even-row-color all-classes-table all-classes-table-tab3">&nbsp;</div>
<div class="col-first odd-row-color all-classes-table all-classes-table-tab3"><a href="com/marklogic/flux/api/SslHostnameVerifier.html" title="enum class in com.marklogic.flux.api">SslHostnameVerifier</a></div>
<div class="col-last odd-row-color all-classes-table all-classes-table-tab3">&nbsp;</div>
<div class="col-first even-row-color all-classes-table all-classes-table-tab1"><a href="com/marklogic/flux/api/StructuredDataImporter.html" title="interface in com.marklogic.flux.api">StructuredDataImporter</a>&lt;<a href="com/marklogic/flux/api/StructuredDataImporter.html" title="type parameter in StructuredDataImporter">T</a> extends <a href="com/marklogic/flux/api/StructuredDataImporter.html" title="interface in com.marklogic.flux.api">StructuredDataImporter</a>&lt;<a href="com/marklogic/flux/api/StructuredDataImporter.html" title="type parameter in StructuredDataImporter">T</a>&gt;&gt;</div>
<div class="col-last even-row-color all-classes-table all-classes-table-tab1">
<div class="block">Base interface for importers that read structured data with column-based schemas.</div>
</div>
<div class="col-first odd-row-color all-classes-table all-classes-table-tab1"><a href="com/marklogic/flux/api/StructuredDataImporter.GroupByOptions.html" title="interface in com.marklogic.flux.api">StructuredDataImporter.GroupByOptions</a>&lt;<a href="com/marklogic/flux/api/StructuredDataImporter.GroupByOptions.html" title="type parameter in StructuredDataImporter.GroupByOptions">T</a> extends <a href="com/marklogic/flux/api/StructuredDataImporter.GroupByOptions.html" title="interface in com.marklogic.flux.api">StructuredDataImporter.GroupByOptions</a>&lt;<a href="com/marklogic/flux/api/StructuredDataImporter.GroupByOptions.html" title="type parameter in StructuredDataImporter.GroupByOptions">T</a>&gt;&gt;</div>
<div class="col-last odd-row-color all-classes-table all-classes-table-tab1">
<div class="block">Options for configuring aggregation behavior when grouping rows.</div>
</div>
<div class="col-first even-row-color all-classes-table all-classes-table-tab1"><a href="com/marklogic/flux/api/TdeOptions.html" title="interface in com.marklogic.flux.api">TdeOptions</a></div>
<div class="col-last even-row-color all-classes-table all-classes-table-tab1">&nbsp;</div>
<div class="col-first odd-row-color all-classes-table all-classes-table-tab1"><a href="com/marklogic/flux/api/WriteDocumentsOptions.html" title="interface in com.marklogic.flux.api">WriteDocumentsOptions</a>&lt;<a href="com/marklogic/flux/api/WriteDocumentsOptions.html" title="type parameter in WriteDocumentsOptions">T</a> extends <a href="com/marklogic/flux/api/WriteDocumentsOptions.html" title="interface in com.marklogic.flux.api">WriteDocumentsOptions</a>&gt;</div>
Expand Down
3 changes: 2 additions & 1 deletion docs/assets/javadoc/allpackages-index.html
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html lang="en">
<head>
<!-- Generated by javadoc (17) -->
<title>All Packages (flux-cli 2.0.0 API)</title>
<title>All Packages (flux-cli 2.1.0 API)</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="description" content="package index">
Expand Down Expand Up @@ -30,6 +30,7 @@
<li><a href="com/marklogic/flux/api/package-summary.html">Package</a></li>
<li>Class</li>
<li><a href="overview-tree.html">Tree</a></li>
<li><a href="deprecated-list.html">Deprecated</a></li>
<li><a href="index-all.html">Index</a></li>
<li><a href="help-doc.html#all-packages">Help</a></li>
</ul>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html lang="en">
<head>
<!-- Generated by javadoc (17) -->
<title>AggregateJsonFilesImporter.ReadJsonFilesOptions (flux-cli 2.0.0 API)</title>
<title>AggregateJsonFilesImporter.ReadJsonFilesOptions (flux-cli 2.1.0 API)</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="description" content="declaration: package: com.marklogic.flux.api, interface: AggregateJsonFilesImporter, interface: ReadJsonFilesOptions">
Expand Down Expand Up @@ -34,6 +34,7 @@
<li><a href="package-summary.html">Package</a></li>
<li class="nav-bar-cell1-rev">Class</li>
<li><a href="package-tree.html">Tree</a></li>
<li><a href="../../../../deprecated-list.html">Deprecated</a></li>
<li><a href="../../../../index-all.html">Index</a></li>
<li><a href="../../../../help-doc.html#class">Help</a></li>
</ul>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html lang="en">
<head>
<!-- Generated by javadoc (17) -->
<title>AggregateJsonFilesImporter (flux-cli 2.0.0 API)</title>
<title>AggregateJsonFilesImporter (flux-cli 2.1.0 API)</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="description" content="declaration: package: com.marklogic.flux.api, interface: AggregateJsonFilesImporter">
Expand Down Expand Up @@ -34,6 +34,7 @@
<li><a href="package-summary.html">Package</a></li>
<li class="nav-bar-cell1-rev">Class</li>
<li><a href="package-tree.html">Tree</a></li>
<li><a href="../../../../deprecated-list.html">Deprecated</a></li>
<li><a href="../../../../index-all.html">Index</a></li>
<li><a href="../../../../help-doc.html#class">Help</a></li>
</ul>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html lang="en">
<head>
<!-- Generated by javadoc (17) -->
<title>AggregateXmlFilesImporter.ReadXmlFilesOptions (flux-cli 2.0.0 API)</title>
<title>AggregateXmlFilesImporter.ReadXmlFilesOptions (flux-cli 2.1.0 API)</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="description" content="declaration: package: com.marklogic.flux.api, interface: AggregateXmlFilesImporter, interface: ReadXmlFilesOptions">
Expand Down Expand Up @@ -34,6 +34,7 @@
<li><a href="package-summary.html">Package</a></li>
<li class="nav-bar-cell1-rev">Class</li>
<li><a href="package-tree.html">Tree</a></li>
<li><a href="../../../../deprecated-list.html">Deprecated</a></li>
<li><a href="../../../../index-all.html">Index</a></li>
<li><a href="../../../../help-doc.html#class">Help</a></li>
</ul>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html lang="en">
<head>
<!-- Generated by javadoc (17) -->
<title>AggregateXmlFilesImporter (flux-cli 2.0.0 API)</title>
<title>AggregateXmlFilesImporter (flux-cli 2.1.0 API)</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="description" content="declaration: package: com.marklogic.flux.api, interface: AggregateXmlFilesImporter">
Expand Down Expand Up @@ -34,6 +34,7 @@
<li><a href="package-summary.html">Package</a></li>
<li class="nav-bar-cell1-rev">Class</li>
<li><a href="package-tree.html">Tree</a></li>
<li><a href="../../../../deprecated-list.html">Deprecated</a></li>
<li><a href="../../../../index-all.html">Index</a></li>
<li><a href="../../../../help-doc.html#class">Help</a></li>
</ul>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html lang="en">
<head>
<!-- Generated by javadoc (17) -->
<title>ArchiveFilesExporter.ReadArchiveDocumentOptions (flux-cli 2.0.0 API)</title>
<title>ArchiveFilesExporter.ReadArchiveDocumentOptions (flux-cli 2.1.0 API)</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="description" content="declaration: package: com.marklogic.flux.api, interface: ArchiveFilesExporter, interface: ReadArchiveDocumentOptions">
Expand Down Expand Up @@ -34,6 +34,7 @@
<li><a href="package-summary.html">Package</a></li>
<li class="nav-bar-cell1-rev">Class</li>
<li><a href="package-tree.html">Tree</a></li>
<li><a href="../../../../deprecated-list.html">Deprecated</a></li>
<li><a href="../../../../index-all.html">Index</a></li>
<li><a href="../../../../help-doc.html#class">Help</a></li>
</ul>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html lang="en">
<head>
<!-- Generated by javadoc (17) -->
<title>ArchiveFilesExporter.WriteArchiveFilesOptions (flux-cli 2.0.0 API)</title>
<title>ArchiveFilesExporter.WriteArchiveFilesOptions (flux-cli 2.1.0 API)</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="description" content="declaration: package: com.marklogic.flux.api, interface: ArchiveFilesExporter, interface: WriteArchiveFilesOptions">
Expand Down Expand Up @@ -34,6 +34,7 @@
<li><a href="package-summary.html">Package</a></li>
<li class="nav-bar-cell1-rev">Class</li>
<li><a href="package-tree.html">Tree</a></li>
<li><a href="../../../../deprecated-list.html">Deprecated</a></li>
<li><a href="../../../../index-all.html">Index</a></li>
<li><a href="../../../../help-doc.html#class">Help</a></li>
</ul>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html lang="en">
<head>
<!-- Generated by javadoc (17) -->
<title>ArchiveFilesExporter (flux-cli 2.0.0 API)</title>
<title>ArchiveFilesExporter (flux-cli 2.1.0 API)</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="description" content="declaration: package: com.marklogic.flux.api, interface: ArchiveFilesExporter">
Expand Down Expand Up @@ -34,6 +34,7 @@
<li><a href="package-summary.html">Package</a></li>
<li class="nav-bar-cell1-rev">Class</li>
<li><a href="package-tree.html">Tree</a></li>
<li><a href="../../../../deprecated-list.html">Deprecated</a></li>
<li><a href="../../../../index-all.html">Index</a></li>
<li><a href="../../../../help-doc.html#class">Help</a></li>
</ul>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html lang="en">
<head>
<!-- Generated by javadoc (17) -->
<title>ArchiveFilesImporter.ReadArchiveFilesOptions (flux-cli 2.0.0 API)</title>
<title>ArchiveFilesImporter.ReadArchiveFilesOptions (flux-cli 2.1.0 API)</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="description" content="declaration: package: com.marklogic.flux.api, interface: ArchiveFilesImporter, interface: ReadArchiveFilesOptions">
Expand Down Expand Up @@ -34,6 +34,7 @@
<li><a href="package-summary.html">Package</a></li>
<li class="nav-bar-cell1-rev">Class</li>
<li><a href="package-tree.html">Tree</a></li>
<li><a href="../../../../deprecated-list.html">Deprecated</a></li>
<li><a href="../../../../index-all.html">Index</a></li>
<li><a href="../../../../help-doc.html#class">Help</a></li>
</ul>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html lang="en">
<head>
<!-- Generated by javadoc (17) -->
<title>ArchiveFilesImporter.WriteArchiveDocumentsOptions (flux-cli 2.0.0 API)</title>
<title>ArchiveFilesImporter.WriteArchiveDocumentsOptions (flux-cli 2.1.0 API)</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="description" content="declaration: package: com.marklogic.flux.api, interface: ArchiveFilesImporter, interface: WriteArchiveDocumentsOptions">
Expand Down Expand Up @@ -34,6 +34,7 @@
<li><a href="package-summary.html">Package</a></li>
<li class="nav-bar-cell1-rev">Class</li>
<li><a href="package-tree.html">Tree</a></li>
<li><a href="../../../../deprecated-list.html">Deprecated</a></li>
<li><a href="../../../../index-all.html">Index</a></li>
<li><a href="../../../../help-doc.html#class">Help</a></li>
</ul>
Expand Down Expand Up @@ -101,7 +102,12 @@ <h2>Method Summary</h2>
<div class="table-header col-last">Description</div>
<div class="col-first even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab3"><code><a href="ArchiveFilesImporter.WriteArchiveDocumentsOptions.html" title="interface in com.marklogic.flux.api">ArchiveFilesImporter.WriteArchiveDocumentsOptions</a></code></div>
<div class="col-second even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab3"><code><a href="#documentType(com.marklogic.flux.api.DocumentType)" class="member-name-link">documentType</a><wbr>(<a href="DocumentType.html" title="enum class in com.marklogic.flux.api">DocumentType</a>&nbsp;documentType)</code></div>
<div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab3">&nbsp;</div>
<div class="col-last even-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab3">
<div class="block">Force a document type for any document with an unrecognized URI extension.</div>
</div>
<div class="col-first odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab3"><code><a href="ArchiveFilesImporter.WriteArchiveDocumentsOptions.html" title="interface in com.marklogic.flux.api">ArchiveFilesImporter.WriteArchiveDocumentsOptions</a></code></div>
<div class="col-second odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab3"><code><a href="#streamingTransformBinaryWithExtension(java.lang.String)" class="member-name-link">streamingTransformBinaryWithExtension</a><wbr>(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>&nbsp;commaDelimitedExtensions)</code></div>
<div class="col-last odd-row-color method-summary-table method-summary-table-tab2 method-summary-table-tab3">&nbsp;</div>
</div>
</div>
</div>
Expand All @@ -123,6 +129,23 @@ <h2>Method Details</h2>
<section class="detail" id="documentType(com.marklogic.flux.api.DocumentType)">
<h3>documentType</h3>
<div class="member-signature"><span class="return-type"><a href="ArchiveFilesImporter.WriteArchiveDocumentsOptions.html" title="interface in com.marklogic.flux.api">ArchiveFilesImporter.WriteArchiveDocumentsOptions</a></span>&nbsp;<span class="element-name">documentType</span><wbr><span class="parameters">(<a href="DocumentType.html" title="enum class in com.marklogic.flux.api">DocumentType</a>&nbsp;documentType)</span></div>
<div class="block">Force a document type for any document with an unrecognized URI extension.</div>
</section>
</li>
<li>
<section class="detail" id="streamingTransformBinaryWithExtension(java.lang.String)">
<h3>streamingTransformBinaryWithExtension</h3>
<div class="member-signature"><span class="return-type"><a href="ArchiveFilesImporter.WriteArchiveDocumentsOptions.html" title="interface in com.marklogic.flux.api">ArchiveFilesImporter.WriteArchiveDocumentsOptions</a></span>&nbsp;<span class="element-name">streamingTransformBinaryWithExtension</span><wbr><span class="parameters">(<a href="https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/lang/String.html" title="class or interface in java.lang" class="external-link">String</a>&nbsp;commaDelimitedExtensions)</span></div>
<dl class="notes">
<dt>Parameters:</dt>
<dd><code>commaDelimitedExtensions</code> - A comma-delimited list of file extensions intended to be used when streaming
entries from an archive and also using a REST transform to convert documents
with format=BINARY and one of the given file extensions into a binary. A
REST transform is required to convert a document to a binary when its URI
extension matches a MarkLogic mimetype that is not binary.</dd>
<dt>Since:</dt>
<dd>2.1.0</dd>
</dl>
</section>
</li>
</ul>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html lang="en">
<head>
<!-- Generated by javadoc (17) -->
<title>ArchiveFilesImporter (flux-cli 2.0.0 API)</title>
<title>ArchiveFilesImporter (flux-cli 2.1.0 API)</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="description" content="declaration: package: com.marklogic.flux.api, interface: ArchiveFilesImporter">
Expand Down Expand Up @@ -34,6 +34,7 @@
<li><a href="package-summary.html">Package</a></li>
<li class="nav-bar-cell1-rev">Class</li>
<li><a href="package-tree.html">Tree</a></li>
<li><a href="../../../../deprecated-list.html">Deprecated</a></li>
<li><a href="../../../../index-all.html">Index</a></li>
<li><a href="../../../../help-doc.html#class">Help</a></li>
</ul>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
<html lang="en">
<head>
<!-- Generated by javadoc (17) -->
<title>AuthenticationType (flux-cli 2.0.0 API)</title>
<title>AuthenticationType (flux-cli 2.1.0 API)</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="description" content="declaration: package: com.marklogic.flux.api, enum: AuthenticationType">
Expand Down Expand Up @@ -34,6 +34,7 @@
<li><a href="package-summary.html">Package</a></li>
<li class="nav-bar-cell1-rev">Class</li>
<li><a href="package-tree.html">Tree</a></li>
<li><a href="../../../../deprecated-list.html">Deprecated</a></li>
<li><a href="../../../../index-all.html">Index</a></li>
<li><a href="../../../../help-doc.html#class">Help</a></li>
</ul>
Expand Down
Loading
Loading