Skip to content

Commit 548325a

Browse files
JingsongLiJingsongLi
committed
[doc] Separate PyPaimon documentations
1 parent 8706428 commit 548325a

9 files changed

Lines changed: 369 additions & 194 deletions

File tree

‎docs/content/concepts/rest/overview.md‎

Lines changed: 0 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -66,11 +66,3 @@ RESTCatalog supports multiple access authentication methods, including the follo
6666
## REST Open API
6767

6868
See [REST API]({{< ref "concepts/rest/rest-api" >}}).
69-
70-
## REST Java API
71-
72-
See [REST Java API]({{< ref "program-api/rest-api" >}}).
73-
74-
## REST Python API
75-
76-
See [REST Python API]({{< ref "program-api/python-api" >}}).

‎docs/content/maintenance/_index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ title: Maintenance
33
icon: <i class="fa fa-wrench title maindish" aria-hidden="true"></i>
44
bold: true
55
bookCollapseSection: true
6-
weight: 95
6+
weight: 94
77
---
88
<!--
99
Licensed to the Apache Software Foundation (ASF) under one

‎docs/content/program-api/_index.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ title: Program API
33
icon: <i class="fa fa-briefcase title maindish" aria-hidden="true"></i>
44
bold: true
55
bookCollapseSection: true
6-
weight: 96
6+
weight: 95
77
---
88
<!--
99
Licensed to the Apache Software Foundation (ASF) under one

‎docs/content/pypaimon/_index.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
---
2+
title: PyPaimon
3+
icon: <i class="fa fa-briefcase title maindish" aria-hidden="true"></i>
4+
bold: true
5+
bookCollapseSection: true
6+
weight: 96
7+
---
8+
<!--
9+
Licensed to the Apache Software Foundation (ASF) under one
10+
or more contributor license agreements. See the NOTICE file
11+
distributed with this work for additional information
12+
regarding copyright ownership. The ASF licenses this file
13+
to you under the Apache License, Version 2.0 (the
14+
"License"); you may not use this file except in compliance
15+
with the License. You may obtain a copy of the License at
16+
17+
http://www.apache.org/licenses/LICENSE-2.0
18+
19+
Unless required by applicable law or agreed to in writing,
20+
software distributed under the License is distributed on an
21+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
22+
KIND, either express or implied. See the License for the
23+
specific language governing permissions and limitations
24+
under the License.
25+
-->

‎docs/content/pypaimon/data-evolution.md‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
---
2+
title: "Data Evolution"
3+
weight: 5
4+
type: docs
5+
aliases:
6+
- /pypaimon/data-evolution.html
7+
---
8+
9+
<!--
10+
Licensed to the Apache Software Foundation (ASF) under one
11+
or more contributor license agreements. See the NOTICE file
12+
distributed with this work for additional information
13+
regarding copyright ownership. The ASF licenses this file
14+
to you under the Apache License, Version 2.0 (the
15+
"License"); you may not use this file except in compliance
16+
with the License. You may obtain a copy of the License at
17+
18+
http://www.apache.org/licenses/LICENSE-2.0
19+
20+
Unless required by applicable law or agreed to in writing,
21+
software distributed under the License is distributed on an
22+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
23+
KIND, either express or implied. See the License for the
24+
specific language governing permissions and limitations
25+
under the License.
26+
-->
27+
28+
# Data Evolution
29+
30+
PyPaimon for Data Evolution mode. See [Data Evolution]({{< ref "append-table/data-evolution" >}}).
31+
32+
## Update Columns By Row ID
33+
34+
You can create `TableUpdate.update_by_arrow_with_row_id` to update columns to data evolution tables.
35+
36+
The input data should include the `_ROW_ID` column, update operation will automatically sort and match each `_ROW_ID` to
37+
its corresponding `first_row_id`, then groups rows with the same `first_row_id` and writes them to a separate file.
38+
39+
```python
40+
simple_pa_schema = pa.schema([
41+
('f0', pa.int8()),
42+
('f1', pa.int16()),
43+
])
44+
schema = Schema.from_pyarrow_schema(simple_pa_schema,
45+
options={'row-tracking.enabled': 'true', 'data-evolution.enabled': 'true'})
46+
catalog.create_table('default.test_row_tracking', schema, False)
47+
table = catalog.get_table('default.test_row_tracking')
48+
49+
# write all columns
50+
write_builder = table.new_batch_write_builder()
51+
table_write = write_builder.new_write()
52+
table_commit = write_builder.new_commit()
53+
expect_data = pa.Table.from_pydict({
54+
'f0': [-1, 2],
55+
'f1': [-1001, 1002]
56+
}, schema=simple_pa_schema)
57+
table_write.write_arrow(expect_data)
58+
table_commit.commit(table_write.prepare_commit())
59+
table_write.close()
60+
table_commit.close()
61+
62+
# update partial columns
63+
write_builder = table.new_batch_write_builder()
64+
table_update = write_builder.new_update().with_update_type(['f0'])
65+
table_commit = write_builder.new_commit()
66+
data2 = pa.Table.from_pydict({
67+
'_ROW_ID': [0, 1],
68+
'f0': [5, 6],
69+
}, schema=pa.schema([
70+
('_ROW_ID', pa.int64()),
71+
('f0', pa.int8()),
72+
]))
73+
cmts = table_update.update_by_arrow_with_row_id(data2)
74+
table_commit.commit(cmts)
75+
table_commit.close()
76+
77+
# content should be:
78+
# 'f0': [5, 6],
79+
# 'f1': [-1001, 1002]
80+
```

‎docs/content/pypaimon/overview.md‎

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
---
2+
title: "Overview"
3+
weight: 1
4+
type: docs
5+
aliases:
6+
- /pypaimon/overview.html
7+
---
8+
<!--
9+
Licensed to the Apache Software Foundation (ASF) under one
10+
or more contributor license agreements. See the NOTICE file
11+
distributed with this work for additional information
12+
regarding copyright ownership. The ASF licenses this file
13+
to you under the Apache License, Version 2.0 (the
14+
"License"); you may not use this file except in compliance
15+
with the License. You may obtain a copy of the License at
16+
17+
http://www.apache.org/licenses/LICENSE-2.0
18+
19+
Unless required by applicable law or agreed to in writing,
20+
software distributed under the License is distributed on an
21+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
22+
KIND, either express or implied. See the License for the
23+
specific language governing permissions and limitations
24+
under the License.
25+
-->
26+
27+
# Overview
28+
29+
PyPaimon is a Python implementation for connecting Paimon catalog, reading & writing tables. The complete Python
30+
implementation of the brand new PyPaimon does not require JDK installation.
31+
32+
## Environment Settings
33+
34+
SDK is published at [pypaimon](https://pypi.org/project/pypaimon/). You can install by
35+
36+
```shell
37+
pip install pypaimon
38+
```
39+
40+
## Build From Source
41+
42+
You can build the source package by executing the following command:
43+
44+
```commandline
45+
python3 setup.py sdist
46+
```
47+
48+
The package is under `dist/`. Then you can install the package by executing the following command:
49+
50+
```commandline
51+
pip3 install dist/*.tar.gz
52+
```
53+
54+
The command will install the package and core dependencies to your local Python environment.

0 commit comments

Comments
 (0)