Skip to content

Commit 1775a0d

Browse files
graczhualgraczhual
committed
docs: use template to generate rst files in "Examples"
1 parent c57ea4b commit 1775a0d

13 files changed

Lines changed: 641 additions & 201 deletions

File tree

‎.gitignore‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,14 @@ instance/
7171
# Sphinx documentation
7272
docs/_build/
7373

74+
# Sphinx examples rst files which is generated by the template
75+
docs/source/quick_start/examples/BSTLD.rst
76+
docs/source/quick_start/examples/DogsVsCats.rst
77+
docs/source/quick_start/examples/LeedsSportsPose.rst
78+
docs/source/quick_start/examples/Newsgroups20.rst
79+
docs/source/quick_start/examples/NeolixOD.rst
80+
docs/source/quick_start/examples/THCHS30.rst
81+
7482
# PyBuilder
7583
target/
7684

‎docs/code/LeedsSportsPose.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,10 @@
4343
dataset = Dataset("LeedsSportsPose", gas)
4444
""""""
4545

46+
"""Read Dataset / list segment names"""
47+
dataset.keys()
48+
""""""
49+
4650
"""Read Dataset / get segment"""
4751
segment = dataset[0]
4852
""""""

‎docs/code/NeolixOD.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,6 +42,10 @@
4242
dataset = Dataset("NeolixOD", gas)
4343
""""""
4444

45+
"""Read Dataset / list segment names"""
46+
dataset.keys()
47+
""""""
48+
4549
"""Read Dataset / get segment"""
4650
segment = dataset[0]
4751
""""""

‎docs/source/__init__.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/usr/bin/env python3
2+
#
3+
# Copyright 2021 Graviti. Licensed under MIT License.
4+
#
5+
6+
"""source."""

‎docs/source/_templates/__init__.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/usr/bin/env python3
2+
#
3+
# Copyright 2021 Graviti. Licensed under MIT License.
4+
#
5+
6+
"""template."""

‎docs/source/_templates/examples.py‎

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
"""The template for example rst files."""
2+
3+
EXAMPLES_TEMPLATE = '''
4+
###################
5+
{dataset_name}
6+
###################
7+
8+
This topic describes how to manage the `{dataset_name} Dataset <https://gas.graviti.cn/dataset/
9+
data-decorators/{file_name}>`_, which is a dataset with
10+
:ref:`reference/label_format/{label_type}:{label_type}` label
11+
{figure_description}
12+
13+
*****************************
14+
Authorize a Client Instance
15+
*****************************
16+
17+
An :ref:`reference/glossary:accesskey` is needed to authenticate identity when using TensorBay.
18+
19+
.. literalinclude:: ../../../../docs/code/{file_name}.py
20+
:language: python
21+
:start-after: """Authorize a Client Instance"""
22+
:end-before: """"""
23+
24+
****************
25+
Create Dataset
26+
****************
27+
28+
.. literalinclude:: ../../../../docs/code/{file_name}.py
29+
:language: python
30+
:start-after: """Create Dataset"""
31+
:end-before: """"""
32+
33+
******************
34+
Organize Dataset
35+
******************
36+
37+
Normally, ``dataloader.py`` and ``catalog.json`` are required to organize the "{dataset_name}"
38+
dataset into the :class:`~tensorbay.dataset.dataset.Dataset` instance.
39+
In this example, they are stored in the same directory like::
40+
41+
{dataset_name}/
42+
catalog.json
43+
dataloader.py
44+
45+
Step 1: Write the Catalog
46+
=========================
47+
48+
A :ref:`reference/dataset_structure:catalog` contains all label information of one dataset, which
49+
is typically stored in a json file like ``catalog.json``.
50+
{catalog_description}
51+
52+
{category_attribute_description}
53+
54+
.. note::
55+
56+
By passing the path of the ``catalog.json``, :func:`~tensorbay.dataset.dataset.DatasetBase.
57+
load_catalog` supports loading the catalog into dataset.
58+
59+
.. important::
60+
61+
See :ref:`catalog table <reference/dataset_structure:catalog>` for more catalogs with different
62+
label types.
63+
64+
Step 2: Write the Dataloader
65+
============================
66+
67+
A :ref:`reference/glossary:dataloader` is needed to organize the dataset into a :class:`~tensorbay.
68+
dataset.dataset.Dataset` instance.
69+
70+
.. literalinclude:: ../../../../tensorbay/opendataset/{file_name}/loader.py
71+
:language: python
72+
:name: {file_name}-dataloader
73+
:linenos:
74+
75+
See :ref:`{label_type} annotation <reference/label_format/{label_type}:{label_type}>` for more
76+
details.
77+
78+
There are already a number of dataloaders in TensorBay SDK provided by the community.
79+
Thus, instead of writing, importing an available dataloader is also feasible.
80+
81+
.. literalinclude:: ../../../../docs/code/{file_name}.py
82+
:language: python
83+
:start-after: """Organize dataset / import dataloader"""
84+
:end-before: """"""
85+
86+
.. note::
87+
88+
Note that catalogs are automatically loaded in available dataloaders, users do not have to write
89+
them again.
90+
91+
.. important::
92+
93+
See :ref:`dataloader table <reference/glossary:dataloader>` for dataloaders with different label
94+
types.
95+
96+
*******************
97+
Visualize Dataset
98+
*******************
99+
100+
Optionally, the organized dataset can be visualized by **Pharos**, which is a TensorBay SDK plug-in.
101+
This step can help users to check whether the dataset is correctly organized.
102+
Please see :ref:`features/visualization:Visualization` for more details.
103+
104+
****************
105+
Upload Dataset
106+
****************
107+
108+
The organized "{dataset_name}" dataset can be uploaded to TensorBay for sharing, reuse, etc.
109+
110+
.. literalinclude:: ../../../../docs/code/{file_name}.py
111+
:language: python
112+
:start-after: """Upload Dataset"""
113+
:end-before: """"""
114+
115+
.. note::
116+
Set ``skip_uploaded_files=True`` to skip uploaded data.
117+
The data will be skiped if its name and segment name is the same as remote data.
118+
119+
Similar with Git, the commit step after uploading can record changes to the dataset as a version.
120+
If needed, do the modifications and commit again.
121+
Please see :ref:`features/version_control/index:Version Control` for more details.
122+
123+
**************
124+
Read Dataset
125+
**************
126+
127+
Now "{dataset_name}" dataset can be read from TensorBay.
128+
129+
.. literalinclude:: ../../../../docs/code/{file_name}.py
130+
:language: python
131+
:start-after: """Read Dataset / get dataset"""
132+
:end-before: """"""
133+
134+
Get the segment names by listing them all.
135+
136+
.. literalinclude:: ../../../../docs/code/{file_name}.py
137+
:language: python
138+
:start-after: """Read Dataset / list segment names"""
139+
:end-before: """"""
140+
141+
Get a segment by passing the required segment name.
142+
143+
.. literalinclude:: ../../../../docs/code/{file_name}.py
144+
:language: python
145+
:start-after: """Read Dataset / get segment"""
146+
:end-before: """"""
147+
148+
In the :ref:`reference/dataset_structure:segment`, there is a sequence of
149+
:ref:`reference/dataset_structure:data`, which can be obtained by index.
150+
151+
.. literalinclude:: ../../../../docs/code/{file_name}.py
152+
:language: python
153+
:start-after: """Read Dataset / get data"""
154+
:end-before: """"""
155+
156+
In each :ref:`reference/dataset_structure:data`,
157+
there is a sequence of :ref:`reference/label_format/{label_type}:{label_type}` annotations,
158+
which can be obtained by index.
159+
160+
.. literalinclude:: ../../../../docs/code/{file_name}.py
161+
:language: python
162+
:start-after: """Read Dataset / get label"""
163+
:end-before: """"""
164+
165+
There is only one label type in "{dataset_name}" dataset, which is ``{label_type}``.
166+
{information_description}
167+
168+
****************
169+
Delete Dataset
170+
****************
171+
172+
.. literalinclude:: ../../../../docs/code/{file_name}.py
173+
:language: python
174+
:start-after: """Delete Dataset"""
175+
:end-before: """"""
176+
'''

‎docs/source/conf.py‎

Lines changed: 133 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,11 +15,14 @@
1515
# documentation root, use os.path.abspath to make it absolute, like shown here.
1616
#
1717
"""Configuration file for the Sphinx documentation builder."""
18+
import os
1819
import sys
1920
from pathlib import Path
2021

2122
sys.path.insert(0, str(Path(__file__).parents[2]))
22-
23+
from docs.source._templates.examples import ( # noqa: E402 # pylint: disable=wrong-import-position
24+
EXAMPLES_TEMPLATE,
25+
)
2326

2427
# -- Project information -----------------------------------------------------
2528

@@ -79,3 +82,132 @@
7982
# relative to this directory. They are copied after the builtin static files,
8083
# so a file named "default.css" will overwrite the builtin "default.css".
8184
# html_static_path = ["_static"]
85+
86+
source_path = os.path.dirname(os.path.abspath(__file__))
87+
example_path = os.path.join(source_path, "quick_start", "examples")
88+
dataset_names = (
89+
"Dogs Vs Cats",
90+
"20 Newsgroups",
91+
"BSTLD",
92+
"Neolix OD",
93+
"Leeds Sports Pose",
94+
"THCHS-30",
95+
)
96+
label_types = (
97+
"Classification",
98+
"Classification",
99+
"Box2D",
100+
"Box3D",
101+
"Keypoints2D",
102+
"Sentence",
103+
)
104+
file_names = ("DogsVsCats", "Newsgroups20", "BSTLD", "NeolixOD", "LeedsSportsPose", "THCHS30")
105+
106+
dataset_with_images = ("BSTLD", "Neolix OD", "Leeds Sports Pose")
107+
108+
figure_description = """(:numref:`Fig. %s <example-{file_name}>`).
109+
110+
.. _example-{file_name}:
111+
112+
.. figure:: ../../images/example-{label_type}.png
113+
:scale: 50 %
114+
:align: center
115+
116+
The preview of a cropped image with labels from "{dataset_name}".
117+
"""
118+
119+
category_attribute_descriptions = {}
120+
category_attribute_descriptions[
121+
"BSTLD"
122+
] = """
123+
The only annotation type for "{dataset_name}" is
124+
:ref:`reference/label_format/{label_type}:{label_type}`, and there are 13
125+
:ref:`reference/label_format/CommonLabelProperties:category` types and one
126+
:ref:`reference/label_format/CommonLabelProperties:attributes` type.
127+
"""
128+
129+
category_attribute_descriptions[
130+
"Dogs Vs Cats"
131+
] = """
132+
The only annotation type for "{dataset_name}" is
133+
:ref:`reference/label_format/{label_type}:{label_type}`, and there are 2
134+
:ref:`reference/label_format/CommonLabelProperties:category` types.
135+
"""
136+
137+
category_attribute_descriptions[
138+
"Leeds Sports Pose"
139+
] = """
140+
The only annotation type for "{dataset_name}" is
141+
:ref:`reference/label_format/{label_type}:{label_type}`.
142+
"""
143+
144+
category_attribute_descriptions[
145+
"Neolix OD"
146+
] = """
147+
The only annotation type for "{dataset_name}" is
148+
:ref:`reference/label_format/{label_type}:{label_type}`, and there are 15
149+
:ref:`reference/label_format/CommonLabelProperties:category` types and 3
150+
:ref:`reference/label_format/CommonLabelProperties:attributes` type.
151+
"""
152+
153+
category_attribute_descriptions[
154+
"20 Newsgroups"
155+
] = """
156+
The only annotation type for "{dataset_name}" is
157+
:ref:`reference/label_format/{label_type}:{label_type}`, and there are 20
158+
:ref:`reference/label_format/CommonLabelProperties:category` types
159+
"""
160+
161+
category_attribute_descriptions["THCHS-30"] = ""
162+
163+
# from docs.source._templates.examples import EXAMPLES_TEMPLATE
164+
for dataset_name, label_type, file_name in zip(dataset_names, label_types, file_names):
165+
if dataset_name == "THCHS-30":
166+
catalog_description = """However the catalog of THCHS-30 is too large, instead of
167+
reading it from json file, we read it by mapping from subcatalog that is loaded by
168+
the raw file. Check the :ref:`dataloader <THCHS30-dataloader>` below for more details.
169+
"""
170+
information_description = """It contains ``sentence``, ``spell`` and ``phone`` information.
171+
See :ref:`Sentence <reference/label_format/{label_type}:{label_type}>` label format for
172+
more details.
173+
"""
174+
else:
175+
catalog_description = """
176+
.. literalinclude:: ../../../../tensorbay/opendataset/{file_name}/catalog.json
177+
:language: json
178+
:name: {file_name}-catalog
179+
:linenos:
180+
"""
181+
information_description = """The information stored in
182+
:ref:`reference/label_format/CommonLabelProperties:category` is one of the names in "categories"
183+
list of :ref:`catalog.json <{file_name}-catalog>`. The information stored in
184+
:ref:`reference/label_format/CommonLabelProperties:attributes` is one or several of
185+
the attributes in "attributes" list of :ref:`catalog.json <{file_name}-catalog>`.
186+
See :ref:`reference/label_format/{label_type}:{label_type}` label format for more details.
187+
"""
188+
189+
if dataset_name in dataset_with_images:
190+
figure_description_tmp = figure_description.format(
191+
dataset_name=dataset_name, file_name=file_name, label_type=label_type
192+
)
193+
else:
194+
figure_description_tmp = ""
195+
catalog_description_tmp = catalog_description.format(file_name=file_name)
196+
information_description_tmp = information_description.format(
197+
label_type=label_type, file_name=file_name
198+
)
199+
category_attribute_description = category_attribute_descriptions[dataset_name].format(
200+
dataset_name=dataset_name, label_type=label_type
201+
)
202+
with open(os.path.join(example_path, f"{file_name}.rst"), "w", encoding="utf-8") as fp:
203+
fp.write(
204+
EXAMPLES_TEMPLATE.format(
205+
dataset_name=dataset_name,
206+
file_name=file_name,
207+
label_type=label_type,
208+
figure_description=figure_description_tmp,
209+
catalog_description=catalog_description_tmp,
210+
category_attribute_description=category_attribute_description,
211+
information_description=information_description_tmp,
212+
)
213+
)

0 commit comments

Comments
 (0)