Skip to content

Commit 735ae17

Browse files
MTecknologyMTecknology
committed
Add more information via readme files
1 parent 700a29c commit 735ae17

5 files changed

Lines changed: 158 additions & 49 deletions

File tree

‎README.md‎

Lines changed: 38 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -4,67 +4,61 @@ Recovery Source: Services
44
This repository provides a "standardized" solution to test, deploy, and maintain
55
various web "services" provided by [Recovery Source](https://handbook.recoverysource.net/).
66

7-
Repository Structure
8-
--------------------
9-
10-
Important directories:
7+
**Repository Structure:**
118

129
- **data**: Source data for all services
10+
- **sync**: Synchronize source and remote data with other resources
1311
- **web_index**: [Directory listing of 12-Step groups](https://sober.page/)
1412
- **nameserver**: Configuration for [DNS](https://handbook.recoverysource.net/essentials/websites.html#domain-name-system) services
1513
- **forwarder**: Configuration for "HTTP Redirector" service
16-
- **sync**: Python module used to synchronize data
1714
- **test**: Data used for automated testing
1815

19-
Web Index
20-
---------
21-
22-
Sober Pages service as a directory services for 12-step focused websites and
23-
aims to provide support for basic maintenance tasks.
24-
25-
See this website running at https://sober.page/.
26-
2716
Source Data
2817
-----------
2918

30-
Everything the Recovery Source project knows about 12-Step groups.
19+
[Source Data](https://github.com/recoverysource/services/tree/master/data)
20+
holds our collection of known 12-Step groups with publicly available meeting
21+
information. This directory contains another README file with addition details.
22+
23+
Synchronize Data
24+
----------------
3125

32-
### Data Format
26+
The [sync](https://github.com/recoverysource/services/tree/master/sync) utility
27+
is a python3 package that uses ``Source Data`` to scrape meeting data from known
28+
sources and assemble them all into a single standardized collection .
3329

34-
**Location:** ``data/domains/<group>.yaml``
30+
This script is responsible for keeping this data synchronized with other services,
31+
such as CloudFlare DNS, a Managed VPS, or into a format suitable for the
32+
``Web Index``. Developer notes are available in sync/README.
3533

36-
**Format [[YAML](https://handbook.recoverysource.net/essentials/yaml.html)]:**
34+
Help text is available using:
3735
```
38-
<subdomain>:
39-
title: website title
40-
keywords: list, of, regions
41-
target: <URL>
42-
type: forward OR cname
43-
feed: <type>^<URL>[^<options>]
36+
$ cd services && python3 -m sync -h
4437
```
4538

46-
**Required:** subdomain, title, target
39+
Web Index
40+
---------
4741

48-
**Rules:**
42+
Sober Pages service as a directory services for 12-step focused websites and
43+
aims to provide support for basic maintenance tasks.
4944

50-
- ``data/domains/*.yaml`` MUST be [valid YAML](https://yaml-online-parser.appspot.com/)
51-
- [``title``, ``keywords``] may be mixed-case, all other fields must be lower-case
52-
- ``target`` should be the shortest functional URL (without
53-
[path/query/fragment](https://handbook.recoverysource.net/essentials/websites.html#url))
54-
- ``target`` should include www if upstream redirects to this address
55-
- ``feed`` may be a [YAML list](https://handbook.recoverysource.net/essentials/yaml.html)
56-
of feed locations (type+url)
57-
- ``subdomain`` is limited to one (1) per ``target``
58-
- ``subdomain`` format should follow ``[type][area]-[district]`` (e.g. aa0-5)
59-
- ``subdomain`` may append characters to resolve conflicts (e.g. aa1-4north)
60-
- ``subdomain`` should use the lowest represented district as canonical
61-
- ``subdomain`` can be used as a SP alias to redirect additional-represented districts
62-
- ``feed/type`` must be one of [``aamod``, ``tsml``]
45+
See this website running at https://sober.page/.
6346

64-
Sync
65-
----
47+
Forwarder
48+
---------
6649

67-
Data synchronization is done using the ``sync`` python module:
68-
```
69-
$ cd services && python3 -m sync -h
70-
```
50+
Every 12-Step website that we list includes a link back that website. We also
51+
provide a subdomain that redirects all requests to that domain. This has been
52+
used for creating short/friendly URLs on flyers, but the options are endless.
53+
54+
Name Server
55+
-----------
56+
57+
For any 12-Step group that does not have a website and finds the cost to be
58+
a burden, our subdomain can be used as a primary website.
59+
60+
Automatic Testing
61+
-----------------
62+
63+
We utilize automatic testing to prevent as many potential issues as possible.
64+
This is used by automatic release processes as well as developers.

‎data/README.md‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
Recovery Source: Data
2+
=====================
3+
4+
This is the source data used by [Recovery Source Services](
5+
https://github.com/recoverysource/services) projects. This is everything the
6+
Recovery Source project knows about 12-Step groups.
7+
8+
This data was orginally created for Hugo, however this framework proved to be
9+
unstable and the future format should be based on i18n+framework.
10+
11+
Data Format
12+
-----------
13+
14+
**Location:** ``data/domains/<group>.yaml``
15+
16+
**Format [[YAML](https://handbook.recoverysource.net/essentials/yaml.html)]:**
17+
```
18+
<subdomain>:
19+
title: website title
20+
keywords: list, of, regions
21+
target: <URL>
22+
type: forward OR cname
23+
feed: <type>^<URL>[^<options>]
24+
```
25+
26+
**Required:** subdomain, title, target
27+
28+
**Rules:**
29+
30+
- ``data/domains/*.yaml`` MUST be [valid YAML](https://yaml-online-parser.appspot.com/)
31+
- [``title``, ``keywords``] may be mixed-case, all other fields must be lower-case
32+
- ``target`` should be the shortest functional URL (without
33+
[path/query/fragment](https://handbook.recoverysource.net/essentials/websites.html#url))
34+
- ``target`` should include www if upstream redirects to this address
35+
- ``feed`` may be a [YAML list](https://handbook.recoverysource.net/essentials/yaml.html)
36+
of feed locations (type+url)
37+
- ``subdomain`` is limited to one (1) per ``target``
38+
- ``subdomain`` format should follow ``[type][area]-[district]`` (e.g. aa0-5)
39+
- ``subdomain`` may append characters to resolve conflicts (e.g. aa1-4north)
40+
- ``subdomain`` should use the lowest represented district as canonical
41+
- ``subdomain`` can be used as a SP alias to redirect additional-represented districts
42+
- ``feed/type`` must be one of [``aamod``, ``tsml``]

‎sync/README.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
Data Synchronization
2+
====================
3+
4+
This ``sync`` directory is a python3 package that is used to synchronize data
5+
across various sources and destinations. This is essentially the "data processing"
6+
portion of [Recovery Source Services](https://github.com/recoverysource/services)
7+
workflows.
8+
9+
Processing
10+
----------
11+
12+
**Input:**
13+
14+
[Source Data](https://github.com/recoverysource/services/tree/master/data)
15+
is the root of the Recovery Source (Web) Index. This data was originally
16+
formatted for Hugo and imported via ``sync.hugo``.
17+
18+
This ``Source Data`` includes links to feeds where meeting information can be
19+
collected. Fetching this remote data is handled using ``sync.collect``.
20+
21+
**Cache:**
22+
23+
Collected data is stored using ``sync.db``; this aims to provide an efficient
24+
mechanism to verify freshness, generate output data, and share assembled data.
25+
26+
**Output:**
27+
28+
- ``sync.nginx``: Creates a map file for Nginx
29+
- ``sync.named``: Creates a zone file for Bind9
30+
- ``sync.cloudflare``: Synchronize ``Source Data`` with CloudFlare DNS
31+
32+
Usage
33+
-----
34+
35+
Current help text:
36+
```
37+
services$ python3 -m sync --help
38+
39+
usage: python3 -m sync [-h] [actions] <options>
40+
41+
Synchronize sober.page data to/from various sources
42+
43+
options:
44+
-h, --help show this help message and exit
45+
-H <path> Path to hugo file containing DNS data
46+
-w <path> Local workspace used for importing/caching data
47+
-l <level> Log level (DEBUG, INFO, WARNING*, ERROR)
48+
49+
actions[*]:
50+
-c Collect meeting data from remote feeds
51+
-n <path> Create an nginx map file and store at <path>
52+
-z <path> Create a bind9 zone file and store at <path>
53+
54+
[*] At least one script action must be specified.
55+
```
56+
57+
Create a bind9 zone file:
58+
```
59+
python3 -m sync -z /etc/named/db.managed
60+
```
61+
62+
Create an nginx map file:
63+
```
64+
python3 -m sync -n /etc/nginx/canonical_redirects.map
65+
```
66+
67+
Collect meeting data from remote feeds:
68+
```
69+
python3 -m sync -c
70+
```

‎sync/__main__.py‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -31,13 +31,13 @@ def main():
3131
source_data = sync.hugo.normalize(hugo_data)
3232

3333
# Connect to database if needed by action(s) [-c,]
34-
# if (options.collect):
35-
# sync.db.open(f'{options.local_data}/cache.db')
34+
if (options.collect):
35+
sync.db.open(f'{options.local_data}/cache.db')
3636

3737
# [-c] Collect meeting data from remote feeds
38-
# if options.collect:
39-
# logging.info('Collecting meeting data')
40-
# sync.collect.fetch_all(source_data)
38+
if options.collect:
39+
logging.info('Collecting meeting data')
40+
sync.collect.fetch_all(source_data)
4141

4242
# [-n] Generate an nginx map file
4343
if options.mapfile:

‎web_index/config.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,9 @@ title: Sober Page
55
languageCode: en-us
66
theme: [aamod, mainroad]
77

8+
ignorefiles:
9+
- "data/README.md"
10+
811
Params:
912
description: Index of 12-step Websites
1013
source: https://github.com/recoverysource/services

0 commit comments

Comments
 (0)