Skip to content

Commit 35cd3b8

Browse files
delasdelas
committed
Added real world sources
1 parent c163d0d commit 35cd3b8

File tree

5 files changed

+142
-0
lines changed

5 files changed

+142
-0
lines changed

‎docs/pybeamline/algorithms.md‎

Whitespace-only changes.

‎docs/pybeamline/sources-real/ais_source.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# ais_source
2+
3+
???+ warning "Extra package required!"
4+
To use this source you need to install package `pybeamline-real-sources` with:
5+
```
6+
pip install pybeamline-real-sources
7+
```
8+
9+
The automatic identification system (AIS) is an automatic tracking system that uses transceivers on ships and is used by vessel traffic services. This source considers the [MMSI](https://en.wikipedia.org/wiki/Maritime_Mobile_Service_Identity) as the case id and the [navigation status (when available) as the activity](https://en.wikipedia.org/wiki/Automatic_identification_system#Broadcast_information). While it is possible connect to any AIS data provider (by passing `host` and `port` parameters), by default, the source connects to the Norwegian Coastal Administration server, which publishes data for the from vessels within the Norwegian economic zone and the protection zones off Svalbard and Jan Mayen (see <https://www.kystverket.no/en/navigation-and-monitoring/ais/access-to-ais-data/>).
10+
11+
**ATTENTION:** while a lot of events are produced by this source, traces are very short and it can take a long time before two events with the same case id are actually observed.
12+
13+
14+
## Parameters
15+
16+
* **host** `str` default: `'153.44.253.27'`
17+
IP Address of the AIS provider.
18+
19+
* **port** `int` default: `5631`
20+
Port of the AIS provider.
21+
22+
23+
## Example
24+
25+
```python
26+
from pybeamline.sinks.print_sink import print_sink
27+
from pybeamline_real_sources.ais import ais_source
28+
29+
ais_source().pipe(
30+
31+
).subscribe(print_sink())
32+
```
33+
34+
Output:
35+
36+
```
37+
(Undefined, 258007330, AIS, 2020-00-00 14:49:31.101265 - {'mmsi': 258007330, 'second': 29, 'heading': 511, 'lon': 5.312967, 'accuracy': False, 'speed': 0.0, 'course': 326.1, 'lat': 60.392442} - {} - {})
38+
(Moored, 257215400, AIS, 2020-00-00 14:49:31.101265 - {'mmsi': 257215400, 'second': 63, 'heading': 511, 'lon': 181.0, 'accuracy': False, 'speed': 102.3, 'course': 360.0, 'lat': 91.0} - {} - {})
39+
(UnderWayUsingEngine, 257264900, AIS, 2020-00-00 14:49:31.102276 - {'mmsi': 257264900, 'second': 29, 'heading': 511, 'lon': 9.027405, 'accuracy': True, 'speed': 0.0, 'course': 5.1, 'lat': 63.652173} - {} - {})
40+
(UnderWayUsingEngine, 257033030, AIS, 2020-00-00 14:49:31.102276 - {'mmsi': 257033030, 'second': 29, 'heading': 335, 'lon': 17.335573, 'accuracy': False, 'speed': 0.0, 'course': 215.0, 'lat': 68.723927} - {} - {})
41+
(UnderWayUsingEngine, 257153000, AIS, 2020-00-00 14:49:31.102276 - {'mmsi': 257153000, 'second': 29, 'heading': 237, 'lon': 15.819412, 'accuracy': False, 'speed': 2.3, 'course': 245.3, 'lat': 69.479643} - {} - {})
42+
(UnderWayUsingEngine, 257198000, AIS, 2020-00-00 14:49:31.102276 - {'mmsi': 257198000, 'second': 30, 'heading': 279, 'lon': 5.076527, 'accuracy': False, 'speed': 0.0, 'course': 66.1, 'lat': 60.201458} - {} - {})
43+
(UnderWayUsingEngine, 257076900, AIS, 2020-00-00 14:49:31.102276 - {'mmsi': 257076900, 'second': 30, 'heading': 511, 'lon': 15.11809, 'accuracy': False, 'speed': 0.0, 'course': 171.8, 'lat': 67.657533} - {} - {})
44+
(UnderWayUsingEngine, 259031700, AIS, 2020-00-00 14:49:31.102276 - {'mmsi': 259031700, 'second': 29, 'heading': 68, 'lon': 4.97066, 'accuracy': True, 'speed': 6.7, 'course': 66.8, 'lat': 61.239747} - {} - {})
45+
(Moored, 258364000, AIS, 2020-00-00 14:49:31.102276 - {'mmsi': 258364000, 'second': 29, 'heading': 141, 'lon': 5.259447, 'accuracy': False, 'speed': 0.0, 'course': 244.1, 'lat': 59.411605} - {} - {})
46+
(Moored, 257158400, AIS, 2020-00-00 14:49:31.102276 - {'mmsi': 257158400, 'second': 30, 'heading': 511, 'lon': 10.757402, 'accuracy': True, 'speed': 0.0, 'course': 360.0, 'lat': 59.902088} - {} - {})
47+
(EngagedInFishing, 258247000, AIS, 2020-00-00 14:49:31.103287 - {'mmsi': 258247000, 'second': 31, 'heading': 205, 'lon': 10.121003, 'accuracy': False, 'speed': 8.7, 'course': 209.0, 'lat': 64.159853} - {} - {})
48+
49+
```

‎docs/pybeamline/sources-real/rejseplanen_source.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# rejseplanen_source
2+
3+
???+ warning "Extra package required!"
4+
To use this source you need to install package `pybeamline-real-sources` with:
5+
```
6+
pip install pybeamline-real-sources
7+
```
8+
9+
This source provides the data from the Danish railway system. Traces are represented as individual trains and events are trains reaching a certain station. The data is continuously generate (updated every 5 seconds, see <https://www.rejseplanen.dk/bin/help.exe/mn?L=vs_dot.vs_livemap&tpl=fullscreenmap>). The current source retrieves information about regional and light train (letbane).
10+
11+
12+
## Parameters
13+
14+
* *None*
15+
16+
17+
## Example
18+
19+
```python
20+
from pybeamline.sinks.print_sink import print_sink
21+
from pybeamline_real_sources.rejseplanen import rejseplanen_source
22+
23+
rejseplanen_source().pipe(
24+
25+
).subscribe(print_sink())
26+
```
27+
28+
Output:
29+
30+
```
31+
(Skibstrup St., 84/46798/18/19/86, Rejseplanen, 2020-00-00 14:53:54.051580 - {'train-delay': 2, 'train-previous-stop': 'Saunte St.'} - {'destination': 'Helsingør St.', 'train-name': 'Lokalbane 940R'} - {})
32+
(Favrholm St., 84/23923/18/27/86, Rejseplanen, 2020-00-00 14:53:54.051580 - {'train-delay': 4, 'train-previous-stop': 'Skævinge St.'} - {'destination': 'Hillerød St.', 'train-name': 'Lokalbane 920E'} - {})
33+
(Vig St., 84/23881/18/24/86, Rejseplanen, 2020-00-00 14:53:54.051580 - {'train-delay': 1, 'train-previous-stop': 'Grevinge St.'} - {'destination': 'Nykøbing Sj St.', 'train-name': 'Lokalbane 510R'} - {})
34+
(Nr. Asmindrup St., 84/23879/18/24/86, Rejseplanen, 2020-00-00 14:53:54.051580 - {'train-delay': 0, 'train-previous-stop': 'Sommerland Sj. St.'} - {'destination': 'Holbæk St.', 'train-name': 'Lokalbane 510R'} - {})
35+
(Vallø St., 84/23818/18/38/86, Rejseplanen, 2020-00-00 14:53:54.051580 - {'train-delay': 2, 'train-previous-stop': 'Grubberholm St.'} - {'destination': 'Roskilde St.', 'train-name': 'Lokalbane 210R'} - {})
36+
(Li.Linde St., 84/23794/18/38/86, Rejseplanen, 2020-00-00 14:53:54.051580 - {'train-delay': 2, 'train-previous-stop': 'Karise St.'} - {'destination': 'Køge St.', 'train-name': 'Lokalbane 110R'} - {})
37+
(Lille Kregme St., 84/23663/18/27/86, Rejseplanen, 2020-00-00 14:53:54.051580 - {'train-delay': 1, 'train-previous-stop': 'Kregme St.'} - {'destination': 'Hundested Havn St.', 'train-name': 'Lokalbane 920R'} - {})
38+
(Brede St., 84/23622/18/47/86, Rejseplanen, 2020-00-00 14:53:54.051580 - {'train-delay': 0, 'train-previous-stop': 'Fuglevad St.'} - {'destination': 'Nærum St.', 'train-name': 'Lokalbane 910'} - {})
39+
(Jyderup St., 84/1883/18/20/86, Rejseplanen, 2020-00-00 14:53:54.051580 - {'train-delay': 0, 'train-previous-stop': 'Svebølle St.'} - {'destination': 'Østerport St.', 'train-name': 'Re 1540'} - {})
40+
(Humlebæk St., 84/1697/18/19/86, Rejseplanen, 2020-00-00 14:53:54.051580 - {'train-delay': 0, 'train-previous-stop': 'Espergærde St.'} - {'destination': 'Holbæk St.', 'train-name': 'Re 4561'} - {})
41+
(Hellerup St., 84/1681/18/19/86, Rejseplanen, 2020-00-00 14:53:54.051580 - {'train-delay': 0, 'train-previous-stop': 'Klampenborg St.'} - {'destination': 'Holbæk St.', 'train-name': 'Re 2559'} - {})
42+
(Trekroner St., 84/1631/18/19/86, Rejseplanen, 2020-00-00 14:53:54.051580 - {'train-delay': 0, 'train-previous-stop': 'Roskilde St.'} - {'destination': 'Helsingør St.', 'train-name': 'Re 4536'} - {})
43+
(København H, 84/1430/18/19/86, Rejseplanen, 2020-00-00 14:53:54.051580 - {'train-delay': 0, 'train-previous-stop': 'Nørreport St.'} - {'destination': 'Næstved St.', 'train-name': 'Re 2257'} - {})
44+
(Vedbæk St., 84/1426/18/19/86, Rejseplanen, 2020-00-00 14:53:54.051580 - {'train-delay': 0, 'train-previous-stop': 'Rungsted Kyst St.'} - {'destination': 'Næstved St.', 'train-name': 'Re 4259'} - {})
45+
(Bedsted Thy St., 84/48192/18/19/86, Rejseplanen, 2020-00-00 14:53:54.051580 - {'train-delay': 0, 'train-previous-stop': 'Hurup Thy St.'} - {'destination': 'Thisted St.', 'train-name': 'RA 5541'} - {})
46+
```

‎docs/pybeamline/sources-real/wikimedia_source.md‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
# wikimedia_source
2+
3+
???+ warning "Extra package required!"
4+
To use this source you need to install package `pybeamline-real-sources` with:
5+
```
6+
pip install pybeamline-real-sources
7+
```
8+
9+
Source that connects to the stream of recent change operations happening on the Media Wiki websites (see <https://wikitech.wikimedia.org/wiki/Event_Platform/EventStreams_HTTP_Service> and <https://www.mediawiki.org/wiki/Manual:RCFeed>).
10+
11+
**ATTENTION:** it is advisable to apply a filter operation to consider only events relevant to one of the websites.
12+
13+
14+
## Parameters
15+
16+
* *None*
17+
18+
19+
## Example
20+
21+
```python
22+
from pybeamline.filters import retains_on_event_attribute_equal_filter
23+
from pybeamline.sinks.print_sink import print_sink
24+
from pybeamline_real_sources.wikimedia import wikimedia_source
25+
26+
wikimedia_source().pipe(
27+
retains_on_event_attribute_equal_filter("wiki", ["dewiki"])
28+
).subscribe(print_sink())
29+
```
30+
31+
Output:
32+
33+
```
34+
(edit, f7ed88659d276caef4e00a0fabb6c3bd, dewiki, 2020-00-00 15:23:19.704655 - {'user': 'Tohma', 'revision': {'old': 263529160, 'new': 263529200}, 'comment': '/* Redakteur des verschwörungstheoretischen Blogs NachDenkSeiten. */', 'server_name': 'de.wikipedia.org', 'server_url': 'https://de.wikipedia.org', 'namespace': 1, 'timestamp': 1769005398, 'wiki': 'dewiki', 'bot': False, 'length': {'old': 5727, 'new': 5843}, 'title': 'Diskussion:Florian Warweg'} - {} - {})
35+
(edit, 5d668c42da72ea67bcc61ed3209efdde, dewiki, 2020-00-00 15:23:22.484338 - {'user': 'Urmelbeauftragter', 'revision': {'old': 263529080, 'new': 263529197}, 'comment': '/* 56. Jahrestreffen 2026 */ Update', 'server_name': 'de.wikipedia.org', 'server_url': 'https://de.wikipedia.org', 'namespace': 0, 'timestamp': 1769005389, 'wiki': 'dewiki', 'bot': False, 'length': {'old': 244004, 'new': 245149}, 'title': 'Weltwirtschaftsforum'} - {} - {})
36+
(edit, 73a2f591a258857bc8c56334a1df47dc, dewiki, 2020-00-00 15:23:26.428809 - {'user': 'Kriddl', 'revision': {'old': 263529045, 'new': 263529201}, 'comment': '/* Oscarverleihung 1988 */', 'server_name': 'de.wikipedia.org', 'server_url': 'https://de.wikipedia.org', 'namespace': 100, 'timestamp': 1769005404, 'wiki': 'dewiki', 'bot': False, 'length': {'old': 27431, 'new': 27440}, 'title': 'Portal:Film und Fernsehen/Fehlende Oscar-Artikel'} - {} - {})
37+
(edit, 2667808e53e036566cd4f1a4792a7bd6, dewiki, 2020-00-00 15:23:28.048902 - {'user': '~2026-45554-3', 'revision': {'old': 262793578, 'new': 263529202}, 'comment': '', 'server_name': 'de.wikipedia.org', 'server_url': 'https://de.wikipedia.org', 'namespace': 0, 'timestamp': 1769005405, 'wiki': 'dewiki', 'bot': False, 'length': {'old': 328988, 'new': 328980}, 'title': 'Liste griechischer Sagen'} - {} - {})
38+
(edit, 1d119f1541ad69b9ebdddfbc21ca717f, dewiki, 2020-00-00 15:23:29.800785 - {'user': 'Spürnase2013', 'revision': {'old': 263529171, 'new': 263529203}, 'comment': '', 'server_name': 'de.wikipedia.org', 'server_url': 'https://de.wikipedia.org', 'namespace': 2, 'timestamp': 1769005408, 'wiki': 'dewiki', 'bot': False, 'length': {'old': 806, 'new': 2114}, 'title': 'Benutzer:Spürnase2013/Mailand–Sanremo 1957'} - {} - {})
39+
(edit, 227de05cfe03a8b4bfe29cdfd9103a7a, dewiki, 2020-00-00 15:23:35.704019 - {'user': '~2026-35153-1', 'revision': {'old': 263528936, 'new': 263529204}, 'comment': 'Amt ihres Bruders hinzugefügt', 'server_name': 'de.wikipedia.org', 'server_url': 'https://de.wikipedia.org', 'namespace': 0, 'timestamp': 1769005414, 'wiki': 'dewiki', 'bot': False, 'length': {'old': 11176, 'new': 11212}, 'title': 'Désirée Silfverschiöld'} - {} - {})
40+
(categorize, dd49cdd04189d39f52fbac66c34e022e, dewiki, 2020-00-00 15:23:36.159790 - {'user': 'Urmelbeauftragter', 'comment': '[[:Weltwirtschaftsforum]] zur Kategorie hinzugefügt', 'server_name': 'de.wikipedia.org', 'server_url': 'https://de.wikipedia.org', 'namespace': 14, 'timestamp': 1769005389, 'wiki': 'dewiki', 'bot': False, 'title': 'Kategorie:Wikipedia:Vorlagenfehler/Vorlage:Internetquelle'} - {} - {})
41+
(edit, 23e6825eec7f2b30aa0f9abb80299990, dewiki, 2020-00-00 15:23:36.695967 - {'user': 'Bernd Rohlfs', 'revision': {'old': 263529131, 'new': 263529205}, 'comment': '/* Strafverfahren */', 'server_name': 'de.wikipedia.org', 'server_url': 'https://de.wikipedia.org', 'namespace': 0, 'timestamp': 1769005414, 'wiki': 'dewiki', 'bot': False, 'length': {'old': 22962, 'new': 23095}, 'title': 'Christina Block'} - {} - {})
42+
43+
```

‎mkdocs.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,10 @@ nav:
1717
- pybeamline/sources/mqtt_source.md
1818
- pybeamline/sources/ocel2_log_source_from_file.md
1919
- pybeamline/sources/dict_test_ocel_source.md
20+
- Real world sources:
21+
- pybeamline/sources-real/ais_source.md
22+
- pybeamline/sources-real/rejseplanen_source.md
23+
- pybeamline/sources-real/wikimedia_source.md
2024
- Sinks:
2125
- pybeamline/sinks/print_sink.md
2226
- pybeamline/sinks/graphviz_sink.md

0 commit comments

Comments
 (0)