diff --git a/.travis.yml b/.travis.yml deleted file mode 100644 index 62890f9..0000000 --- a/.travis.yml +++ /dev/null @@ -1,10 +0,0 @@ -language: python -cache: pip -python: - - "3.5" - - "3.6" -install: - - pip install -r requirements.txt - - pip install pytest -script: - - python -m pytest atgmlogger/tests -v diff --git a/LICENSE.txt b/LICENSE.txt index be95694..4bfbf68 100644 --- a/LICENSE.txt +++ b/LICENSE.txt @@ -1,4 +1,4 @@ -Copyright 2018 Zachery P. Brady +Copyright 2016-2020 Zachery P. Brady Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: diff --git a/MANIFEST.in b/MANIFEST.in index cc6debb..caf0c35 100644 --- a/MANIFEST.in +++ b/MANIFEST.in @@ -1,3 +1,3 @@ include README.md include tools/send.py -include atgmlogger/install/* +include atgmlogger/atgmlogger.json diff --git a/README.md b/README.md index 42d4668..f7539a3 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,5 @@ -Dynamic Gravity Systems - Serial Data Recorder -============================================== +Dynamic Gravity Systems - Serial Data Recorder v0.6.0 +===================================================== 1. Dependencies: - Python v3.5 or 3.6 or later, and the following modules: @@ -9,96 +9,29 @@ Dynamic Gravity Systems - Serial Data Recorder - ntfs-3g - exfat-fuse - exfat-utils -2. Preparing the Raspberry Pi: - 1. Installing Python3.6 from source: - - Download Python3.6 source tarball from https://www.python.org - - Install the required development libraries to build the source: - - make - - build-essential - - libssl-dev - - zliblg-dev - - libbz2-dev - - libreadline-dev - - libsqlite3-dev - - wget - - curl - - llvm - - libncurses5-dev - - libncursesw5-dev - - xz-utils - - tk-dev - ```commandline - sudo apt-get install -y make build-essential libssl-dev zlib1g-dev - sudo apt-get install -y libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm - sudo apt-get install -y libncurses5-dev libncursesw5-dev xz-utils tk-dev - ``` - - Extract, Configure and Build: - ```commandline - tar -xzf python-3.6.2.tgz - cd python-3.6.2 - ./configure - make - sudo make altinstall - ``` - - 2. Install Python3.6 from Debian SID (experimental) repository: - - add the following line to /etc/apt/sources.list: - - ```commandline - deb http://http.us.debian.org/debian sid main - ``` - - configure apt-pinning to prevent the system from using experimental repo for system-wide updates - - ```commandline - vim /etc/apt/preferences.d/pinning - - Package: * - Pin: release a=stable - Pin-Priority: 700 - ``` - - 4. **Configure Raspberry Pi GPIO Console:** - - By default the Raspberry Pi GPIO console is enabled as a TTY terminal, this needs to be disabled to allow it - to be used as a Serial Data input. - - Modify /boot/cmdline.txt removing the section similar to: 'console=serial0,115200' - - ```commandline(bash) - # All Commands executed as Root (sudo) - sed -i -e s/console=serial0,115200//g /boot/cmdline.txt - echo 'enable_uart=1' >> /boot/config.txt - systemctl stop serial-getty@ttyS0.service - systemctl disable serial-getty@ttyS0.service - ``` - -3. Installation: - - The atgmlogger utility is now packaged as a Python Wheel and can be installed via pip: - ```commandline - sudo pip3 install atgmlogger-0.3.1-py3-none-any.whl - ``` - This method will create a commandline script in /usr/bin/atgmlogger, which can be invoked to run the logger. - - - -Manual Installation: --------------------- - -Installation Directories (copy the following files to the specified destinations): - - 90-removable-usb.rules -> /etc/udev/rules.d/90-removable-usb.rules - - media-removable.mount -> /etc/systemd/system/media-removable.mount - - SerialLogger.service -> /etc/systemd/system/SerialLogger.service +2. Installation: + + - The installation of atgmlogger on a Raspberry Pi requires several supporting configuration changes and updates; + to automate the process an ansible playbook is provided: [ATGMLogger Ansible Playbook](https://github.com/DynamicGravitySystems/atgmlogger-ansible) + + - In general the following steps are taken to prepare the system: + 1. Configure values in /boot/cmdline.txt and /boot/config.txt to enable UART and disable the TTY on the serial GPIO + 2. Install dependencies that enable the use of NTFS/FAT/EXFAT formatted external devices (for data retrieval) + 3. Add a UDEV rule and systemd mount unit to auto-mount any removable block device (e.g. USB drive) in order to allow copying of logged data for retrieval + 4. Add a logrotate configuration to automatically rotate data and application logs + 5. Install python3 and the ATGMLogger python application + 6. Install a systemd service unit file to control the automatic startup of the ATGMLogger application + 7. Secure the raspberry Pi by changing the default 'pi' user password, and adding an authorized key for factory maintenance usage + -After installing .mount and .service files run the following commands: -```commandline -sudo systemctl daemon-reload -sudo systemctl enable media-removable.mount -sudo systemctl enable SerialLogger.service -``` +3. Execution: -Explanation: -- 90-removable-usb.rules creates a UDEV rule that adds a symbolic link to /dev/usbstick when a usb block device (hdd) -is inserted. This symlink is used by the following mount file to mount the filesystem. -- media-removable.mount is a systemd mount unit which instructs systemd to mount /dev/usbstick to /media/removable when -it detects the device 'dev-usbstick.device'. The unit will also dismount the device when it becomes unavailable. -- SerialLogger.service is a systemd service unit which executes the Serial Logging python script. When this unit is -enabled (see above command - systemctl enable SerialLogger.service) the program will be executed upon system startup. + - Once installed with the ansible deploy playbook, ATGMLogger will automatically start every time the system is booted + via a systemd service unit. + + - ATGMLogger can otherwise be manually executed (e.g. for debugging purposes) with the following command + + ```commandline + /usr/bin/python3 -m atgmlogger -vvv + ``` diff --git a/TODO.md b/TODO.md deleted file mode 100644 index d23ab91..0000000 --- a/TODO.md +++ /dev/null @@ -1,7 +0,0 @@ -ToDo: -===== - -1. Change from using Global log to module level instances for hierarchy -2. Add install method to update RPi /boot/cmdline and /boot/config (sed) -3. Method to infer frequency of data from incoming rate -4. Partition argparse interface using subcommands diff --git a/atgmlogger/__init__.py b/atgmlogger/__init__.py index 834c8f8..5dcc383 100644 --- a/atgmlogger/__init__.py +++ b/atgmlogger/__init__.py @@ -3,13 +3,11 @@ import sys import logging -__all__ = ['LOG_LVLMAP', 'LOG_FMT', 'DATE_FMT', 'POSIX', - '__version__', '__description__'] +__all__ = ['LOG_LVLMAP', 'LOG_FMT', 'DATE_FMT', 'POSIX', '__version__', '__description__'] -__version__ = '0.4.5' +__version__ = '0.6.0' __description__ = "Advanced Technology Gravity Meter - Serial Data Logger" - LOG_LVLMAP = {0: logging.CRITICAL, 1: logging.ERROR, 2: logging.WARNING, @@ -18,6 +16,8 @@ # Application level root logger - all other loggers should branch from this APPLOG = logging.getLogger('atgmlogger') LOG_FMT = "%(levelname)8s::%(asctime)s - (%(funcName)s) %(message)s" +TRACE_LOG_FMT = "%(levelname)8s::%(asctime)s - (%(name)s/%(funcName)s:%(lineno)d " \ + "- thread: %(threadName)s) %(message)s" DATE_FMT = "%Y-%m-%d::%H:%M:%S" _stderr_hdlr = logging.StreamHandler(sys.stderr) _stderr_hdlr.setFormatter(logging.Formatter(LOG_FMT, diff --git a/atgmlogger/__main__.py b/atgmlogger/__main__.py index 7d10900..078a70d 100644 --- a/atgmlogger/__main__.py +++ b/atgmlogger/__main__.py @@ -1,9 +1,9 @@ #! /usr/bin/python3 # -*- encoding: utf-8 -*- -import sys -import logging import argparse +import logging +import sys from pathlib import Path from . import __description__, __version__, LOG_LVLMAP @@ -28,41 +28,18 @@ def parse_args(argv=None): parser.add_argument('--trace', action='store_true', help="Enable detailed trace info in log messages.") - # Sub-Parser Groups - sub_parsers = parser.add_subparsers(dest='command', help="Subcommands to run/install/uninstall ATGMLogger") - - install_parser = sub_parsers.add_parser('install', help='Install system files for ATGMLogger', allow_abbrev=True) - install_parser.add_argument('--service', action='store_true', default=True, help='Install ATGMLogger as a Systemd ' - 'Service') - install_parser.add_argument('--dependencies', action='store_true', help='Install system dependencies using apt.') - install_parser.add_argument('--configure', action='store_true', help='Run RaspberryPi configuration scripts.') - install_parser.add_argument('--check-install', action='store_true', help='Verify installed components.') - install_parser.add_argument('--logrotate', action='store_true', default=True, help='Install logrotate ' - 'configuration') - install_parser.add_argument('--with-mqtt', action='store_true', help='Install MQTT plugin configuration and AWS ' - 'IoT dependencies') - - uninst_parser = sub_parsers.add_parser('uninstall', help='Uninstall ATGMLogger system files and configurations') - uninst_parser.add_argument('--keep-config', action='store_true', help='Retain configuration after uninstalling ' - 'ATGMLogger') - - chkinst_parser = sub_parsers.add_parser('chkinstall', help="Check ATGMLogger installation and depdendencies") - - run_parser = sub_parsers.add_parser('run', help='Run atgmlogger') - run_parser.add_argument('-d', '--device', action='store', + # Runtime options + parser.add_argument('-d', '--device', action='store', help="Serial device path") - run_parser.add_argument('-l', '--logdir', action='store') - run_parser.add_argument('-m', '--mountdir', action='store', + parser.add_argument('-l', '--logdir', action='store') + parser.add_argument('-m', '--mountdir', action='store', help="Specify custom USB Storage mount path. " "Overrides path configured in configuration.") - run_parser.add_argument('-c', '--config', action='store', + parser.add_argument('-c', '--config', action='store', help="Specify path to custom JSON configuration.") - run_parser.add_argument('--nogpio', action='store_true', + parser.add_argument('--nogpio', action='store_true', help="Disable GPIO output (LED notifications).") - # This fails if we specify global positional args before the command - # if args[0].lower() not in ['install', 'uninstall', 'run']: - # args.insert(0, 'run') return parser.parse_args(args) @@ -75,37 +52,6 @@ def initialize(args): log_level = LOG_LVLMAP.get(args.verbose, logging.INFO) LOG.setLevel(log_level) - if args.command in {'install', 'uninstall', 'chkinstall'}: - from . import install - method = getattr(install, args.command, None) - if method is None: - LOG.error("Command %s is not implemented", args.command) - sys.exit(1) - sys.exit(method(args)) - - - # if args.command == 'install': - # try: - # from .install import install - # sys.exit(install(args)) - # except (ImportError, OSError): - # LOG.exception("Exception occurred trying to install system " - # "files.") - # sys.exit(1) - # elif args.command == 'uninstall': - # try: - # from .install import uninstall - # sys.exit(uninstall(args)) - # except (ImportError, OSError): - # LOG.exception("Exception occurred uninstalling system files.") - # elif args.command == 'chkinstall': - # try: - # from .install import chkinstall - # sys.exit(chkinstall(args)) - # - # except (ImportError, OSError): - # pass - # Set overrides from arguments from .runconfig import rcParams @@ -132,3 +78,7 @@ def entry_point(): from .atgmlogger import atgmlogger sys.exit(atgmlogger(args)) + + +if __name__ == "__main__": + entry_point() diff --git a/atgmlogger/install/atgmlogger.json b/atgmlogger/atgmlogger.json similarity index 100% rename from atgmlogger/install/atgmlogger.json rename to atgmlogger/atgmlogger.json diff --git a/atgmlogger/atgmlogger.py b/atgmlogger/atgmlogger.py index d0bdcd5..30ead64 100644 --- a/atgmlogger/atgmlogger.py +++ b/atgmlogger/atgmlogger.py @@ -131,7 +131,6 @@ def decode(bytearr, encoding='utf-8'): return decoded -# TODO: Move some/all of this functionality into __main__ initialize function def _configure_applog(log_format): logdir = Path(rcParams['logging.logdir']) if not logdir.exists(): @@ -207,11 +206,7 @@ def atgmlogger(args, listener=None, handle=None, dispatcher=None): # Init Performance Counter t_start = time.perf_counter() - # TODO: Again candidate to move into __main__::initialize - fmt = LOG_FMT - if args.trace: - fmt = TRACE_LOG_FMT - _configure_applog(fmt) + _configure_applog(TRACE_LOG_FMT if args.trace else LOG_FMT) if listener is None: listener = SerialListener(handle or _get_handle()) @@ -228,7 +223,6 @@ def atgmlogger(args, listener=None, handle=None, dispatcher=None): # Note: Signal handler must be defined in main thread signal.signal(signal.SIGHUP, lambda sig, frame: dispatcher.log_rotate()) dispatcher.start() - # print(logging.Logger.manager.loggerDict.keys()) listener() except KeyboardInterrupt: LOG.info("Keyboard Interrupt intercepted, cleaning up and exiting.") diff --git a/atgmlogger/install/90-removable-storage.rules b/atgmlogger/install/90-removable-storage.rules deleted file mode 100644 index cfd0f1b..0000000 --- a/atgmlogger/install/90-removable-storage.rules +++ /dev/null @@ -1 +0,0 @@ -ACTION=="add", KERNEL=="sd?1", SUBSYSTEMS=="block", SYMLINK+="usbstorage", TAG+="systemd", ENV{SYSTEMD_WANTS}="media-removable.mount" diff --git a/atgmlogger/install/__init__.py b/atgmlogger/install/__init__.py deleted file mode 100644 index 309b1a2..0000000 --- a/atgmlogger/install/__init__.py +++ /dev/null @@ -1,186 +0,0 @@ -# -*- coding: utf-8 -*- -# This file is part of ATGMLogger https://github.com/bradyzp/atgmlogger - -import os -import shlex -import logging -import subprocess -import pkg_resources -from textwrap import dedent -from pathlib import Path - -from .. import POSIX - -__all__ = ['install', 'uninstall'] - -BASEPKG = __name__.split('.')[0] -PREFIX = '' - -LOG = logging.getLogger(__name__) -# LOG.propagate = True -LOG.setLevel(logging.WARNING) -if POSIX: - _install_log_path = 'install.log' -else: - _install_log_path = 'install.log' - -LOG.addHandler(logging.FileHandler(_install_log_path, - encoding='utf-8')) - - -_file_map = { - '.atgmlogger': '%s/etc/%s/.atgmlogger' % (PREFIX, BASEPKG), - 'media-removable.mount': - '%s/lib/systemd/system/media-removable.mount' % PREFIX, - '90-removable-storage.rules': - '%s/etc/udev/rules.d/90-removable-storage.rules' % PREFIX, - 'atgmlogger.service': '%s/lib/systemd/system/atgmlogger.service' % PREFIX -} - - -def write_bytes(path: str, bytearr, mode=0o644): - try: - fd = os.open(path, os.O_WRONLY | os.O_CREAT, mode) - os.write(fd, bytearr) - os.close(fd) - except OSError: - LOG.exception("Exception writing template to file: %s", str(path)) - - -def sys_command(cmd, verbose=True): - try: - if verbose: - LOG.info("Executing system command: '%s'", cmd) - return subprocess.check_output(shlex.split(cmd)) - except (OSError, subprocess.SubprocessError, subprocess.CalledProcessError): - if verbose: - LOG.exception("Exception encountered executing command: '%s'", cmd) - else: - LOG.warning("Exception encountered executing command: '%s'", cmd) - return -1 - - -def _install_logrotate_config(log_path=None): - # Create atgmlogger logrotate file in /etc/logrotate.d/atgmlogger - # If atgmlogger config is dropped above, no further action needed as - # there should already be a daily logrotate cron entry - dest_path = Path('%s/etc/logrotate.d/%s' % (PREFIX, BASEPKG)) - if log_path is not None: - log_path = Path(log_path) - else: - log_path = Path('/var/log/%s' % BASEPKG) - if not log_path.exists(): - log_path.mkdir() - - postscript = """ - postrotate - if [ -x /usr/bin/killall ]; then - killall -HUP atgmlogger - fi - endscript - """ - config = """ - {logpath}/*.log {{ - missingok - weekly - dateext - dateyesterday - dateformat .%Y-%m-%d - rotate 30 - compress - }} - {logpath}/*.dat {{ - missingok - daily - dateext - dateyesterday - dateformat .%Y-%m-%d - rotate 30 - {postrotate} - }} - """.format(logpath=str(log_path.resolve()), postrotate=postscript) - try: - LOG.info("Installing logrotate configuration in %s", str(dest_path)) - fd = os.open(str(dest_path), os.O_WRONLY | os.O_CREAT, mode=0o640) - hdl = os.fdopen(fd, mode='w') - hdl.write(dedent(config)) - except IOError: - LOG.exception("Exception creating atgmlogger logrotate config.") - - -def configure_rpi(): - """Check/set parameters in /boot/config.txt and /boot/cmdline.txt to - configure Raspberry Pi for GPIO serial IO. - Specifically this requires adding `enable_uart=1` to the end of config.txt - and removing a clause from the cmdline.txt file to disable TTY over the GPIO - serial interface.""" - - sys_command("sed -i -r 's/console=serial0,115200 //' /boot/cmdline.txt") - - try: - with open('/boot/config.txt', 'r') as fd: - if 'enable_uart=1' in fd.read(): - LOG.info("enable_uart is already set in config.txt, no action taken.") - return - with open('/boot/config.txt', 'a') as fd: - fd.write("enable_uart=1\n") - LOG.critical("enable_uart set, system reboot required for configuration to take effect.") - except (IOError, OSError): - LOG.exception("Error reading/writing from /boot/config.txt") - - -def install(verbose=True): - if verbose: - LOG.setLevel(logging.DEBUG) - if not POSIX: - LOG.warning("Invalid system platform for installation.") - return 1 - - df_mode = 0o640 - for src, dest in _file_map.items(): - LOG.info("Installing source file: %s to %s", src, dest) - parent = os.path.split(dest)[0] - if not os.path.exists(parent): - try: - os.mkdir(parent, df_mode) - except OSError: - LOG.exception("Error creating directory: %s" % parent) - continue - try: - src_bytes = pkg_resources.resource_string(__name__, src) - write_bytes(dest, src_bytes, df_mode) - except (FileNotFoundError, OSError): - LOG.exception("Error writing resource to dest file.") - _install_logrotate_config() - - # Try to install dependencies for USB removable storage formats - sys_command('apt-get update') - sys_command('apt-get install -y ntfs-3g exfat-fuse exfat-utils') - - sys_command('systemctl daemon-reload') - sys_command('systemctl enable media-removable.mount') - sys_command('systemctl enable atgmlogger.service') - configure_rpi() - LOG.critical("Installation of atgmlogger completed successfully.") - return 0 - - -def uninstall(verbose=True): - if verbose: - LOG.setLevel(logging.DEBUG) - LOG.info("Stopping and disabling services.") - sys_command('systemctl stop atgmlogger.service') - sys_command('systemctl disable media-removable.mount') - sys_command('systemctl disable atgmlogger.service') - - for src, dest in _file_map.items(): - try: - LOG.info("Removing file: %s", dest) - os.remove(dest) - except (IOError, OSError): - if verbose: - LOG.exception("Unable to remove installed file: %s", dest) - else: - LOG.warning("Unable to remove installed file: %s", dest) - LOG.info("Successfully completed uninstall.") - return 0 diff --git a/atgmlogger/install/atgmlogger-mqtt.json b/atgmlogger/install/atgmlogger-mqtt.json deleted file mode 100644 index a8df84c..0000000 --- a/atgmlogger/install/atgmlogger-mqtt.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "version": "0.4.1-alpha.1", - "serial": { - "port": "/dev/serial0", - "baudrate": 57600, - "bytesize": 8, - "parity": "N", - "stopbits": 1 - }, - "logging": { - "logdir": "/var/log/atgmlogger" - }, - "usb": { - "mount": "/media/removable", - "copy_level": "debug" - }, - "plugins": { - "gpio": { - "mode": "board", - "data_pin": 11, - "usb_pin": 13, - "freq": 0.03 - }, - "usb": { - "mountpath": "/media/removable", - "logdir": "/var/log/atgmlogger", - "patterns": ["*.dat", "*.log", "*.gz"] - }, - "timesync": { - "interval": 1000 - }, - "mqtt": { - "sensorid": "", - "topicid": "", - "topic_pfx": "sensor", - "endpoint": "", - "rootca": "aws.root.crt", - "prikey": "iot.pem.key", - "devcert": "iot.pem.crt", - "interval": 1, - "fields": [ - "gravity", "long", "cross", "temp", "pressure", "latitude", "longitude", "datetime" - ] - } - } -} diff --git a/atgmlogger/install/media-removable.mount b/atgmlogger/install/media-removable.mount deleted file mode 100644 index 0790b68..0000000 --- a/atgmlogger/install/media-removable.mount +++ /dev/null @@ -1,10 +0,0 @@ -[Unit] -Description=Mount usb storage removable device to /media/removable - -[Mount] -What=/dev/usbstorage -Where=/media/removable -ForceUnmount=True - -[Install] -WantedBy=multi-user.target diff --git a/atgmlogger/logger.py b/atgmlogger/logger.py index a28c8f9..ce7da90 100644 --- a/atgmlogger/logger.py +++ b/atgmlogger/logger.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# This file is part of ATGMLogger https://github.com/bradyzp/atgmlogger +# This file is part of ATGMLogger https://github.com/DynamicGravitySystems/atgmlogger import io import logging @@ -59,8 +59,6 @@ def run(self): LOG.exception("Error opening file for writing.") return - # TODO: Take sample of data, find the mode (freq) of - # transmission to set Blink frequency while not self.exiting: try: item = self.get(block=True, timeout=None) diff --git a/atgmlogger/plugins/mqtt.py b/atgmlogger/plugins/mqtt.py deleted file mode 100644 index 92f4bc1..0000000 --- a/atgmlogger/plugins/mqtt.py +++ /dev/null @@ -1,244 +0,0 @@ -# -*- coding: utf-8 -*- -# This file is part of ATGMLogger https://github.com/bradyzp/atgmlogger - -import json -import logging -from datetime import datetime -from uuid import uuid4 - -LOG = logging.getLogger(__name__) -try: - from AWSIoTPythonSDK.MQTTLib import AWSIoTMQTTClient -except ImportError: - LOG.exception("AWSIoTPythonSDK not available. Run pip install AWSIoTPythonSDK in the ATGMLogger environment.") - raise - -from atgmlogger.plugins import PluginInterface -from atgmlogger.runconfig import rcParams - - -""" -MQTTClient Plugin (mqtt) - -Logger plugin to publish messages to an AWS IoT MQTT Message queue. - -Configurations can be applied to this plugin under the 'mqtt' directive in -the configuration json file. See the available options below. -This implementation connects to the AWS IoT device manager via port 8883, -authenticating with a X509 certificate/private-key. - -Dependencies ------------- -AWSIoTPythonSDK (available from PyPi via pip) - - >>> pip install AWSIoTPythonSDK - -Notes ------ -Full line of data is approx 124 bytes, or approx 160 bytes when wrapped with JSON and device metadata - -TODO ----- -Option to batch send data as Json list of maps (reduce IoT cost if each message < 5kb) - - -MQTT Plugin configuration options: ----------------------------------- -sensorid : String - Unique identifier for this device -topicid : String, optional - Optional topicid to publish messages to, will use sensorid by default -topic_pfx : String, optional - Optional prefix branch to publish message to, uses gravity by default -endpoint : String, required - Required, AWSIoT Endpoint URL -rootca : String - Optional, path to root CA Certificate for AWS IoT. Relative to the configuration - path. Defaults to root-CA.crt -prikey : String - Optional name of private key file for this IoT device. Relative to config path. - Defaults to iot.private.key -devcert : String - Optional name of device certificate (PEM) file for this IoT device. Relative to - config path. Defaults to iot.cert.pem -interval : Number, optional - Optional interval of ticks at which to send a data line, e.g. with default 1, every data line is sent, - with interval of 10, every 10th data line is sent. - -""" - - -def join_cfg(path): - """Append the application base config directory to given path.""" - return str(rcParams.path.parent.joinpath(path)) - - -def convert_gps_time(gpsweek, gpsweekseconds): - """ - Converts a GPS time format (weeks + seconds since 6 Jan 1980) to a UNIX - timestamp (seconds since 1 Jan 1970) without correcting for UTC leap - seconds. - - Static values gps_delta and gpsweek_cf are defined by the below functions - (optimization) gps_delta is the time difference (in seconds) between UNIX - time and GPS time. - - gps_delta = (dt.datetime(1980, 1, 6) - dt.datetime(1970, 1, 1)).total_seconds() - - gpsweek_cf is the coefficient to convert weeks to seconds - gpsweek_cf = 7 * 24 * 60 * 60 # 604800 - - Parameters - ---------- - gpsweek : int - Number of weeks since beginning of GPS time (1980-01-06 00:00:00) - - gpsweekseconds : float - Number of seconds since the GPS week parameter - - Returns - ------- - float - UNIX timestamp (number of seconds since 1970-01-01 00:00:00) without - leapseconds subtracted - """ - gps_delta = 315964800.0 - gpsweek_cf = 604800 - gps_ticks = (float(gpsweek) * gpsweek_cf) + float(gpsweekseconds) - - return gps_delta + gps_ticks - - -""" -Marine Data Fields/Sample - -$UW,20083,-1369,-940,5104887,252,466,212,4502,400,-24,-16,4430,5128453,39.9092261667,-105.0747506667,0.0040330.9400,20171206173348 -$UW,-9696,-7781,4628,4118252,257,409,199,32888,128,73,-42,4362,4139315,39.9092032667,-105.0747801500,0.0000,0.0000,00000000000022 - - -""" - - -def get_timestamp(fields): - timestamp = datetime.utcnow().timestamp() - if len(fields) == 13: - # Assume Marine data - GPS week/second format - try: - week = int(fields[11]) - seconds = float(fields[12]) - timestamp = convert_gps_time(week, seconds) - except ValueError: - LOG.exception("Exception resolving timestamp from fields: " + ','.join(fields)) - elif len(fields) == 19: - # Assume Airborne data - YMD format - date = str(fields[18]) - fmt = "%Y%m%d%H%M%S" - try: - timestamp = datetime.strptime(date, fmt).timestamp() - except ValueError: - timestamp = datetime.utcnow().timestamp() - - return timestamp - - -class MQTTClient(PluginInterface): - options = ['sensorid', 'topicid', 'topic_pfx', 'endpoint', 'rootca', 'prikey', 'devcert', 'batch', 'interval', - 'fields'] - topic_pfx = 'gravity' - endpoint = None - rootca = 'root-CA.crt' - prikey = 'iot.private.key' - devcert = 'iot.cert.pem' - interval = 1 - fields = ['gravity', 'long', 'cross', 'latitude', 'longitude', 'datetime'] - - # Ordered list of marine fields - _marine_fieldmap = ['header', 'gravity', 'long', 'cross', 'beam', 'temp', 'pressure', 'etemp', 'vcc', 've', 'al', - 'ax', 'status', 'checksum', 'latitude', 'longitude', 'speed', 'course', 'datetime'] - _airborne_fieldmap = [] - - def __init__(self): - super().__init__() - # Set AWSIoTPythonSDK logger level from default - logging.getLogger('AWSIoTPythonSDK').setLevel(logging.WARNING) - self.client = None - self.tick = 0 - self.sensorid = None - self._errcount = 0 - - @classmethod - def extract_fields(cls, data: str, fieldmap=_marine_fieldmap): - extracted = {} - data = data.split(',') - for i, field in enumerate(fieldmap): - if field.lower() in cls.fields: - extracted[field] = data[i] - return extracted - - @staticmethod - def consumer_type() -> set: - return {str} - - # def _batch_process(self): - # # TODO: Implement batch publish feature, perhaps collect items in a queue until a limit is reached - # # then publish as a list of json maps - # sendqueue = [] - # limit = 10 - - def configure_client(self): - if self.endpoint is None: - raise ValueError("No endpoint provided for MQTT Plugin.") - try: - self.sensorid = getattr(self, 'sensorid', str(uuid4())[0:8]) - topicid = getattr(self, 'topicid', self.sensorid) - - self.client = AWSIoTMQTTClient(self.sensorid, useWebsocket=False) - self.client.configureEndpoint(self.endpoint, 8883) - self.client.configureOfflinePublishQueueing(10000) - self.client.configureConnectDisconnectTimeout(10) - self.client.configureCredentials(join_cfg(self.rootca), - join_cfg(self.prikey), - join_cfg(self.devcert)) - self.client.configureDrainingFrequency(2) - self.client.configureMQTTOperationTimeout(5) - self.client.connect() - - topic = '/'.join([self.topic_pfx, topicid]) - except AttributeError: - LOG.exception("Missing attributes from configuration for MQTT plugin.") - raise - return topic - - def run(self): - topic = self.configure_client() - while not self.exiting: - item = self.get(block=True, timeout=None) - self.tick += 1 - if item is None or item == "" or self.tick % self.interval: - self.task_done() - continue - else: - try: - self.tick = 0 # reset tick count - fields = item.split(',') - timestamp = get_timestamp(fields) - if not len(fields): - continue - - data_dict = self.extract_fields(item) - - item_json = json.dumps({'d': self.sensorid, 't': timestamp, 'v': data_dict}) - # Note: returns bool value on success/fail of publish (maybe useful to know) - self.client.publish(topic, item_json, 0) - self.task_done() - except: - LOG.exception("Exception occured in mqtt-run loop. Item value: %s", item) - self._errcount += 1 - if self._errcount > 10: - # Terminate MQTT if errors accumulate - raise - - self.client.disconnect() - - -__plugin__ = MQTTClient diff --git a/atgmlogger/runconfig.py b/atgmlogger/runconfig.py index 58afee0..8d34c70 100644 --- a/atgmlogger/runconfig.py +++ b/atgmlogger/runconfig.py @@ -41,12 +41,9 @@ def __init__(self, config: Dict=None, path=None): else: LOG.warning("No configuration file could be located, " "attempting to load default.") - LOG.warning("Execute with --install option to install " - "default configuration files.") try: import pkg_resources as pkg - rawfd = pkg.resource_stream(_base + '.install', - self.cfg_name) + rawfd = pkg.resource_stream(_base, self.cfg_name) text_wrapper = TextIOWrapper(rawfd, encoding='utf-8') self.load_config(text_wrapper) diff --git a/atgmlogger/tests/conftest.py b/atgmlogger/tests/conftest.py index ee4b2be..392af01 100644 --- a/atgmlogger/tests/conftest.py +++ b/atgmlogger/tests/conftest.py @@ -1,25 +1,55 @@ # -*- coding: utf-8 -*- -import pytest -import json -import serial import threading from pathlib import Path -from atgmlogger.runconfig import _ConfigParams +import pytest +import serial + from atgmlogger.dispatcher import Dispatcher +from atgmlogger.runconfig import _ConfigParams -@pytest.fixture(scope="module", params=["atgmlogger/install/atgmlogger.json"]) -def cfg_dict(request): - with open(request.param, 'r') as fd: - config = json.load(fd) - return config +@pytest.fixture(scope="module") +def cfg_dict(): + return { + "version": 0.4, + "serial": { + "port": "/dev/serial0", + "baudrate": 57600, + "bytesize": 8, + "parity": "N", + "stopbits": 1 + }, + "logging": { + "logdir": "/var/log/atgmlogger" + }, + "usb": { + "mount": "/media/removable", + "copy_level": "debug" + }, + "plugins": { + "gpio": { + "mode": "board", + "data_pin": 11, + "usb_pin": 13, + "freq": 0.04 + }, + "usb": { + "mountpath": "/media/removable", + "logdir": "/var/log/atgmlogger", + "patterns": ["*.dat", "*.log", "*.gz", "*.dat.*"] + }, + "timesync": { + "interval": 1000 + } + } + } @pytest.fixture def rcParams(): - return _ConfigParams(path='atgmlogger/install/atgmlogger.json') + return _ConfigParams(path='atgmlogger/atgmlogger.json') @pytest.fixture() @@ -37,6 +67,7 @@ def __init__(self): def log(self, level, data): self.accumulator.append(data) + return CustomLogger() @@ -65,5 +96,3 @@ def logpath(request): def mountpoint(tmpdir): path = tmpdir.mkdir('mount') return Path(str(path)) - - diff --git a/atgmlogger/tests/test_argparser.py b/atgmlogger/tests/test_argparser.py index 1eface7..e0e09f1 100644 --- a/atgmlogger/tests/test_argparser.py +++ b/atgmlogger/tests/test_argparser.py @@ -2,6 +2,7 @@ import shlex from argparse import Namespace + import pytest from atgmlogger.__main__ import parse_args @@ -13,45 +14,6 @@ def namespace(): return Namespace(debug=False, trace=False, verbose=0) -@pytest.fixture -def inst_namespace(namespace): - """Namespace defaults for 'install' sub-command""" - namespace.command = "install" - namespace.service = True - namespace.dependencies = False - namespace.configure = False - namespace.check_install = False - namespace.logrotate = True - namespace.with_mqtt = False - return namespace - - -def test_install_parse_args(inst_namespace): - test_args = "install --dependencies --configure" - - result = parse_args(argv=shlex.split(test_args)) - inst_namespace.dependencies = True - inst_namespace.configure = True - - assert inst_namespace == result - -def test_install_override_defaults(inst_namespace): - """Test override syntax for default_true arguments""" - test_args = "install --logrotate=false" - - inst_namespace.logrotate = False - - -def test_uninstall_parse_args(namespace): - test_args = "uninstall" - - result = parse_args(shlex.split(test_args)) - namespace.command = "uninstall" - namespace.keep_config = False - - assert namespace == result - - def test_run_command_parse(): """Test insertion of default command when non specified (run)""" test_args = "run --device com1 --logdir /etc/atgmlogger" diff --git a/atgmlogger/tests/test_data_logging.py b/atgmlogger/tests/test_data_logging.py index efddcca..c11e573 100644 --- a/atgmlogger/tests/test_data_logging.py +++ b/atgmlogger/tests/test_data_logging.py @@ -1,10 +1,8 @@ # -*- coding: utf-8 -*- -import pytest -import datetime from pathlib import Path -from atgmlogger.logger import DataLogger +from atgmlogger.logger import DataLogger LINE = "$UW,81242,-1948,557,4807924,307,872,204,6978,7541,-70,305,266," \ "4903912,0.000000,0.000000,0.0000,0.0000,{idx}" @@ -16,11 +14,8 @@ def blink(self, *args, **kwargs): def test_simple_logger(tmpdir): - test_dir = Path(str(tmpdir.mkdir('logs'))) log_file = test_dir.joinpath('gravdata.dat') - # print("Logging test_dir: ", test_dir) - # print("Initial grav file: ", log_file) logger = DataLogger() logger.set_context(MockAppContext()) @@ -45,44 +40,3 @@ def test_simple_logger(tmpdir): with log_file.open('r') as fd: for i, line in enumerate(fd): assert accumulator[i] == line.strip() - - -@pytest.mark.skip("Changed behavior of log_rotate to notify of system event") -def test_logger_rotate(tmpdir): - test_dir = Path(str(tmpdir.mkdir('logs'))) - log_file = test_dir.joinpath('gravdata.dat') - - logger = DataLogger() - _params = dict(logfile=log_file) - logger.configure(**_params) - - accumulator = [] - logger.start() - rot_time = None - rot_idx = 500 - - for i in range(1000): - if i == rot_idx: - logger.queue.join() - logger.log_rotate() - rot_time = datetime.datetime.now().strftime('%Y%m%d-%H%M') - item = LINE.format(idx=i) - accumulator.append(item) - logger.put(item) - - logger.exit(join=True) - assert not logger.is_alive() - print("Logger has exited in test_logger_rotate") - - orig_file = test_dir.joinpath('gravdata.dat.'+rot_time) - assert orig_file.exists() - # print(orig_file) - assert log_file.exists() - # print(log_file) - with orig_file.open('r') as fd: - for i, line in enumerate(fd): - assert accumulator[i] == line.strip() - - with log_file.open('r') as fd: - for i, line in enumerate(fd): - assert accumulator[rot_idx+i] == line.strip() diff --git a/atgmlogger/tests/test_mqtt.py b/atgmlogger/tests/test_mqtt.py deleted file mode 100644 index 21f733e..0000000 --- a/atgmlogger/tests/test_mqtt.py +++ /dev/null @@ -1,25 +0,0 @@ -# -*- coding: utf-8 -*- - -from atgmlogger.plugins.mqtt import MQTTClient as client - -data = '$UW,20083,-1369,-940,5104887,252,466,212,4502,400,-24,-16,4430,5128453,39.9092261667,-105.0747506667,' \ - '0.0040330.9400,20171206173348' - - -def test_extract_fields_ordered(): - client.fields = ['gravity', 'long', 'cross'] - - expected = {"gravity": '20083', "long": '-1369', "cross": '-940'} - - extracted = client.extract_fields(data) - - assert expected == extracted - - -def test_extract_fields_unordered(): - client.fields = ['long', 'longitude', 'cross', 'latitude', 'gravity'] - expected = {'gravity': '20083', 'long': '-1369', 'cross': '-940', 'latitude': '39.9092261667', 'longitude': - '-105.0747506667'} - result = client.extract_fields(data) - - assert expected == result diff --git a/requirements.txt b/requirements.txt index d46ecc3..ab49a03 100644 --- a/requirements.txt +++ b/requirements.txt @@ -1,3 +1,3 @@ -pyserial>=3.3 -RPi.GPIO==0.6.3 -pytest>=3.3 +pyserial==3.4 +RPi.GPIO==0.7.0 +pytest==5.4.3 diff --git a/setup.cfg b/setup.cfg index b7e4789..07f110d 100644 --- a/setup.cfg +++ b/setup.cfg @@ -1,2 +1,5 @@ [aliases] test=pytest + +[metadata] +license_files = LICENSE.txt \ No newline at end of file diff --git a/setup.py b/setup.py index e84f832..732730e 100644 --- a/setup.py +++ b/setup.py @@ -1,7 +1,8 @@ # -*- coding: utf-8 -*- # setup.py for atgmlogger # -# (C) 2016-2018 Zachery P. Brady +# (C) 2016-2020 Zachery P. Brady +# (C) 2016-2020 Dynamic Gravity Systems from setuptools import setup @@ -9,8 +10,8 @@ requirements = [ 'setuptools >= 38.5.1', - 'pyserial >= 3.3', - 'RPi.GPIO >= 0.6.3' + 'pyserial == 3.4', + 'RPi.GPIO == 0.7.0' ] setup( @@ -19,7 +20,6 @@ packages=['atgmlogger', 'atgmlogger.plugins', 'atgmlogger.tests', 'atgmlogger.tests.plugins'], url='https://github.com/bradyzp/atgmlogger', - license='', author='Zachery Brady', author_email='bradyzp@dynamicgravitysystems.com', description="Serial Data Recording Utility for Linux/RaspberryPi devices.", @@ -36,7 +36,7 @@ setup_requires=['pytest-runner'], tests_require=['pytest'], classifiers=[ - 'Development Status :: 3 - Alpha', + 'Development Status :: 4 - Beta', 'Environment :: Console', 'Environment :: No Input/Output (Daemon)', 'License :: OSI Approved :: MIT License', @@ -45,6 +45,8 @@ 'Operating System :: POSIX :: Linux', 'Programming Language :: Python :: 3.5', 'Programming Language :: Python :: 3.6', + 'Programming Language :: Python :: 3.7', + 'Programming Language :: Python :: 3.8', 'Programming Language :: Python :: Implementation :: CPython', 'Topic :: Scientific/Engineering :: GIS', 'Topic :: Terminals :: Serial', diff --git a/system-packages.txt b/system-packages.txt deleted file mode 100644 index dafcf16..0000000 --- a/system-packages.txt +++ /dev/null @@ -1,15 +0,0 @@ -sudo apt-get install -ntfs-3g -exfat-fuse -exfat-utils - - -# To download/cache the deb packages needed -apt-get download PACKAGE && apt-cache depends -i PACKAGE | awk '/Depends:/ {print $2}' | xargs apt-get download - -# if downloading from non arm(pi) system: -dpkg --add-architecture armhf -apt-get update -apt-get download :armhf -# same format for apt-cache depends -apt-cache depends -i :armhf \ No newline at end of file