Technology Stack and Approach

• Agile development
• Data Model is being developed
• Continuous Integration: Jenkins

See the entire technical stack below along with the reasoning behind choosing particular components in italics. Unless specified, all software is Free and/or Open Source.

Table of Contents DB ETL Tools and Infrastructure Documentation

DB

• Postgres DB: PostgreSQL 9.6.x on CentOS Linux 7.3.1611
  • HBase HDInsight cluster as an alternative is not beneficial.

ETL

• CentOS Linux 7.3.1611
• Python 2.7, Anaconda2
  • More flexibility than provided by specialized ETL tools, e.g. Pentaho Community Edition ETL. Community edition has significant limitations.
• [] TODO. Anti-virus: ClamAV

Tools and Infrastructure

• Version Control: Git
• Continuous Integration: Jenkins
• Tickets: JIRA
• Automated Testing: George will develop queries to be tested using Jenkins and TAP.
  • pgTap
  • PL/SQL TAP
  • Jenkins TAP Plugin

Documentation

• Web of Science Web Services