TODO.md

downloadable items

• zrc datasets:{pull,import,rm}: list, pull, import and delete datasets archives
• zrc checkpoints:{pull,import,rm}: list pull, import and delete checkpoint archives
• zrc samples:{pull,import,rm}: list pull and, import delete samples archives

benchmarks

• zrc benchmarks: list existing benchmarks
• zrc benchmarks:info: details on each benchmark
• zrc benchmarks:run <name> <submission_dir>: run a benchmark on a submission
• zrc benchmarks:run <name> -t <task_name>: run only some tasks of a benchmark on a submission
• zrc benchmarks:run <name> -s <set_name>: run a benchmark only on specific subsets of a submission

Index

• auto-update: auto update: conditions: if local file is older than a week, if remote file has been updated
• manual update zrc reset-index

submissions

• zrc submission:init <benchmark_name> <submission_dir>: create a submission directory
  • TODO: deduce benchmark from meta.yaml (remove it as argument to the command)
• zrc submission:params <submission_dir>: show current parameters
• zrc submission:verify <submission_dir>: validate a submission directory

Submit

• zrc submit <submission_dir> : upload a submission from a directory
  • submission verification
  • leaderboard generation
  • zip_submission
  • upload zip (with resume function)
  • TODO: connect to backend

user

• zrc user
• zrc user:login
• zrc user:clear

Available Benchmark list

• sLM21
• abxLS
• abx17
• tde17
• tts019
  • TODO: eval implementation
  • Add dataset
  • Add random submission
  • TODO: Benchmark & task integration
  • TODO: Submission
  • TODO: Submission Validation
  • TODO: Leaderboard & ScoreDir

potential extensions & plugins

• extension 0: vocolab_extension:

  • implement for leaderboard validation/management
  • TODO: implement for submission validation

• extension 1 : extractors --> implement some basic extractor for the most used models Extractor for CPC, Bert, LSTM, etc...

• extension 2 : infSim adaptor wrapper package Wrapper module that allows to use this API to allow running benchmarks on infSim architecture