Checking the number of rows in all columns is same (or less-than-or-equals?) as number of headers (per file) (error-reporting)

[aplaice]

The user should have CSV files which have all columns labeled (i.e. no row has more columns than there are headers in the given file). If this is the case, then the error described here won't occur. (i.e. I'm not describing a bug in BrainBrew — if the inputs are correct, then everything is fine — just sub-optimal error-reporting.)

Unfortunately, while this will usually be the case, it won't always be, as the user might add some text to the right of the rightmost column (as temporary annotation) or forget to add the relevant column header. If it isn't the case, then a rather confusing message is displayed (e.g. see here):

Details

$ pipenv run build
INFO:root:Builder file recipes/source_to_anki.yaml is ✔ good
INFO:root:Attempting to generate Guids
INFO:root:Generate guids complete
Traceback (most recent call last):
  File "/home/munksgaard/.local/share/virtualenvs/ultimate-geography-wQmoXBY1/bin/brain_brew", line 8, in <module>
    sys.exit(main())
  File "/home/munksgaard/.local/share/virtualenvs/ultimate-geography-wQmoXBY1/lib/python3.10/site-packages/brain_brew/main.py", line 19, in main
    command.execute()
  File "/home/munksgaard/.local/share/virtualenvs/ultimate-geography-wQmoXBY1/lib/python3.10/site-packages/brain_brew/commands/run_recipe/run_recipe.py", line 15, in execute
    recipe = TopLevelBuilder.parse_and_read(self.recipe_file_name, self.verify_only)
  File "/home/munksgaard/.local/share/virtualenvs/ultimate-geography-wQmoXBY1/lib/python3.10/site-packages/brain_brew/commands/run_recipe/top_level_builder.py", line 60, in parse_and_read
    return cls.from_list(recipe_data)
  File "/home/munksgaard/.local/share/virtualenvs/ultimate-geography-wQmoXBY1/lib/python3.10/site-packages/brain_brew/commands/run_recipe/recipe_builder.py", line 20, in from_list
    tasks = cls.read_tasks(data)
  File "/home/munksgaard/.local/share/virtualenvs/ultimate-geography-wQmoXBY1/lib/python3.10/site-packages/brain_brew/commands/run_recipe/recipe_builder.py", line 68, in read_tasks
    task_or_tasks = [matching_task.from_repr(task_arguments)]
  File "/home/munksgaard/.local/share/virtualenvs/ultimate-geography-wQmoXBY1/lib/python3.10/site-packages/brain_brew/commands/run_recipe/parts_builder.py", line 40, in from_repr
    return cls.from_list(data)
  File "/home/munksgaard/.local/share/virtualenvs/ultimate-geography-wQmoXBY1/lib/python3.10/site-packages/brain_brew/commands/run_recipe/recipe_builder.py", line 20, in from_list
    tasks = cls.read_tasks(data)
  File "/home/munksgaard/.local/share/virtualenvs/ultimate-geography-wQmoXBY1/lib/python3.10/site-packages/brain_brew/commands/run_recipe/recipe_builder.py", line 68, in read_tasks
    task_or_tasks = [matching_task.from_repr(task_arguments)]
  File "/home/munksgaard/.local/share/virtualenvs/ultimate-geography-wQmoXBY1/lib/python3.10/site-packages/brain_brew/build_tasks/csvs/notes_from_csvs.py", line 56, in from_repr
    file_mappings=rep.get_file_mappings(),
  File "/home/munksgaard/.local/share/virtualenvs/ultimate-geography-wQmoXBY1/lib/python3.10/site-packages/brain_brew/build_tasks/csvs/shared_base_csvs.py", line 22, in get_file_mappings
    return list(map(FileMapping.from_repr, self.file_mappings))
  File "/home/munksgaard/.local/share/virtualenvs/ultimate-geography-wQmoXBY1/lib/python3.10/site-packages/brain_brew/transformers/file_mapping.py", line 76, in from_repr
    derivatives=list(map(cls.from_repr, rep.derivatives)) if rep.derivatives is not None else [],
  File "/home/munksgaard/.local/share/virtualenvs/ultimate-geography-wQmoXBY1/lib/python3.10/site-packages/brain_brew/transformers/file_mapping.py", line 68, in from_repr
    csv.read_file()
  File "/home/munksgaard/.local/share/virtualenvs/ultimate-geography-wQmoXBY1/lib/python3.10/site-packages/brain_brew/representation/generic/csv_file.py", line 47, in read_file
    self._data.append({key.lower(): row[key] for key in row})
  File "/home/munksgaard/.local/share/virtualenvs/ultimate-geography-wQmoXBY1/lib/python3.10/site-packages/brain_brew/representation/generic/csv_file.py", line 47, in <dictcomp>
    self._data.append({key.lower(): row[key] for key in row})
AttributeError: 'NoneType' object has no attribute 'lower'

It's possible to track down the source of the error with a small amount of debugging, but it's not very user-friendly.

If BrainBrew checked that there are no rows with too many columns (i.e. more columns than there are headers in the given file), then I believe that this class of issues could be caught with nicer warning messages.

This is obviously not very urgent/important and I haven't really thought about whether there are better solutions/whether my proposed solution would not slow BB down etc. — I'm mainly writing this down here for reference! :)