Preprocessing a file for csvbase.Importer requires acrobatics with temporary files

[mlell]

As CSV files downloaded from banks are notoriously messy, one might need to pre-process them beyond overwriting the header and footer members of 'csvbase.CSVReader`. For example. there might be a variable number of header lines. The common solution is to read in the file as a string, do the preprocessing (e.g. skip all lines up to a keyword, e.g. the first column header), then provide the updated string to the usual processing.

But csvbase.CSVReader.read() expects a file name as argument, not a file-like object or a content string. Therefore, one must do something like this

class MyMessyBank:
  
  def read(self, filepath):
    with open(filepath, 'r') as f:
      content = f.read()
    content = my_preprocessing(content)
    with tempfile.NamedTemporaryFile as t:
      t.writelines(content)
      for row in super().read(t.name):
        yield row

it would be easier if one could just use StringIO() or similar to provide modified data to CSVReader.read().

There are multiple options that come to my mind

• Allow read(str, ...) as well as read(file, ...)
• Introduce read() usage like read(None, ... , content: <file>)
• Add a open(filename) -> file function to importers.Importer that can be overwritten by subclasses

[sim642]

I just found it easiest to not bother with csvbase and wrote importers from scratch using Python's csv module. It's cleaner than piling various hacks around csvbase to make it work for strange CSV files it was never intended to handle.

[mlell]

@sim642 I see that this might lead to code which is easier to understand to someone who knows Python but is new to Beangulp. However, csvbase.CSVReader has a lot of options (header, footer, etc.) and is built in an extremely modular and extensible way (e.g. you could not only subclass the Importer, but also csvbase.Column to read a particularly strange column format), so I think it does target also very strange CSV files. Therefore it would be a small addition with a big impact to also allow to extend the raw file handling.