Skip to content

Error on sentences without "normal" tokens #11

Description

@universome

When you pass an empty string or string without normal tokens (for example, ascii emoticons), exception is raised:

>>> import ner
>>> extractor = ner.Extractor()
>>> list(extractor(':)')) # works fine — empty list is returned
>>> list(extractor('')) # fails
>>> list(extractor('|*!*|')) # fails

Maybe these cases should return an empty generator too? Trace:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/universome/pyvenvs/zoo/lib/python3.6/site-packages/ner/extractor/extractor.py", line 54, in __call__
    tags = self.network.predict_for_token_batch([tokens_lemmas])[0]
  File "/home/universome/pyvenvs/zoo/lib/python3.6/site-packages/ner/network.py", line 379, in predict_for_token_batch
    batch_x, _ = self.corpus.tokens_batch_to_numpy_batch(tokens_batch)
  File "/home/universome/pyvenvs/zoo/lib/python3.6/site-packages/ner/corpus.py", line 205, in tokens_batch_to_numpy_batch
    max_token_len = max([len(token) for utt in batch_x for token in utt])
ValueError: max() arg is an empty sequence

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions