Skip to content

Unable to Modify Tokenization Logic #169

@Andrew03

Description

@Andrew03

Tried to follow the wiki for changing linguistic resources and got to this step: make -f Makefile.gnu lang=wbd all, which fails on Linux (ubuntu 20.04) VM. The output appears to match the wiki's output up until the first error:

fa_build_conf \
  --in=wbd/ldb.conf.small \
  --out=wbd/tmp/ldb.mmap.small.txt
fa_fsm2fsm_pack --type=mmap \
  --in=wbd/tmp/ldb.mmap.small.txt \
  --out=wbd/tmp/ldb.conf.small.dump \
  --auto-test
fa_build_lex --dict-root=. --full-unicode --in=wbd/wbd.lex.utf8 \
  --tagset=wbd/wbd.tagset.txt --out-fsa=wbd/tmp/wbd.rules.fsa.txt \
  --out-fsa-iwmap=wbd/tmp/wbd.rules.fsa.iwmap.txt \
  --out-map=wbd/tmp/wbd.rules.map.txt
Assigning non-zero to $[ is no longer possible at fa_preproc_WS11bVXJ line 2.
Assigning non-zero to $[ is no longer possible at fa_preproc_PgtJwpNP line 2.
Assigning non-zero to $[ is no longer possible at fa_preproc_QfbtHErz line 2.
Segmentation fault (core dumped)
ERROR: Input/Output error. in /home/azureuser/BlingFire/blingfirecompile.library/src/FAAutIOTools.cpp at line 47 in program fa_fsm2fsm_iwec
ERROR: Input/Output error. in /home/azureuser/BlingFire/blingfirecompile.library/src/FAAutIOTools.cpp at line 313 in program fa_nfa2dfa
Segmentation fault (core dumped)
Segmentation fault (core dumped)
ERROR: Input/Output error. in /home/azureuser/BlingFire/blingfirecompile.library/src/FAAutIOTools.cpp at line 84 in program fa_fsm2fsm
fa_fsm2fsm_pack --alg=triv --type=moore-dfa --remap-iws --use-iwia --in=wbd/tmp/wbd.rules.fsa.txt --iw-map=wbd/tmp/wbd.rules.fsa.iwmap.txt --out=wbd/tmp/wbd.fsa.small.dump
ERROR: Input/Output error. in /home/azureuser/BlingFire/blingfirecompile.library/src/FAAutIOTools.cpp at line 449 in program fa_fsm2fsm_pack
make: *** [Makefile.gnu:233: wbd/tmp/wbd.fsa.small.dump] Error 2

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions