Core dump - 'vtslibs::storage::BadFileFormat'

[Tekn1ck]

Hi @vaclavblazek

I am having a big issue with the VEF2VTS tool. The VEF2vts tool loads all the windows and splits the tiles fine (it seems) but then about 1-2% of the way through generating the tiles it just throws the error below and crashes.

2021-11-09 22:48:13 I3 [850(tile:16-58907-56488)]: Generated tile #1905 of 254506 ( 0.75 % done): 16-58907-56488 (epsg27700, extents: 541321.192481,179184.067720:541331.920247,179202.757555) [mesh,atlas]. {encoder.cpp:process():304}
terminate called after throwing an instance of 'vtslibs::storage::BadFileFormat'
what(): File "/mnt/VTS_Processing/vef2vts_output/Project/tmp/0/c5/16-0001842-0001766.tiles:18,0,0" is not a VTS simplemesh file. @{tmptileset.cpp:loadSimpleMesh():180}
Aborted (core dumped)

I have done all the usual checks - manifest has the correct number of tiles versus LODS and the number of tile folders is correct and everything matches. I am now going through and trying to do the tiles in smaller batches.

Can you give me any pointers as to what may be the reason for this and what I can do to troubleshoot the issue and isolate the cause?

Thanks

[vaclavblazek]

Pls, can you send me the file mention in the exception, i.e.: /mnt/VTS_Processing/vef2vts_output/Project/tmp/0/c5/16-0001842-0001766.tiles in this case.

[Tekn1ck]

It is a bit large, like 5gb and we are under NDA so it is hard to send this to you.
I just did a smaller number of tiles and it worked - so I guess it could be something in the model files themselves. Could it be tiles with not enough geometry at the lowest LOD level for the splitting to have worked properly?

[vaclavblazek]

No, this is something wrong in the temporary data that are created during conversion. Which is very, very strange.

[Tekn1ck]

hmm - really hard - I will see if I can replicate the issue on a smaller dataset and see if I can get you that to work with - how would I get it to you?

[vaclavblazek]

Well, first, I guess the tiles file mentioned in the error is 5 GB, it should be way, way less. You may send it via drobpox or some other service.
What distro and vef2vts tool version do you use?

[Tekn1ck]

Can I send a private message on here?

Ubuntu 18.04 LTS
Latest version of vef2vts I believe I did an apt update before I started processing.

[vaclavblazek]

I'm not sure there is a PM support on github.
You may send me an email (my name.surname@melowntech.com).

We have not made recent packages for bionic (18.04), due to some problems we switched to focal (20.04). How (in)convenient is to upgrade for you?

[Tekn1ck]

ok great - so it is not too hard - we are just running it on a VM. If we did an upgrade to 20.04 would that work - or better to do a fresh install?

Will pop you an email.

[vaclavblazek]

IMHO fresh install would be better -- no need to mess with different repos and their incompatibilities.

[Tekn1ck]

OK thanks, Well I will wait first to see if you find anything before we do a bit overhaul. Nick [note: deleted personal info]

[vaclavblazek]

Aha, now I know 🤦‍♂️.
The underlying file format uses 32-bit indices and your's has 4.3 GB. Which cannot fit in 32-bit integer.

Now, the question is: why is this file so enormous? It contains up to 1024 tiles (cannot tell details now since the pointer to content index is nonsense).

How big are other .tiles files?

NB: upgrade is not gonna help here, but I'd recommend it anyway.

[Tekn1ck]

They vary from a fes mb through to gigabytes of data. It is late here today. But I can send you a directory tree tomorrow so you can see.

…

On Wed, 10 Nov 2021, 23:51 Václav Blažek, ***@***.***> wrote: Aha, now I know 🤦‍♂️. The underlying file format uses 32-bit indices and your's has 4.3 GB. Which cannot fit in 32-bit integer. Now, the question is: why is this file so enormous? It contains up to 1024 tiles (cannot tell details now since the pointer to content index is nonsense). How big are other .tiles files? NB: upgrade is not gonna help here, but I'd recommend it anyway. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQGXPSK5D2LMG5YJ6TUORZLULJTF3ANCNFSM5HW6DFSQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[vaclavblazek]

Hi, can you pls install package vts-tools version 1.99 from our repository (focal only). There's a brand new option --tweak.binaryOrder that sets size of single archive file in tiles (square, with edge 2^binaryOrder, i.e. total area in tiles is 4^binaryOrder). Try setting it to 4. I hope it'd help, at least to process data.

[Tekn1ck]

Thank you Vaclav, I will let you know how I get on Nick

…

On Mon, 22 Nov 2021 at 23:04, Václav Blažek ***@***.***> wrote: Hi, can you pls install package vts-tools version 1.99 from our repository (focal only). There's a brand new option --tweak.binaryOrder that sets size of single archive file in tiles (square, with edge 2^binaryOrder, i.e. total area in tiles is 4^binaryOrder). Try setting it to 4. I hope it'd help, at least to process data. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#35 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AQGXPSJVEOZVB24AM4CBQJLUNIWTBANCNFSM5HW6DFSQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

-- *Nick Pickop* Group Director Unit A, 2/F, Gold King Industrial Building, 35-41 Tai Lin Pai Road, Kwai Chung, New Territories, Hong Kong Mob Primary(AUS) +61 412 731 029 Mob(Hong Kong): +852 9370 7836 Mob(UK): +44 7790 904579 Mob(Singapore): +65 9650 3805 LONDON • LISBON • HONG KONG • SINGAPORE www.tekuchi.com