[Linux, VS Code] Fails to read ITD_RESS.PAK; `pakInfo.compressionFlag` out of bounds

[jmortiger]

Problem

On Linux, loadPak fails to correctly load ITD_RESS.PAK.

Specifically, pakInfo.compressionFlag is supposed to be either 0, 1, or 4; however, it's read as 52.
It also appears that some other values might be wrong.

• I'd imagine pakInfo.discSize should be smaller than pakInfo.uncompressedSize; it isn't
• I'd also imagine pakInfo.offset isn't supposed to be negative; it is

How to replicate

Follow the instructions for building on Linux locally using Vs Code on my readme

[jmortiger]

It's possible, though I haven't had the time to verify, that this might be the result of taking Windows/NTFS files and moving/copying them to a Linux/ex4 context (e.g. improperly handling & resolving NTFS compression), or it might be an endianness problem.
If any other Linux users have the time to replicate & verify, that'd be great. If you don't have the game files, I'm not gonna give you the whole game, but you only need 2 files (really 1) to replicate this; you need a file named LISTBOD2.PAK for the engine to detect the game as AITD1 (it doesn't need to be the actual file, it just needs that name), and you need ITD_RESS.PAK to actually try and read data from. I'd be perfectly ok with giving anyone willing to try this my copy of ITD_RESS.PAK; just tell me your email and I'll send you the file. In fact, if anyone on Windows or macOS wants to try using my copy of ITD_RESS.PAK, that might help determine if it's the file that's bad, or the environment that's reading it; that would also be a big help. Windows would help determine if it's the file, macOS (being Unix derived) might help determine if it's the environment.
If a Windows user who already has the whole game & can successfully build & run it wants to try, be sure to keep track of my file vs your OG working files; maybe keep a backup archive of your files before downloading mine.

Additionally, though I'm personally quite skeptical of this, it's also possible that this is because of my toolchain; I'm using Visual Studio Code w/ the CMake plugin & gcc as the C compiler & g++ as the C++ compiler. If anyone on Windows with VSC & the required Visual Studio compilation files wants to try building using VSC, that'd also be a big help & I'll update the readme in my PR; just be sure to clean the configuration files before building to make sure the files are being generated by the same toolchain. I might try clang locally as well to see if it works.

[jmortiger]

If anyone would like to build for Windows & send me the compiled executable, I could test my local files and see if they work correctly that way as well.

[jmortiger]

Ok, here's the troubleshooting I've done today:

1. Boot into Windows & compile & run the binary against the files; worked fine
2. Ensured the files do not have NTFS compression enabled; they didn't
3. Booted back into Linux & used the same validated files on the native Linux build; same error
4. Ran the compiled Windows binary through Wine with the same validated files; worked fine.
   At this point, I have to assume there's either a serious problem or it's a simple oversight, like 1 system defaulting to a different endianness.

[jmortiger]

It appears to be the endianness. This resource says ITD_RESS.PAK has 19 subfiles in it; adding debug lines that change the endianness yield the following:

PAK_getNumFiles: Initial: ITD_RESS has 31586263236627 files (Raw value: 126345052946516)
	LE: ITD_RESS has 31586263236627 files (Raw value: 126345052946516)
	BE: ITD_RESS has 19 files (Raw value: 84)

[jmortiger]

I've figured it out. All of the #ifdef MACOSX that are related to endianness need to be changed to #ifndef WIN32. Both macOS & Linux need the endianness changed; makes sense as they're both Unix based. I've gotten the 2D armadillo to load, but it now crashes on the title screen, I assume for the same reason.

[yaz0r]

Endianness are nothing to do with them being Linux based, but of the CPU you are using. You can have a big or a little endian Linux. Although in this day and age, most common architectures are little endian. All this code was written in the age of PowerPC mac and the assumption that MACOSX == big endian is just plain wrong. I will do a pass on this quickly, but I don't really have anything big endian on hand where I can test that.

[jmortiger]

I'm dualbooting Linux & Windows 10, and the same branches ran on both; after changing it so the same branches taken on macOS were taken on Linux, it's been fixed. I know it's supposed to be processor dependent, but platform does matter. I even tried running the binary I compiled on Windows (before my changes) on Linux with Wine; worked fine, on the same OS, with the same CPU. Here's the lscpu command's output on Linux:

Architecture:             x86_64
  CPU op-mode(s):         32-bit, 64-bit
  Address sizes:          39 bits physical, 48 bits virtual
  Byte Order:             Little Endian

[jmortiger]

Whether it's a tooling problem or not, it is manually correctable in software. It's seemingly not consistent either; my fix isn't complete (sounds are wildly screwed up). I'm still trying to figure it all out, and I have to try rebuilding & launching on Windows to make sure I didn't break that.

[jmortiger]

I've found some documentation supporting this

• Linux appears to have changed endianness at some point (?) (third paragraph down)
• Apparently, some operating systems - including Unix - allowed "the same code to be compiled for platforms with different internal representations". Additionally, the Unix C compiler ...stored 32-bit "double precision integer long" values with the 16-bit halves swapped from the expected little-endian order... which seems similar to a bug I'm running into, where it's not straight LE nor BE.

[jmortiger]

The only other thing I can think of is that some/all of the program is 32 bit on the Windows workflow and 64 bit with my existing Linux workflow or vice versa. I'll see if I can test it, but I'm not super confident in that explanation.

[yaz0r]

If you're dual booting the same computer on windows and linux, it will have the same endianness (likely little endian).
Let me take a look at this when I have a minute. I know how to fix all this properly, I just need to find the time.

[jmortiger]

It's little endian. There's no rush; I'm working on a diagnostic thing anyways, might help.