Venix binaries on TLVC?

[Mellvik]

An old idea that may be coming to fruition - not binaries directly but 'decompiled' and converted. Why is this a good idea? Maybe it isn't, but a K&R compiler suite with very low resource profile is within reach.

Just started, and the first test-object is almost ready. But there is a cross tool challenge: how to ia16-elf-link a pure asm file with a single global label, main, to become runnable on ELKS/TLVC? It needs crt0.o, that's it. Suggestions @ghaerr?

[ghaerr]

not binaries directly but 'decompiled' and converted.

Hmmm... what exactly does decompiled and converted mean? You mean disassembled, then assembled again using ia16-elf-as? There will be lots more to do than that: system calls have to be converted, possibly with glue to ensure any structures are matched (e.g. struct stat, usually different between any OS). In addition, it gets very tricky changing any data locations or offsets, as the original program may depend its ordering. For these reasons, sometimes it's better to do all of this "conversion" in the kernel itself, by swapping the syscall vector and performing structure conversions at runtime in the kernel.

There are also other problems, for instance knowing the value of the end of BSS, so that sbrk/brk work to support malloc, etc.

how to ia16-elf-link a pure asm file

You mean link an object file? I assume the object file is output from ia16-elf-as? In that case, easiest answer is just to write a simple crt0.s stub with .global _start which the linker already looks for, but and then calls main. But that's also problematic since some user-mode C startup expects other code to be executed, e.g. automatic stdio init routines, etc. These would be linked into separate sections. Not sure what/how Venix dealt with all that. As above, sometimes easier to write a simple loader for the .text and .data sections, as then all the above details have already been worked out - by the original linker, then emulate the syscalls.

Good luck!

[ghaerr]

a K&R compiler suite with very low resource profile is within reach.

Nice, which compiler suite is this, what's it run on? Is there source for it?

[Mellvik]

a K&R compiler suite with very low resource profile is within reach.

Nice, which compiler suite is this, what's it run on? Is there source for it?

If we had the source for it, this exercise wouldn't be useful, would it? Presumably, venix being a v7 port, that's where the compiler suite originates. But it's simplified and a lot smaller (and faster) - tiny and small model only, no overlays which the pdp11 version was (in)famous for etc. we'll find out, I'm still in the tool testing mode. It's crazy but kind of fun.

[Mellvik]

About the how to assemble question: assume you have a .s file. What it does is insignificant in this context, the question is generic, but for the example assume it has a main entry, calls sync() by stuffing the sync syscall # into the correct register, then int 0x80, then exit.

The venix crt0 code is being removed in the conversion process, to be replaced by the ELKS crt0, but I hit a wall when trying to link a .o file. I may need a special crt file but the problem is the same: what is the command to create a proper ELKS/TLVC binary from asm? I looked at unreal16 which has its own crt0 replacement, which may be a hint as to the way to go. The conversion tool can easily replace crt0 with something instead of just removing it.

Btw, the tool is using syscalls.dat to map syscall numbers and process (move around) the correct number of parameters. That's when doing syscall level conversion. The other alternative - yet to be tested - is library level: recognize libc calls, remove the original code and call the native libc instead. Having both is an opportunity to compare. That situation will require a different linker setup.

[ghaerr]

what is the command to create a proper ELKS/TLVC binary from asm?

$ ia16-elf-gcc -melks-libc -mcmodel=small -nostdlib asmfile.s

The above command will assemble asmfile.s into asmfile.o then attempt creating an a.out ELKS executable from asmfile.o. This will work only if asmfile.s has the _start symbol defined, as described above. If not, then the link process usually finds _start defined in the C library's crt0.o file, which drags it in, which then references and calls main for C programs.

[Mellvik]

Thanks @ghaerr. I was missing the -melks part. This works.

Now, has the tiny memory model ever been used with ELKS? Most Venix binaries use the tiny model and some relocation issues would go away if the same model could be used. What I'm getting then telling gcc to use 'tiny' is
´´´
crt0.o(.discard+0x0): error: undefined reference to 'CTOR_LIST'
´´´
... type errors. Do you know anything about this?

[ghaerr]

crt0.o(.discard+0x0): error: undefined reference to 'CTOR_LIST'

CTOR_LISTs are part of the somewhat-complex crt0.S startup used in ELKS. I thought you were going to use a stripped-down crt0.S, or none at all if -nostdlib used? As mentioned above, if you present a global _start symbol in your ASM program, no startup code is required. Otherwise you'll need to cut back libc/crt0.S to your liking and specify that in the command line.

has the tiny memory model ever been used with ELKS?

No. IIRC the tiny model overlaps CS and DS (i.e. CS == DS) prohibiting and complicating the implementation of shared text segments, which is standard. A new or modified loader would have to be written to execute them. I don't recommend trying to use tiny model, but you may have to given the limitations I discussed earlier. There may be assumptions that brk/sbrk takes with t_enddata/t_endbrk etc that would have to be considered or rewritten given the in-core memory layout isn't a.out format.

What I'm getting then telling gcc to use 'tiny' is

You can always try rewriting the linker script(s), they control the linking process, segment allocations, entry symbols, etc. They live in elks/*.ld.

[Mellvik]

Thanks, @ghaerr - that's what I figured (the back of my head says this was discussed aeons ago). Fine - no reason to do anything about that, just good to know. Most Venix binaries are 'tiny', but the interesting ones are small. And when going via disassembly to reassembly and linking, that's not a problem.

Crazy as it is, this side project is an educating exercise exposing all kinds of subtle surprises. Such as how to discover and handle cases like movw $L_1a54,L_1a54 - which needs two relocations - automatically

I decided to use the standard crt0.o by the way, rip out the Venix equivalent and (minimally) adapt main to match. Small binaries convert fine and are running at this point. Buffered IO turned out to be a real challenge. And hand-assembled libraries. As I progress (and acknowledge that only small model programs are interesting), I'm still wondering whether a wrapper in TLVC exec.c and running the binaries as they are would be better. The syscall mapping turned out to be easy. termio (tty IOCTLs) not so. And running programs that deal with file system structure is out of the question anyway. We'll see.

[ghaerr]

Glad to hear the project is going well, impressive you have Venix programs running already. It'd be great to see a writeup with more details about what you've done when you're ready.

how to discover and handle cases like movw $L_1a54,L_1a54 - which needs two relocations

My understanding is that the above construct may need object file relocations but never needs a runtime relocation (i.e. relocation entries in executable), since both relocations are fixed DS offsets when linked.

Buffered IO turned out to be a real challenge.

Are you running the stdio that was linked into Venix binaries, or does Venix support dynamic linking? If the former, everything should just work running the Venix linked-in stdio re-assembled. With DL, I can see that could be ugly, unless Venix libc headers can be found.

wondering whether a wrapper in TLVC exec.c and running the binaries as they are would be better.

Depending on the need for runtime relocations (all of which should be unneeded in small model) and dynamic libraries, writing a new loader should be pretty straightforward. If DL is required or the memory layout strays too far from the current MINIX model (separate I&D, DS ordered as .data, .bss, heap then stack) things could get complicated requiring other kernel mods.

termio (tty IOCTLs) not so.

Even with a Venix loader, structure incompatibility would seemingly require a syscall trampoline approach, where a new Venix syscall is created that then maps the Venix structure into the kernel's structure then normal syscall. This could get a bit complicated but might work by just calling sys_xxx from the new Venix syscall, if the syscall arguments are passed directly to the function. If they are passed via user-mode pointers, that may require splitting the affected kernel function into two, with the top half retrieving arguments and the other half executing the function.

[Mellvik]

how to discover and handle cases like movw $L_1a54,L_1a54 - which needs two relocations

My understanding is that the above construct may need object file relocations but never needs a runtime relocation (i.e. relocation entries in executable), since both relocations are fixed DS offsets when linked.

The thing here is that while the source is syntactically correct, gas will generate incorrect code since only one relocation will be generated. The instruction is equivalent to static int *p = &p and pops up in settings like buffered stdio where sentinel variables are commonly (?) used. Anyway, I had to check what gcc does in such cases and found that it does what I ended up doing in the conversion script: Change the code to use an intermediate register.

Buffered IO turned out to be a real challenge.

Are you running the stdio that was linked into Venix binaries, or does Venix support dynamic linking?

I'm trying hard to change as little as possible in the binaries. Everything is statically linked, there was no dynamic linking back then, and I don't think the system had shared text either. The problem with stdio is probably malloc related, I haven't done the deep dive yet - or more like it: I'm finding new and interesting things on the way down. There is generally no symbol table, but my test object (uuencode) was compiled by me 40+ years ago and not stripped, so symbols are in place and the patterns from libc routines found. Instrumenting the code for debugging is cumbersome and tedious, but possible.

wondering whether a wrapper in TLVC exec.c and running the binaries as they are would be better.

Depending on the need for runtime relocations (all of which should be unneeded in small model) and dynamic libraries, writing a new loader should be pretty straightforward. If DL is required or the memory layout strays too far from the current MINIX model (separate I&D, DS ordered as .data, .bss, heap then stack) things could get complicated requiring other kernel mods.

That's what I've been thinking too now that I'm getting to know the inside of the binaries. syscalls are easy, and only small model binaries are interesting, but I need to take closer look at the native crt0 to see how the heap and stack are organized.

termio (tty IOCTLs) not so.

Even with a Venix loader, structure incompatibility would seemingly require a syscall trampoline approach, where a new Venix syscall is created that then maps the Venix structure into the kernel's structure then normal syscall. This could get a bit complicated but might work by just calling sys_xxx from the new Venix syscall, if the syscall arguments are passed directly to the function. If they are passed via user-mode pointers, that may require splitting the affected kernel function into two, with the top half retrieving arguments and the other half executing the function.

Agreed, that is indeed possible. For now, and like I mentioned before, I'm staying away from binaries that need 'system level structures' like file system operations outside open/read/write/close. ioctl is a different story, the mentioned stty/gtty (really TIOCGETP/TIOCSETP) being a prime example. I decided to handle that one the 'hard' way by adding the necessary code to ntty.c/ntty.h - mapping termios<->sgttyb. There may be more of those along the way.