replace TREE with PDTREE

[mateuszviste]

SvarDOS comes currently with TREE v3.7.2 from the FreeDOS project, itself being a 1995 commercial code by Dave Dunfield, GPL-ized somewhere in the 2000s.

There is also pdTree: a public domain version written by @PerditionC
https://web.archive.org/web/20011212151852/http://www.darklogic.org/fdos/projects/tree/
https://github.com/FDOS/tree

Glancing at the source code it appears to be a very clean and memory-efficient implementation. Perhaps SvarDOS could use it instead of the current GPL tree after some minor work:

• "port" it from TCC to OpenWatcom, this should be a formality, code looks very nice and standard
• replace CPP parts with ANSI C (did not investigate this, but main file is CPP so I guess there must be some C++ hidden somewhere)
• create a proper makefile
• drop all Windows-related stuff
• maybe remove LFN support, if it leads to any significant simplification
• replace CATGETS with SvarLANG
• add a few translations (DE, FR, PL, TR, ...)
• remove (/reimplement) all GPL-tainted code (DB.C, DB.H, GET_LINE.C ? perhaps these are only CATGETS dependencies)
• drop libc, reimplement necessary (few) std functions and link with WMINCRT
• add some documentation/notes that explains where this TREE comes from and to avoid any confusion with KJD's original

[boeckmann]

Well, that bullet list doesn't exactly look like "minor work". But if you think its worth it, go for it. Maybe we should put a counter anywhere which counts the packages still being GPL licensed, and try to converge that towards zero :)

[PerditionC]

it should build with ow given it builds with msvc, but I will check it this week. note I have a pd implementation of cats that I plan on switching out the lgl one with and releasing as fd tree 4.0. It's been a while, I'm guessing it builds with a batch file. Adding a make based build for ow should be trivial, so I will check on it. I guess I need to make sure all translations are updated, both versions use same files for translations. There is no c++ other than maybe some variable scoping, I believe it was a naming conflict resolution. The api is based on win32 findfile, so removing lfn support likely won't really reduce size much.

I have too look, may need to add a findfile using ow pragma aux.

(off topic, but fd format is switching back to small model, but small penalty to copy strings from far buffer to near accessible one).

I will try to get some updates on GitHub later this week.

[mateuszviste]

Maybe we should put a counter anywhere which counts the packages still being GPL licensed, and try to converge that towards zero :)

I did think about it for a short time, but ultimately dropped the idea because I do not want to give the impression that "fighting" with GPL is one of SvarDOS goals.

I plan on switching out the lgl one with and releasing as fd tree 4.0.

So pdTree will soon become the default FreeDOS tree, replacing the 3.7.2 version from Dave Dunfield? Cool. Is there any practical reason for it?

BTW the FreeDOS listing is somewhat confusing: it lists TREE 3.7.2 by Dave Dunfield, but provides a link to your PD version:
https://www.ibiblio.org/pub/micro/pc-stuff/freedos/files/distributions/1.2/repos/pkg-html/tree.html

I have too look, may need to add a findfile using ow pragma aux.

It's exactly what I was planning to do :)
Great to hear that you are still interested in this 20-years old code!

[mateuszviste]

I started doing some maintenance & trimming on pdTree here:
http://svn.svardos.org/log.php?repname=SvarDOS&path=%2Ftree%2Ftrunk%2F&isdir=1

So far I removed most of Windows-specific stuff, utf8 output, CATGETS, and wide characters support. I have also replaced all C++-isms by ANSI C equivalents, so this fork of pdTree is 100% C now.

[mateuszviste]

After some more hours of hacking, slashing and reformatting, SvarDOS TREE is finally functional.
It compiles with OpenWatcom with a proper makefile, loads its translations via SvarLANG and appears to work reliably as far as I can tell. I removed a LOT of code (no LFN, no Windows support, no Unicode...), simplified many parts of it and replaced some functions with OpenWatcom-specific calls.

UPXed it is a 11K COM file now. Not as small as I want it to be, but there is still hope to get a binary smaller than the GPL FreeDOS TREE (10K) once I get rid of printf. I will probably not go the WMINCRT route, though, since I rely heavily on OpenWatcom's libc now.

Two things that I have yet to do:

• write some (light) document that explains the history of this TREE
• review the translation strings, there are at least some unused strings, removing them will help decreasing the footprint

[boeckmann]

Well done :) We could spare some bytes by not writing the default language to the .lng file. We could adapt tlumacz to at least optionally skip outputting the default language. For FDISK, for example, that would save over 10k bytes. And I am sure that I will recompile the program when I have to update the default language anyway.

[mateuszviste]

We could spare some bytes by not writing the default language to the .lng file.

Yes but then it would not be possible to re-load EN from within the application (if the application supportes such reloads). This is used by the SvarDOS installer.

[mateuszviste]

a win would be to have each language in the LNG file optionally compressed. It's human language, lots of redundancy. There must be some algorithm nowadays that is capable of in-place depacking with a reasonably small depacking code.

[boeckmann]

I took the opportunity to make the makefile compatible with Linux. Should still work fine under DOS (at least in DosBox it does).

[ecm-pushbx]

a win would be to have each language in the LNG file optionally compressed. It's human language, lots of redundancy. There must be some algorithm nowadays that is capable of in-place depacking with a reasonably small depacking code.

You may want to look at my use of heatshrink in my Extensions for lDebug, extpak.eld and list.eld. You need a buffer the size of the depack window. If you access the compressed file in a linear way then you can re-use the same depacker state across several calls into the depacker.

[ecm-pushbx]

Main depacker is in https://hg.pushbx.org/ecm/ldebug/file/a35f88de973a/source/eld/depack.asm

[mateuszviste]

Main depacker is in https://hg.pushbx.org/ecm/ldebug/file/a35f88de973a/source/eld/depack.asm

Seems a bit complicated. I was thinking about something much simpler. Maybe a custom algorithm that would read bytes from the data file and when spotting a special marker (say, 0xFF O S), it would know that it has to copy now S bytes from offset -O in past stream. Probably not the most efficient approach, but should be easy to implement in any language, and might provide good enough results for text strings. I might investigate this in incoming days, once I am done with SvarTREE.

[mateuszviste]

I committed an experimental change to SvarLANG: TLUMACZ outputs now two versions of the LNG file: one as usual (OUT.LNG) and the other one compressed (OUTC.LNG). SvarLANG is able to read both. Older versions of SvarLANG will not crash on it, just ignore the compressed languages.

The compression scheme is very simple, I invented it during a coffee break and implemented within an evening hour. It is not meant to be state of the art - just somewhat better than uncompressed text and simplest possible to decompress with no extra memory buffer. It assumes highly redundancy of comoressed data, should work well with text, but attempting to compress binary things is likely to produce "compressed" data twice larger than the original.

It works on a simple example, I will proceed with further tests tomorrow or next week. There is also one or two tweaks I'd like to check.

[mateuszviste]

Some preliminary results:

• compressing TREE.LNG yields no benefit, the size of the strings is almost the same (8.7K uncompressed, 8.8K compressed).
• compressing FDISK.LNG makes the string resources over twice smaller (72K uncompressed, 35K compressed).

TREE still works after compression. FDISK I could not check because for some reasons I am not able to compile it (unrelated to SvarLANG).