tsconfig notes

[tve]

I've been spending quite some time trying to get my typescript infrastructure sorted out. It's not easy... This (long!) post is mainly to record what I learned for future reference. Hopefully it helps others avoid some of the hours of frustration. If someone has suggestions or corrections please chime in!

Goal

• manifest.jsons that work and properly type check typescript code
• tsconfig.json for use in an editor (e.g. vscode) that properly type checks (and suggests) code
• minimize absolute paths in tsconfig.json for portability
• keep manifests and tsconfigs as concise as possible
• support apps (full bootable firmware) as well as mods (incrementally loaded code)

Difficulties

1. There are two parallel typing infrastructures:

   • the tsconfig auto-generated by mcconfig from manifests that is used when building an application
   • the static tsconfig in an application's root directory used by the editor's language server

   both need to be (more or less) in sync but have different constraints

2. ECMA-419 uses colons in module names (e.g. embedded:io/digital) and Windows apparently doesn't like colons in file names so all those modules don't have a direct module name to filepath mapping and necessitate paths declarations in tsconfig files (this is not a problem per-se but compounds other problems)

3. tsconfig files do not support any (environment) variable substitution, so anything outside of . necessitates absolute paths (or chains of ../ which is similarly non-portable)

4. tconfig files have an extends feature such that the firmware's tsconfig.json can include the Moddable SDK's tsconfig.json and thereby pick-up a lot defined there, except that stuff defined in the firmware tsconfig replaces the included parts. This is particularly annoying for paths because the Moddable SDK can have a nicely crafted paths clause but as soon as the firmware's tsconfig also has a paths clause the SDK's one is effectively lost.

5. When writing a host app and mods in typescript one has to refer to the .ts files when building the host (so the code gets built into the firmware) but one has to refer to corresponding .d.ts files when building the mod otherwise the module gets loaded twice. E.g., suppose the host has a module util in util.ts then the hosts' firmware needs to include that file but if a mod also want to use the utils it should refer to a util.d.ts so it gets just the type declarations and not all the implementation again.

Solutions

The building of application firmware is relatively straight-forward thanks to the fact that mcconfig produces a tsconfig that merges all paths defined in all manifests.

Getting the tsconfig for the IDE to work properly is more tricky. First I created a base tsconfig.json located in the Moddable typings dir. Conveniently is has no absolute paths:

{
  "compilerOptions": {
    "forceConsistentCasingInFileNames": true,
    "lib": ["es2022"],
    "module": "es2022",
    "target": "es2022",
    "sourceMap": true,
    "strict": true,
    "noImplicitAny": true,
    "typeRoots": [ // root to find *.d.ts typings
      ".",
    ],
    "types": [ // files to include without being referenced
      "./global",
      "../xs/includes/xs",
    ],
    "paths": { // modules whose name doesn't match the file path
      "embedded:io/*": ["./embedded_io/*"],
      "embedded:network/*": [ "./embedded_network/*" ],
      "embedded:provider/*": [ "./embedded_provider/*" ],
    }
  },
}

Next, this can be included in an application's tsconfig.json, which can be as short as

{
  "extends": ["/path/to/moddable/typings/tsconfig.json"],
  "include": [ "./main.ts" ]
}

(The include isn't even necessary if you don't mind all .ts in the subtree being included.)

But as soon as the application needs a paths declaration it has to repeat the SDK's paths and has to use absolute paths for that. For example, my app defines an embedded:network/something module:

{
  "extends": ["/path/to/moddable/typings/tsconfig.json"],
  "compilerOptions": {
    "paths": { // these _override_ all paths in the base tsconfig.json (ugh)
      "embedded:io/*": ["/path/to/moddable/typings/embedded_io/*"],
      "embedded:provider/*": [ "/path/to/moddable/typings/embedded_provider/*" ],
      "embedded:network/*": [
          "./embedded_network/*",  // <===== pick up my app's embedded:network modules
          "/path/to/moddable/typings/embedded_network/*",
      ],
    },
  },
  "include": [ "./main.ts" ]
}

Now mods are not obvious at all. First I tackled the compilation. For this, we need .d.ts files for all modules in the host firmware that may be used in the mod. For modules in the SDK this is easy because these files all exist in the typings dir (this would change if someday the SDK includes .ts files). For typescript modules in the application one can ask the typescript compiler to produce .d.ts files by adding this into the application's manifest:

  "typescript": {
    "tsconfig": {
      "compilerOptions": {
        "declaration": true,
        "declarationDir": "$(SRC)/typings"
      }
    }
  }

This provides a directory tree that the manifest for the mod can now include. The best way I found is to post-process all the paths in the manifest generated by mcconfig for the host because that lists everything in the host and thus everything the mod could import. The result is a manifest that the mod can include. The manifest for the mod looks something like this and expects the HOST environment variable to be set appropriately:

{
  "include": ["$(MODDABLE)/examples/manifest_mod.json", "$(HOST)/manifest_host.json"],
  "modules": {
    "*": ["./*.ts"],
  }
}

In my case the paths in manifest_host.json undergo a bunch of transformations to replace leading path components by $(MODDABLE) or $(HOST) and to deal with some cases where the .d.ts file doesn't end up where it's needed. I also had to edit some of the generated .d.ts files to remove a /// reference line.

The tsconfig for the IDE is a bit more tricky as it needs to reference the typings produced for the host firmware. This necessitates a paths declaration, which means that a repeat of all paths of the SDK and the host is necessary. I ended up with something like:

{
    "extends": [
        "/path/to/moddable/typings/tsconfig.json",
    ],
    "compilerOptions": {
        "paths": { // these _override_ all paths in the base tsconfig.json (ugh)
            "embedded:io/*": ["/path/to//moddable/typings/embedded_io/*"],
            "embedded:provider/*": [ "/path/to//moddable/typings/embedded_provider/*" ],
            "embedded:network/*": [
                "/path/to/host/typings/embedded_network/*", // <===== my host's embedded:network modules
                "/path/to//moddable/typings/embedded_network/*",
            ],
            "*" : ["/path/to/host/typings/*"], // <==== host typings produced by tsc
        },
      },
    "include": [ "./**/*.ts" ],
}

Note the non-portable absolute paths, which could be substituted by ../ relative paths and mandating a parent directory layout.

Tricks

• to see the result of extends run tsc --noEmit --showConfig -p .
• to see which files tsc pulls into the application add "listFiles": true to the compilerOptions and run tsc --noEmit
• to see where tsc goes to look for imported modules add "traceResolution": true to compilerOptions
• in some cases, to avoid the paths declarations it's possible to add a "include": "./**/*.d.ts" to the Moddable SDK's tsconfig. This works because all the files include a declare module statement which puts the declarations in the right place as "ambient modules", so even the ECMA-419 names with colons work. But this breaks down as soon as your app/mod has its own "include" since that overrides the SDK's.

[phoddie]

Thanks for writing up all of this knowledge to share! It is amazing what is possible. The challenge is to make that convenient to use, presumably with support tools support. Is there a practical starting point for that?

[tve]

I feel that everything I have is pretty raw still, it's not like I'm a TypeScript or tsconfig expert... The main difficulty is that there are a slew of options resulting in a bunch of different approaches one can take.