Automatic code extraction - top level utils

[maiieul]

What is it about?

Automatically extracting code – namely top level utils – besides $ boundaries.

What's the motivation for this proposal?

Problems you are trying to solve:

The optimizer can produce segments code that contain both component code and shared helpers. This can lead to over-preloading on user-interaction or – more problematically – on visible tasks (e.g. instead of ~4-5 bundles to preload, it can easily become ~50-100). This happens because the optimizer doesn't split helpers into separate segments. So in practice if a topLevelHelper function is declared in a separate file, the optimizer will leave it untouched, but if it is colocated in the same file as the component code, then the optimizer will output it in the same facade segment as the componentQrl, meaning that when the topLevelHelper is imported by another module, not only the topLevelHelper but also the component and all its transitive dependencies will be eagerly preloaded (with 100% probability) as well.

============================= test.js ==

import { componentQrl } from "@qwik.dev/core";
import { qrl } from "@qwik.dev/core";
const i_uFrpfTnm3bA = ()=>import("./test.tsx_TestButton_component_uFrpfTnm3bA");

export const topLevelHelper = ()=>{ // -> helper function which might be imported anywhere else in the application as `import { topLevelHelper } from "test.js";`
    console.log("topLevelHelper");
};

export const TestButton = /*#__PURE__*/ componentQrl(/*#__PURE__*/ qrl(i_uFrpfTnm3bA, "TestButton_component_uFrpfTnm3bA"));

Now this means that if an on-click segment imports { topLevelHelper } from "./test.js" and TestButton_component_uFrpfTnm3bA has a lot of transitive dependencies, then the the preloader will end up eagerly preloading (100% probability) TestButton_component_uFrpfTnm3bA and all its transitive dependencies, even though we only needed to get a hold of topLevelHelper.

Goals you are trying to achieve:

• Prevent over-preloading on visible tasks -> Faster preloading

---

Solution n°1

What do you propose?

Extract everything outside of component$ into a "test_helpers_hash.js" segment, and then do a side-effect import "./test_utils_hash.js" in the component segment (test_hash.js) to keep the same initialization order. Then any consumer of helper will import from test_utils_hash.js instead of from test_hash.js.

Code examples

For

let globEnv = isDev ? "dev" : "prod";
globEnv = "special";
export const helper = () => {
  if (globEnv === "special") console.log("helper");
};

export const TestButton = component$(() => {
  ...
});

We would extract the globals into a separate file. See the diff here.

For this to work, we need to be able to resolve the aliases and verify if the function is a component or a util.

For a barrel file, the current optimizer would produce

============================= index_CCrpfTnm3CC.js ==


export { TestButton } from './test.js';
export { topLevelHelper } from './test.js';

but now it would have to be

============================= index_CCrpfTnm3CC.js ==


export { TestButton } from './test_components_AArpfTnm3AA.js';
export { topLevelHelper } from './test_utils_BBrpfTnm3BB.js';

which is not easy to achieve.

How can the optimizer know whether topLevelHelper belongs in test_utils_BBrpfTnm3BB or in test_components_AArpfTnm3AA?

Same with

import { topLevelHelper } from "@alias/lib";

Now it would have to be

import { topLevelHelper } from "./test_utils_BBrpfTnm3BB.js";

We can resolve the alias to "../../../path-to-helpher/test.js" but how can we know if it belongs in belongs in test_utils_BBrpfTnm3BB or in test_components_AArpfTnm3AA.

We can use the PascalCase convention to know that TestButton should be imported from the _components_ path, but it's quite brittle. A better solution is to resolve the aliases and process them to verify what the functions are but it is more expensive. We would need to do a first pass, where we'd create a Map containing the alias, the resolved path and the metadata of the functions imported by aliases.

This is challenging with OXC and even more so with SWC.

diff here

Solution n°2

Alternatively we could extract each top level helper into its own segment with one globals segment, then we can simply deduce the import path by resolving the alias (no need to verify what the function is): export { topLevelHelper } from './test_topLevelHelper_EErpfTnm3EE'; and have each helper segment import from the globals segment as well as the component import the globals file as a side-effect.

Similarly we need to resolve import { topLevelHelper } from "@alias/lib"; into import { topLevelHelper } from "../../../path-to-helper/test"; because the developer could create an alias that points directly to the file itself such as import { topLevelHelper } from "@alias/path-to-helper/test"; in which case we can't guess that it's a barrel file alias import.

Pros:

• optimizer logic is much simpler and less error prone, probably faster too
• helper segments can then be bundled back together with the bundler in any way we want (e.g. all of them together, group those that have a common parent, etc.)
  Cons:
• More strain on the Rollup side -> Probably fine with Rolldown. We are doing it for every $() wrapper already.
• We need to be surgical in the way we group things back together to avoid circular dependencies or having too many chunks

This might work better but requires every cog in the machinery to be perfectly in place.

See this stackblitz for a minimal improved repro of Qwik's bundling. We currently use manualChunks but for a better result we could use emitFile with implicitlyLoadedAfterOneOf to specify which segments are entries (like in the input array), and which can be bundled together with those entries.