Consider immutable array buffers for `type: bytes`

[lucacasonato]

For our implementation it would be very beneficial for this proposal to expose an immutable byte buffer, because the user visible ArrayBuffer could then be backed by the same underlying bytes that the module loader initially loads from disk / network.

If a mutable ArrayBuffer is returned, we have to take a copy of the underlying buffer, and would have to keep both copies in memory in perpetuity for two reasons:

• to avoid different underlying bytes for import(specifier, { type: "text" }) and import(specifier, { type: "bytes" }), we need to keep the underlying bytes around
• for JS modules we have to keep around the original source text for error display + source mapping, and without a significant refactor the same would apply to bytes / text modules in our implementation

The first point likely also applies to browsers, depending on the HTML integration.

As such, I'd advocate that we pipeline type: bytes after the immutable ArrayBuffer proposal (currently Stage 2.7), and return an immutable buffer for this type of import.

[styfle]

Makes sense, thanks!

It looks like you work at Deno which just shipped a mutable version of this feature: https://deno.com/blog/v2.4#importing-text-and-bytes

Is that because Deno doesn't support immutable yet?

[lucacasonato]

Yes indeed - it is still flagged in Deno. You need to use --unstable-raw-imports. We'd like to move to immutable buffers once we have them.

[styfle]

Are you thinking that import photo from "./photo.png" with { type: "bytes" } would mean that photo is ArrayBuffer?

If so then maybe we should change the type to buffer, like import photo from "./photo.png" with { type: "buffer" }

[phoddie]

I agree with @lucacasonato (and similar comments from @kriskowal elsewhere) because returning an immutable ArrayBuffer would work best for scenarios where RAM is limited (e.g. resource-constrained embedded systems).

As background, the Moddable SDK includes a Resource class which instantiates immutable buffers associated with a name (e.g. new Resource("image.bmp");). The proposed import is interesting to us as a standard way to express this capability. Because the buffers returned by the Resource constructor are immutable, we are able to return a reference to read-only SPI flash memory (more or less ROM) rather than making a copy. This allows for instantaneous instantiation and uses no RAM (important because it isn't uncommon to have resources that would otherwise use much of the limited RAM).

In addition to immutability, we prefer returning an ArrayBuffer over a view. There are many different kinds of views in ECMAScript because there are many different ways that one might reasonably want to access the content of the buffer. Preferring one particular view in import seems inappropriate since the data being imported is arbitrary.

[erights]

In further support for using an Immutable ArrayBuffer, consider the consequences of using a normal mutable ArrayBuffer (or equivalently enough, a Uint8Array on a mutable ArrayBuffer)

Let's say both x.js and y.js import foo.data as bytes. Because of module semantics, they must both get the same value. By assumption, let's say this is a mutable ArrayBuffer.

x.js wishes to share the data in this ArrayBuffer over postMessage, so that it is shared zero-copy on implementations that support zero-copy sharing of Immutable ArrayBuffers over postMessage. Since x.js is trying to avoid the cost of a copy, it does const passableFoo = foo.transferToImmutable(), moving the data zero-copy into the passableFoo Immutable ArrayBuffer, leaving the foo ArrayBuffer detached.

y.js wishes to use the data it just imported from the same foo.data, but finds that this import only gives it a detached ArrayBuffer. Not good.

[erights]

Or, if x.s doesn't know about transferToImmutable, it could just transfer to contents directly using postMessage, detaching the foo ArrayBuffer that both x.js and y.js hold. This is still zero-copy. x.js is not surprised that foo is detached. But this is a nasty surprise for y.js.

[styfle]

Thanks all!

I'll change this proposal to use immutable arraybuffer 👍

[styfle]

Based on feedback from the meeting today, we switched back to Uint8Array but it will be backed by an immutable ArrayBuffer.

• See PR change back to Uint8Array (backed by immutable ArrayBuffer) #12

[yisibl]

Please note that in FileAPI, a Blob.bytes() has been added to return Uint8Array. The motivation behind this may be useful as a reference here.

https://groups.google.com/a/chromium.org/g/blink-dev/c/klTrgVxlYYA