Mobile WebView `Raycasters.castRay()` returns `null` while `FragmentsManager.raycast()` hits

[ynnob]

Summary

In our Flutter WebView runtime, OBC.Raycasters.get(world).castRay() returns null for our loaded fragment models, while a direct FragmentsManager.raycast(...) at the same pointer position returns a valid hit.

The same viewer/model works on desktop, and the official ThatOpen Raycasters example also works when opened inside the same mobile WebView. We have not identified the exact difference yet.

Environment

• @thatopen/components: 3.4.3
• @thatopen/components-front: 3.4.3
• @thatopen/fragments: 3.4.5
• camera-controls: 3.1.2
• web-ifc: 0.77
• Runtime with issue: Flutter WebView on mobile
• Desktop/browser runtime: works

Observed Behavior

On mobile WebView, pointer/touch events fire correctly and the mouse coordinates appear valid. However:

const caster = components.get(OBC.Raycasters).get(world);
const result = await caster.castRay();

returns null.

At the same time, this succeeds:

const result = await fragments.raycast({
  camera: world.camera.three,
  dom: world.renderer.three.domElement,
  mouse: caster.mouse.rawPosition,
});

This result includes a valid localId, itemId, point, normal, and distance.

Debug Findings

We created a stripped-down test host based on the Raycasters example and logged both paths.

Touch Pick In Our Viewer

When tapping an element in the mobile WebView, input and camera-controls events are received correctly. The orbit/proximity logic can also hit via direct fragments raycast, but caster.castRay() returns null:

[Camera2:controlstart] { currentAction: 64, domTouchAction: 'none' }
[Camera2:touchstart] { touches: 1, first: {...}, second: null }
[Camera2:pickOrbitPoint:start]
[Camera2:pickOrbitPoint:miss] null

[IfcHost:hittest:start] { mousePosition: _Vector2, rawMousePosition: _Vector2 }
[IfcHost:hittest:miss] null

[Camera2:proximity:start] {
  enabled: true,
  hasRenderer: true,
  hasPointer: true,
  pointerClientX: 216.53366088867188,
  pointerClientY: 322.4615478515625
}

[Camera2:controlend] { currentAction: 0 }
[Camera2:touchend] { remainingTouches: 0, changedTouches: 1 }

[Camera2:mousedown] {
  button: 0,
  buttons: 1,
  clientX: 216,
  clientY: 322
}

[Camera2:pickOrbitPoint:start]
[Camera2:pickOrbitPoint:miss] null

[IfcHost:hittest:start] { mousePosition: _Vector2, rawMousePosition: _Vector2 }
[IfcHost:hittest:miss] null

[Camera2:proximity:hit] {
  hitDistance: 4.586507817493689,
  positionNdc: _Vector2,
  pointerPx: _Vector2
}

The important part is that the proximity path hits at the same pointer location, while the raycaster path misses.

Minimal TestHost Logs

For our loaded model in mobile WebView:

[TestHost:pointerup-tap:start] {
  event: {...},
  worldMeshes: 0,
  fragmentsInitialized: true,
  fragmentsList: 1
}

[TestHost:pointerup-tap:castRay] null

[TestHost:pointerup-tap:fragments.raycast] {
  point: _Vector3,
  normal: _Vector3,
  distance: 99.17305739690951,
  rayDistance: undefined,
  itemId: 460,
  localId: ...
}

[TestHost:pointerup-tap:miss] caster.castRay returned no localId

We also observed that the model object added to the scene can have no visible child meshes in the scene graph at the time of inspection:

[TestHost:model-added] { modelId: 'model', childCount: 0 }

[TestHost:pick-geometry-stats] {
  meshes: 0,
  visibleMeshes: 0,
  meshesWithId: 0,
  visibleMeshesWithId: 0,
  meshesWithoutId: 0
}

That explains why the post-GPU-pick fallback to world.meshes does not help in our setup: world.meshes is empty.

Possible Relation To Recent Raycaster Change

This may be related to the change introduced in:

de31964

From that change, non-snap SimpleRaycaster.castRay() now uses the GPU picker fast path by default. If the GPU pick path returns no hit, the remaining fallback appears to test world.meshes, not FragmentsManager.raycast(...).

In our mobile WebView case, direct FragmentsManager.raycast(...) succeeds, but castRay() returns null.

Open Question

Why does the official Raycasters example work inside the same mobile WebView, while our implementation with our loaded models does not?

Potential differences we are still investigating:

• Differences in fragment file content or generation version.
• Differences in how streamed fragments expose renderable tile meshes under model.object.
• Differences between the official hosted sample fragment and our fragment files.
• Differences in scene/camera/renderer initialization timing.
• Mobile WebView-specific behavior in the GPU picker render/readPixels path.

We do not currently know which of these is the root cause.

Workaround Implemented

We implemented a local LegacyRaycaster component that mirrors the pre-GPU default raycast behavior: it calls FragmentsManager.raycast(...) for fragment models, then optionally compares with intersections from world.meshes.

Simplified behavior:

const fragments = components.get(OBC.FragmentsManager);

const fragmentResult = await fragments.raycast({
  camera: world.camera.three,
  dom: world.renderer.three.domElement,
  mouse: raycaster.mouse.rawPosition,
  snappingClasses,
});

const itemsResult = raycaster.castRayToObjects(items, position);

return nearest(fragmentResult, itemsResult);

In our IfcHost, we install this only for the Flutter WebView runtime:

const raycaster = components.get(OBC.Raycasters).get(world);

if (environment === "flutter-webview") {
  const legacyCaster = components.get(LegacyRaycaster).get(world);

  raycaster.castRay = async (options) => legacyCaster.castRay({
    ...options,
    position: options?.position ?? raycaster.mouse.position,
    rawPosition: raycaster.mouse.rawPosition,
  });
}

This lets existing consumers such as OBF.Highlighter keep calling OBC.Raycasters.get(world).castRay(), while mobile WebView uses the fragments-worker raycast path internally.

Reproduction

I attached a fragment as example. In my test it did not matter if the fragment was generated with ifc-web 0.75 or with the new ifc-web 0.77.

ifc_and_frag_example.zip

import * as THREE from "three";
import * as OBC from "@thatopen/components";
import * as FRAGS from "@thatopen/fragments";
import { EnvironmentHelper } from "@zis/core";
import fragmentsPackage from "../../../../../node_modules/@thatopen/fragments/package.json" with { type: "json" };

type TestWorld = OBC.SimpleWorld<OBC.SimpleScene, OBC.OrthoPerspectiveCamera, OBC.SimpleRenderer>;

export class TestHost  {

    private world!: TestWorld;
    private readonly container: HTMLElement;

    constructor(private readonly components: OBC.Components) {
        const container = document.getElementById(EnvironmentHelper.getCanvasContainerId());
        if (!container) throw new Error("Viewer container not found");
        this.container = container;
    }

    async init(): Promise<void> {
        const worlds = this.components.get(OBC.Worlds);
        this.world = worlds.create<OBC.SimpleScene, OBC.OrthoPerspectiveCamera, OBC.SimpleRenderer>();
        this.world.scene = new OBC.SimpleScene(this.components);
        this.world.scene.setup();
        this.world.renderer = new OBC.SimpleRenderer(this.components, this.container);
        this.world.camera = new OBC.OrthoPerspectiveCamera(this.components);
    }

    async loadModel(): Promise<void> {
        const fragments = this.components.get(OBC.FragmentsManager);
        fragments.init(`./webworker/${fragmentsPackage.version}/worker.mjs`);

        this.world.camera.controls.addEventListener("update", () => fragments.core.update());
        fragments.list.onItemSet.add(({ value: model }) => {
            model.useCamera(this.world.camera.three);
            this.world.scene.three.add(model.object);
            fragments.core.update(true);
        });

        const url = `/MODEL_URL`;
        const response = await fetch(url);
        const model = await fragments.core.load(await response.arrayBuffer(), { modelId: "model" });
        await model.setLodMode(FRAGS.LodMode.ALL_GEOMETRY);

        const box = new THREE.Box3().copy(model.box);
        const sphere = box.getBoundingSphere(new THREE.Sphere());
        await this.world.camera.controls.fitToSphere(sphere, false);
        fragments.core.update(true);

        const caster = this.components.get(OBC.Raycasters).get(this.world);
        this.container.addEventListener("click", async () => {
            const castRayResult = await caster.castRay();
            const fragmentsRaycastResult = await (fragments as any).raycast({
                camera: this.world.camera.three,
                dom: this.world.renderer!.three.domElement,
                mouse: caster.mouse.rawPosition,
            });

            console.info("[raycast-repro]", {
                castRayResult,
                fragmentsRaycastResult,
                worldMeshes: this.world.meshes.size,
                fragmentsList: fragments.list.size,
                modelObjectChildren: model.object.children.length,
            });
        });

        console.info("[raycast-repro:ready] click model to compare castRay vs fragments.raycast");
    }
}

Suggested Improvement

Would it make sense for SimpleRaycaster.castRay() to fall back to FragmentsManager.raycast(...) when the GPU picker returns null, before falling back to world.meshes?

That would preserve the new GPU fast path where it works, while keeping compatibility for cases where the GPU pick path cannot resolve a hit but the fragments raycast can.

[agviegas]

This is a bug. Can you try @thatopen/components@3.4.4?

[ynnob]

Looks good to me! Thanks