Skip to content

E2E: CCloud tests can land on the wrong environment and cluster #3397

@shouples

Description

@shouples

The VS Code extension's E2E suite signs in to CCloud as a real user and then takes .first() of the rendered Resources tree to pick an environment and a cluster. Because the extension sorts environments alphabetically and the test user has access to multiple environments (e.g., mcp-test-env and vscode-test-env), .first() lands on mcp-test-env. That's how e2e-produce-* topics and flink-vscode-* statements recently ended up in the mcp-confluent test environment.

The cluster-side leak already has a partial mitigation via E2E_KAFKA_CLUSTER_NAME, but it only kicks in for one branch:

let clusterLabel: string | RegExp | undefined;
if (topicConfig.produce && connectionType === ConnectionType.Ccloud) {
clusterLabel = topicConfig.clusterLabel ?? process.env.E2E_KAFKA_CLUSTER_NAME!;
}

The environment-side leak has no mitigation at all.

Leak sites

  • tests/e2e/objects/views/ResourcesView.ts - getEnvironment(Ccloud) defaults to ccloudEnvironments.first() when no label is provided.
    async getEnvironment(connectionType: ConnectionType, label?: string | RegExp): Promise<Locator> {
    let environment: Locator;
    switch (connectionType) {
    case ConnectionType.Ccloud: {
    try {
    // shorter timeout to allow refresh/retry below if needed
    await expect(this.ccloudEnvironments, "should see CCloud environment(s)").not.toHaveCount(
    0,
    { timeout: 3000 },
    );
    } catch (error) {
    // we've only seen this fail when VS Code fails to render the "connected" status update
    // for the CCloud connection, so try refreshing the view once and re-checking
    console.warn(
    `No CCloud environments found in Resources view; refreshing connections and retrying...`,
    { error },
    );
    await this.clickRefresh();
    await expect(this.ccloudEnvironments).not.toHaveCount(0);
    }
    environment = label
    ? this.ccloudEnvironments.filter({ hasText: label }).first()
    : this.ccloudEnvironments.first();
    break;
    }
    case ConnectionType.Direct: {
    await expect(this.directConnections).not.toHaveCount(0);
    environment = label
    ? this.directConnections.filter({ hasText: label }).first()
    : this.directConnections.first();
    break;
    }
    case ConnectionType.Local: {
    await expect(this.localItem).not.toHaveCount(0);
    environment = this.localItem;
    break;
    }
    default:
    throw new Error(`Unsupported connection type: ${connectionType}`);
    }
  • tests/e2e/utils/flinkStatement.ts:23-24, 27, 57 - ccloudEnvironments.first(), ccloudFlinkComputePools.first(), and the cluster quickpick .first(), all bypassing the helper.
    const resourcesView = new ResourcesView(page);
    // First, expand a CCloud env
    await expect(resourcesView.ccloudEnvironments).not.toHaveCount(0);
    await resourcesView.ccloudEnvironments.first().click();
    // Then click on a Flink compute pool to open the Flink Statements view
    await expect(resourcesView.ccloudFlinkComputePools).not.toHaveCount(0);
    const computePoolLocator = resourcesView.ccloudFlinkComputePools.first();
    await computePoolLocator.click();
    const computePoolItem = new FlinkComputePoolItem(page, computePoolLocator);
    const computePoolName = await computePoolItem.copyName();
    // Open the fixture file
    await stubMultipleDialogs(electronApp, [
    {
    method: "showOpenDialog",
    value: {
    filePaths: [fixtureFileName],
    },
    },
    ]);
    await page.keyboard.press("ControlOrMeta+O");
    const flinkSqlDoc = new TextDocument(page, path.basename(fixtureFileName));
    // Wait for the CodeLens actions to show up
    await expect(flinkSqlDoc.tab).toBeVisible();
    const codeLens = flinkSqlDoc.codeLensActions;
    // Select the Flink compute pool
    await codeLens.getByText("Set Compute Pool").click();
    const computePoolQuickpick = new Quickpick(page);
    await computePoolQuickpick.selectItemByText(computePoolName);
    await expect(codeLens.getByText(computePoolName)).toBeVisible();
    // Select a Kafka cluster (auto-filters to cloud provider/region matching the pool)
    await codeLens.getByText("Set Catalog & Database").click();
    const kafkaClusterQuickpick = new Quickpick(page);
    await kafkaClusterQuickpick.items.first().click();
  • tests/e2e/specs/scaffold.spec.ts:74-75, 80 - same pattern for the scaffold-from-Flink test.
  • tests/e2e/baseTest.ts:306-309 - clusterLabel is only set when topicConfig.produce && Ccloud, so every non-produce CCloud topic test calls loadTopics(..., undefined) and lands on whichever cluster the extension renders first.
  • tests/e2e/specs/produceMessage.spec.ts:70 - sets topicConfig: { name } without produce: true for the CCloud variant, falling into the same default-first path. This is the direct cause of e2e-produce-* topics in the mcp-confluent cluster.

Suggested approach

  • Introduce E2E_CCLOUD_ENV_NAME (default: vscode-test-env) and wire it through CI/vault next to E2E_KAFKA_CLUSTER_NAME. Document both in the E2E README.
  • Update ResourcesView.getEnvironment() so the Ccloud branch reads from E2E_CCLOUD_ENV_NAME when no label is passed, and throws (with a message listing the env names that were visible) if neither is set. Local/Direct branches keep .first() since they're inherently single-env.
  • Replace the three direct .first() calls (flinkStatement.ts:24, scaffold.spec.ts:75, and the default branch of ResourcesView.ts:282) with calls through the helper.
  • In baseTest.ts:306-309, set clusterLabel unconditionally for CCloud, not just when produce is true.
  • Apply the same "require a label for CCloud" rule to the Flink compute-pool selection in utils/flinkStatement.ts:27 and specs/scaffold.spec.ts:80, plus the Kafka cluster quickpick in submitFlinkStatement (flinkStatement.ts:57). Introduce E2E_CCLOUD_COMPUTE_POOL_NAME if needed.

Follow-up hardening (uniform resource naming, post-suite cleanup) lives in a companion issue and can land independently.

Metadata

Metadata

Labels

testingInvolves the test suite, click-testing, or tests running in CI

Type

No fields configured for Task.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions