[🐞] Cloudflare Pages: Ability to run cloudflare pages dev server with vite

[sujith97]

Which component is affected?

Qwik Rollup / Vite plugin

Describe the bug

The current Cloudflare pages setup makes it hard to develop qwik-city with Cloudflare page functions. Cloudflare has D1, KV etc objects that should be accessible during the development mode but at the moment we should first build. The build drops the static files to dist folder which can then be accessed using Cloudflare pages. Ideally, we should be able to have the HMR and wrangler reload during local development.

Can we provide guidance on how to develop locally with Cloudflare and qwik-city vite? (https://developers.cloudflare.com/pages/platform/functions/local-development/)

export const useListLoader = routeLoader$(async (context: any) => {
  const ps = (context.env.get("cloudflaredb") as any)?.prepare(
    "SELECT * FROM users"
  );
  if (ps) {
    const users = await ps.first();
    return {
      users,
    };
  } else {
    console.log("No access to D1", context.env.get("cloudflaredb"));
  }

  return {
    users: [],
  };
});

Reproduction

https://stackblitz.com/edit/qwik-starter-joopk8?file=src%2Froutes%2Findex.tsx

Steps to reproduce

No response

System Info

System:
    OS: macOS 13.0
    CPU: (8) arm64 Apple M1 Pro
    Memory: 279.02 MB / 32.00 GB
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 18.15.0 - ~/.volta/tools/image/node/18.15.0/bin/node
    Yarn: 1.22.19 - ~/.volta/tools/image/yarn/1.22.19/bin/yarn
    npm: 9.5.0 - ~/.volta/tools/image/node/18.15.0/bin/npm
  Browsers:
    Chrome: 111.0.5563.64
    Safari: 16.1
  npmPackages:
    @builder.io/qwik: 0.20.1 => 0.20.1
    @builder.io/qwik-city: 0.5.2 => 0.5.2
    undici: 5.20.0 => 5.20.0
    vite: 4.1.4 => 4.1.4

Additional Information

No response

[bolt-new-by-stackblitz]

Start a new pull request in StackBlitz Codeflow.

• qwik-starter-joopk8 (Open reproduction)

[sujith97]

Solid start's Cloudflare adapter supports these options in vite configuration: link

+  plugins: [
+   solid({
+     adapter: cloudflare({
+       kvNamespaces: ["MY_KV"],
+       d1Databases: ["MY_DB"]
+     }),
+   }),
+  ],

We should have something similar to Qwik's Cloudflare pages adapter

[sujith97]

Workaround

1. Install following dev dependencies:
   npm i @miniflare/d1 @miniflare/shared

2. Create a separate file to fetch the DB instance:

let getDevDb: any = () => {};

if (import.meta.env.DEV) {
  const { D1Database, D1DatabaseAPI } = await import("@miniflare/d1");
  const { createSQLiteDB } = await import("@miniflare/shared");

  let devDb: any;

  getDevDb = async () => {
    if (!devDb) {
      const sqlLite = await createSQLiteDB(
        ".wrangler/state/d1/cloudflared1db.sqlite3"
      );
      devDb = new D1Database(new D1DatabaseAPI(sqlLite));
    }
    return devDb;
  };
}

const getDb = async (context: any) => {
  if (context.env.get("cloudflared1db")) {
    return context.env.get("cloudflared1db");
  }

  return getDevDb();
};

export default getDb;

3. Ignore these @miniflare deps in build

/adapters/cloudflare-pages/vite.config.ts

rollupOptions: {
      input: ['src/entry.cloudflare-pages.tsx', '@qwik-city-plan'],
+     external: ['@miniflare/d1', '@miniflare/shared']
},

That's it. Now whereever you need db instance just fetch it using await getDb(). Local dev server will use miniflare (local sqllite db) whereas the production builds use the actual cloudflare instances.

export const useListLoader = routeLoader$(async (context: any) => {
  const db = await getDb(context);

  if (db) {
    const ps = db.prepare("select * from users");
    const firstUser = (await ps.all())?.results?.[0];
    return {
      list,
      user: { // DB objects have some issue if we directly send them so create a new object
        ...firstUser,
      },
    };
  }
});

[mattp0123]

I have the same problem with KV. here's the workaround modified from @sujith97's:

// src/lib/kv.ts
import type { KVNamespace } from '@miniflare/kv';

let getDevKVNamespace: () => KVNamespace;

if (import.meta.env.DEV) {
  const { KVNamespace } = await import('@miniflare/kv');
  const { MemoryStorage } = await import('@miniflare/storage-memory');

  let kvNamespace: KVNamespace;

  getDevKVNamespace = () => {
    if (!kvNamespace) {
      kvNamespace = new KVNamespace(new MemoryStorage());
    }
    return kvNamespace;
  };
}

export const getKVNamespace = (
  platform: QwikCityPlatform,
  KVBindingKey: string
): KVNamespace => {
  if (platform.env) {
    return platform.env[KVBindingKey] as KVNamespace;
  }
  return getDevKVNamespace();
};

// src/routes/index.tsx
export const useData = routeLoader$(async ({ platform }) => {
  const MY_KV = getKVNamespace(platform, 'MY_KV');
  return await MY_KV.list();
});

[lyonsbp]

Has anyone been able to get this working with R2 yet as well or only KV?

[austin-raey]

I'd like to note that from the workaround comment above for D1, https://github.com/BuilderIO/qwik/issues/3345#issuecomment-1475385715

For whatever reason, my builds began to fail recently which had something to do with the miniflare package,

Putting this in the main vite.config.ts solved the build issue. :)

	build: {
		rollupOptions: {
			external: ['@miniflare/d1', '@miniflare/shared'],
		},
	},

As far as I can tell miniflare code is not being bundled, bundle sizes do not change even if I just comment out all the code between if (import.meta.env.DEV) stuff. So I'm not really entirely sure what is going on 🤔

For instance, if I were to completely remove the build.rollupOptions.external configuration in both places, and replace if (import.meta.env.DEV) { with if (false) { ..., vite seems to know not to bundle the miniflare code and there is no error. With import.meta.env.DEV there is one.

It seems I just have a gap in my knowledge for this lmao

[serbyxyz]

I also can not get this to work, the main issue I believe is the qwik-auth package, I have installed from npm run qwik add auth

this is the original way I had it set up :

export const { onRequest, useAuthSession, useAuthSignin, useAuthSignout } =
  serverAuth$(({ env, platform }) => ({
    secret: env.get("PRIVATE_AUTH_SECRET"),
    adapter: DrizzleAdapter(platform.env?.DB),

Is there a way to set default "Env" or "PLATFORM" to be of the type needed by miniflare or any other package ? instead of needing to call
const db = await getDb(platform.env.YOUR_DB)

So that when the Vite npm run dev starts up it check for the import.meta.DEV so it loads with the PLATFORM necessary ?

[mhevery]

Qwik-auth is very simple: https://github.com/BuilderIO/qwik/blob/main/packages/qwik-auth/src/index.ts But I am not an export on it. What needs to be changed there? Can you propose a PR?

[austin-raey]

@mhevery If I ever find the time I wouldn't mind looking into such a thing, although it does seem a little complex.

If you look at the source code for what solid-start does for Cloudflare Pages, it seems they apply some deeper integration with Miniflare.

This is what allows for the platform variable to have the appropriate Cloudflare bindings like D1, R2, KV etc in the dev environment i.e. npm run dev, like what @serbyxyz is asking.

Such a fix is not necessarily specific to qwik-auth, rather, it would likely need to be applied somehow with Qwik's Cloudflare adapter. (@builder.io/qwik-city/adapters/cloudflare-pages/vite I believe)

Currently it seems Qwik just wants you to npm run build then wrangler pages dev ./dist, but this is not ideal for rapid development.

As an aside, it may be possible to run wrangler pages dev and vite dev at the same time and get access to platform, but I haven't really looked to much into this. Integrating miniflare to the Cloudflare Pages adapter seems like the most elegant solution.

[serbyxyz]

yes how @austin-meadows explained it. it has nothing to do with qwik-auth just realized it when I had to modify qwik-auth to change the drizzle adapter from drizzle D1 to cloudflare D1. Which would work by just adding {env, platform} to the plugin. Instead of using the getDB() inside of the plugin

[serbyxyz]

okay so if you add this to package.json or just run it in terminal :
@austin-meadows
npx wrangler pages dev --proxy 5173 --compatibility-date=2023-10-06 -- npm run dev
( if you don't add the --proxy it runs but the returned page is just blank..)

also just running the npm run build and then the npm run serve would work but you need to add the wrangler.toml,
it doesn't "dev mode" --reload though

assuming you have wrangler v3+ it will do all the miniflare "V3" / workerd ... stuff for you and allows you to do the platform.env anywhere. just need to add a wrangler.toml with your bindings etc.

The docs are very mixed between versions like the run flags --local etc but there is a list of deprecated ones
https://developers.cloudflare.com/workers/wrangler/deprecations/

... now to figure out what all this proxy errors are with it (the authjs adapter d1-adapter not drizzle) also complains about some file
that isnt getting bundled in the ... -- npm run dev.

I tried in the plugin@auth.ts doing this instead of using the getDb() in there since its much less modifications

serverAuth$(({ env, platform }) => ({
    secret: env.get("PRIVATE_AUTH_SECRET"),
    adapter: DrizzleAdapter(drizzle(platform.env?.DB, { schema: sqliteDB })),

it doesn't give the bundle error just the proxy error which it seems like might be something with the provider callback ( which I've tried all the ip addresses 127.0.0.1:8788 :5173... localhost: ...)

but that's as far as I got so far

[serbyxyz]

I actually found something interesting when messing about with Hono and Cloudflare workers ( very similar to FastAPI which I know well) … but hono does something that is typescript specific with global variables. That maybe could be added to the cloudflare or any other adapter. The platform works with the wrangler v3 + with the adapter. @mhevery , But I’m not sure if the adapter is looking for wrangler.toml file , in the case of cloudflare , not sure what vercel or others have . But if the adapter reads the wrangler.toml file and in some way “loads” or caches the bindings into the env, ( similar to how in tsconfig you can add the types to cloudflare and it’s “globally” available). If the adapter reads the wrangler.toml and makes the bindings available in that “global” way then I think it might solve a lot of the “type”ing problems.

for example it seems it’s pretty standard to make a src/db directory similar to the qwik insights (using libSQL/Turso based on env vars). I noticed it has this getDb() function/ method …

So with most using drizzle for example purpose ,
You can do the same IE

Export const db=drizzle(client,{schema:myschema})

But with cloudflare D1 client = platform.env.MyBinding

but platform can’t be used or “type”d in a seperate directory it always needs to be called inside an event / fetch handler style function by using the platform that’s provided in the method / function.

So in every routeLoader or routAction the db needs to be initiated for d1 , vs importing db from src/db …

I could totally be missing something here that might be common knowledge that I’m missing out on, that lead me down this path.( I have no problems with any other way of doing this except on cloudflare d1 … yes it’s beta so there is not much docs examples to go off).

I did a test run of installing qwik through the cloudflare C3 method and it does add extra functions to the package.json but I think the main thing that’s missing is the loading or parsing of the wrangler.toml file which could add the bindings as global vars or something like name space? In typescript ? I’m not very good at typescript so I can be off on that.

[mhevery]

A good investigation, but I have to be honest with you it's ower my head. Any chance someone could propose a PR?

[serbyxyz]

I think if your going to investigate , from what I have found out, looking at the qwik insights repo, make a branch off that. Last time I looked at it you have the getDB for turso / libSQL already in place and you already have the Qwik-auth plug-in installed. But I’m not sure if I’m production mode you have an adapter connected to it or not?

Here is some findings I tried and maybe you can reproduce results or get better “insight” into the subject ( for anyone that is stuck on this ).

Before even starting install qwik cloudflare plug-in via the builder.io - qwik adapters method and then install it via the cloudflare -C3 method , then compare the difference between the package.json “scripts”, first you will notes a “pages:dev” that is similar to the qwik - cloudflare adapters run dev that points to the .entry files. Second you will notice that the C3 create method adds a wrangler.toml file which is needed for the newer wrangler v3 which automatically has mini flare when in NODE_ENV= (!’not’ “production”)
So you can do .local or .dev whatever with your env files ( it gets confusing in the wrangler.toml) when you set( see below ) [env.production] [env.dev] … or [vars] [vars.production] still confused about dev.vars file instead of .env file I think it’s the same or Vite will load the env for you but not the dev.vars .
But one thing you will find in cloudflare docs is that your [d1database.bindings] needs a preview_id=“can be same as database_id”

[env.production]
NODE_ENV= “production”
[env.dev]
NODE_ENV= “development”

^^^^
But for example Vite has automatic like search or parse for .env .env.local so I tried add
[env.local]
NODE_ENV= “development”
[env.local.d1database]
database_name= “database”
database_id= “your db id”
database_binding = “ your DB binding
preview_id= “ I think you can make it whatever you want but I copied the database_id here”

but in env.local file I also added NODE_ENV=development

According to the wrangler v3 docs you don’t need to use —local anymore when running pages:dev , I had issue with qwik page loading using both install methods without —proxy “(Vite port)5173” then it would load the page
And at the end of the pages:dev command adding — npm run dev ( I think I posted link / command in previous post)

but with the wrangler.toml file and running npm run pages:dev you will notice a new .sqlite file under the .wrangler/cache( or something)/{the preview_id you gave it} in the root of your project folder

So I think the main thing is the adapter install or add command just needs to be modified by adding / creating a wrangler.toml file to it similar to how C3 does it (that’s why I mention installing both for investigation) to compare what’s different ( I think it’s just what’s different in the updated wrangler v3)

But for me the database did not get migrated even by running the migrate commands for d1 with the —local flag in wrangler d1 migrate —local ( I think it’s like that) it says it migrates but in sqlite viewer extension still says 0 tables but if you run migrate again it says nothing to migrate …

so what I ended up doing was just using drizzle to push:sqlite -> that .wrangler/cache… .sqlite file using better-sqlite adapter and ended up just making the getDB() function when NODE_ENV ===“development” use drizzle( better-sqlite3) and when production use drizzle( d1 )

but you can’t just add better-sqlite3 with top-level import because when you push to GitHub ( and cloudflare runs build) it will complain about better-sqlite ( even if you have node_compat= true in cloudflare settings )… the drizzle adapter is fine it’s just the import of Database from /better-sqlite

so a work around was to make second function that you do like const Database = await import (“better-sqlite”).default but since Database is a class and it expects it to be async it needs to be awaited that’s why it needs to be in its own function and then called from inside of getDB ( when I get to pc I will paste example) I’m not very good with typescript so maybe it can be refined , it’s mainly that it will complain inside of the auth-js adapter that type of Database is not compatable but by sticking it in its own function then calling it in getDB it goes away ( this still doesn’t solve the fact that d1 is not running local on mini flare mainly due to the fact the local db doesn’t want to migrate the tables, so it does work without better-sqlite3) but the tables are not there so it throws error that table can not be found … so only way I found was using better-sqlite, ( which is similar to how qwik-insights uses libSQL for isLocal and turso for production)

** another thing to note is that the drizzle d1 adapter already has mini flare added to it …