v2: Allow custom extensions to be passed into the Webpack file loader

[mattisdada]

🚀 Feature

The ability to provide your own list of file extensions to be loaded via file loader or a specific directory that will always be loaded in via the file loader.

Have you read the Contributing Guidelines on issues?

Yes

Motivation

I currently have a YAML file that needs to be downloaded via the static file server, currently the only way to get download paths are by import or require, but the file loader is limited to pdf|doc|docx|xls|xlsx|zip|rar (reference) currently with no way of adding to that list. This will cause the Webpack loader to panic with a You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. error.

Handling files that are not in this very limited list are impossible to use with Docusaurus for generating download links, this limits the functionality greatly.

Example

import InsomniaUrl from "@site/static/downloads/insomnia.yml";
// You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file.

Pitch

There should be a configuration option in docusaurus.config.js to allow setting a custom list (or appending to the existing list) of allowed file types to be used by the file loader. Alternatively a path (ie static/downloads) where all files get handled by the file loader, this path should be configurable.

[mattisdada]

My workaround for this issue was to do the following:

import useBaseUrl from "@docusaurus/useBaseUrl";

export const InsomniaButton = () => {
  const fileUrl = "/downloads/insomnia.yml";
  const url = `${location.protocol}//${location.host}${useBaseUrl(fileUrl)}`;
  const encodedUrl = encodeURIComponent(url);
// ...

There might be some other Docusaurus hook that allows generating the full url without relying on location, but I wasn't able to find it...

But this approach is just string juggling which means we don't get the safety of using the file loader and requires using BrowserOnly as a workaround

[slorber]

Handling files that are not in this very limited list are impossible to use with Docusaurus for generating download links, this limits the functionality greatly.

That's not entirely true, you can add whatever you want to the webpack config with configurreWebpack, including file-loader configs, so if we add this, it would only be a "shortcut" for something that is already possible today.

Something like this might work:

module.exports = {
  configureWebpack: function () {
    return {
      module: {
        rules: [
          {
            test: /\.yaml$/,
            use: [
              {
                loader: 'file-loader',
                options: {name: 'assets/files/[name]-[hash].[ext]'},
              },
            ],
          },
        ],
      },
    };
  },
};

Note we have multiple output folders for static assets currently. If we enabled to configure a list of extensions, do we want all these files to be in build/files?

[mattisdada]

@slorber Thanks for that, I looked around and I couldn't find the Webpack escape hatch in the docs 😅.

I think it's still worth while for something that is a bit more systematic for this, maybe I overestimate how often docs are generated with downloaded files?

Otherwise I'm perfectly happy for this issue to be just closed, your answer is a good solution.

[slorber]

Thanks for the feedback

I think it's still worth while for something that is a bit more systematic for this, maybe I overestimate how often docs are generated with downloaded files?

This is the first time anyone request that, so I'll just close for now, and see if this is requested again in the future

[khusamov]

@slorber

Hello! I tried to place configureWebpack in the docusaurus.config file.js
And got an error:

[ERROR] Error: These field(s) ("configureWebpack",) are not recognized in docusaurus.config.js.
If you still want these fields to be in your configuration, put them in the "customFields" field.

Where should I put the configureWebpack?

[Josh-Cena]

@khusamov configureWebpack is a plugin lifecycle. See:

• https://docusaurus.io/docs/advanced/plugins#function-definition
• https://docusaurus.io/docs/api/plugin-methods/lifecycle-apis#configureWebpack

If you want to modify docusaurus.config.js instead of creating a separate plugin file, you would write something like:

module.exports = {
  plugins: [
    () => ({
      configureWebpack() {
        return {...};
      },
    }),
  ],
};

[benjie]

I needed to import a file from the filesystem into the documentation to be included raw, to do so I added the following Docusaurus plugin to the list of plugins in docusaurus.config.js:

const webpackAssetSourcePlugin = () => ({
  configureWebpack() {
    return {
      module: {
        rules: [
          {
            resourceQuery: /raw/,
            type: "asset/source",
          },
        ],
      },
    };
  },
}),

The plugin uses Webpack 5's asset/source type for any imports with ?raw at the end. Then in my .mdx I could import the content by being sure to append ?raw:

import Mermaid from '@theme/Mermaid';
import chart from '../path/to/file.mermaid?raw';

# Blah

Blah blah blah

<Mermaid chart={chart}/>

Thanks for all the guidance above ❤️

[slorber]

great to know that you made this work

I tried to migrate Docusaurus from file-loader to the assets system but there were some annoying edge cases that prevented from completing this migration (#4708)

also discussed here: #8106

[erzz]

@slorber (and anyone else interested!) - I would love something to be easier to configure here. Frankly as a relative n00b coming to webpack5 I barely understand what you guys say in terms of solutions here :)

I come from Hugo where the idea of a data/ directory is just automatically accessible with a simple inline command like {{ $data := (index somefile.yml) }}

Literally all I want to do is load content from .yml files in the site (its actually a collection of gihub workflows) and render various parts.

Even a more laborious method (yaml-js + fs) isn't working because also webpack doesnt seem to like the fs package

There are many I am sure that have a local datasource that is a bunch of YAML or JSON files and to be honest its a whole world of pain for noobs like me to get something working

The only little success I have had is with something like:

import wf from "yaml-loader!@site/.github/workflows/artillery.yml";

export default function JobList(props) {
  return (
    <div>
      <h3>Workflow</h3>
      <pre>
        {JSON.stringify(wf.jobs, null, 2)}
      </pre>
    </div>
  );
}

But that is difficult to work with in a component way as I need to pass it in from props

But to return to the issue - surely command data exchange formats such as YAML should be in the OOTB list?

[slorber]

@erzz there's not a single way to import a yaml file:

• file-loader (what other users wanted): the imported yaml becomes a public URL string
• yaml-loader (what you want?): the imported yaml becomes a json object

We can't satisfy both users at the same time, and in any case you can configure this yourself with the configureWebpack hook

I'm open to supporting Yaml OOTB better. But if we supported Yaml out of the box, but not in the way you though we should (because there's not a single use-case), it could even make it harder for you to revert our OOTB yaml support and add your own support, and vice versa. I hope you understand my concerns about doing so and that enabling you to bring your own support is the safest option for us, even if it's less convenient for a few users.