Create mixer to multiplex LSP requests to backing LSP servers

[robinp]

Context

The idea is to have a live and a static LSP server started on the user's workstation. The live responsible for actively edited code, and static for all code in the repository.

A specific usecase will have haskell-ide-engine (hie in short) for the live server. Kythe's language server will serve the static index.

Note: what is considered live can differ per user.

Multi-project support in hie.

Now hie doesn't have a good multi-project support. While changing between projects [1] might be possible, crossreferences between them certainly doesn't work. There's plan to improve that this summer [2], but in the mean time we assume a single-project hie.

Rationale

For hie, or any other live code server, to support multiple projects, those projects need to be loaded into memory. Switching between many large projects [3] is uncanny. Since most projects don't change and are just navigated, it makes sense to compile a static index upfront on them.

Here I choose Kythe due to familiarity and prospective features. Using LSIF or HieDb [2] are alternatives.

Considerations

Navigating into files of the static set

When editing a live project locally, project sources of the static set might not be available. But the text editor is supposed to be able to open those files.

For this, I previously developed a small utility kythefs, with which user can mount a remote Kythe server under a local read-only directory. The Kythe langserver can be configured to redirect static file locations into this directory (or actual local sources if available).

Determining which backend to use

LSP requests send source code positions.

If the source file belongs to a project in the live set, the live LSP server is asked to resolve the entity at the given position.

Otherwise the static server is asked. Note that the file itself might have changed since the static indexing, but Kythe's langserver supports diffpatching results on the unchanged portions of the file.

The modified / newly added portions of the static set will be invisible for referencing purposes. Usually this is not a big problem: from the perspective of a user focused on some projects, the non-live parts of the codebase either change rarely, or are not interesting.

Mapping the resolved entitiy between backends.

With some additional complexity, more accurate results can be achieved when finding backreferences. Up until now, if the user adds a new use-site to an entity (say function) defined in the static set, that reference won't show up when the static server is queried.

To fix this, first we need to map the static backend's entity into the live backend's entity somehow, and then execute a search for usages in the live backend.

Similarly, if the user wants to find static-set references to a function defined in the live set, we need to map the live entity to the static entity, and execute a search in the static backend.

Mapping through LSP info

A complication is that if we intend to communicate with the backends purely through LSP, we have to gather and map based on the information returned over LSP (like a entity name in the hover response).

The tradeoff is between fragility of mining entity from hover text, vs the additional complexity of some server-specific communication protocol.

Until proved insufficient, I prefer texty mapping.

Plan

• Create a pure LSP forwarder service, to have the plumbing running.
• Serve LSP requests (hover, go-to-def, backrefs) from appropriate backing server (based on which set the source file belongs to).
• Define how to map resolved entity between backends (GHC Name to Kythe uri)
• Implement cross-backend merging, for example refs from both live and static set.

[1]: A project might correspond to stack/cabal projects, or rules_haskell targets, among others. See https://github.com/mpickering/haskell-ide-engine/tree/hie-bios.

[2]: See discussion on haskell/haskell-ide-engine#1126 (comment).

[3]: No specific measurements were made, but is folk lore.