Cycle through suggestions

[snobu]

Is there a way to cycle through code completion suggestions? You know, similar to what GitHub Copilot can do with Alt (Option) + ] and Alt (Option) + [, and then Tab to accept it.

[gergap]

No, this is a copilot feature. Regular LLMs don't provide multiple solutions.
You can cancel the completion and try again. Based on the configured tempreature setting the response is more "creative" or more determinisitc (reproducible). I prefer the later, which is why I normally set temperature=0.

In theory we could generate multiple solutions as well with prompt engineering, but I have not tried id yet. What the plugin does is using FIM tokens to complete the current code fragment only. Nothing more.

[gergap]

Another idea could be to use multiple different LLMs simultanously, so we could cycle through the different solutions of multiple models. The question will be how many LLMs can you load into VRAM on consumer hardware, though. Completion models are smaller than the chat models normally, so this could work.

[snobu]

I think prompt engineering would be easier to implement while still maintaining sane requirements (running a single model instance).
We could have a key binding that asks the model for another code block than the one already suggested - "[code context] + provide another suggestion, that either improves on the current or uses a different approach: [current code suggestion block]".

We could use a list to store subsequent suggestions so we could mimic the GitHub Copilot behavior.

Don't get me wrong, i absolutely enjoy this project and am grateful it exists, but this seems like a feature that is necessary in a real development environment.

[mcelrath]

I think prompt engineering is the wrong way to go about this. Small LLMs are much worse at following instructions. Instead just provide an easy way to generate a new suggestion with temperature > 0 IMHO.

[gergap]

Instead just provide an easy way to generate a new suggestion with temperature > 0 IMHO.

I agree, and this should be easy to implement. Small code completions models are fast. We could really just trigger the completion again.

Another solution would be to always generate the next completion in the background, so that cycling through is faster, but this would increase the load on the system and the results are not used anyway in 99% of all cases.
IMO a code completion model should work (with GPU) <1s, then on demand generation for the next solution should work fast enough.

[mcelrath]

A configurable set of completions would be good. I've set up a dedicated compute ollama server that does nothing but generate this stuff for me. So I'd like it to generate and cache 5 suggestions or so. ;-)

[gergap]

In the past I set temperature=0 to get deterministic completions. Because we can now specify a seed value in the ollama REST API, wen can use higher temperature values and it is still deterministic. By increasing/decrementing seed values I can cycle through different solutions. This requires a temperature>0, though. At the moment I'm unsure about what the optimal temp would be. Maybe it's different for different models.

[snobu]

For some models temperature goes up to 2.0, whatever you would pick as default needs to be configurable, maybe through an additional .vimrc entry, like:

"vim-ollama                                                                    
let g:ollama_host = 'http://localhost:11434'                                   
let g:ollama_model = 'qwen2.5-coder:3b'                                        
let g:ollama_model_temp = 0.5

According to "The Internet",

Temperature 0: Best for factual responses and consistent outputs
Temperature 0.1-0.5: Good for structured generation
Temperature 0.7-1: Ideal for creative tasks
Temperature >1: Experimental use cases

[gergap]

It is already configurable through g:ollama_model_options:

        let g:ollama_model_options = {
                \ 'temperature': 0.5,
                \ 'top_p': 0.95,
                \ 'num_predict': 256
                \ }

You can change all the model's options this way.