Completions may be incorrect or missing and throw ClassCastExceptions in v4.2.1.21

[willplatt]

Consider this very simple grammer:

namespace org.example.brackets.agl

grammar Brackets {
	sentence = '[' 'x' ']';
	
	skip WHITESPACE = "\s+";
}

And this Java program that uses it to get completions on some example strings:

import net.akehurst.language.agl.Agl;
import net.akehurst.language.agl.simple.ContextWithScope;
import net.akehurst.language.api.processor.CompletionItem;
import net.akehurst.language.api.processor.LanguageProcessor;
import net.akehurst.language.asm.api.Asm;

import java.io.InputStream;
import java.util.List;
import java.util.Scanner;

import static java.nio.charset.StandardCharsets.UTF_8;

public class BracketsSuggestions {
	private static final String GRAMMAR_STRING = readResource(BracketsSuggestions.class.getResourceAsStream("Brackets.agl"));
	private static final LanguageProcessor<Asm, ContextWithScope<Object, Object>> LANGUAGE_PROCESSOR = Agl.INSTANCE.processorFromStringSimpleJava(
			GRAMMAR_STRING, null, null, null, null, null,
			Agl.INSTANCE.configurationSimple(), null).getProcessor();
	
	public static void main(String[] args) {
		System.out.print("Initial suggestions:          ");
		printSuggestions("", 0);
		System.out.print("Suggestions after '[':        ");
		printSuggestions("[", 1);
		System.out.print("Suggestions after '[ ':       ");
		printSuggestions("[ ", 2);
		System.out.print("Suggestions after '[x':       ");
		printSuggestions("[x", 2);
		System.out.print("Suggestions within '[ ':      ");
		printSuggestions("[ ", 1);
		System.out.print("Suggestions within '[]':      ");
		printSuggestions("[]", 1);
		System.out.print("Suggestions within '[  ]':    ");
		printSuggestions("[  ]", 3);
		System.out.print("Suggestions after '[x]':      ");
		printSuggestions("[x]", 3);
		System.out.print("Suggestions after '[x] ':     ");
		printSuggestions("[x] ", 4);
	}
	
	private static String readResource(InputStream resourceStream) {
		return new Scanner(resourceStream, UTF_8).useDelimiter("\\A").next();
	}
	
	private static void printSuggestions(String str, int position) {
		System.out.println(String.join(", ", getSuggestions(str, position)));
	}
	
	private static List<String> getSuggestions(String str, int position) {
		try {
			return LANGUAGE_PROCESSOR.expectedItemsAt(str, position, LANGUAGE_PROCESSOR.optionsDefault()).getItems().stream().map(CompletionItem::getText)
					.toList();
		} catch (ClassCastException e) {
			e.printStackTrace();
			return List.of();
		}
	}
}

Hopefully this is an appropriate way to initialise the LanguageProcessor; I don't know if I should be providing more parameters or using some other configuration.

This is the kind of output I would expect ("[x]" would be acceptable in place of "[", as would "x]" in place of "x"; indeed, that may even be preferable):

Initial suggestions:          [
Suggestions after '[':        x
Suggestions after '[ ':       x
Suggestions after '[x':       ]
Suggestions within '[ ':      x
Suggestions within '[]':      x
Suggestions within '[  ]':    x
Suggestions after '[x]':      
Suggestions after '[x] ':     

But here is the actual output:

Initial suggestions:          [ x ], [
Suggestions after '[':        [ x ], [
Suggestions after '[ ':       x
Suggestions after '[x':       
Suggestions within '[ ':      [ x ], [
Suggestions within '[]':      [ x ], [
Suggestions within '[  ]':    x
Suggestions after '[x]':      
Suggestions after '[x] ':     

Note the clearly incorrect suggestion of "[" after an existing "[" and the missing suggestions of "]" and "x". The suggestion of "[ x ]" after a "[" also appears incorrect, although it could be interpreted as suggesting the existing string be replaced by "[ x ]".

The problem may be related to whitespace; whitespace appears to be expected between each literal, although the grammar doesn't require it (parsing works as expected). But removing the line from the grammar that makes whitespace skippable results in almost identical suggestions (still incorrect and lacking).

ClassCastException

Finally, note that I have to catch ClassCastException in the program above, because with another grammar like below it throws this exception (for every attempt to get completions except the last one):

java.lang.ClassCastException: class net.akehurst.language.grammar.asm.SimpleListDefault cannot be cast to class net.akehurst.language.grammar.api.TangibleItem (net.akehurst.language.grammar.asm.SimpleListDefault and net.akehurst.language.grammar.api.TangibleItem are in unnamed module of loader 'app')
	at net.akehurst.language.agl.completionProvider.SpineDefault.expectedNextLeafNonTerminalOrTerminal_delegate$lambda$1(CompletionProviderAbstract.kt:69)
	at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:83)
	at net.akehurst.language.agl.completionProvider.SpineDefault.getExpectedNextLeafNonTerminalOrTerminal(CompletionProviderAbstract.kt:64)
	at net.akehurst.language.agl.simple.CompletionProviderSimple.provide(CompletionProviderSimple.kt:66)
	at net.akehurst.language.agl.processor.LanguageProcessorAbstract.expectedItemsAt(LanguageProcessorAbstract.kt:296)
	at BracketsSuggestions.getSuggestions(BracketsSuggestions.java:50)
	at BracketsSuggestions.printSuggestions(BracketsSuggestions.java:45)
	at BracketsSuggestions.main(BracketsSuggestions.java:21)

Grammar causing the exception:

namespace org.example.bracketscomplex.agl

grammar BracketsComplex {
	sentence = word | bracketed;
	bracketed = '[' word* ']';
	word = NAME;
	
	leaf NAME = NAME_CHAR+;
	leaf NAME_CHAR = "[-.0-9]";
	
	skip WHITESPACE = "\s+";
}

[dhakehurst]

Thank you for the extensive feedback, and clear description of the issue.

There seem to be two issues,

1. The result of 'expectedItemsAt' is not what you are expecting.
2. There is a ClassCastException

I will first address (1)
consider a simple variation of your first grammar.

namespace org.example.brackets.agl

grammar Brackets {
    sentence = '[' 'uvwxyz' ']';
    
    skip WHITESPACE = "\s+";
}

what would you expect to see as CompletionItems at the end of a sentence '[ uv' ?

The function is designed to give you the completion of the terminal that is partially complete.
it provides 'uvwxyz' as a completion item.

Now due to the complexities of the algorithm, at the end of the terminal, i.e. end of sentence '[ uvwxyz'
it does not know if the string uvwxyz is a finished terminal or part of something else
(perhaps I could improve this, with some additional investigation - but it is non trivial)

thus the completion items you are given, also include the terminal immediately to the left.
If there is a skip (whitespace) terminal added, then the algorithm knows it is finished.

Thus,

• '[ uv' will give the completion 'uvwxyz'
• '[ uvwxyz' will also include the completion 'uvwxyz'
• '[ uvwxyz ' will not include 'uvwxyz' due to the whitespace.

You can remove the unexpected-just-completd-terminal by checking backwards in the sentence until
a skip terminal is found. At this point you can check if any offered completion is matched from this point forward
and remove it.

My DSL Editor will do this (https://github.com/dhakehurst/net.akehurst.language.editor)

I will look into whether it is possible to do this in general and include it by default, however currently, in order to be
able to offer the completion for 'partial' terminals, it is included.

[dhakehurst]

Though, having just said all that and written some more tests around it (i.e. trying out your second grammar)
there is some inconsistency between literals and patterns as terminals.
I will do my best to find time to fix the inconsistency.

[dhakehurst]

Regarding (2) the ClassCastException, I do not get this in the latest code base.
Perhaps it is something I already fixed.
which version of AGL are you using?

[willplatt]

Hi David, thanks for looking into this.

I see my expectations were a bit beyond what you had intended for the suggestions to do yet then. The grammar I am actually using in my software is along the lines of the second grammar I included, not having many fixed keywords (like 'uvwxyz' in your example), so the completion of partially typed literals is unfortunately not that useful to me. But even suggestions for those keywords don't seem to be working for me for whatever reason.

What I could really use are suggestions of what rules the next character may fall under (e.g. word, NAME_CHAR, ']'), and I can translate those into suggestions for the user (e.g. words that have special meanings or that match what the user has typed so far, or simply ']'). Ideally, each suggestion from AGL would also tell me whether it's a continuation of the last character's rule or not (i.e. a different rule or new application of the same rule), but usually I would be able to work this out.

I was getting the ClassCastException with the latest release, v.4.2.1.21.

I'm still getting some use out of the suggestions in my software by accounting for their quirks, although the range and specificity of the suggestions I'm able to offer could be better and occasionally they can be incorrect.

[dhakehurst]

ok,
I'm working on a new release (before Xmas with luck)
the Exception is no longer an issue, I tested your example.
I may be able to add a flag that indicates if it is a 'terminal-completion', good idea.

The rule indication may also be possible, I do something like that already.

[willplatt]

Sounds great, thanks!

[dhakehurst]

I'll aim for a result along these lines

interface ExpectedAtResult {
    /**
     * The position in the sentence that the completions were requested for.
     */
    val requestedPosition: Int
    
    /**
     * The position in the sentence that the completions are being offered for.
     */
    val offeredPosition: Int
    
    /**
     * true if the offered completions are for completing a terminal.
     */
    val isTerminalCompletion: Boolean
    
    /**
     * The number of characters from the requested position to the offered position.
     */
    val offset: Int

    /**
     * the completion items being offered
     */
    val items: List<CompletionItem>
    
    /**
     * Any issues found whilst trying to provide completions.
     */
    val issues: IssueCollection<LanguageIssue>
}

[dhakehurst]

What I could really use are suggestions of what rules the next character may fall under

An indication of this is given with the 'label' text.
However, I agree more could be usful.
I will try adding a 'detail' property to a CompletionItem that can contain the 'RuleItem' the suggestion is offered for.
Will sometimes be null, and may change in future releases.
From a RuleItem you should be able to navigate to its owningRule.

[dhakehurst]

There are also some ParseOption that might help you get what you want

    /**
     * A regular expression pattern used for identifying the start of a word
     * while looking for the nextExpected Tokens
     * (in the reverse direction, i.e. a sentence position going backwards towards the start of the sentence).
     *
     * This allows for handling of partial-words. The 'next-expected' could be partially written/covered by a position.
     *
     * This property is optional and can be set to null, in which case the parser
     * may resort to a default behavior or other mechanisms for determining word boundaries.
     *
     * The Regex should identify the first character (i.e. not part of a word) after which the search for the next expected tokens should start.
     *
     * Default value: `null`.
     */
    var reverseFindWordStartRegex: String?

    /**
     * Indicates whether the parser should use the skip rules of the grammar to determine a word's start position
     * while looking for the nextExpected Tokens
     * (in the reverse direction, i.e. a sentence position going backwards towards the start of the sentence).
     *
     * Default value: `true`.
     */
    var reverseFindWordStartBySkipRules: Boolean

[willplatt]

If you're able to provide what rule is being used for the CompletionItem and whether it's a rule already begun then I think that would be both more general and specific than isTerminalCompletion.

[dhakehurst]

provide what rule is being used for the CompletionItem
that would be the 'detail' property to a CompletionItem I mention above

[willplatt]

Okay, and I'd be able to tell if the suggestion is for finishing a rule already started because the offeredPosition would be before the requestedPosition?

[dhakehurst]

    override val isTerminalCompletion: Boolean get() = requestedPosition != offeredPosition
    override val offset: Int get() = requestedPosition - offeredPosition

[willplatt]

So isTerminalCompletion will be true for the completion of any rule already started? I thought it would just be for simple strings and maybe regular expressions.

[dhakehurst]

Just for terminals that are partially complete. The offset should indicate where the terminal starts. Not for rules

…

On Mon, 10 Nov 2025 at 21:57, William Platt ***@***.***> wrote: *willplatt* left a comment (dhakehurst/net.akehurst.language#57) <#57 (comment)> So isTerminalCompletion will be true for the completion of any rule already started? I thought it would just be for simple strings and maybe regular expressions. — Reply to this email directly, view it on GitHub <#57 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABK3WZOL2M3AJK4FFD4H5ZD34D32DAVCNFSM6AAAAACC7V5ZSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKMJTHA3DGNZVGM> . You are receiving this because you commented.Message ID: <dhakehurst/net. ***@***.***>