Add tokenIndex to SymbolNode by matthew-canestraro · Pull Request #2796 · josdejong/mathjs

matthew-canestraro · 2022-09-28T17:02:38Z

This is a small POC to go with #2795

I kept this minimal, only adding tokenIndex to SymbolNodes specifically, but I could see the value in extending this to most/all nodes

Will mark as draft until the discussion resolves

josdejong

This is shaping up really nicely, and it looks well tested!

I made a few inline comments, can you have a look at those? Besides that, we should write a section in the documentation explaining source.

src/expression/parse.js

josdejong · 2023-03-17T15:29:02Z

test/unit-tests/expression/parse.test.js

+      // should have a source for brackets and item delimiters
+      const expected = [
+        { index: 0, text: '[' },
+        { index: 2, text: ',' },


If we parser a large matrix, I can imagine that listing all comma's impacts the performance. I think we should double check if this is an issue or not.

Can do! Are you worried about performance during parsing or evaluation or somewhere else?

Yes I was thinking about performance of parsing, there is a lot of extra objects created then. I think it cannot impact evaluation performance since the sources are not used during evaluation.

We can set up a small benchmark in the /test/benchmark folder to test if there is a performance impact or not, if you want I can do that.

If you're willing to do that, I'm glad for the help, thank you! Otherwise, I'll tackle it after finishing everything else

I'll do that 👍

I've run the benchmark test/benchmark/expression_parser.js and compared the develop branch with this PR (you can try it out yourself to see the differences).

The evaluate benchmark is just as fast in both cases, that is most important: about 1.7 million ops/sec

The parse benchmark is slower, goes from 16000 ops/second to 11000 ops/sec. I think that is acceptable, and simply a consequence of the parse step having to do more work.

Shall we leave it at that?

I'm getting the following results after rebasing the changes on the latest develop branch.
No changes:

expression: 2 + 3 * sin(pi / 4) - 4x scope: Map(1) { 'x' => 2 } result: -3.878679656440358 (plain js) evaluate 0.04 µs ±0.00% (mathjs) evaluate 0.27 µs ±31.61% (mathjs) parse, compile, evaluate 4.98 µs ±7.58% (mathjs) parse, compile 4.95 µs ±10.21% (mathjs) parse 3.68 µs ±8.69%

With changes from the task branch:

expression: 2 + 3 * sin(pi / 4) - 4x scope: Map(1) { 'x' => 2 } result: -3.878679656440358 (plain js) evaluate 0.04 µs ±0.00% (mathjs) evaluate 0.25 µs ±0.23% (mathjs) parse, compile, evaluate 6.15 µs ±14.67% (mathjs) parse, compile 6.05 µs ±13.45% (mathjs) parse 5.17 µs ±29.06%

@josdejong is this acceptable?

Yes I think that's acceptable. Thanks for testing this 👍

test/unit-tests/expression/parse.test.js

josdejong

Thanks for the updates @matthew-canestraro! I made a few comments, mostly about thinking through the exact API for meta regarding optional arguments/properties.

josdejong · 2023-05-02T08:05:00Z

docs/expressions/expression_trees.md

 All nodes have the following methods:

-   `clone() : Node`
+-   `clone(options: MetaOptions) : Node`


The options are optional, can you describe that here and for all constructors in the docs?

josdejong · 2023-05-02T08:09:05Z

test/unit-tests/function/algebra/resolve.test.js

+
+function emptySourcesFromTree (tree) {
+  tree.traverse((node) => {
+    node.sources = []


How about using the immutable methods .clone({ sources: [] }) and .transform() in these helper functions to try out whether the immutable API is workable? (eat your own dog foo 😉 )

Something like this @josdejong ?

function emptySourcesFromTree (tree) { return tree.clone().transform(function (node) { node.sources = [] return node }) }

Not exactly, I think I meant something like:

function emptySourcesFromTree (tree) { return tree.transform(node) => node.clone({ sources: [] })) }

josdejong · 2023-05-02T08:19:43Z

src/expression/node/AccessorNode.js

     */
-    clone () {
-      return new AccessorNode(this.object, this.index)
+    clone (meta = {}) {


The default value for meta, an empty object {}, does not correspond with the official TypeScript definition where sources is a require property.

I think it is best to have sources are required property. I would also like to get rid of mutating the input meta via meta.sources = ..., so I think we can rewrite the clone methods to something like:

/** * Create a clone of this node, a shallow copy * @param {MetaOptions} [meta] object with additional options for cloning this node * @return {AccessorNode} */ clone (meta) { const cloned = new AccessorNode(this.object, this.index, meta ?? { sources: this.sources }) return cloned }

Having to construct a default meta object via { sources: this.sources } is a bit odd. Would it make sense to just keep the meta object as it is instead of destructuring it in the Node constructor, and having to re-construct it in the methods clone before we can pass it to another constructor? I.e we could think about:

const defaultMetaOptions = { sources = [] } class Node { /** * @constructor Node * A generic node, the parent of other AST nodes * @param {MetaOptions} [meta] object with additional options for building this node */ constructor (meta = defaultMetaOptions) { this.meta = meta } // ... }

josdejong · 2023-05-02T08:22:10Z

test/unit-tests/expression/parse.test.js

+      // should have a source for brackets and item delimiters
+      const expected = [
+        { index: 0, text: '[' },
+        { index: 2, text: ',' },


I'll do that 👍

josdejong · 2023-05-02T08:34:16Z

src/expression/node/AccessorNode.js

     * @param {Node} object                 The object from which to retrieve
     *                                      a property or subset.
     * @param {IndexNode} index             IndexNode containing ranges
+     * @param {MetaOptions} object with additional options for building this node


Just for clarity: to make this valid JSDoc, the lines like:

@param {MetaOptions} object with additional options for building this node

Should be changed to:

@param {MetaOptions} meta object with additional options for building this node

And if meta is optional:

@param {MetaOptions} [meta] object with additional options for building this node

josdejong · 2023-09-13T15:32:27Z

@matthew-canestraro we never finished this PR. Are you still interested in getting it done?

matthew-canestraro · 2023-09-18T15:24:33Z

@matthew-canestraro we never finished this PR. Are you still interested in getting it done?

Hi @josdejong, sorry for dropping the ball on this, priorities shifted a lot over the past few months. That being said, I still believe this will be a good contribution to MathJS and it would be a shame to lose the work done. I'll do a final pass on the PR this week and ping you when I think it's ready

josdejong · 2023-09-20T06:45:28Z

Thanks @matthew-canestraro . Also if you can't find time for it just let me know, maybe someone else can finish your work then.

Prior to this commit it was not possible to trace back the parsed node location in the original string. This information could be used to highlight the syntax of a math expression displayed in a rich editor (e.g. a spreadsheet or a calculator). This commit is based on the PR josdejong#2796 that has been abandoned for 2 years. I rebased the task branch and fixed the issues reported in the original PR. In the nutshell, each parsed node stores an array of sources (`SourceMapping[]`) that are set during parsing (see `tokenSource` and its usages for details). The node's constructor and clone method are adjusted to take an optional `MetaOptions` object containing the source mappings. In the future `MetaOptions` could be extended to store more information. Benchmarks showed no change in `evaluate`. `parse` became slower, from 3.68µs to 5.17µs with the changes from this commit. Closes josdejong#2795

serso · 2025-10-17T13:47:50Z

I have opened a new PR with the changes based on this PR, see #3557
I think this PR could be closed. I've rebased the changes on the latest develop branch and adjusted the code with the suggestions from this PR.

josdejong · 2025-10-22T14:06:28Z

Awesome @serso, that's good news!

I'll close this PR now in favor of #3557

matthew-canestraro force-pushed the develop branch from 9c24329 to edd789f Compare February 7, 2023 21:47

Add source mappings for all nodes on parse()

742e662

matthew-canestraro force-pushed the develop branch from ad50348 to 742e662 Compare March 15, 2023 19:28

Merge branch 'develop' of github.com:josdejong/mathjs into develop

32793d7

matthew-canestraro marked this pull request as ready for review March 15, 2023 19:53

josdejong requested changes Mar 17, 2023

View reviewed changes

matthew-canestraro added 5 commits April 4, 2023 12:36

Merge branch 'develop' of github.com:josdejong/mathjs into develop

b722237

Use constructors instead of setSources()

173b20a

Add type for MetaOptions

e3706b8

Clean lint

fea1cbe

Update docs

c169f7d

josdejong reviewed May 2, 2023

View reviewed changes

josdejong mentioned this pull request Sep 2, 2025

Add range information to SymbolNode and ConstantNode #3522

Closed

serso mentioned this pull request Oct 17, 2025

Add information about node's original location in the parsed string #3557

Open

josdejong closed this Oct 22, 2025

Uh oh!

Conversation

matthew-canestraro commented Sep 28, 2022

Uh oh!

josdejong left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

josdejong left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

josdejong commented Sep 13, 2023

Uh oh!

matthew-canestraro commented Sep 18, 2023

Uh oh!

josdejong commented Sep 20, 2023

Uh oh!

serso commented Oct 17, 2025

Uh oh!

josdejong commented Oct 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants