Performance Issue: Slow Table Iteration

[thierry-f-78]

A significant performance degradation has been observed when iterating over large tables in Gopher-lua compared to the standard Lua implementation. The iteration time for a table with 50,000 entries is approximately 10,000 times slower in Gopher-lua.

Environment for first test

• Operating System: MacOs
• Gopher-lua version: v0.0.0-20240527182111-9ab1540f3f5f
• Go version: go version go1.22.3 darwin/arm64
• Standard Lua version (for comparison): Lua 5.4.7 Copyright (C) 1994-2024 Lua.org, PUC-Rio

Environment for last test

• Operating System: Linuc
• CPU: Intel(R) Atom(TM) CPU C2338 @ 1.74GHz
• Standard Lua version (for comparison): Lua 5.1.5 Copyright (C) 1994-2012 Lua.org, PUC-Rio

Steps to Reproduce

1. Create a Lua script (perf_test.lua) with the following content:

local function get_time()
    return os.clock()
end

local table = {}
local start_time = get_time()

for i = 100000000, 100049999 do
    local key = tostring(i) .. "_MySuperKey"
    table[key] = i
end

local insertion_time = get_time() - start_time
print(string.format("Insertion time: %.6f seconds", insertion_time))

local count = 0
start_time = get_time()

for _ in pairs(table) do
    count = count + 1
end

local iteration_time = get_time() - start_time
print(string.format("Iteration time: %.6f seconds", iteration_time))
print(string.format("Count: %d", count))

2. Run the script using both the standard Lua interpreter and Gopher-lua.

There is the go code to execute the script lua_executor.go:

package main

import "bytes"
import "fmt"
import "io/ioutil"
import "os"

import "github.com/Shopify/go-lua"

func main() {
	if len(os.Args) < 2 {
		fmt.Println("Usage: go run lua_executor.go <path_to_lua_script>")
		os.Exit(1)
	}

	scriptPath := os.Args[1]
	scriptContent, err := ioutil.ReadFile(scriptPath)
	if err != nil {
		fmt.Printf("Error reading script file: %v\n", err)
		os.Exit(1)
	}

	L := lua.NewState()
	lua.OpenLibraries(L)

	err = L.Load(bytes.NewReader(scriptContent), scriptPath, "t")
	if err != nil {
		fmt.Printf("Error executing Lua script: %v\n", err)
		os.Exit(1)
	}

	L.ProtectedCall(0, 0, 0)
}

Observed Results

Gopher-lua

Insertion time: 0.069506 seconds
Iteration time: 32.081409 seconds
Count: 50000

Standard Lua 5.4

Insertion time: 0.047063 seconds
Iteration time: 0.003339 seconds
Count: 50000

Standard Lua 5.1

Insertion time: 0.294900 seconds
Iteration time: 0.016081 seconds
Count: 50000

Expected Results

The iteration time for Gopher-lua should be comparable to that of standard Lua, allowing for some performance differences due to the Go implementation. However, a difference of this magnitude (32 seconds vs 0.003 seconds) suggests a significant issue.

Additional Information

• The insertion time is relatively comparable between the two implementations.
• The iteration time in Gopher-lua is approximately 9,608 times slower than in standard Lua for this test case.
• This performance issue could significantly impact applications that rely on frequent iterations over large tables.

Possible Causes

The extreme slowdown during iteration might be due to:

1. Inefficient implementation of the pairs() function in Gopher-lua.
2. Suboptimal internal representation of tables in Gopher-lua.
3. Possible memory management issues during iteration.

Impact

This performance issue severely limits the usability of lua Go stack for applications that require frequent iterations over large tables, which is a common operation in many Lua scripts.

We would greatly appreciate your investigation into this matter and any insights you could provide on potential optimizations or workarounds.