What sparks will arise when Golang and Lua meet?

This article is introduced to you by the go language tutorial column. Golang and Lua, I hope it will be helpful to friends in need!

While playing around on GitHub, I accidentally discovered gopher-lua, which is a Lua virtual machine implemented in pure Golang. We know that Golang is a static language, while Lua is a dynamic language. Golang's performance and efficiency are very good among other languages, but in terms of dynamic capabilities, it is definitely not comparable to Lua. So if we can combine the two, we can combine their respective strengths (manual funny.

In the project Wiki, we can know that the execution efficiency and performance of gopher-lua are only better than those implemented in C Bindings are poor. Therefore, from a performance perspective, this should be a very good virtual machine solution.

Hello World

Here is a simple Hello World program. We first created a new virtual machine, then performed DoString(...) on it to interpret and execute the lua code, and finally closed the virtual machine. When we execute the program, we will see the characters "Hello World" on the command line String.

package main
import (
"github.com/yuin/gopher-lua"
)
func main() {
l := lua.NewState()
defer l.Close()
if err := l.DoString(`print("Hello World")`); err != nil {
panic(err)
}
}
// Hello World

Compiled in advance

After looking at the call chain of the above DoString(...) method, we found that every time DoString(...) or DoFile(...) will execute parse and compile once each.

func (ls *LState) DoString(source string) error {
if fn, err := ls.LoadString(source); err != nil {
return err
} else {
ls.Push(fn)
return ls.PCall(0, MultRet, nil)
}
}
func (ls *LState) LoadString(source string) (*LFunction, error) {
return ls.Load(strings.NewReader(source), "<string>")
}
func (ls *LState) Load(reader io.Reader, name string) (*LFunction, error) {
chunk, err := parse.Parse(reader, name)
// ...
proto, err := Compile(chunk, name)
// ...
}

Considering from this point, the same Lua code will be executed multiple times (such as in http server, each request will execute the same Lua code) scenario, if we can compile the code in advance, we should be able to reduce the overhead of parse and compile (if this is hotpath code). According to the Benchmark results, early compilation can indeed reduce unnecessary overhead.

package glua_test
import (
"bufio"
"os"
"strings"
lua "github.com/yuin/gopher-lua"
"github.com/yuin/gopher-lua/parse"
)
// 编译 lua 代码字段
func CompileString(source string) (*lua.FunctionProto, error) {
reader := strings.NewReader(source)
chunk, err := parse.Parse(reader, source)
if err != nil {
return nil, err
}
proto, err := lua.Compile(chunk, source)
if err != nil {
return nil, err
}
return proto, nil
}
// 编译 lua 代码文件
func CompileFile(filePath string) (*lua.FunctionProto, error) {
file, err := os.Open(filePath)
defer file.Close()
if err != nil {
return nil, err
}
reader := bufio.NewReader(file)
chunk, err := parse.Parse(reader, filePath)
if err != nil {
return nil, err
}
proto, err := lua.Compile(chunk, filePath)
if err != nil {
return nil, err
}
return proto, nil
}
func BenchmarkRunWithoutPreCompiling(b *testing.B) {
l := lua.NewState()
for i := 0; i < b.N; i++ {
_ = l.DoString(`a = 1 + 1`)
}
l.Close()
}
func BenchmarkRunWithPreCompiling(b *testing.B) {
l := lua.NewState()
proto, _ := CompileString(`a = 1 + 1`)
lfunc := l.NewFunctionFromProto(proto)
for i := 0; i < b.N; i++ {
l.Push(lfunc)
_ = l.PCall(0, lua.MultRet, nil)
}
l.Close()
}
// goos: darwin
// goarch: amd64
// pkg: glua
// BenchmarkRunWithoutPreCompiling-8         100000             19392 ns/op           85626 B/op         67 allocs/op
// BenchmarkRunWithPreCompiling-8           1000000              1162 ns/op            2752 B/op          8 allocs/op
// PASS
// ok      glua    3.328s

Virtual machine instance pool

In the scenario where the same Lua code is executed, in addition to using advance compilation to optimize performance, we can also introduce a virtual machine instance pool.

Because creating a new Lua virtual machine involves a large number of memory allocation operations, if the method of re-creating and destroying each run is adopted, a large amount of resources will be consumed. Introducing a virtual machine instance pool can reuse virtual machines , reducing unnecessary overhead.

func BenchmarkRunWithoutPool(b *testing.B) {
for i := 0; i < b.N; i++ {
l := lua.NewState()
_ = l.DoString(`a = 1 + 1`)
l.Close()
}
}
func BenchmarkRunWithPool(b *testing.B) {
pool := newVMPool(nil, 100)
for i := 0; i < b.N; i++ {
l := pool.get()
_ = l.DoString(`a = 1 + 1`)
pool.put(l)
}
}
// goos: darwin
// goarch: amd64
// pkg: glua
// BenchmarkRunWithoutPool-8          10000            129557 ns/op          262599 B/op        826 allocs/op
// BenchmarkRunWithPool-8            100000             19320 ns/op           85626 B/op         67 allocs/op
// PASS
// ok      glua    3.467s

Benchmark results show that the virtual machine instance pool can indeed reduce a lot of memory allocation operations.

The instance pool implementation provided by the README is given below, but note that In the initial state, this implementation did not create enough virtual machine instances (initially, the number of instances was 0), and there was a problem of dynamic expansion of slices. These are areas worthy of improvement.

type lStatePool struct {
    m     sync.Mutex
    saved []*lua.LState
}
func (pl *lStatePool) Get() *lua.LState {
    pl.m.Lock()
    defer pl.m.Unlock()
    n := len(pl.saved)
    if n == 0 {
        return pl.New()
    }
    x := pl.saved[n-1]
    pl.saved = pl.saved[0 : n-1]
    return x
}
func (pl *lStatePool) New() *lua.LState {
    L := lua.NewState()
    // setting the L up here.
    // load scripts, set global variables, share channels, etc...
    return L
}
func (pl *lStatePool) Put(L *lua.LState) {
    pl.m.Lock()
    defer pl.m.Unlock()
    pl.saved = append(pl.saved, L)
}
func (pl *lStatePool) Shutdown() {
    for _, L := range pl.saved {
        L.Close()
    }
}
// Global LState pool
var luaPool = &lStatePool{
    saved: make([]*lua.LState, 0, 4),
}

Module call

gopher-lua supports Lua to call Go modules. Personally, I think this is a very exciting feature, because in Golang program development, we may design many commonly used Modules, this cross-language calling mechanism, enable us to reuse code and tools.

Of course, in addition, there is also Go calling Lua module, but I personally feel that the latter is not necessary, so the latter is not covered here.

package main
import (
"fmt"
lua "github.com/yuin/gopher-lua"
)
const source = `
local m = require("gomodule")
m.goFunc()
print(m.name)
`
func main() {
L := lua.NewState()
defer L.Close()
L.PreloadModule("gomodule", load)
if err := L.DoString(source); err != nil {
panic(err)
}
}
func load(L *lua.LState) int {
mod := L.SetFuncs(L.NewTable(), exports)
L.SetField(mod, "name", lua.LString("gomodule"))
L.Push(mod)
return 1
}
var exports = map[string]lua.LGFunction{
"goFunc": goFunc,
}
func goFunc(L *lua.LState) int {
fmt.Println("golang")
return 0
}
// golang
// gomodule

Variable pollution

When we use instance pools to reduce overhead, another thorny problem will be introduced: because the same virtual machine may be executed multiple times Lua code, and then change the global variables in it. If the code logic relies on global variables, unpredictable running results may occur (this smells a bit like "non-repeatable reads" in database isolation).

Global variables

If we need to restrict Lua code to only use local variables, then from this starting point, we need to restrict global variables. So the question is, how to achieve it?

We know that Lua is compiled into bytecode and then interpreted and executed. Then, we can restrict the use of global variables during the bytecode compilation stage. After checking the Lua virtual machine instructions, I found that there are two instructions involving global variables: GETGLOBAL (Opcode 5) and SETGLOBAL (Opcode 7).

At this point, we already have a general idea: we can limit the use of global variables in the code by judging whether the bytecode contains GETGLOBAL and SETGLOBAL. As for obtaining the bytecode, you can get the FunctionProto of the Lua code by calling CompileString(...) and CompileFile(...), and the Code attribute is the bytecode slice, of type []uint32.

In the virtual machine implementation code, we can find a tool function that outputs the corresponding OpCode according to the bytecode.

// 获取对应指令的 OpCode
func opGetOpCode(inst uint32) int {
return int(inst >> 26)
}

With this tool function, we can check global variables.

package main
// ...
func CheckGlobal(proto *lua.FunctionProto) error {
for _, code := range proto.Code {
switch opGetOpCode(code) {
case lua.OP_GETGLOBAL:
return errors.New("not allow to access global")
case lua.OP_SETGLOBAL:
return errors.New("not allow to set global")
}
}
// 对嵌套函数进行全局变量的检查
for _, nestedProto := range proto.FunctionPrototypes {
if err := CheckGlobal(nestedProto); err != nil {
return err
}
}
return nil
}
func TestCheckGetGlobal(t *testing.T) {
l := lua.NewState()
proto, _ := CompileString(`print(_G)`)
if err := CheckGlobal(proto); err == nil {
t.Fail()
}
l.Close()
}
func TestCheckSetGlobal(t *testing.T) {
l := lua.NewState()
proto, _ := CompileString(`_G = {}`)
if err := CheckGlobal(proto); err == nil {
t.Fail()
}
l.Close()
}

Modules

In addition to variables that may be contaminated, imported Go modules may also be tampered with during runtime. Therefore, we need a mechanism to ensure that modules imported into the virtual machine are not tampered with, that is, the imported objects are read-only.

After consulting relevant blogs, we can modify the __newindex method of Table and set the module to read-only mode.

package main
import (
"fmt"
"github.com/yuin/gopher-lua"
)
// 设置表为只读
func SetReadOnly(l *lua.LState, table *lua.LTable) *lua.LUserData {
ud := l.NewUserData()
mt := l.NewTable()
// 设置表中域的指向为 table
l.SetField(mt, "__index", table)
// 限制对表的更新操作
l.SetField(mt, "__newindex", l.NewFunction(func(state *lua.LState) int {
state.RaiseError("not allow to modify table")
return 0
}))
ud.Metatable = mt
return ud
}
func load(l *lua.LState) int {
mod := l.SetFuncs(l.NewTable(), exports)
l.SetField(mod, "name", lua.LString("gomodule"))
// 设置只读
l.Push(SetReadOnly(l, mod))
return 1
}
var exports = map[string]lua.LGFunction{
"goFunc": goFunc,
}
func goFunc(l *lua.LState) int {
fmt.Println("golang")
return 0
}
func main() {
l := lua.NewState()
l.PreloadModule("gomodule", load)
    // 尝试修改导入的模块
if err := l.DoString(`local m = require("gomodule");m.name = "hello world"`); err != nil {
fmt.Println(err)
}
l.Close()
}
// <string>:1: not allow to modify table

Written at the end

The integration of Golang and Lua has broadened my horizons: It turns out that static language and dynamic language can be integrated in this way, and the static language runs at a high speed Efficiency, combined with the high efficiency of dynamic language development, makes me excited just thinking about it (escape.

I searched online for a long time and found that there was no technical sharing about Go-Lua. I only found a slightly related article (Continuous Architecture Optimization of JD.com’s Level 3 List Page—Golang Lua (OpenResty) Best Practices). And in this article, Lua still runs on C. Due to the lack of information and my (student party) lack of development experience, I cannot well evaluate the feasibility of this solution in actual production. Therefore, this article can only be regarded as a "casual article", haha.

The above is the detailed content of What sparks will arise when Golang and Lua meet?. For more information, please follow other related articles on the PHP Chinese website!