Out of Memory when using Golang method in Python

Question

I have been following this https://medium.com/learning-the-go-programming-language/calling-go-functions-from-other-languages-4c7d8bcc69bf

Now, what I am doing is:-

Pass a Json String
Use Json Unmarshall with a Go Struct Reference & the string
Do Business Logic & Operations
Marshal the struct to JSON String
Return

Now, the issue I am facing is that I am getting an Out-of-Memory exception when running it through python. If I run it through normal Go, then it works good.

Also if I dont unmarshal and just initialize the structure normally then I do not have an issue, and get a json string back. But I need to pass a structured data between both.

The issue occurs in the Unmarshal Line

func BuildCRMLinkFromJSON(jsonstring string) CRMLinkBO {
    var crmLink CRMLink = CRMLink{}
    json.Unmarshal([]byte(jsonstring), &crmLink)
    var clb = CRMLinkBO{crmLink: crmLink}
    return clb
}

The runtime error is:

runtime: out of memory: cannot allocate 140328041390080-byte block (66781184 in use)
fatal error: out of memory

or

Shauryas-MacBook-Pro:go-python-interfacing xuryax$ python pythonclient.py 
unexpected fault address 0x210c1a334
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x1 addr=0x210c1a334 pc=0x1087b0c75]

Minimal Reproduce: usego.go

package main

import (
    "encoding/json"
    "fmt"
)

import "C"

// Data Model

type CustomField struct {
    Field string `json:"field"`
    Value string `json:"value"`
}

type PersonDetails struct {
    Person_id string        `json:"person_id"`
    Detail    []CustomField `json:"detail"`
}

func New(pid string) PersonDetails {
    var p = PersonDetails{"HARDCODED PID IN GO", []CustomField{}}
    return p
}

func BuildFromJSON(jsonString string) PersonDetails {
    var person_detail PersonDetails = PersonDetails{}
    json.Unmarshal([]byte(jsonString), &person_detail)
    return person_detail
}

func ConvertToJSON(p PersonDetails) string {
    fmt.Println(p)
    var je, _ = json.Marshal(p)
    return string(je)
}

func AddValue(p PersonDetails) PersonDetails {
    var CustField = CustomField{"Hardcoded Field in Go", "Hardcoded Value in Gos"}
    p.Detail = append(p.Detail, CustField)
    return p
}

//export DoJSONOperation
func DoJSONOperation(jsonString string) *C.char {
    var p = BuildFromJSON(jsonString)
    p = AddValue(p)
    var nstr = ConvertToJSON(p)
    return C.CString(nstr)
}

//export DoNormalOperation
func DoNormalOperation(jsonString string) *C.char {
    var p = New(jsonString)
    p = AddValue(p)
    var nstr = ConvertToJSON(p)
    fmt.Println("Before Return: ", string(nstr))
    return C.CString(nstr)
}

func main() {
    // DO noting
    // var jsonStr = `{"person_id":"JsonID","detail":[{"field":"json passed field", "value":"json passed value"}]}`
}

PythonClient.py

from ctypes import *
import json

def main():
    lib = cdll.LoadLibrary("/Users/xuryax/work/repos/research/go-python-interfacing/usego.so")

    lib.DoJSONOperation.argtypes = [c_char_p]
    lib.DoJSONOperation.restype = c_char_p

    json_string = """{"person_id":"PythonJsonID","detail":[{"field":"json passed field", "value":"json passed value"}]}"""
    normal_pid = "Python ID"
    updated_json = lib.DoNormalOperation(normal_pid)

    print(updated_json)


if __name__=="__main__":
    main()

Weird thing I found in if I use the one with hardcoded it is returning Pointer(integer) but On my previous experiment, I did get a string back.

you should try to simplify your question by including minimal complete code ... something we can execute to exhibit the issue ... How to create a Minimal, Complete, and Verifiable example stackoverflow.com/help/mcve — John Scott Stensland
– John Scott Stensland, Commented Apr 20, 2019 at 10:56
Are you sure jsonstring is what you think it is? The stack trace indicates that it's the byte slice conversion that attempts to allocate 130ish terra bytes of memory. An mcve would certainly help a lot. — Peter
– Peter, Commented Apr 20, 2019 at 12:24
Added a minimum code . Yea thats what was weird that it was trying to allocatee 130 TB :S — Shaurya Chaudhuri
– Shaurya Chaudhuri, Commented Apr 20, 2019 at 12:59

Shaurya Chaudhuri · Accepted Answer · 2019-04-20 15:19:26Z

0

The answer seems quite simple actually. It is a problem with typecasting which I missed.

The method should accept *C.char

//export DoJSONOperation
func DoJSONOperation(jsonString *C.char) *C.char {
    var GoString = C.GoString(jsonString)
    var p = BuildFromJSON(GoString)
    p = AddValue(p)
    var nstr = ConvertToJSON(p)
    return C.CString(nstr)
}

And this seems to solve the problem.

In Python we need to typecast as well. And instead of sending Python String we need to send. c_char_p(jsonString)

answered Apr 20, 2019 at 15:19

Shaurya Chaudhuri

3,8728 gold badges32 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Out of Memory when using Golang method in Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related