Callback on include

[edhoedt]

Python interface for pull request VirusTotal/yara#727

Example usage:

import yara
import sys
if sys.version_info >= (3, 0):
    import urllib.request as urllib
else:
    import urllib as urllib

def mycallback(requested_filename, filename, namespace):
    if requested_filename == 'req.yara':
        uf = urllib.urlopen('https://pastebin.com/raw/siZ2sMTM')
        sources = uf.read()
        if sys.version_info >= (3, 0):
            sources = str(sources, 'utf-8')
        return sources
    else:
        raise Exception(filename+": Can't fetch "+requested_filename)

rules = yara.compile(source='include "req.yara" rule r{ condition: true }',
                     include_callback=mycallback)

[plusvic]

The string returned by the callback is never freed. When PyObject_CallFunctionObjArgs returns the reference count for result is >=1 and your code has the ownership for that string. In order to let the garbage collector free the string you must do a Py_DECREF at some point.

The problem here is that you can't do it in yara_include_callback, because then you are returning to YARA a string that could be freed by the garbage collector at any time. But if you can't do it here, then where?

[plusvic]

You must do a Py_DECREF(py_incl_name) when you are done with the Python string in order to let the garbage collector free the memory. The same applies to py_calling_fn and py_calling_ns.

[plusvic]

You must do Py_INCREF(callback) before PyObject_CallFunctionObjArgs and Py_DECREF(callback) after it.

[plusvic]

Before calling any Python function you must acquire the global interpreter lock (GIL) with PyGILState_Ensure and release it with PyGILState_Release.

[plusvic]

@edhoedt please check #67

[plusvic]

Closing this PR as #67 fix conflicts and some other issues.