如何释放由与Cython模块接口的外部C库分配的内存，最终将内存返回给Python进程?

问题描述：

我对Cython还是陌生的，但基本上，我的应用程序需要大幅提高性能，因此我和我的团队正在尝试重写Cython和C语言中的瓶颈.

I am brand new to Cython, but basically I have this application that needs significant performance increases, and so my team and I are trying to rewrite our bottlenecks in Cython and in C.

对于我们应用程序中最慢的部分，我编写了一些C代码，这些代码被编译到库中，并且cdef extern导入到Cython模块中，我认为这是一个.pyx文件.本质上，pyx文件中的代码基本上只是一个包装，该包装返回对C库函数的调用.最后，有一个Python进程(主应用程序)导入pyx文件中定义的所有函数并使用这些结果.

For the slowest part of our application, I wrote some C code that gets compiled into a library and cdef extern imported into a Cython module, which I believe is a .pyx file. Essentially, the code in the pyx file is basically just a wrapper that returns calls to the C library functions. Finally, there is a Python process (the main application) that imports all of the functions defined in the pyx file and uses these results.

我相信我有内存泄漏，因为在C代码中，我需要传递给Python进程的结果有时是动态分配的.我的问题是，一旦Python进程使用了内存，我将不知道如何释放它.

I believe I have a memory leak because in the C code, the results that I need to pass to the Python process are at times dynamically allocated. My issue is that I don't know how to free this memory once the Python process has made use of it.

Python示例代码

from examplecython import *

def foo(data):
    context = data.context
    value = call_pyx_function(context, data)
    return value

def bar(results):
    for data in results:
        res = foo(data)
        do_something_with_res(res)
        # I want to free here

Cython代码示例

cdef extern from "my_lib.h"
    char * my_function(const char * context, int data)

def call_pyx_function(context: bytes, int x):
    return my_function(context, x)

示例C代码


#define BUFSIZE 256

char *
my_function(const char * context, int x) {
    char * retbuf;
    int res;

    retbuf = (char *)malloc(BUFSIZE * sizeof(char));

    res = do_some_math(x, context);

    int length = snprintf(retbuf, BUFSIZE, "%d", res);
    if (length >= BUFSIZE) {
        exit(EXIT_FAILURE);
    }

    return retbuf;
}

如果有人对我如何以及在何处可以释放此内存有任何建议，将不胜感激.

If anyone has any suggestions for how and where I can free this memory, that would be very much appreciated.

答

您可以直接从libc.stdlib导入free:

from libc.stdlib cimport free

def bar(results):
    for data in results:
        res = foo(data)
        try:
            do_something_with_res(res)
        finally:
            free(res)

(请注意，您需要try/finally，因为即使某些情况引发异常，您也希望将其释放)

(Note you need the try/finally because you want it to be freed even if something throws an exception)

您可以使用上下文管理器或在__del__/__dealloc__中删除的包装器来简化此操作:

You can make this easier with a context manager or a wrapper that deletes in __del__ / __dealloc__:

@contextlib.contextmanager
def freeing(res):
    try:
        yield res
    finally:
        free(res)

def bar(results):
    for data in results:
        with freeing(foo(data)) as res:
            do_something_with_res(res)

或者(可能会释放得很晚，可能会更慢，但是(几乎)保证最终会被释放)

Or (Might get freed much later, probably slower, but (almost) guaranteed to be freed eventually)

# (in pyx file)
cdef class MallocedResource:
    cdef void* res;

    def __init__(self, res):
        # Note: This "steals" res. Don't free `res`
        # as it is freed when this class's storage is freed
        self.res = <void *>res

    def __dealloc__(self):
        free(self.res)

def call_pyx_function(context: bytes, int x):
    return MallocedResouce(my_function(context, x))

# No need to change python code, so you can't forget to use try/finally.

如何释放由与Cython模块接口的外部C库分配的内存，最终将内存返回给Python进程?

相关推荐