带有Struct和Numpy的Python内存泄漏

问题描述：

使用binascii，zlib，struct的Python内存泄漏的延续，以及numpy ，但示例代码可以正确说明我遇到的问题.

Continuation of Python Memory Leak Using binascii, zlib, struct, and numpy but with example code that correctly illustrates the issue I have.

import struct
import zlib
import binascii
import numpy as np
import os
import psutil
import gc

l = list()
l.append('eJztwYEAAAAAw6D5U1/gCFUB' + 'A'*161 + 'McA6vIAAQ==')
l.append('eJztwYEAAAAAw6D5U1/hAFUB' + 'A'*3957 + 'HwGw6IAAQ==')
l.append('eJztwYEAAAAAw6D5U1/hAFUB' + 'A'*3957 + 'LwGw6QAAQ==')

process = psutil.Process(os.getpid())
for s in l:
    print (process.get_memory_info()[0] / float(2 ** 20))
    byte_array = zlib.decompress(binascii.a2b_base64(s))
    array = np.array(struct.unpack('%dB' % (len(byte_array)), byte_array))
    del byte_array
    del array
    gc.collect()
    print (process.get_memory_info()[0] / float(2 ** 20))

del l
del s
gc.collect()
print (process.get_memory_info()[0] / float(2 ** 20))

它打印:

22.37109375
25.83203125
25.83203125
95.65625
95.65625
166.69140625
166.69140625

为什么使用的内存继续增加?为什么即使删除变量后，脚本末尾还是要使用大量内存?谢谢.

Why does the memory used continue to increase? Why is so much memory used at the end of the script even after the variables are deleted? Thank you.

答

此链接 http://bugs.python. org/issue14596 非常有帮助.问题出在struct模块缓存格式字符串上.如果我明确创建了一个Struct对象，请使用它，然后再删除它，问题就消失了.

This link http://bugs.python.org/issue14596 was very helpful. The issue was with the struct module caching format strings. If i explicitly create a Struct object, use it, and then delete it the issue goes away.

import struct
import zlib
import binascii
import os
import psutil
import gc


def print_memory(string):
    print string + str(process.get_memory_info()[0] / float(2 ** 20))

l = list()
l.append('eJztwYEAAAAAw6D5U1/gCFUB' + 'A'*161 + 'McA6vIAAQ==')
l.append('eJztwYEAAAAAw6D5U1/hAFUB' + 'A'*3957 + 'HwGw6IAAQ==')
l.append('eJztwYEAAAAAw6D5U1/hAFUB' + 'A'*3957 + 'LwGw6QAAQ==')

process = psutil.Process(os.getpid())
for s in l:
    print_memory('Before inflating: ')
    byte_array = zlib.decompress(binascii.a2b_base64(s))
    _struct = struct.Struct('%dB' % (len(byte_array)))
    array = _struct.unpack(byte_array)
    del byte_array
    del array
    del _struct
    gc.collect()
    print_memory('After inflating and deleting: ')

del l
del s
gc.collect()
print_memory('After deleting everything: ')

带有Struct和Numpy的Python内存泄漏

相关推荐