不能腌制 <type 'instancemethod'>使用多处理 Pool.map() 时
我正在尝试使用 multiprocessing
的 Pool.map()
函数来同时划分工作.当我使用以下代码时,它工作正常:
I'm trying to use multiprocessing
's Pool.map()
function to divide out work simultaneously. When I use the following code, it works fine:
import multiprocessing
def f(x):
return x*x
def go():
pool = multiprocessing.Pool(processes=4)
print pool.map(f, range(10))
if __name__== '__main__' :
go()
但是,当我以更面向对象的方法使用它时,它不起作用.它给出的错误信息是:
However, when I use it in a more object-oriented approach, it doesn't work. The error message it gives is:
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
__builtin__.instancemethod failed
当以下是我的主程序时会发生这种情况:
This occurs when the following is my main program:
import someClass
if __name__== '__main__' :
sc = someClass.someClass()
sc.go()
以下是我的 someClass
类:
import multiprocessing
class someClass(object):
def __init__(self):
pass
def f(self, x):
return x*x
def go(self):
pool = multiprocessing.Pool(processes=4)
print pool.map(self.f, range(10))
有人知道可能是什么问题,或者有什么简单的方法可以解决吗?
Anyone know what the problem could be, or an easy way around it?
问题是多处理必须pickle 东西才能在进程间sling 它们,而绑定的方法是不能picklable 的.解决方法(无论您认为它容易"与否;-)是将基础结构添加到您的程序中以允许腌制此类方法,并使用 copy_reg 标准库方法.
The problem is that multiprocessing must pickle things to sling them among processes, and bound methods are not picklable. The workaround (whether you consider it "easy" or not;-) is to add the infrastructure to your program to allow such methods to be pickled, registering it with the copy_reg standard library method.
例如,Steven Bethard 对此线程的贡献(在线程的末尾)展示了一种非常可行的方法来允许通过 copy_reg
进行方法pickling/unpickling.
For example, Steven Bethard's contribution to this thread (towards the end of the thread) shows one perfectly workable approach to allow method pickling/unpickling via copy_reg
.