不能腌制< type'instancemethod'>使用多重处理Pool.map()时
我正在尝试使用multiprocessing
的Pool.map()
函数同时划分工作.当我使用以下代码时,它可以正常工作:
I'm trying to use multiprocessing
's Pool.map()
function to divide out work simultaneously. When I use the following code, it works fine:
import multiprocessing
def f(x):
return x*x
def go():
pool = multiprocessing.Pool(processes=4)
print pool.map(f, range(10))
if __name__== '__main__' :
go()
但是,当我以一种更加面向对象的方式使用它时,它不起作用.它给出的错误消息是:
However, when I use it in a more object-oriented approach, it doesn't work. The error message it gives is:
PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup
__builtin__.instancemethod failed
当以下是我的主程序时,会发生这种情况:
This occurs when the following is my main program:
import someClass
if __name__== '__main__' :
sc = someClass.someClass()
sc.go()
,以下是我的someClass
类:
import multiprocessing
class someClass(object):
def __init__(self):
pass
def f(self, x):
return x*x
def go(self):
pool = multiprocessing.Pool(processes=4)
print pool.map(self.f, range(10))
任何人都知道问题可能是什么,或解决问题的简便方法?
Anyone know what the problem could be, or an easy way around it?
问题在于,多处理必须对程序进行腌制以使它们在进程之间挂起,并且绑定的方法不可腌制.解决方法(无论您是否认为它简单" ;-)是向您的程序中添加基础结构以允许对此类方法进行腌制,并在
The problem is that multiprocessing must pickle things to sling them among processes, and bound methods are not picklable. The workaround (whether you consider it "easy" or not;-) is to add the infrastructure to your program to allow such methods to be pickled, registering it with the copy_reg standard library method.
例如,史蒂文·贝萨德(Steven Bethard)对此线程(在线程末尾)显示了一种完全可行的方法,允许通过copy_reg
进行方法酸洗/取消酸洗.
For example, Steven Bethard's contribution to this thread (towards the end of the thread) shows one perfectly workable approach to allow method pickling/unpickling via copy_reg
.