python进程可以共享活动对象吗?

问题描述:

我有一个多进程python应用程序(进程由uwsgi生成),需要将变量存储在RAM中,然后从几个不同的进程中读取和更新这些变量。我知道有很多可用的缓存选项,但是我发现的所有缓存选项只能存储字符串。

I have a multi-process python application (processes are spawned by uwsgi) that needs to store variables in RAM, then read and update those variables from several different processes. I know there are a lot caching options available, but all the ones I've found can only store strings. Is it possible for different python processes to access the same virtual memory, and thus share data without ever converting it or even copying it?

是否可以使不同的python进程访问相同的虚拟内存,从而共享数据而无需进行转换甚至复制? p>除了 POSH,Python共享对象,它至少可以完成您想要做的一部分(放置Python SvsV-IPC中的对象共享内存并从多个进程进行修改),并且可以作为开发自己的扩展模块的起点,以适应wsgi生成的服务器进程的需求,Python世界中没有太多其他东西(我知道...)在进程之间共享对象时不依赖于酸洗/取消酸洗。

Besides POSH, Python Shared Objects, which at least does a part of what you want to do (place Python Objects in SvsV-IPC shared memory and modify them from multiple processes) and could serve as a starting point on developing your own extension module to fit the need you have with wsgi-spawned server processes, there's not much else in the Python world (that I know of...) that doesn't rely on pickling/unpickling objects when sharing them between processes.

想到的另一件事是 Pyro ,它确实在进程之间通过任意网络连接进行共享(因此,也可以通过Unix域套接字共享),并以我自己的经验比内置多重处理提供的(代理)对象管理更灵活。

One other thing that also comes to mind is Pyro, which does sharing over an arbitrary network connections between processes (so, can also share via Unix-Domain sockets), and is in my own experience more flexible than what the builtin multiprocessing can offer for (proxy) object management.

您可能还会看到的是,是否可以使驱动WSGI应用程序的Web服务器不派生进程,而是使用线程。这样,您只需要为共享对象缓存使用标准的Python全局数据,然后就可以从所有产生的WSGI处理程序线程中进行访问。线程化的WSGI服务器是例如CherryPy-builtin wsgiserver , m用于一个完全满足您需求的项目。 mod_wsgi 也适用于您的上下文,以防您使用工作程序MPM配置Apache(以便Apache线程,并且不会派生新进程。)

What you also might have a look at is whether you can get the webserver that's driving your WSGI-application to not fork processes, but rather to use threading; this way, you'd simply need to use standard Python global data for the shared object cache which you can then access from all spawned WSGI handler threads. A threaded WSGI-server is for example the CherryPy-builtin wsgiserver, which I'm using for a project which has exactly the demand you're having. mod_wsgi also works for your context in case you configure Apache with the worker MPM (so that Apache threads, and doesn't fork new processes).

如果所有这些都不可选项,那么如何提取您现在在网络服务器中正在执行的实际处理,外部过程,网页通过某种形式的RPC机制与之通信以推送工作请求并提取数据?这样,后端处理服务器可以是一个简单的多线程Python进程,它通过标准库 SimpleXMLRPCServer 或类似的东西。

If all of those aren't an option, how about extracting the actual processing you're now doing in the webserver to an external process, which the webpages communicate with via some form of RPC mechanism to push work requests and pull data? The "backend" processing server can then be a simple multithreaded Python process, which offers an XML-RPC interface through the standard library SimpleXMLRPCServer or some thing similar.