Java mmap在Android上失败,“mmap failed:ENOMEM(Out of memory)”

问题描述:

使用Java在Android上映射大型文件的内存效果很好。但是,即使多次映射调用,总共映射超过~1.5GB也会失败:

Memory mapping a large file on Android in Java works good. But when mapping more than ~1.5GB in total even with multiple mapping calls it fails with:

mmap failed: ENOMEM (Out of memory)

参见完整讨论这里。注意:它在服务器Linux上不会失败。为应用程序启用了android:largeHeap =true。

See the full discussion here. Note: It does not fail on a server Linux. The android:largeHeap="true" is enabled for the application.

以下Java代码被称为几百次,每次调用请求~1MB:

The following Java code is called a few hundred times requesting ~1MB per call:

ByteBuffer buf = raFile.getChannel().map(allowWrites ? FileChannel.MapMode.READ_WRITE : FileChannel.MapMode.READ_ONLY, offset, byteCount);

以避免请求一个通常难以找到的大型连续内存块。查看完整代码这里。请记住,将段大小(即单个地图调用的大小)加倍无效,这意味着它会停在相似的内存位置。另外值得注意的是,两个略低于限制的应用程序执行正常(暗示每个进程限制)。

to avoid requesting one large contiguous memory chunk which is often harder to be found. See the full code here. Keep in mind that doubling the 'segment size' (i.e. the size of a single map call) has no effect, which means it stops at the similar memory position. Also it is important to note that 2 apps with both slightly under the limit are executing fine (hinting to a per process limit).

相关问题是在其中 where 这里这里这里这里这里这里

Related questions are here, here, here, here, here, here, here and here.

使用多个文件而不是一个有多个映射的文件会有帮助吗?

Will using multiple files instead of one file with multiple mappings help?

我读过这可能是虚拟地址空间的每个进程限制 。哪里可以找到更多相关信息?我可以使用NDK更改此设置,例如如何调用 ulimit ?可以 madvise 对我有所帮助吗?

I've read that this could be a per process limit for the virtual address space. Where can I find more about this? Can I change this setting with NDK e.g. how to call ulimit? Could madvise help me a bit here?

更新

查看我的回答这里是一个可在Java中使用的mmap工具

See my answer here for a mmap tool usable in Java

你的问题肯定是由虚拟地址引起的空间耗尽。可能你的问题在32位Android设备上重现,用户地址空间可用于物理上限制为2GB且无法碰撞。 (虽然它可能是3GB(不太可能),并且它是在OS构建过程中配置的)。可能~500 MB用于系统库,JVM及其堆。并且~1.5 GB可供您使用。

Your problem is surely caused by virtual address space exhausting. Probably your problem reproduces on 32-bit Android devices, where available to user address space is physically limited to 2GB and cannot be bumped. (Although it may be 3GB (unlikely so) and it is configured during OS build process). Probably ~500 MB is used for system libraries, JVM and its heap. And ~1.5 GB is available for you.

在这种情况下唯一的方法IMO - 继续仅映射现在真正使用的文件部分,并尽快取消映射未使用的文件部分可能。您可以使用某种滑动窗口,其中只有一小部分文件将被映射到内存,当您完成时 - 取消映射该部分,推进您的窗口位置并映射该更新的窗口,等等。

The only way in this situation IMO - keep being mapped only file sections that are really used now and unmap unused ones as soon as possible. You can utilize some kind of sliding window where only small part of file will be mapped to memory, and when you finish - unmap that part, advance your window position and map that updated window, so on.

当你映射整个大文件时 - 你的过程成为系统内存杀手的一个有吸引力的受害者。因为当你读取这样的映射文件时 - 物理内存的消耗会增加,并且在某些时刻进程将被终止。

Also when you map whole large file - your process becomes an attractive victim for system's Out-Of-Memory killer. Because when you read such mapped file - consumption of physical memory raises and at some moment process will be killed.