tag:blogger.com,1999:blog-3531969847869413627.post3123749370437932503..comments2023-07-05T07:05:39.753-04:00Comments on Boduch's Blog: Trying to break Python memory consumptionAdam Boduchhttp://www.blogger.com/profile/00484502507931628726noreply@blogger.comBlogger4125tag:blogger.com,1999:blog-3531969847869413627.post-29439018877183360812009-03-12T13:25:00.000-04:002009-03-12T13:25:00.000-04:00Sadly my montecarlo analysis app receives MemoryEr...Sadly my montecarlo analysis app receives MemoryError exceptions all the time. :( I hit the 3GB (not 2) limit for a 32bit app under Linux. You don't say how much memory your computer has but I'm guessing that you're swapping before hitting 3G which is making something else break before you hit the limit. (Which probably means you should be worrying about other issues before this one most likely)<BR/><BR/>To generate the exception run on a 4G machine and just create a list, copy the list to a new one, append them together, repeat. Shouldn't take more than a few minutes on a fast machine with psyc like mine. Maybe longer without psyc.<BR/><BR/>Good luck!proteusguyhttps://www.blogger.com/profile/06205203744072448163noreply@blogger.comtag:blogger.com,1999:blog-3531969847869413627.post-41413429306264650292009-03-10T08:20:00.000-04:002009-03-10T08:20:00.000-04:00Your process is probably getting killed by the Out...Your process is probably getting killed by the Out Of Memory killer (oom killer) in the kernel. Check in /var/log/messages, you should see a message from the oom killer.Scott Wimerhttps://www.blogger.com/profile/07398480470740937873noreply@blogger.comtag:blogger.com,1999:blog-3531969847869413627.post-43652626901802635722009-03-09T16:26:00.000-04:002009-03-09T16:26:00.000-04:00I strongly suspect that the process is getting a s...I strongly suspect that the process is getting a signal of some sort, perhaps SIGSEGV. You might have greater luck in setting up a signal handler.Anonymoushttps://www.blogger.com/profile/11442714620753651697noreply@blogger.comtag:blogger.com,1999:blog-3531969847869413627.post-25674930867445595512009-03-09T15:57:00.000-04:002009-03-09T15:57:00.000-04:00Linux generally follows a policy of overallocation...Linux generally follows a policy of overallocation. What this means is that malloc() will almost always succeed, but pages won't actually be allocated until you attempt to write to them. By that point however, it's too late to signal an error - the malloc is long past, so the kernel has to take drastic measures, by killing a process in the hopes of freeing up memory.<BR/><BR/>If you'd allocated it as zeroed memory, rather than attempting to write to it. eg mem = [0] * 2**24, you'd have succeeded until you attempted to write a non-zero value to the memory.<BR/><BR/>There is one case where this will return an error however, and that's when you run out of address space, rather than physical memory. On a 32 bit processor, you won't be able to address more than 2 GiB, so allocations of that amount will probably fail immediately with a MemoryError. (In fact, since memory will fragment, even much smaller ranges may fail as no contiguous block exists in that range)Anonymousnoreply@blogger.com