When doing large data array operations in MATLAB(like in Image and Video processing), it's largely possible to come across the "Out of Memory" error. Matlab eats about 800 Megabytes of memory just to start and be idle. If you want to do a memory-killing computation, you'd better increasing the maximum memory limit for MATLAB in Windows. (Note: Only tried on 32bit Windows XP, no Linux experience on this. I would like to hear if someone did the same thing on Linux or other platforms.)
Read through this article first. It tells you how to avoid the "Out of Memory" Error : http://www.mathworks.com/support/tech-notes/1100/1107.html#_Toc170182654
This is the article tells you how to increase the maximum memory limit on Windows http://technet.microsoft.com/en-us/library/bb124810.aspx (Note: Windows XP is not mentioned, but it's applicable. Take the procedure of: To Set the /3GB Startup Switch in Windows Server 2003 ) When I did the modification on my Windows and restarted the machine, my graphic card driver got lost somehow, and some default startup options got changed. No big deal.. Reinstall and reset, and you are all set.
Don't forget to read through this: http://www.mathworks.com/company/newsletters/news_notes/june07/patterns.html, which gives you programming tips in Matlab to minimize memory usage.