R - Memory issue on allocating large size data to a vector


#1

Hi all,

I want to process a large dataset typically in matrix format in R. I am facing memory issue “Error: cannot allocate vector of size 446.8 Mb”

How can i handle this?

Regards,
Karthikeyan P


How to resolve error in allocation of memory while applying linear regression?
#2

Hi @karthe1,

This is probably because your C drive or where you have R installed is low on memory.R does give memory issues but not on 446.8Mb(otherwise it won’t be really that popular, :P).
One more reason can be that the matrix dimensions are too huge for R to handle.I had faced this issue sometime back while trying matrix rotations by 10,15 degrees.R gave the same error to me.
Then I did two rotations of 5 each and it worked.I guess you have to try something similar.
So either reduce the size of the matrix or do multiple small operations.
Hope this helps!!


#3

Hi @shuvayan,

Thanks for the reply. I dont know the concept of rotation of matrix. What is meant by that? I know taking transpose of a matrix. But what will rotating 5 times will have an effect on the memory size?

Regards.
Karthikeyan P


#4

You can look at this thread for some additional methods:

http://discuss.analyticsvidhya.com/t/what-are-the-ways-to-handle-huge-data-in-r/141


#5

also see this package

http://www.inside-r.org/packages/cran/bigmemory/docs/read.big.matrix


#6

Karthikeyan,
You can also try something like “ff” or “data.table” packages which are meant to handle big data. Most of the function still work in these packages.

Hope this helps.
Tavish