Re: Error on importing previously saved data file: Get::bigfile
- To: mathgroup at smc.vnet.net
- Subject: [mg122435] Re: Error on importing previously saved data file: Get::bigfile
- From: A Retey <awnl at gmx-topmail.de>
- Date: Fri, 28 Oct 2011 05:36:37 -0400 (EDT)
- Delivered-to: l-mathgroup@mail-archive0.wolfram.com
- References: <j89ugu$t7k$1@smc.vnet.net>
Hi,
> I have saved a data set (Save["filename",data]) - which is about 3.5
> MB on the harddisk (a table with numbers and images,
> ByteCount=610117226).
> When I try to read it again into Memory I get "Get::bigfile: "The file
> "filename" is too large to be read into memory."
>
> I can work around it by splitting the data in smaller junks, but there
> should be a better way since the data set should not be too large
>
> Any ideas?
is it important for you to save in a text format? If all you want to do
is to store the data and read it back into Mathematica at a later time I
would suggest to use a binary format:
In[45]:= data =
Table[{#, ArrayPlot[#]} &@RandomReal[{0, 1}, {100, 100}], {30}];
In[46]:= ByteCount[data]
Out[46]= 4838920
In[48]:= Timing[
Export[ToFileName[{$HomeDirectory, "Desktop"}, "tst.mx"], data]]
Out[48]= {0.047, "C:\\Users\\albert.retey\\Desktop\\tst.mx"}
In[51]:= Timing[
datain = Import[ToFileName[{$HomeDirectory, "Desktop"}, "tst.mx"]];]
Out[51]= {0.047, Null}
If you need it to be transfered to another machine, there is the
wdx-format, which has no problem with files of that size but is
relatively slow:
In[54]:= Timing[
Export[ToFileName[{$HomeDirectory, "Desktop"}, "tst.wdx"], data]]
Out[54]= {6.927, "C:\\Users\\albert.retey\\Desktop\\tst.wdx"}
In[55]:= Timing[
datain = Import[ToFileName[{$HomeDirectory, "Desktop"}, "tst.wdx"]];]
Out[55]= {3.089, Null}
hth,
albert