Services & Resources / Wolfram Forums / MathGroup Archive
-----

MathGroup Archive 2011

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: Error on importing previously saved data file: Get::bigfile

  • To: mathgroup at smc.vnet.net
  • Subject: [mg122435] Re: Error on importing previously saved data file: Get::bigfile
  • From: A Retey <awnl at gmx-topmail.de>
  • Date: Fri, 28 Oct 2011 05:36:37 -0400 (EDT)
  • Delivered-to: l-mathgroup@mail-archive0.wolfram.com
  • References: <j89ugu$t7k$1@smc.vnet.net>

Hi,

> I have saved a data set (Save["filename",data]) - which is about 3.5
> MB on the harddisk (a table with numbers and images,
> ByteCount=610117226).
> When I try to read it again into Memory I get "Get::bigfile: "The file
> "filename" is too large to be read into memory."
>
> I can work around it by splitting the data in smaller junks, but there
> should be a better way since the data set should not be too large
>
> Any ideas?

is it important for you to save in a text format? If all you want to do 
is to store the data and read it back into Mathematica at a later time I 
would suggest to use a binary format:

In[45]:= data =
   Table[{#, ArrayPlot[#]} &@RandomReal[{0, 1}, {100, 100}], {30}];

In[46]:= ByteCount[data]

Out[46]= 4838920

In[48]:= Timing[
  Export[ToFileName[{$HomeDirectory, "Desktop"}, "tst.mx"], data]]

Out[48]= {0.047, "C:\\Users\\albert.retey\\Desktop\\tst.mx"}

In[51]:= Timing[
  datain = Import[ToFileName[{$HomeDirectory, "Desktop"}, "tst.mx"]];]

Out[51]= {0.047, Null}

If you need it to be transfered to another machine, there is the 
wdx-format, which has no problem with files of that size but is 
relatively slow:

In[54]:= Timing[
  Export[ToFileName[{$HomeDirectory, "Desktop"}, "tst.wdx"], data]]

Out[54]= {6.927, "C:\\Users\\albert.retey\\Desktop\\tst.wdx"}

In[55]:= Timing[
  datain = Import[ToFileName[{$HomeDirectory, "Desktop"}, "tst.wdx"]];]

Out[55]= {3.089, Null}

hth,

albert



  • Prev by Date: Re: Another basic (?) question about RecurrenceTable and replacement
  • Next by Date: Re: Another basic (?) question about RecurrenceTable and replacement
  • Previous by thread: Re: Error on importing previously saved data file: Get::bigfile
  • Next by thread: Coin Toss Sim