MathGroup Archive 2005

[Date Index] [Thread Index] [Author Index]

Search the Archive

Exporting and importing arbitrary expressions efficiently

  • To: mathgroup at smc.vnet.net
  • Subject: [mg60048] Exporting and importing arbitrary expressions efficiently
  • From: "J. McKenzie Alexander" <jalex at lse.ac.uk>
  • Date: Tue, 30 Aug 2005 04:43:06 -0400 (EDT)
  • Sender: owner-wri-mathgroup at wolfram.com

Mathematica 5.1 says it has highly optimized import and export of  
binary data, plus automated encoding and decoding of .gz files.  If I  
have Mathematica-generated data that consists of complex lists of  
(effectively) arbitrary expressions, what is the best way to save  
this to a data file and  read that data file back in later?

The best solution I've found for *exporting* is to use Export, saving  
to a .m.gz file.  For example,

> In[1]:=
> <<DiscreteMath`Combinatorica`
>
> In[2]:=
> Timing[
>   Export["~/filetest.m.gz",
>     Table[{RandomGraph[10,.3],Table[Random[],{25}]},{1000}]
>     ]
>   ]
>
> Out[2]=
> {2.63919 Second,~/filetest.m.gz}

I don't know if that's the highly optimized export that was touted,  
but it's certainly perfectly fine. But what's the "highly optimized"  
way to *import* this data?  Using Import is painfully slow.  For  
example,

> In[3]:=
> Timing[
>   data=Import["~/filetest.m.gz"];
>   ]
>
> Out[3]=
> {41.3851 Second,Null}

If I gunzip the file by hand, Get pulls the file in about 80 times  
*faster* than import.

> In[4]:=
> Timing[
>   data1=Get["~/filetest.m"];
>   ]
>
> Out[4]=
> {0.500501 Second,Null}

Granted, Get doesn't have to gunzip the file, but there's no way that  
should take so long. Timing gunzip on the same file from the command  
line says it takes only a fraction of a second:

> evolve: ~
> $time gunzip filetest.m.gz
>
> real    0m0.049s
> user    0m0.025s
> sys     0m0.015s

Any pointers on what I'm doing wrong would be appreciated.  I'm  
running v 5.1 on Mac OS X 10.4.

Cheers,

Jason
--
J. McKenzie Alexander
Department of Philosophy, Logic and Scientific Method
London School of Economics and Political Science
Houghton Street, London WC2A 2AE



  • Prev by Date: Re: List Replace Problems
  • Next by Date: Re: Re: How to solve this type of equation in Mathematica?
  • Previous by thread: Re: Filemaker 7 database
  • Next by thread: Pictures into Mathematica