Exporting and importing arbitrary expressions efficiently
- To: mathgroup at smc.vnet.net
- Subject: [mg60048] Exporting and importing arbitrary expressions efficiently
- From: "J. McKenzie Alexander" <jalex at lse.ac.uk>
- Date: Tue, 30 Aug 2005 04:43:06 -0400 (EDT)
- Sender: owner-wri-mathgroup at wolfram.com
Mathematica 5.1 says it has highly optimized import and export of
binary data, plus automated encoding and decoding of .gz files. If I
have Mathematica-generated data that consists of complex lists of
(effectively) arbitrary expressions, what is the best way to save
this to a data file and read that data file back in later?
The best solution I've found for *exporting* is to use Export, saving
to a .m.gz file. For example,
> In[1]:=
> <<DiscreteMath`Combinatorica`
>
> In[2]:=
> Timing[
> Export["~/filetest.m.gz",
> Table[{RandomGraph[10,.3],Table[Random[],{25}]},{1000}]
> ]
> ]
>
> Out[2]=
> {2.63919 Second,~/filetest.m.gz}
I don't know if that's the highly optimized export that was touted,
but it's certainly perfectly fine. But what's the "highly optimized"
way to *import* this data? Using Import is painfully slow. For
example,
> In[3]:=
> Timing[
> data=Import["~/filetest.m.gz"];
> ]
>
> Out[3]=
> {41.3851 Second,Null}
If I gunzip the file by hand, Get pulls the file in about 80 times
*faster* than import.
> In[4]:=
> Timing[
> data1=Get["~/filetest.m"];
> ]
>
> Out[4]=
> {0.500501 Second,Null}
Granted, Get doesn't have to gunzip the file, but there's no way that
should take so long. Timing gunzip on the same file from the command
line says it takes only a fraction of a second:
> evolve: ~
> $time gunzip filetest.m.gz
>
> real 0m0.049s
> user 0m0.025s
> sys 0m0.015s
Any pointers on what I'm doing wrong would be appreciated. I'm
running v 5.1 on Mac OS X 10.4.
Cheers,
Jason
--
J. McKenzie Alexander
Department of Philosophy, Logic and Scientific Method
London School of Economics and Political Science
Houghton Street, London WC2A 2AE