Mathematica 9 is now available
Student Support Forum
-----
Student Support Forum: 'Importing large amounts of data' topicStudent Support Forum > General > "Importing large amounts of data"

Next Comment >Help | Reply To Topic
Author Comment/Response
David
08/01/02 1:11pm

Hi, I have about 1 gigabyte of data to import and process, and I'm wondering if there's some way in mathematica to speed up this process.

Each file is only about 6-7 megabytes large, and I'm currently using the:

data = Import["day117_361.dat"];

to get the data. It's taking about 15 minutes to import one file, which is quite tedious. The file is organized into a table, 19 columns and 50,000+ rows of numbers less than 200.
Here's a sample of my data:

1 4.29 10.22 0.36 0.27 0.45 0.27 53.00 87. 19. 146.
2 2.24 2.70 12.64 18.64 16.09 0.64 60.00 170. 3. 255.
3 0.58 0.72 3.27 6.82 28.64 131.45 53.00 8. 18. 12.
4 0.15 0.40 27.45 2.73 0.00 0.00 30.00 8. 22. 135.
5 2.18 2.92 32.82 0.09 1.27 0.27 55.00 7. 1. 142.

Of course, with 19 columns, but it won't fit on this page and show up correctly.

I'm wondering if there's some way to save this data table as a Mathematica file so that it will load faster, or something. I know that when I try to open the actual data file in Mathematica by hand, it takes about 10 seconds max to open.

Is there any way to load my files in a quicker fashion? thanks

URL: ,

Subject (listing for 'Importing large amounts of data')
Author Date Posted
Importing large amounts of data David 08/01/02 1:11pm
Re: Importing large amounts of data Forum Modera... 08/06/02 06:09am
Next Comment >Help | Reply To Topic