Re: Importing data
- To: mathgroup at smc.vnet.net
- Subject: [mg113424] Re: Importing data
- From: David Bailey <dave at removedbailey.co.uk>
- Date: Thu, 28 Oct 2010 04:26:28 -0400 (EDT)
- References: <ia8qp5$1vb$1@smc.vnet.net>
On 27/10/10 10:18, David Higgins wrote:
> Hi all,
>
> Relatively new user here. I have a text file containing 540,000 lines of
> data in 30 columns, fixed width. Not all columns are populated for all data
> lines.
>
> When I import, Mathematica creates a list and populates the list elements
> from 1 to the number of data elements it finds in each line, ignoring the
> empty columns so for data lines with only, say 20 columns populated, it has
> populated data elements 1 to 20 instead of skipping the missing ones. Is
> there a way to import data that allows me to define a template for the data
> elements so that the right data ends up in the right elements in the list?
>
> Cheers
>
> David
>
>
The format you describe is certainly not ideally suited for input into
Mathematica. Comma separated format (.CSV) where the empty elements were
marked by adjacent commas, would be easy to read with Import.
However, assuming you can't easily change the way the data is formatted,
you can read it in via strings. Here I am using a data file with 3
columns, 2 rows, with data aligned with 10 characters per item:
12.1 4.5 9.1
16.9 8.8
res = ReadList["~/xxx/test.dat", String];
res = Map[(Partition[PadRight[Characters[#], 30, " "], 10]) &, res];
res=Map[ToExpression[StringJoin @@ #] &, res, {2}]
{{12.1, 4.5, 9.1}, {16.9, Null, 8.8}}
ToExpression converts a string into a Mathematica expression, and it
converts an all-blank string into Null. If you want the missing items to
be stored as something else, such as 0.0, you can change them afterwards:
res=res/.Null->0.0
Notice the 10 and the 30 in the above code, and adjust as required.
David Bailey
http://www.dbaileyconsultancy.co.uk