[Date Index] [Thread Index] [Author Index]

Re: Importing large file into table and calculating takes a long time. How to improve efficiency?

• To: mathgroup at smc.vnet.net
• Subject: [mg126333] Re: Importing large file into table and calculating takes a long time. How to improve efficiency?
• From: Šerých Jakub <Serych at panska.cz>
• Date: Tue, 1 May 2012 14:58:14 -0400 (EDT)
• Delivered-to: l-mathgroup@mail-archive0.wolfram.com

I haven't studied your code, but definitely the first error I can see is, that normal asignment is "=" not "==" in Mathematica.


Jakub

> -----Original Message-----
> From: Gangerolv [mailto:ipschka at yahoo.com]
> Sent: Tuesday, May 01, 2012 11:24 AM
> To: mathgroup at smc.vnet.net
> Subject: Importing large file into table and calculating takes a
> long time. How to improve efficiency?
>
> (First my disclaimer, I'm new to mathematica)
>
> I'm importing a file with three values, x,y,z in hexadecimal.
> Sample of the input data: {0., 24, "009d"}, {0., 28, 9}, {0., 28, 99}, {"00dc",
> 27, 98}, {0., 29, 95},...
>
> This set is converted to integer: {0., 36, 157}, {0., 40, 9}, {0., 40, 153}, {220,
> 39, 152}, {0., 41, 149},...
>
> A final vector is calculated using: g==sqrt(x^2 + y^2 + z^2) for each data set.
>
> The data is then plotted.
>
> All of this is good an works great for a small file (2000 data sets). But when I
> try to import and calculate a larger file (over 100k data sets), it seems to take
> forever. Either my methods are not efficient (use of Table), or I'm not using
> correct settings for importing of the data. The file is only 2Mb so I know
> mathematica should be able to handle it.
>
> Here's what I'm doing:
>
> =========================================================================
> dataHex ==
>  Import["C:\\Projects\\Mathematica\\test.csv"]
> points == Length[dataHex]
> dataDec == Table[
>   ToExpression["16^^" <> #] & /@ {
>     ToString[dataHex[[i, 1]]],
>     ToString[dataHex[[i, 2]]],
>     ToString[dataHex[[i, 3]]]},
>   {i, 1, points}]
> dataDecCompl == Table[{
>     If[dataDec[[i, 1]] > 32768, dataDec[[i, 1]] - 65536, dataDec[[i, 1]]],
>     If[dataDec[[i, 2]] > 32768, dataDec[[i, 2]] - 65536, dataDec[[i, 2]]],
>     If[dataDec[[i, 3]] > 32768, dataDec[[i, 2]] - 65536, dataDec[[i, 3]]]},
>    {i, 1, points}];
>
> dataG == MovingAverage[Table[
>     Sqrt[dataX[[i]]^2 + dataY[[i]]^2 + dataZ[[i]]^2] // N,
>     {i, 1, points - filter}],
>    3];
> plotG == ListLinePlot[{dataG}, PlotRange -> All];
> =========================================================================
>
> Any suggestions would be highly appreciated. Using M 8.0.4 on Win7.
>
> Cheers!
>
> jan
>