Re: Importing large file into table and calculating takes a long time. How to improve efficiency?

*To*: mathgroup at smc.vnet.net*Subject*: [mg126340] Re: Importing large file into table and calculating takes a long time. How to improve efficiency?*From*: Bob Hanlon <hanlonr357 at gmail.com>*Date*: Wed, 2 May 2012 05:45:47 -0400 (EDT)*Delivered-to*: l-mathgroup@mail-archive0.wolfram.com

data = {{0., 24, "009d"}, {0., 28, 9}, {0., 28, 99}, {"00dc", 27, 98}, {0., 29, 95}}; dataHex = data /. {0. -> "0", x_?NumericQ :> ToString[x]}; dataDec = Map[FromDigits[#, 16] &, dataHex, {2}]; dataDecCompl = Map[If[# > 32768, # - 65536, #] &, dataDec, {2}]; dataG = MovingAverage[Norm /@ dataDecCompl // N, 3]; plotG = ListLinePlot[dataG, PlotRange -> All] Bob Hanlon On Tue, May 1, 2012 at 3:55 PM, j p <ipschka at yahoo.com> wrote: > Thanks Bob for your suggestions. I certainly like the Map and Norm functions > and I knew there had to be a better way to convert from hex to decimal. You > were right about your assumption about not defining dataX, dataY, and dataZ, > I had left those out by mistake from the code I included in my original > question. > > But when I use the line > > Map[FromDigits[#, 16] &, dataHex, {2}] > > I get the error message: > > FromDigits::nlst: The expression 0.` is not a list of digits or a > string of valid digits. > > The input data may look something like this: > > {{0., 24, "009d"}, {0., 28, 9}, {0., 28, 99}, {"00dc", 27, 98}, {0., > 29, 95}, > > where there is a decimal point when the value is zero. So I added the > following to the import function to replace all occurrences of "0.": > . > dataHex = Import["C:\\Projects\\Mathematica\\Trainings\\test.csv"] //. > {0. -> 0} > > But that did not work, I still get the message: > > FromDigits::nlst: The expression 0 is not a list of digits or a string > of valid digits. > > It turns out I get the same error for the number 24 as well (in the sample > data above), but not the hex number 009d. So there must be a difference in > the numeric vs non-numeric inputs. > > Thanks again, > > Jan Ingsen > Univ of Colorado > > > > ----- Original Message ----- > From: Bob Hanlon <hanlonr357 at gmail.com> > To: Gangerolv <ipschka at yahoo.com> > Cc: mathgroup at smc.vnet.net > Sent: Tuesday, May 1, 2012 10:45 AM > Subject: Re: Importing large file into table and calculating > takes a long time. How to improve efficiency? > > data = RandomInteger[100000, {10, 3}]; > > dataHex = IntegerString[data, 16]; > > dataDec = Map[FromDigits[#, 16] &, dataHex, {2}]; > > dataDecCompl = Map[If[# > 32768, # - 65536, #] &, dataDec, {2}]; > > You did not define dataX, dataY, dataZ, and filter. The following > assumes that dataX, dataY, and dataZ are the components of elements of > dataDecCompl and that filter =0. If not, modify as required. > > Since > > Norm[{x, y, z}] > > Sqrt[Abs[x]^2 + Abs[y]^2 + > Abs[z]^2] > > then > > dataG = MovingAverage[Norm /@ dataDecCompl // N, 3]; > > plotG = ListLinePlot[dataG, PlotRange -> All] > > > Bob Hanlon > > > On Tue, May 1, 2012 at 5:24 AM, Gangerolv <ipschka at yahoo.com> wrote: >> (First my disclaimer, I'm new to mathematica) >> >> I'm importing a file with three values, x,y,z in hexadecimal. >> Sample of the input data: {0., 24, "009d"}, {0., 28, 9}, {0., 28, 99}, >> {"00dc", 27, 98}, {0., 29, 95},... >> >> This set is converted to integer: {0., 36, 157}, {0., 40, 9}, {0., 40, >> 153}, {220, 39, 152}, {0., 41, 149},... >> >> A final vector is calculated using: g=sqrt(x^2 + y^2 + z^2) for each data >> set. >> >> The data is then plotted. >> >> All of this is good an works great for a small file (2000 data sets). But >> when I try to import and calculate a larger file (over 100k data sets), it >> seems to take forever. Either my methods are not efficient (use of Table), >> or I'm not using correct settings for importing of the data. The file is >> only 2Mb so I know mathematica should be able to handle it. >> >> Here's what I'm doing: >> >> ========================================================================= >> dataHex = >> Import["C:\\Projects\\Mathematica\\test.csv"] >> points = Length[dataHex] >> dataDec = Table[ >> ToExpression["16^^" <> #] & /@ { >> ToString[dataHex[[i, 1]]], >> ToString[dataHex[[i, 2]]], >> ToString[dataHex[[i, 3]]]}, >> {i, 1, points}] >> dataDecCompl = Table[{ >> If[dataDec[[i, 1]] > 32768, dataDec[[i, 1]] - 65536, dataDec[[i, 1]]], >> If[dataDec[[i, 2]] > 32768, dataDec[[i, 2]] - 65536, dataDec[[i, 2]]], >> If[dataDec[[i, 3]] > 32768, dataDec[[i, 2]] - 65536, dataDec[[i, 3]]]}, >> {i, 1, points}]; >> >> dataG = MovingAverage[Table[ >> Sqrt[dataX[[i]]^2 + dataY[[i]]^2 + dataZ[[i]]^2] // N, >> {i, 1, points - filter}], >> 3]; >> plotG = ListLinePlot[{dataG}, PlotRange -> All]; >> >> ========================================================================= >> >> Any suggestions would be highly appreciated. Using M 8.0.4 on Win7. >> >> Cheers! >> >> jan >>