Re: Importing large file into table and calculating takes a long time. How to improve efficiency?
- To: mathgroup at smc.vnet.net
- Subject: [mg126340] Re: Importing large file into table and calculating takes a long time. How to improve efficiency?
- From: Bob Hanlon <hanlonr357 at gmail.com>
- Date: Thu, 3 May 2012 04:33:33 -0400 (EDT)
- Delivered-to: l-mathgroup@mail-archive0.wolfram.com
data = {{0., 24, "009d"}, {0., 28, 9}, {0., 28, 99}, {"00dc", 27,
98}, {0., 29, 95}};
dataHex = data /. {0. -> "0", x_?NumericQ :> ToString[x]};
dataDec = Map[FromDigits[#, 16] &, dataHex, {2}];
dataDecCompl = Map[If[# > 32768, # - 65536, #] &, dataDec, {2}];
dataG = MovingAverage[Norm /@ dataDecCompl // N, 3];
plotG = ListLinePlot[dataG, PlotRange -> All]
Bob Hanlon
On Tue, May 1, 2012 at 3:55 PM, j p <ipschka at yahoo.com> wrote:
> Thanks Bob for your suggestions. I certainly like the Map and Norm functions
> and I knew there had to be a better way to convert from hex to decimal. You
> were right about your assumption about not defining dataX, dataY, and dataZ,
> I had left those out by mistake from the code I included in my original
> question.
>
> But when I use the line
>
> Map[FromDigits[#, 16] &, dataHex, {2}]
>
> I get the error message:
>
> FromDigits::nlst: The expression 0.` is not a list of digits or a
> string of valid digits.
>
> The input data may look something like this:
>
> {{0., 24, "009d"}, {0., 28, 9}, {0., 28, 99}, {"00dc", 27, 98}, {0.,
> 29, 95},
>
> where there is a decimal point when the value is zero. So I added the
> following to the import function to replace all occurrences of "0.":
> .
> dataHex = Import["C:\\Projects\\Mathematica\\Trainings\\test.csv"] //.
> {0. -> 0}
>
> But that did not work, I still get the message:
>
> FromDigits::nlst: The expression 0 is not a list of digits or a string
> of valid digits.
>
> It turns out I get the same error for the number 24 as well (in the sample
> data above), but not the hex number 009d. So there must be a difference in
> the numeric vs non-numeric inputs.
>
> Thanks again,
>
> Jan Ingsen
> Univ of Colorado
>
>
>
> ----- Original Message -----
> From: Bob Hanlon <hanlonr357 at gmail.com>
> To: Gangerolv <ipschka at yahoo.com>
> Cc: mathgroup at smc.vnet.net
> Sent: Tuesday, May 1, 2012 10:45 AM
> Subject: Re: Importing large file into table and calculating
> takes a long time. How to improve efficiency?
>
> data = RandomInteger[100000, {10, 3}];
>
> dataHex = IntegerString[data, 16];
>
> dataDec = Map[FromDigits[#, 16] &, dataHex, {2}];
>
> dataDecCompl = Map[If[# > 32768, # - 65536, #] &, dataDec, {2}];
>
> You did not define dataX, dataY, dataZ, and filter. The following
> assumes that dataX, dataY, and dataZ are the components of elements of
> dataDecCompl and that filter =0. If not, modify as required.
>
> Since
>
> Norm[{x, y, z}]
>
> Sqrt[Abs[x]^2 + Abs[y]^2 +
> Abs[z]^2]
>
> then
>
> dataG = MovingAverage[Norm /@ dataDecCompl // N, 3];
>
> plotG = ListLinePlot[dataG, PlotRange -> All]
>
>
> Bob Hanlon
>
>
> On Tue, May 1, 2012 at 5:24 AM, Gangerolv <ipschka at yahoo.com> wrote:
>> (First my disclaimer, I'm new to mathematica)
>>
>> I'm importing a file with three values, x,y,z in hexadecimal.
>> Sample of the input data: {0., 24, "009d"}, {0., 28, 9}, {0., 28, 99},
>> {"00dc", 27, 98}, {0., 29, 95},...
>>
>> This set is converted to integer: {0., 36, 157}, {0., 40, 9}, {0., 40,
>> 153}, {220, 39, 152}, {0., 41, 149},...
>>
>> A final vector is calculated using: g=sqrt(x^2 + y^2 + z^2) for each data
>> set.
>>
>> The data is then plotted.
>>
>> All of this is good an works great for a small file (2000 data sets). But
>> when I try to import and calculate a larger file (over 100k data sets), it
>> seems to take forever. Either my methods are not efficient (use of Table),
>> or I'm not using correct settings for importing of the data. The file is
>> only 2Mb so I know mathematica should be able to handle it.
>>
>> Here's what I'm doing:
>>
>> =========================================================================
>> dataHex =
>> Import["C:\\Projects\\Mathematica\\test.csv"]
>> points = Length[dataHex]
>> dataDec = Table[
>> ToExpression["16^^" <> #] & /@ {
>> ToString[dataHex[[i, 1]]],
>> ToString[dataHex[[i, 2]]],
>> ToString[dataHex[[i, 3]]]},
>> {i, 1, points}]
>> dataDecCompl = Table[{
>> If[dataDec[[i, 1]] > 32768, dataDec[[i, 1]] - 65536, dataDec[[i, 1]]],
>> If[dataDec[[i, 2]] > 32768, dataDec[[i, 2]] - 65536, dataDec[[i, 2]]],
>> If[dataDec[[i, 3]] > 32768, dataDec[[i, 2]] - 65536, dataDec[[i, 3]]]},
>> {i, 1, points}];
>>
>> dataG = MovingAverage[Table[
>> Sqrt[dataX[[i]]^2 + dataY[[i]]^2 + dataZ[[i]]^2] // N,
>> {i, 1, points - filter}],
>> 3];
>> plotG = ListLinePlot[{dataG}, PlotRange -> All];
>>
>> =========================================================================
>>
>> Any suggestions would be highly appreciated. Using M 8.0.4 on Win7.
>>
>> Cheers!
>>
>> jan
>>