Re: Working with Indeterminate in large numerical lists

*To*: mathgroup at smc.vnet.net*Subject*: [mg100227] Re: [mg100207] Working with Indeterminate in large numerical lists*From*: Bob Hanlon <hanlonr at cox.net>*Date*: Thu, 28 May 2009 19:33:17 -0400 (EDT)*Reply-to*: hanlonr at cox.net

$HistoryLength = 0; n = 1000; x = RandomSample[Join[RandomReal[1, 100 n], ConstantArray[Indeterminate, n]]]; Timing[a1 = Select[x, ! NumberQ[#] &];] {0.096885,Null} However, note that NumberQ /@ {1, 1., Pi, E, I} {True,True,False,False,True} NumericQ /@ {1, 1., Pi, E, I} {True,True,True,True,True} So in most cases, you would want to use NumericQ rather than NumberQ Timing[a2 = Select[x, ! NumericQ[#] &];] {0.096093,Null} Timing[a3 = Select[x, # === Indeterminate &];] {0.069981,Null} Oddly, this is faster Timing[a4 = x[[Flatten[Position[x, Indeterminate]]]];] {0.016217,Null} Cases appears to be much more efficient than Select in this case Timing[a5 = Cases[x, Indeterminate];] {0.007767,Null} a1 === a2 === a3 === a4 === a5 === ConstantArray[Indeterminate, n] True However, for valid entries Timing[b1 = Select[x, NumericQ];] {0.035489,Null} Timing[b2 = Cases[x, _?NumericQ];] {0.064408,Null} b1 === b2 True Bob Hanlon ---- pfalloon <pfalloon at gmail.com> wrote: ============= Hi everyone, I'm wondering about the optimal way to work with Indeterminate in large matrices. I've been using this to replace "bad" data points that I want to prevent from polluting calculations involving lists of data, but I'm not sure I'm working as smartly as I could be. As an example, suppose I have a list of machine-precision reals and some Indeterminate elements: x = RandomSample[Join[RandomReal[1, 1000], ConstantArray [Indeterminate, 10]]; If I want to take only the valid entries, the best I have been able to find is something like: Select[x, NumberQ] This seems to work reasonably well. But if I want to specifically select the Indeterminate entries, there doesn't seem to be any function (equivalent to, say, "isnan" from another system), so I have to resort to something less succinct like Select[x, # === Indeterminate &] Does anyone have any suggestions on better ways to do this, or any general tips for working with Indeterminate in this context? I'm particularly keen to do things in the most efficient way possible as I'm working with rather large lists. Thanks, Peter.