Cautionary tale : Remember to close StringToStream...

*To*: mathgroup at smc.vnet.net*Subject*: [mg124767] Cautionary tale : Remember to close StringToStream...*From*: Roger Wilson <rogerhw999 at gmail.com>*Date*: Sat, 4 Feb 2012 06:34:03 -0500 (EST)*Delivered-to*: l-mathgroup@mail-archive0.wolfram.com*Reply-to*: comp.soft-sys.math.mathematica at googlegroups.com

I some code running extremely slow and finally traced the problem to a division by zero which should take very little time... 2/3/12 10:10:15 In[2]:= Timing[1.0/0.0] 2/3/12 10:10:15 During evaluation of In[2]:= Power::infy: Infinite expression 1/0. encountered. >> 2/3/12 10:10:15 Out[2]= {0.,ComplexInfinity} However I was experiencing something like this... 2/3/12 10:11:04 In[4]:= Timing[1.0/0.0] 2/3/12 10:11:12 During evaluation of In[4]:= Power::infy: Infinite expression 1/0. encountered. >> 2/3/12 10:11:12 Out[4]= {7.75,ComplexInfinity} As the Barman said to the Bear "Why the big paws?"... I traced the problem to the way I was reading in my data. I had loaded a big file as a string, then having split this into separate items ran a pattern search for everything which looked like a number and then used Read[StringToStream[x], Number] to convert those items into numbers. You will note that I did not close the streams created by StringToStream. It appears that the time taken to process Message scales approximately as the square of the number of open streams. Not a problem with say 1000 streams open but with 30,000 streams left open, on my machine, results in a few seconds of delay. With hundreds of thousands of open streams my code just stopped. Turning the warning message off... 2/3/12 10:15:43 In[5]:= Off[Power::infy]; Timing[1.0/0.0] 2/3/12 10:15:43 Out[5]= {0.,ComplexInfinity} Sidesteps the problem, so it appears that it is a stream issue which makes some kind of sense as Messages come down a stream of their own. In the future I will be more careful but this amused me (once I'd stopped bashing my head into the desk) so I thought I'd share it. ReadClose[str_String]:=Reap[Apply[(Sow[Read[#,Number]]; Close[#])&,{StringToStream[str]}]][[2,1,1]];