Cautionary tale : Remember to close StringToStream...
- To: mathgroup at smc.vnet.net
- Subject: [mg124767] Cautionary tale : Remember to close StringToStream...
- From: Roger Wilson <rogerhw999 at gmail.com>
- Date: Sat, 4 Feb 2012 06:34:03 -0500 (EST)
- Delivered-to: l-mathgroup@mail-archive0.wolfram.com
- Reply-to: comp.soft-sys.math.mathematica at googlegroups.com
I some code running extremely slow and finally traced the problem to a division by zero which should take very little time...
2/3/12 10:10:15 In[2]:= Timing[1.0/0.0]
2/3/12 10:10:15 During evaluation of In[2]:= Power::infy: Infinite expression 1/0. encountered. >>
2/3/12 10:10:15 Out[2]= {0.,ComplexInfinity}
However I was experiencing something like this...
2/3/12 10:11:04 In[4]:= Timing[1.0/0.0]
2/3/12 10:11:12 During evaluation of In[4]:= Power::infy: Infinite expression 1/0. encountered. >>
2/3/12 10:11:12 Out[4]= {7.75,ComplexInfinity}
As the Barman said to the Bear "Why the big paws?"...
I traced the problem to the way I was reading in my data. I had loaded a big file as a string, then having split this into separate items ran a pattern search for everything which looked like a number and then used Read[StringToStream[x], Number] to convert those items into numbers. You will note that I did not close the streams created by StringToStream.
It appears that the time taken to process Message scales approximately as the square of the number of open streams. Not a problem with say 1000 streams open but with 30,000 streams left open, on my machine, results in a few seconds of delay. With hundreds of thousands of open streams my code just stopped.
Turning the warning message off...
2/3/12 10:15:43 In[5]:= Off[Power::infy]; Timing[1.0/0.0]
2/3/12 10:15:43 Out[5]= {0.,ComplexInfinity}
Sidesteps the problem, so it appears that it is a stream issue which makes some kind of sense as Messages come down a stream of their own.
In the future I will be more careful but this amused me (once I'd stopped bashing my head into the desk) so I thought I'd share it.
ReadClose[str_String]:=Reap[Apply[(Sow[Read[#,Number]]; Close[#])&,{StringToStream[str]}]][[2,1,1]];