Re: Unexpected behaviour of HoldRest

*To*: mathgroup at smc.vnet.net*Subject*: [mg45821] Re: Unexpected behaviour of HoldRest*From*: John Tanner <john at janacek.demon.co.uk>*Date*: Mon, 26 Jan 2004 01:53:28 -0500 (EST)*References*: <200401210954.EAA07748@smc.vnet.net> <buo2ml$h76$1@smc.vnet.net>*Sender*: owner-wri-mathgroup at wolfram.com

Thank you, Andrej and Hartmut, you confirm my suspicions that Sequence is tricky and dangerous to use. I do however still have problems that are not resolved by the alternatives as I understand them at present. I rather regret now focusing on HoldRest as the issue here. There is a rather more general problem with selection from lists. The optimum list selection strategy has been addressed most extensively in a thread on DeleteCases (started March 2001) and the basic options were well aired then, with the main options being: Select DeleteCases (or Cases) Position The primary discrimination between these was in terms of the Timing, and here Select still seems to be marginally preferable to DeleteCases/Cases and distinctly faster than Position. My present problem is however with the internal memory allocation, such as reported by MaxMemoryUsed[], and in this case I thought Sequence would help. The problem comes when there is a large amount of data to be processed, with only selected parts returned. The specific case I have found is when selecting immediately on calculation, so using a similar example to that used previously for DeleteCases: CASE 1 - Sequence[] In[1]:= Unprotect[If]; SetAttributes[If,SequenceHold]; Protect[If]; Timing[SeedRandom[123456]; ranpick1=Table[x=Random[];If[x>0.01,Sequence[],x],{1000000}]; MaxMemoryUsed[]] Out[4]= {31.876 Second,6343872} CASE 2 - Hold[] In[1]:= Timing[SeedRandom[123456]; ranpick2=Table[x=Random[];ReleaseHold[If[x>0.01,Hold[],x]],{1000000}]; MaxMemoryUsed[]] Out[1]= {47.288 Second,35045032} CASE 3 - Select In[1]:= Timing[SeedRandom[123456]; ranpick3=Select[Table[Random[],{1000000}],#<=0.01&]; MaxMemoryUsed[]] Out[1]= {18.486 Second,35112256} CASE 4 - Cases In[1]:= Timing[SeedRandom[123456]; ranpick4=Cases[Table[Random[],{1000000}],_?(#<=0.01&)]; MaxMemoryUsed[]] Out[1]= {24.465 Second,35115264} All of the above of course return identical results, and as UnPacked arrays. Timing results are for a 266MHz Pentium II running Windows NT. advantages over Hold[]. For raw Timing, Select and Cases are quicker than both, but also have very high MaxMemoryUsed[] results. However the use of Sequence[] here does require that If is given the Attribute SequenceHold so, given the limited application of the technique and the risks involved, I have now regretfully abandoned this approach and I am also reviewing the use of Sequence elsewhere. Actually (to be honest) I was hoping for equivalent improvements in MaxMemoryUsed[] when operating on big PackedArrays of stored data, but here the results are far worse when using Sequence[] than when using Select: In[1]:= Unprotect[If]; SetAttributes[If,SequenceHold]; Protect[If]; Timing[SeedRandom[123456]; randata=Table[Random[],{1000000}]; ranpick1a=If[#>0.01,Sequence[],#]& /@ randata; MaxMemoryUsed[]] Out[4]= {26.868 Second,63143384} Basically I do need a new computer with LOTS more memory... John Tanner. In article <buo2ml$h76$1 at smc.vnet.net>, Andrzej Kozlowski <akoz at mimuw.edu.pl> writes >It may be of some interest to point out that one can often use Hold and >ReleaseHold instead of Sequence, with similar effect (and sometimes >with fewer problems). > >For example: > > >ReleaseHold[{{0, 1, 2, 3, 2}, 3, 2} /. 2 -> Hold[]] > > >{{0,1,3},3} > >Carl Woll's OrderedUnion can also be written using this approach: > >OrderedUnion[li_] := ReleaseHold[Block[{i}, > i[n_] := (i[n] = Hold[]; n); i /@ li]] > > >OrderedUnion[{1,4,3,17,3,5,5,17,11,4}] > >{1,4,3,17,5,11} > >(I have not tried comparing its performance with Carl's original >version). > >Andrzej Kozlowski >Chiba, Japan >http://www.mimuw.edu.pl/~akoz/ > > > > >On 21 Jan 2004, at 09:54, Wolf, Hartmut wrote: > >> John, >> >> you'r right, as Sequence[] should be used with care. Up to Version 4 >> it was >> an undocumented function and the only mention to it I found was from >> Roman >> Maeder (who is, or was an insider; and Carl Woll certainly had read his >> books). >> >> Almost ever the use of Sequence can be avoided (although it might come >> handy), and in this group its use has been popularized more than it >> deserves. Even OrderedUnion -- which was a stroke of genius at its >> time -- >> nowadays with the modern versions of Mathematica (packed arrays, >> improved >> Split, Ordering) clearly no longer is the fastest way to do it! >> >> Sequence should be used with care and presupposes an exact knowledge >> and >> understanding of the Mathematica evaluation sequence. >> >> (Further comments interspersed) >> >> >> >>> -----Original Message----- >>> From: John Tanner [mailto:john at janacek.demon.co.uk] To: mathgroup at smc.vnet.net >> To: mathgroup at smc.vnet.net >>> Sent: Monday, January 19, 2004 11:15 AM >>> To: mathgroup at smc.vnet.net >>> Subject: [mg45821] Unexpected behaviour of HoldRest >>> >>> >>> Inspired by Carl Woll's OrderedUnion (for which many, many thanks!) I >>> have found similar uses for Sequence[], such as using >>> replacement rules: >>> >>> In[1]:= {{0,1,2,3,2},3,2} /. 2->Sequence[] >>> >>> Out[1]= {{0, 1, 3}, 3} >>> >> >> The right way to do that: >> >> In[2]:= DeleteCases[{{0, 1, 2, 3, 2}, 3, 2}, 2, 2] >> Out[2]= {{0, 1, 3}, 3} >> >> >>> But I recently found an "unexpected" result: >>> >>> In[1]:= If[1 == 1, Sequence[], 2] >>> >>> Out[1]= 2 >>> >> >> Well, it should be expected, Sequence will be "flattened out" when the >> arguments are evaluated _before_ the expression If itself is being >> evaluated >> (unless it had the SequenceHold or HoldAllComplete attribute, but it >> has >> neither of them) >> >> In[4]:= Attributes[If] >> Out[4]= {HoldRest, Protected} >> >> Such in a first step >> >> If[1 == 1, Sequence[], 2] >> >> is evaluated to >> >> If[True, Sequence[], 2] >> >> now Sequence objects will be flattened out irrespectively of HoldRest, >> therefore the expression is transformed to >> >> If[True, 2] >> >> only then rules for If are applied, hence Out[1] >> >> >>> Now when presented in this form it is "obvious", that Sequence[] is >>> being "flattened out" of the arguments to If. The wanted behaviour of >>> If in this case can be restored if SequenceHold is applied: >>> >>> In[3]:= Unprotect[If]; >>> SetAttributes[If, SequenceHold]; >>> Protect[If]; >>> >>> In[6]:= If[1 == 1, Sequence[], 2] >>> >>> Out[6]= Sequence[] >>> >> >> Dont't do that! We cannot overview the consequences of such a global >> intervention. Instead act locally: >> >> In[7]:= If[1 == 1, Unevaluated[Sequence[]], 2] >> Out[7]= Sequence[] >> >> >> This now allows to do other things, e.g. >> >> In[3]:= Map[If[# === 2, Unevaluated[Sequence[]], #] &, >> {{0, 1, 2, 3, 2}, 3, 2}, 2] >> Out[3]= {{0, 1, 3}, 3} >> >> >> >>> But If has (I now find) the Attribute HoldRest, and Hold does not >>> flatten out Sequence[] so I don't understand why that does not apply >>> here also. >>> >> >> Hold has the attributes >> >> In[6]:= Attributes[Hold] >> Out[6]= {HoldAll, Protected} >> >> hence it flattens out Sequence[..] >> >> In[5]:= Hold[Sequence[], If[1 == 1, Sequence[], 2]] >> Out[5]= Hold[If[1 == 1, Sequence[], 2]] >> >> (Of course there will be no operation for and _within_ the held >> expression.) >> >> >>> In[7]:= xx[1+2,2+3,3+4] >>> >>> Out[7]= xx[3,5,7] >>> >>> In[8]:= SetAttributes[xx,HoldRest] >>> >>> In[9]:= xx[1+2,2+3,Sequence[],3+4] >>> >>> Out[9]= xx[3,2+3,3+4] >>> >> >> You just showed that HoldRest has nothing to do with all that. >> >> >>> Relatively few built-in functions have SequenceHold applied: >>> >>> {AbsoluteTiming, Rule, RuleDelayed, Set, SetDelayed, TagSet, >>> TagSetDelayed, Timing, UpSet, UpSetDelayed} >>> >>> And also relatively few built-in functions have HoldRest applied: >>> >>> {Assuming, DumpSave, If, PatternTest, RuleDelayed, Save, Switch} >>> >> >> Yes, the main reason is for so-called lazy evaluation. Would you >> imagine >> what happend when If lacked this attribute? Not only that many parts >> of the >> expression had to be evaluated in vain, and never used for the result, >> also >> side effects would become completely unpredictable (more than is >> anyway). >> >> >> >>> I will therefore now be much more careful of the use of Sequence! At >>> present I don't want to be more daring and apply these attributes >>> wholesale without understanding why this is happening. There must be >>> other cases where Sequence can have unexpected effects - I now >>> understand some of the problems I had when trying to use to >>> modify $Post - >>> reallyBig needs to cope with Sequence[] and Sequence[__]. >>> >>> -- >>> from- John Tanner email- john at janacek.demon.co.uk >>> mantra- curse Microsoft, curse... web - >>> http%##www.janacek.demon.co.uk/ >>> I hate this 'orrible computer, I really ought to sell it: >>> It never does what I want, but only what I tell it. >>> >> >> >> I agree, see also the Book about evaluation (e.g. §A.4.1) >> >> -- >> Hartmut Wolf >> >> >> > -- from- John Tanner email- john at janacek.demon.co.uk mantra- curse Microsoft, curse... web - http^%%www.janacek.demon.co.uk/ I hate this 'orrible computer, I really ought to sell it: It never does what I want, but only what I tell it.

**Follow-Ups**:**Re: Re: Unexpected behaviour of HoldRest***From:*Daniel Lichtblau <danl@wolfram.com>

**References**:**RE: Unexpected behaviour of HoldRest***From:*"Wolf, Hartmut" <Hartmut.Wolf@t-systems.com>