Re: Unexpected behaviour of HoldRest
- To: mathgroup at smc.vnet.net
- Subject: [mg45821] Re: Unexpected behaviour of HoldRest
- From: John Tanner <john at janacek.demon.co.uk>
- Date: Mon, 26 Jan 2004 01:53:28 -0500 (EST)
- References: <200401210954.EAA07748@smc.vnet.net> <buo2ml$h76$1@smc.vnet.net>
- Sender: owner-wri-mathgroup at wolfram.com
Thank you, Andrej and Hartmut, you confirm my suspicions that Sequence
is tricky and dangerous to use. I do however still have problems that
are not resolved by the alternatives as I understand them at present.
I rather regret now focusing on HoldRest as the issue here. There is a
rather more general problem with selection from lists. The optimum list
selection strategy has been addressed most extensively in a thread on
DeleteCases (started March 2001) and the basic options were well aired
then, with the main options being:
Select
DeleteCases (or Cases)
Position
The primary discrimination between these was in terms of the Timing, and
here Select still seems to be marginally preferable to DeleteCases/Cases
and distinctly faster than Position. My present problem is however with
the internal memory allocation, such as reported by MaxMemoryUsed[], and
in this case I thought Sequence would help.
The problem comes when there is a large amount of data to be processed,
with only selected parts returned. The specific case I have found is
when selecting immediately on calculation, so using a similar example to
that used previously for DeleteCases:
CASE 1 - Sequence[]
In[1]:=
Unprotect[If];
SetAttributes[If,SequenceHold];
Protect[If];
Timing[SeedRandom[123456];
ranpick1=Table[x=Random[];If[x>0.01,Sequence[],x],{1000000}];
MaxMemoryUsed[]]
Out[4]=
{31.876 Second,6343872}
CASE 2 - Hold[]
In[1]:=
Timing[SeedRandom[123456];
ranpick2=Table[x=Random[];ReleaseHold[If[x>0.01,Hold[],x]],{1000000}];
MaxMemoryUsed[]]
Out[1]=
{47.288 Second,35045032}
CASE 3 - Select
In[1]:=
Timing[SeedRandom[123456];
ranpick3=Select[Table[Random[],{1000000}],#<=0.01&];
MaxMemoryUsed[]]
Out[1]=
{18.486 Second,35112256}
CASE 4 - Cases
In[1]:=
Timing[SeedRandom[123456];
ranpick4=Cases[Table[Random[],{1000000}],_?(#<=0.01&)];
MaxMemoryUsed[]]
Out[1]=
{24.465 Second,35115264}
All of the above of course return identical results, and as UnPacked
arrays. Timing results are for a 266MHz Pentium II running Windows NT.
advantages over Hold[]. For raw Timing, Select and Cases are quicker
than both, but also have very high MaxMemoryUsed[] results.
However the use of Sequence[] here does require that If is given the
Attribute SequenceHold so, given the limited application of the
technique and the risks involved, I have now regretfully abandoned this
approach and I am also reviewing the use of Sequence elsewhere.
Actually (to be honest) I was hoping for equivalent improvements in
MaxMemoryUsed[] when operating on big PackedArrays of stored data, but
here the results are far worse when using Sequence[] than when using
Select:
In[1]:=
Unprotect[If];
SetAttributes[If,SequenceHold];
Protect[If];
Timing[SeedRandom[123456];
randata=Table[Random[],{1000000}];
ranpick1a=If[#>0.01,Sequence[],#]& /@ randata;
MaxMemoryUsed[]]
Out[4]=
{26.868 Second,63143384}
Basically I do need a new computer with LOTS more memory...
John Tanner.
In article <buo2ml$h76$1 at smc.vnet.net>, Andrzej Kozlowski
<akoz at mimuw.edu.pl> writes
>It may be of some interest to point out that one can often use Hold and
>ReleaseHold instead of Sequence, with similar effect (and sometimes
>with fewer problems).
>
>For example:
>
>
>ReleaseHold[{{0, 1, 2, 3, 2}, 3, 2} /. 2 -> Hold[]]
>
>
>{{0,1,3},3}
>
>Carl Woll's OrderedUnion can also be written using this approach:
>
>OrderedUnion[li_] := ReleaseHold[Block[{i},
> i[n_] := (i[n] = Hold[]; n); i /@ li]]
>
>
>OrderedUnion[{1,4,3,17,3,5,5,17,11,4}]
>
>{1,4,3,17,5,11}
>
>(I have not tried comparing its performance with Carl's original
>version).
>
>Andrzej Kozlowski
>Chiba, Japan
>http://www.mimuw.edu.pl/~akoz/
>
>
>
>
>On 21 Jan 2004, at 09:54, Wolf, Hartmut wrote:
>
>> John,
>>
>> you'r right, as Sequence[] should be used with care. Up to Version 4
>> it was
>> an undocumented function and the only mention to it I found was from
>> Roman
>> Maeder (who is, or was an insider; and Carl Woll certainly had read his
>> books).
>>
>> Almost ever the use of Sequence can be avoided (although it might come
>> handy), and in this group its use has been popularized more than it
>> deserves. Even OrderedUnion -- which was a stroke of genius at its
>> time --
>> nowadays with the modern versions of Mathematica (packed arrays,
>> improved
>> Split, Ordering) clearly no longer is the fastest way to do it!
>>
>> Sequence should be used with care and presupposes an exact knowledge
>> and
>> understanding of the Mathematica evaluation sequence.
>>
>> (Further comments interspersed)
>>
>>
>>
>>> -----Original Message-----
>>> From: John Tanner [mailto:john at janacek.demon.co.uk]
To: mathgroup at smc.vnet.net
>> To: mathgroup at smc.vnet.net
>>> Sent: Monday, January 19, 2004 11:15 AM
>>> To: mathgroup at smc.vnet.net
>>> Subject: [mg45821] Unexpected behaviour of HoldRest
>>>
>>>
>>> Inspired by Carl Woll's OrderedUnion (for which many, many thanks!) I
>>> have found similar uses for Sequence[], such as using
>>> replacement rules:
>>>
>>> In[1]:= {{0,1,2,3,2},3,2} /. 2->Sequence[]
>>>
>>> Out[1]= {{0, 1, 3}, 3}
>>>
>>
>> The right way to do that:
>>
>> In[2]:= DeleteCases[{{0, 1, 2, 3, 2}, 3, 2}, 2, 2]
>> Out[2]= {{0, 1, 3}, 3}
>>
>>
>>> But I recently found an "unexpected" result:
>>>
>>> In[1]:= If[1 == 1, Sequence[], 2]
>>>
>>> Out[1]= 2
>>>
>>
>> Well, it should be expected, Sequence will be "flattened out" when the
>> arguments are evaluated _before_ the expression If itself is being
>> evaluated
>> (unless it had the SequenceHold or HoldAllComplete attribute, but it
>> has
>> neither of them)
>>
>> In[4]:= Attributes[If]
>> Out[4]= {HoldRest, Protected}
>>
>> Such in a first step
>>
>> If[1 == 1, Sequence[], 2]
>>
>> is evaluated to
>>
>> If[True, Sequence[], 2]
>>
>> now Sequence objects will be flattened out irrespectively of HoldRest,
>> therefore the expression is transformed to
>>
>> If[True, 2]
>>
>> only then rules for If are applied, hence Out[1]
>>
>>
>>> Now when presented in this form it is "obvious", that Sequence[] is
>>> being "flattened out" of the arguments to If. The wanted behaviour of
>>> If in this case can be restored if SequenceHold is applied:
>>>
>>> In[3]:= Unprotect[If];
>>> SetAttributes[If, SequenceHold];
>>> Protect[If];
>>>
>>> In[6]:= If[1 == 1, Sequence[], 2]
>>>
>>> Out[6]= Sequence[]
>>>
>>
>> Dont't do that! We cannot overview the consequences of such a global
>> intervention. Instead act locally:
>>
>> In[7]:= If[1 == 1, Unevaluated[Sequence[]], 2]
>> Out[7]= Sequence[]
>>
>>
>> This now allows to do other things, e.g.
>>
>> In[3]:= Map[If[# === 2, Unevaluated[Sequence[]], #] &,
>> {{0, 1, 2, 3, 2}, 3, 2}, 2]
>> Out[3]= {{0, 1, 3}, 3}
>>
>>
>>
>>> But If has (I now find) the Attribute HoldRest, and Hold does not
>>> flatten out Sequence[] so I don't understand why that does not apply
>>> here also.
>>>
>>
>> Hold has the attributes
>>
>> In[6]:= Attributes[Hold]
>> Out[6]= {HoldAll, Protected}
>>
>> hence it flattens out Sequence[..]
>>
>> In[5]:= Hold[Sequence[], If[1 == 1, Sequence[], 2]]
>> Out[5]= Hold[If[1 == 1, Sequence[], 2]]
>>
>> (Of course there will be no operation for and _within_ the held
>> expression.)
>>
>>
>>> In[7]:= xx[1+2,2+3,3+4]
>>>
>>> Out[7]= xx[3,5,7]
>>>
>>> In[8]:= SetAttributes[xx,HoldRest]
>>>
>>> In[9]:= xx[1+2,2+3,Sequence[],3+4]
>>>
>>> Out[9]= xx[3,2+3,3+4]
>>>
>>
>> You just showed that HoldRest has nothing to do with all that.
>>
>>
>>> Relatively few built-in functions have SequenceHold applied:
>>>
>>> {AbsoluteTiming, Rule, RuleDelayed, Set, SetDelayed, TagSet,
>>> TagSetDelayed, Timing, UpSet, UpSetDelayed}
>>>
>>> And also relatively few built-in functions have HoldRest applied:
>>>
>>> {Assuming, DumpSave, If, PatternTest, RuleDelayed, Save, Switch}
>>>
>>
>> Yes, the main reason is for so-called lazy evaluation. Would you
>> imagine
>> what happend when If lacked this attribute? Not only that many parts
>> of the
>> expression had to be evaluated in vain, and never used for the result,
>> also
>> side effects would become completely unpredictable (more than is
>> anyway).
>>
>>
>>
>>> I will therefore now be much more careful of the use of Sequence! At
>>> present I don't want to be more daring and apply these attributes
>>> wholesale without understanding why this is happening. There must be
>>> other cases where Sequence can have unexpected effects - I now
>>> understand some of the problems I had when trying to use to
>>> modify $Post -
>>> reallyBig needs to cope with Sequence[] and Sequence[__].
>>>
>>> --
>>> from- John Tanner email- john at janacek.demon.co.uk
>>> mantra- curse Microsoft, curse... web -
>>> http%##www.janacek.demon.co.uk/
>>> I hate this 'orrible computer, I really ought to sell it:
>>> It never does what I want, but only what I tell it.
>>>
>>
>>
>> I agree, see also the Book about evaluation (e.g. §A.4.1)
>>
>> --
>> Hartmut Wolf
>>
>>
>>
>
--
from- John Tanner email- john at janacek.demon.co.uk
mantra- curse Microsoft, curse... web - http^%%www.janacek.demon.co.uk/
I hate this 'orrible computer, I really ought to sell it:
It never does what I want, but only what I tell it.
- Follow-Ups:
- Re: Re: Unexpected behaviour of HoldRest
- From: Daniel Lichtblau <danl@wolfram.com>
- Re: Re: Unexpected behaviour of HoldRest
- References:
- RE: Unexpected behaviour of HoldRest
- From: "Wolf, Hartmut" <Hartmut.Wolf@t-systems.com>
- RE: Unexpected behaviour of HoldRest