Re: Scoping with Module
- To: mathgroup at smc.vnet.net
- Subject: [mg109691] Re: Scoping with Module
- From: Leonid Shifrin <lshifr at gmail.com>
- Date: Wed, 12 May 2010 07:33:08 -0400 (EDT)
Hi Patrick,
>From the point of a user I find it very hard to explain to a newbie (who
> started to see the usefulness of lokalized variables) why this works
>
> sol = First@
> NDSolve[{y'[x] == y[x] Cos[x + y[x]], y[0] == 1}, y[x], {x, 0, 30}];
> func[x_] = y[x] /. sol;
> Plot[func[x], {x, 0, 30}]
>
> while this doen't
>
> Module[{sol, y, x, func},
> sol = First@
> NDSolve[{y'[x] == y[x] Cos[x + y[x]], y[0] == 1}, y[x], {x, 0, 30}];
> func[x_] = y[x] /. sol;
> Plot[func[x], {x, 0, 30}]
> ]
>
Yes, I agree that this behavior is not intuitive. But I'd say that what the
novice user should be told up front is that she should *not* expect the
first version of your example or similar constructs to work, in the first
place. In principle, in func[x_] = y[x] /. sol, the system is free to rename
x in all lexical occurrences, it just does not do it. So, I think that in a
sense we should blame our own expectations in such cases - we are just too
used to "good".
I think that the documentation should make much more clear the distinction
between scoping stage and evaluation stage. The fact that your first example
works is a consequence of both the scoping stage behavior (which happens
first), and then the evaluation. In a way, it works only due to the "good
will" of the scoping mechanism which in this case did not rename things. We
should not count on it. So, IMO what must be explained is not why the second
example did not work (this I think is normal), but why the first did.
>
> Additionally, nothing prevents you from doing something like this
>
> t$ = 1;
> Module[{f, g, t},
> g[t_] := Sin[t];
> f[t_] = g[t];
> DownValues[f]
> ]
>
> Out[2]= {HoldPattern[f$560[t$_]] :> Sin[1]}
>
> Since most system variables are named like $... there is maybe someone who
> accidently uses t$ as additional variable and gets some really strange
> behavior.
>
Yes, I think this is another important point. Lexical scoping in Mathematica
is emulated by symbol renaming, and since it is an emulation, it can be
broken. I think there were many discussions of this issue here on the group,
and the bottom line of all these discussions seems to have been to avoid
using symbols with a dollar sign in their names.
>
> The whole point is really not that important but I just thought that there
> is more behind the idea of the scoping inside Module. Why doesn't has Block
> this behavoir?
>
I think you mean cases like this:
In[1]:= Module[{a}, Hold[Function[b, Print[a + b]][c]]]
Out[1]= Hold[Function[b$, Print[a$592 + b$]][c]]
In[2]:= Block[{a}, Hold[Function[b, Print[a + b]][c]]]
Out[2]= Hold[Function[b, Print[a + b]][c]]
My guess is that the implementation of lexical scoping in Mathematica is
well-aware of the difference between dynamic and lexical scoping. Block is
not dangerous to inner lexical scoping constructs really: its effect is
dynamical, that is - it shows up only at evaluation stage which is already
after all lexical bindings (together with name collision resolution etc)
take place. To put it differently, the action of Block is equivalent to a
(temporal) change in the global state and does not affect lexical scoping
per se, only evaluation.
Therefore, for constructs that do not ordinarily produce new symbols (With,
Function,Set,SetDelayed), it is guaranteed to be happening already after
they bind their variables. Rule and RuleDelayed seem to protect their
variables against any (dynamic or lexical) outer scoping construct except
themselves, without renaming. For Module, OTOH, Block is harmless simply
because Module produces unique symbols to which external Block-s can not
normally have access (since Block binds dynamically based on the exact name
occurrences), unless again we take some deliberate steps to break the
scoping by for example feeding special symbols to Block.
Outer lexical scoping constructs are however potentially dangerous to inner
ones, since they bind their variables to the body from outside to inside.
So, my guess would be that, when we start to execute code with nested
lexical scoping constructs, the most external scoping construct first
"looks" at the (held) code(its body) and does the necessary renamings in
inner scoping constructs, then binds its own local variables (after
renamings in inner scoping construsts have happened), and finally executes
its body. At least, this is what I think should be happening (I may be wrong
of course). OTOH, we also know that Rule and RuleDelayed are different from
other lexical scoping constructs in that they do not respect the inner ones
(unlike say With, Module, Function), which IMO has both a good side (ability
to do scope surgery with them), and a bad side (lots of opportunities for
nasty and subtle scoping bugs).
Anyway, to conclude, one thing I am sure of is this: given that the details
of internal implementation of scoping constructs (renaming scheme etc) seem
to be manifesting themselves pretty often in examples like the one you gave,
the documentation could at least be more clear and perhaps give some more
details on what is going on behind the scenes, so that at least some of
these things could be understood logically rather than guessed or memorized
like magic spells (for example, I have no idea what percent of my guesses
above about the scoping mechanism is correct).
Regards,
Leonid
> Cheers
> Patrick
>
> Am May 6, 2010 um 12:37 PM schrieb Leonid Shifrin:
>
>
> Hi Patrick,
>>
>> My understanding is that Mathematica uses aggressive variable renaming
>> (more than is strictly necessary for many cases) in case it sees nested
>> scoping constructs with conflicting variables (variables with the same
>> name). This manifests itself in the renaming happening if these variables
>> occur not just in the declaration lists, but also in the body, like here:
>>
>> scoping1[{a},scoping2[{b},a+b]
>>
>> In such cases, <b> will be often renamed to something like b$ even before
>> the actual binding happens,
>> for example:
>>
>> In[1]:= Function[a, Function[b, Print[a + b]]][c]
>>
>> Out[1]= Function[b$, Print[c + b$]]
>>
>> Note that in many cases this is not strictly necessary, but it is probably
>> easier and / or more efficient to do this excessive renaming than to work on
>> the case-by-case basis. Now, both Set with patterns and Module are scoping
>> constructs. In your starting examples there are no local variables in the
>> Module declaration , so no conflicts of the above type are possible.
>> However, in this one:
>>
>> Module[{r},
>> r = {t :> 1};
>> f[t_] = t /. r;
>> DownValues[f]
>> ]
>>
>> you do have a possibility of a conflict of the above type, and so
>> Mathematica renames a dummy variable in the inner scoping construct - <t> in
>> f[t_] = t /. r; in this case. At the same time, <t> in the
>> r = {t :> 1}; remains global since no patterns / scoping is involved here
>> - thus the result you observed. In your final example:
>>
>> Module[{r},
>> r = {t :> 1};
>> r2 = r;
>> f[t_] = t /. r2;
>> DownValues[f]
>> ]
>>
>> while the situation looks pretty similar from the evaluation viewpoint,
>> <r> does not appear inside the definition f[t_] = t /. r2; lexically (this
>> happens only dynamically), and for lexical scoping constructs (at least as
>> they exist in Mathematica) this makes all the difference. At the time of
>> variable name conflict resolution (this is before any code inside Module
>> starts to execute), Mathematica can no longer sense any danger, and thus no
>> renaming happens.
>>
>> I don't know whether this behavior should be sonsidered satisfactory or
>> not, since it is certainly not very intuitive. I'd say that it's still ok:
>> when we take a line like f[t_] = t /. r;, we should treat <t> as a local
>> lexically scoped variable (the fact that it evaluates to global <t> is
>> related to evaluation, not scoping). Therefore, there is generally no
>> guarantee that injecting there r = {t :> 1}; (which is defined outside the
>> scope of Set in f[t_] = t /. r;) will work. I think that our puzzlement
>> originates from mixing scoping and evaluation. From the fact that Set
>> evaluates the r.h.s and uses global values of <t> for entries of <t> on the
>> r.h.s. (this is what we are used to from our experiences of working with
>> global variables), we subconsciously expect that this is true for all
>> entries of <t> on the r.h.s, while this is only guaranteed for non-lexical
>> entries of <t> - those that were not explicitly (lexically), but rather
>> dynamically, present in the code before evaluation started. The problem is
>> made worse by the fact that in many common cases it does work (like the
>> case of global variables and no surrounding scoping constructs).
>>
>> The bottom line is that IMO the construct you looked at is not reliable
>> from the start in that it attempts to do scope engineering and the results
>> are then necessarily dependent on the details of Mathematica internal
>> implementation of lexical scoping constructs. Then it is probably a matter
>> of taste whether or not to use constructs like this. Personally, I do it
>> from time to time for my own purposes, but I would hesitate to put them into
>> "production" code. Besides, I believe that in most cases there exist more
>> reliable alternatives.
>>
>> Regards,
>> Leonid
>>
>>
>>
>> On Thu, May 6, 2010 at 12:52 PM, Patrick Scheibe <
>> pscheibe at trm.uni-leipzig.de> wrote:
>> Hi,
>>
>> I cannot explain the following behaviour of Module. Maybe someone else
>> can:
>>
>> in the global scope
>>
>> f[t_] = t;
>> DownValues[f]
>>
>> results in {HoldPattern[f[t_]] :> t} as expected. Simply wrapping it
>> with Module, without
>> making f lokal does not change this behaviour.
>>
>> Module[{},
>> f[t_] = t;
>> DownValues[f]
>> ]
>>
>> results still in {HoldPattern[f[t_]] :> t}.
>> Say I want to replace the right side of the definition with a rule
>>
>> r = {t :> 1};
>> f[t_] = t /. r;
>> DownValues[f]
>>
>> and we get {HoldPattern[f[t_]] :> 1}
>>
>> Module[{},
>> r = {t :> 1};
>> f[t_] = t /. r;
>> DownValues[f]
>> ]
>>
>> still the expected result {HoldPattern[f[t_]] :> 1}
>>
>> Now (sorry for the lengthy introduction) making r lokal
>>
>> Module[{r},
>> r = {t :> 1};
>> f[t_] = t /. r;
>> DownValues[f]
>> ]
>>
>> and we get {HoldPattern[f[t$_]] :> t$}. The Pattern t_ was renamed
>> before the replacement could act.
>> Using the local variable r to set a global r2 and it's working again.
>>
>> Module[{r},
>> r = {t :> 1};
>> r2 = r;
>> f[t_] = t /. r2;
>> DownValues[f]
>> ]
>>
>> So there must be at least one thing about Module I did'nt understand.
>> I thought the main difference between Block
>> and Module is the lexical scoping!? But the only thing which was
>> renamed is the lokal usage of r.
>> Any ideas?
>>
>> Cheers
>> Patrick
>>
>> 7.0 for Mac OS X x86 (64-bit) (February 19, 2009)
>>
>>
>>
>