MathGroup Archive 2011

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: Assigning part of indexed object

  • To: mathgroup at
  • Subject: [mg119881] Re: Assigning part of indexed object
  • From: Leonid Shifrin <lshifr at>
  • Date: Mon, 27 Jun 2011 07:33:05 -0400 (EDT)
  • References: <itsjn7$8u7$>


On Sun, Jun 26, 2011 at 2:26 PM, Oleksandr Rasputinov <
oleksandr_rasputinov at> wrote:

> Leonid,
> I fully agree with all of your comments and of course you are right to
> point out that modifying core functions such as Set is a bad idea in
> general due to the likelihood of unintended consequences. Of course,
> the performance issue introduced for setting parts of OwnValues can
> be repaired to some extent:
> indexedQ = Function[sym,
>   Symbol =!= Head@Unevaluated[sym] =!= Unevaluated[sym],
>   HoldFirst
>  ];
> Unprotect[Set];
> Set[(sym_?indexedQ)[[part_]], val_] := (
>    sym = ReplacePart[sym, part -> val];
>    sym[[part]]
>  );
> Protect[Set];
> which restores the O(1) behaviour (albeit at the cost of an increased
> constant factor). But even so, I will join with you in stressing that this
> general approach is dangerous and should be attempted only with
> extreme caution, if at all.

While your latest fix seems to work (except that it does not solve the O(n)
for part assignments of  arrays stored in indexed variables, but that at
least is not possible
to do without this code anyway, so no existing code can be broken. You also
could have included
the multi-index parts), I would not use it anyway.

In Mathematica, Hold-attributes give one the ability to make custom scoping
constructs and
assignment operators, a luxury that few languages offer. Overloading Set
(particularly by adding
DownValues to Set) therefore is, in the vast majority of cases, not a
necessity but just a
notational convenience, which one should weight against completely
unpredictable consequences of
redefing one of the most fundamental operations. And it does not matter how
carefully you use it, since there is a lot of top-level code in Mathematica itself,
and you will
never be able to guarantee that that code won't break. So the conclusion
should be
clear - create your own data type and assignment operators if you need. Too
bad that
Mathematica's grammar is fixed and currently no tools like preprocessor or
reader exist for it to make the code look prettier, but I think one can live
with that.

I must admit that I am myself guilty, having overloaded Set and SetDelayed a
few times, like e.g. here:

but I did it for very special purposes when I needed to "spy" on defintions
given when loading
packages, or modify them on the fly, and these were mostly for functions, so
more like a one-time
affair. And I am not fond of those hacks either.

Besides, Part assignment is anyway special and already overloaded in
since for a generic head h which does not have Hold* Attributes, in an
assignment like

h[x] = value

both the head h, and value x, *are*evaluated, while rules for h[ _ ] are not
(the fact that is not fully reflected in the documentation, which states
this for <x> but not for <h>).
But clearly, this is not what is happening in the case of  Part, since Part
is not HoldFirst and
therefore in expr[[part]] = value, expr should normally have been
evaluated (and then would of
course no longer represent the L - value). So, by overloading Part
assignment, you not only
overload Set, but overload the already overloaded and not completely
specified part of Set definitions.

As an alternative to a custom part assignment operator, one can use a custom
wrapper for a part
assignment, at the expense of a slight notational inconvenience. For
example, like this (using your
indexedQ - which, to be pedantic, better be HoldAllComplete rather than

part /: Set[part[sym_?indexedQ, part__], value_] :=
   sym = ReplacePart[sym, part -> value];
part /: Set[part[sym_, part_], value_] := sym[[part]] = value;
part[expr_, part__] := Part[expr, part];

Since Set does not apply the rules for the head of its l.h.s.,  the
DownValues for
<part> won't fire before UpValues when we use <part> inside Set. Example of

In[111]:= Clear[f];
f[1] = Range[10];
part[f[1], 3]

Out[113]= 3

In[116]:= part[f[1], 3] = 10;
part[f[1], 3]

Out[117]= 10

This immediately removes the main problem, since Set is now "softly"
overloaded via UpValues.
One can get fancier and then make a custom "scoping construct", which would
lexically or dynamically
change Part to part in a piece of code. Inside it, one can then use the
exact same syntax as before,
without any changes. Here is the lexical "scoping construct", for example:

SetAttributes[withCustomPartLex, HoldAll];
withCustomPartLex[code_] := With[{Part = part}, code];

Example of use:

 f[1] = Range[10];
 f[1][[3]]  = 10;

{1, 2, 10, 4, 5, 6, 7, 8, 9, 10}

And all is safe. The O(n) problem for assignments to parts of indexed
variables still not solved, since the same ReplacePart-based implementation
is used, but this is a separate topic. One way to solve it is to use the
temporary symbols generated during the first assignment to an indexed
variable, to store the data. One will have to deallocate those however.

Anyways, my main message is that even within what is available at the
top-level and already parsed Mathematica code, Mathematica metaprogramming
facilities are rich enough that one may, in most cases (including this one
IMO), satisfy one's taste in terms of both syntax and semantics, without
playing games with crucial and heavily used system functions.


> Best,
> O. R.
> On Jun 25, 10:30 am, Leonid Shifrin <lsh... at> wrote:
> > Hi Fabrice,
> >
> > My guess is that the reasons are efficiency and immutability of  expres=
> sions
> > in Mtahematica. By restricting Set
> > to variables or array elements, one can connect it directly to the
> intern=
> al
> > memory layout for that element. Note
> > that all proposed solutions are based on ReplacePart, which copies the
> > entire list to modify a single element.
> > OTOH, constructs like indexed variables are serviced by hash-tables
> > internally. Even if the value stored
> > in the hash table is a pointer to some array, that array will likely be
> > immutable. Whether it would be a
> > good idea to make it mutable I don't know, but the general ideology is
> th=
> at
> > expressions are immutable
> > in Mathematica. In other words, I don't think that the case of indexed
> > variables was overlooked when
> > designing Set - my guess is that the restriction to symbols is quite
> > intentional.
> >
> > Regarding the "fix", I'd consider this not a fix but  a recipe for disa=
> ster,
> > because modifying Set in such
> > a fashion is  dangerous, not to mention that it will often be completel=
> y
> > unacceptable performance-wise. Let us say
> > you store a large array in a symbol and use a For loop with array
> indexin=
> g.
> > Suddenly all your array element assignments
> > become O(n) time, where n is the list size, or even worse for higher
> > dimensional tensors, not to mention increased
> > memory consumption. Here is an illustration:
> >
> > In[18]:= a = Range[10000];
> >
> > In[19]:= Unprotect[Set];
> > Set[sym_[[part_]], val_] := sym = ReplacePart[sym, part -> val];
> > Protect[Set];
> >
> > In[27]:= Do[a[[i]] = a[[i]]^2, {i, Length[a]}] // Timing
> >
> > Out[27]= {1.156, Null}
> >
> > In[28]:= Unprotect[Set];
> > Clear[Set];
> > Protect[Set];
> >
> > In[31]:= Do[a[[i]] = a[[i]]^2, {i, Length[a]}] // Timing
> >
> > Out[31]= {0.047, Null}
> >
> > You may do this for  yourself if you wish (although I'd avoid even that=
> ),
> > but I'd strongly
> > advise against such modifications in production code or any code someone
> > else will be using. Set is one of the
> > most fundamental operations, and its modifications may have very far-
> > reaching and hard to predict consequences.
> > It may, for example, corrupt the system in very subtle ways, which you
> ma=
> y
> > never be able to identify.
> >
> > Unfortunately, the standard mechanism of "soft" system symbols
> overloadin=
> g
> > via UpValues does not
> > quite work with Set (or at least I don't know of a way to make it work)
> > because Set holds its argument and UpValues
> > can only be defined for tags not deeper than level 1 in an expression. In
> > this post:
> >
> >
> >
> > I demonstrated a method, which I also don't like, but which is at least
> > safer because it tests for a
> > specific data type. Generally, the still  safer way is to create your o=
> wn
> > data type and introduce your own
> > assignment operator(s) - in this way, you don't have to modify Set at
> all=

  • Prev by Date: Is Put - Get cycle in Mathematica always deterministic?
  • Next by Date: C code generation
  • Previous by thread: Re: Assigning part of indexed object
  • Next by thread: Re: Assigning part of indexed object