Re: Assigning part of indexed object

*To*: mathgroup at smc.vnet.net*Subject*: [mg119881] Re: Assigning part of indexed object*From*: Leonid Shifrin <lshifr at gmail.com>*Date*: Mon, 27 Jun 2011 07:33:05 -0400 (EDT)*References*: <itsjn7$8u7$1@smc.vnet.net>

Oleksandr, On Sun, Jun 26, 2011 at 2:26 PM, Oleksandr Rasputinov < oleksandr_rasputinov at hmamail.com> wrote: > Leonid, > > I fully agree with all of your comments and of course you are right to > point out that modifying core functions such as Set is a bad idea in > general due to the likelihood of unintended consequences. Of course, > the performance issue introduced for setting parts of OwnValues can > be repaired to some extent: > > indexedQ = Function[sym, > Symbol =!= Head@Unevaluated[sym] =!= Unevaluated[sym], > HoldFirst > ]; > > Unprotect[Set]; > Set[(sym_?indexedQ)[[part_]], val_] := ( > sym = ReplacePart[sym, part -> val]; > sym[[part]] > ); > Protect[Set]; > > which restores the O(1) behaviour (albeit at the cost of an increased > constant factor). But even so, I will join with you in stressing that this > general approach is dangerous and should be attempted only with > extreme caution, if at all. > While your latest fix seems to work (except that it does not solve the O(n) problem for part assignments of arrays stored in indexed variables, but that at least is not possible to do without this code anyway, so no existing code can be broken. You also could have included the multi-index parts), I would not use it anyway. In Mathematica, Hold-attributes give one the ability to make custom scoping constructs and assignment operators, a luxury that few languages offer. Overloading Set (particularly by adding DownValues to Set) therefore is, in the vast majority of cases, not a necessity but just a notational convenience, which one should weight against completely unpredictable consequences of redefing one of the most fundamental operations. And it does not matter how carefully you use it, since there is a lot of top-level code in Mathematica itself, and you will never be able to guarantee that that code won't break. So the conclusion should be clear - create your own data type and assignment operators if you need. Too bad that Mathematica's grammar is fixed and currently no tools like preprocessor or programmable reader exist for it to make the code look prettier, but I think one can live with that. I must admit that I am myself guilty, having overloaded Set and SetDelayed a few times, like e.g. here: http://groups.google.com/group/comp.soft-sys.math.mathematica/browse_thread/thread/c000439b48751078 but I did it for very special purposes when I needed to "spy" on defintions given when loading packages, or modify them on the fly, and these were mostly for functions, so more like a one-time affair. And I am not fond of those hacks either. Besides, Part assignment is anyway special and already overloaded in Mathematica, since for a generic head h which does not have Hold* Attributes, in an assignment like h[x] = value both the head h, and value x, *are*evaluated, while rules for h[ _ ] are not applied (the fact that is not fully reflected in the documentation, which states this for <x> but not for <h>). But clearly, this is not what is happening in the case of Part, since Part is not HoldFirst and therefore in expr[[part]] = value, expr should normally have been evaluated (and then would of course no longer represent the L - value). So, by overloading Part assignment, you not only overload Set, but overload the already overloaded and not completely specified part of Set definitions. As an alternative to a custom part assignment operator, one can use a custom wrapper for a part assignment, at the expense of a slight notational inconvenience. For example, like this (using your indexedQ - which, to be pedantic, better be HoldAllComplete rather than HoldFirst): ClearAll[part]; part /: Set[part[sym_?indexedQ, part__], value_] := ( sym = ReplacePart[sym, part -> value]; sym[[part]] ); part /: Set[part[sym_, part_], value_] := sym[[part]] = value; part[expr_, part__] := Part[expr, part]; Since Set does not apply the rules for the head of its l.h.s., the DownValues for <part> won't fire before UpValues when we use <part> inside Set. Example of use: In[111]:= Clear[f]; f[1] = Range[10]; part[f[1], 3] Out[113]= 3 In[116]:= part[f[1], 3] = 10; part[f[1], 3] Out[117]= 10 This immediately removes the main problem, since Set is now "softly" overloaded via UpValues. One can get fancier and then make a custom "scoping construct", which would lexically or dynamically change Part to part in a piece of code. Inside it, one can then use the exact same syntax as before, without any changes. Here is the lexical "scoping construct", for example: ClearAll[withCustomPartLex]; SetAttributes[withCustomPartLex, HoldAll]; withCustomPartLex[code_] := With[{Part = part}, code]; Example of use: withCustomPartLex[ Clear[f]; f[1] = Range[10]; f[1][[3]] = 10; f[1]] {1, 2, 10, 4, 5, 6, 7, 8, 9, 10} And all is safe. The O(n) problem for assignments to parts of indexed variables still not solved, since the same ReplacePart-based implementation is used, but this is a separate topic. One way to solve it is to use the temporary symbols generated during the first assignment to an indexed variable, to store the data. One will have to deallocate those however. Anyways, my main message is that even within what is available at the top-level and already parsed Mathematica code, Mathematica metaprogramming facilities are rich enough that one may, in most cases (including this one IMO), satisfy one's taste in terms of both syntax and semantics, without playing games with crucial and heavily used system functions. Regards, Leonid > > Best, > > O. R. > > On Jun 25, 10:30 am, Leonid Shifrin <lsh... at gmail.com> wrote: > > Hi Fabrice, > > > > My guess is that the reasons are efficiency and immutability of expres= > sions > > in Mtahematica. By restricting Set > > to variables or array elements, one can connect it directly to the > intern= > al > > memory layout for that element. Note > > that all proposed solutions are based on ReplacePart, which copies the > > entire list to modify a single element. > > OTOH, constructs like indexed variables are serviced by hash-tables > > internally. Even if the value stored > > in the hash table is a pointer to some array, that array will likely be > > immutable. Whether it would be a > > good idea to make it mutable I don't know, but the general ideology is > th= > at > > expressions are immutable > > in Mathematica. In other words, I don't think that the case of indexed > > variables was overlooked when > > designing Set - my guess is that the restriction to symbols is quite > > intentional. > > > > Regarding the "fix", I'd consider this not a fix but a recipe for disa= > ster, > > because modifying Set in such > > a fashion is dangerous, not to mention that it will often be completel= > y > > unacceptable performance-wise. Let us say > > you store a large array in a symbol and use a For loop with array > indexin= > g. > > Suddenly all your array element assignments > > become O(n) time, where n is the list size, or even worse for higher > > dimensional tensors, not to mention increased > > memory consumption. Here is an illustration: > > > > In[18]:= a = Range[10000]; > > > > In[19]:= Unprotect[Set]; > > Set[sym_[[part_]], val_] := sym = ReplacePart[sym, part -> val]; > > Protect[Set]; > > > > In[27]:= Do[a[[i]] = a[[i]]^2, {i, Length[a]}] // Timing > > > > Out[27]= {1.156, Null} > > > > In[28]:= Unprotect[Set]; > > Clear[Set]; > > Protect[Set]; > > > > In[31]:= Do[a[[i]] = a[[i]]^2, {i, Length[a]}] // Timing > > > > Out[31]= {0.047, Null} > > > > You may do this for yourself if you wish (although I'd avoid even that= > ), > > but I'd strongly > > advise against such modifications in production code or any code someone > > else will be using. Set is one of the > > most fundamental operations, and its modifications may have very far- > > reaching and hard to predict consequences. > > It may, for example, corrupt the system in very subtle ways, which you > ma= > y > > never be able to identify. > > > > Unfortunately, the standard mechanism of "soft" system symbols > overloadin= > g > > via UpValues does not > > quite work with Set (or at least I don't know of a way to make it work) > > because Set holds its argument and UpValues > > can only be defined for tags not deeper than level 1 in an expression. In > > this post: > > > > http://stackoverflow.com/questions/5886066/overloading-seta-b-a-b/588... > > > > I demonstrated a method, which I also don't like, but which is at least > > safer because it tests for a > > specific data type. Generally, the still safer way is to create your o= > wn > > data type and introduce your own > > assignment operator(s) - in this way, you don't have to modify Set at > all= >