Re: Re: Passing arrays to MathLink without extra memory allocation
- To: mathgroup at smc.vnet.net
- Subject: [mg98877] Re: [mg98805] Re: Passing arrays to MathLink without extra memory allocation
- From: John Fultz <jfultz at wolfram.com>
- Date: Mon, 20 Apr 2009 19:12:41 -0400 (EDT)
- Reply-to: jfultz at wolfram.com
To further clarify Jens' points...it's a fundamental principle of modern operating system design that separate processes are absolutely protected from each other. If one process is allowed to access another process's memory, then you've set up the preconditions which allow a rogue process to destabilize the OS, or to create considerable security violations. The reason why modern operating systems like Linux, MacOS X, and the Windows NT line (2000, XP, Vista) are so much less likely to crash than operating systems such as Mac System 9 and the Windows 9x line are because of these protections. Obviously, the OS itself and anything mapped into the OS (most notably, hardware drivers) can still cause problems. But modern OS architecture is very good at preventing a single application from bringing down your entire machine. I, for one, remember the days of Win9x and Mac System 9 (or, worse yet, Win 3.1 and System 6), and believe me, we live in a *much* better world, now. There are two abstract possibilities for doing what you want, neither of which can presently be done in Mathematica. One is to use memory-mapped files. This allows you to map a file (a real, properly named file, or some temporary thing in the swap space) into memory, and explicitly give other processes permissions to map the same file into their own memory space. That is, incidentally, one of the methods MathLink uses for communicating (but in a buffered way...it's not directly mapping internal structures). The second is to write your code as a DLL and have Mathematica run your code in its own process space. In both, there are significant risks, as you now open up the list of things that can crash Mathematica to not just issues in Mathematica, but also to bugs which exist in the external code. There are also some other considerable complications which I'm glossing over a bit (Jens rightly points out one of those...the internal data structures may be quite sophisticated). Nonetheless, I wouldn't go so far as to say that the problem is insurmountable or that we'll never have other solutions. But it's certainly not there now. Depending on your problem, though, other solutions may very well be feasible. Sincerely, John Fultz jfultz at wolfram.com User Interface Group Wolfram Research, Inc. On Mon, 20 Apr 2009 01:28:38 -0400 (EDT), Jens-Peer Kuska wrote: > Hi, > > axnavaei at googlemail.com wrote: >> Hi, >> >> Are there any other solutions for passing pointers to big data from >> mathematica to external programs? >> > No > > In this context, an 'external >> program' does not need to be a separate process, it could simply be a >> loadable shared library. The way another system deals with its c- >> extensions is >> a good example of what I mean by this. >> > A MahtLink program can run on a second computer, that will not work > with dynamic librarys > >> >> It seems like those official mathematica functions implemented in >> native code do not have the above constraint and it is possible to >> pass pointers without deep copying, is this correct? >> > What is a official Mathematica function ? > > Why do you assume that the kernel store your data in the same way as > your code does ? Since Mathematica can store symbols, expressions, > integers, multi-precision numbers in its lists your program would never > know what is in the memory location. MathLink is the only way to > unfold the internal structures and to send it to an external program. > > > If so, how is it >> possible to implement a function like this? >> > > *IT IS NOT POSSIBLE* > > And with a little cleverness in cen C-code it is not necessary. > I have C-programs that deal with several GByte of data for visualization > and image processing tasks and I have no idea for what this should be > good for. > > Regards > Jens > >> On 18 Apr, 08:37, Jens-Peer Kuska <ku... at informatik.uni-leipzig.de> >> wrote: >>> Hi, >>> >>> a) MathLink is always buffered and >>> b) it must copy the data into a memory area >>> that bot programs can access >>> c) your C-program can never access the memory space >>> of an other program/process. The operating system is >>> responsible to hinder/prevent such operations >>> >>> Regards >>> Jens >>> >>> axnav... at googlemail.com wrote: >>>> Hi, >>>> When passing a list/array from mathematica to c using mathlink, eg >>>> when calling MLReleaseInteger32List(), it seems mathlink allocates >>>> some memory which later has to be freed, eg using >>>> MLReleaseInteger32List(). The question is, how is it possible to >>>> pass >>>> a list from mathematica to c without this c memory allocation? >>>> This example helps to make the question more clear: Assume >>>> MyFunction >>>> [BigList] to be a function implemented in c using mathlink, where >>>> BigList is a list of integers. Since BigList has been already >>>> created >>>> in mathematica and its corresponding memory has been allocated, we >>>> do >>>> not wish to allocate memory once again in c. It looks like one >>>> option >>>> is to pass BigList by reference, that is: >>>> SetProperties[MyFunction, HoldAll] >>>> In this case, does mathlink still allocate memory and copy BigList >>>> in >>>> the c side?