Mathematica 9 is now available
Services & Resources / Wolfram Forums / MathGroup Archive
-----

MathGroup Archive 2009

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: Re: Passing arrays to MathLink without extra memory allocation

  • To: mathgroup at smc.vnet.net
  • Subject: [mg98877] Re: [mg98805] Re: Passing arrays to MathLink without extra memory allocation
  • From: John Fultz <jfultz at wolfram.com>
  • Date: Mon, 20 Apr 2009 19:12:41 -0400 (EDT)
  • Reply-to: jfultz at wolfram.com

To further clarify Jens' points...it's a fundamental principle of modern
operating system design that separate processes are absolutely protected
from each other.  If one process is allowed to access another process's memory, then you've set up the preconditions which allow a rogue process
to destabilize the OS, or to create considerable security violations.

The reason why modern operating systems like Linux, MacOS X, and the
Windows NT line (2000, XP, Vista) are so much less likely to crash than
operating systems such as Mac System 9 and the Windows 9x line are because
of these protections.  Obviously, the OS itself and anything mapped
into the OS (most notably, hardware drivers) can still cause problems.
But modern OS architecture is very good at preventing a single application
from bringing down your entire machine.

I, for one, remember the days of Win9x and Mac System 9 (or, worse
yet, Win 3.1 and System 6), and believe me, we live in a *much* better
world, now.

There are two abstract possibilities for doing what you want, neither
of which can presently be done in Mathematica.

One is to use memory-mapped files.  This allows you to map a file (a
real, properly named file, or some temporary thing in the swap space)
into memory, and explicitly give other processes permissions to map the
same file into their own memory space.  That is, incidentally, one of
the methods MathLink uses for communicating (but in a buffered way...it's
not directly mapping internal structures).  The second is to write your
code as a DLL and have Mathematica run your code in its own process space.

In both, there are significant risks, as you now open up the list of
things that can crash Mathematica to not just issues in Mathematica,
but also to bugs which exist in the external code.  There are also some
other considerable complications which I'm glossing over a bit (Jens
rightly points out one of those...the internal data structures may be
quite sophisticated).

Nonetheless, I wouldn't go so far as to say that the problem is
insurmountable or that we'll never have other solutions.  But it's
certainly not there now.  Depending on your problem, though, other
solutions may very well be feasible.
Sincerely,
 
John Fultz
jfultz at wolfram.com
User Interface Group
Wolfram Research, Inc.



On Mon, 20 Apr 2009 01:28:38 -0400 (EDT), Jens-Peer Kuska wrote:
> Hi,
>
> axnavaei at googlemail.com wrote:
>> Hi,
>>
>> Are there any other solutions for passing pointers to big data from
>> mathematica to external programs?
>>
> No
>
> In this context, an 'external
>> program' does not need to be a separate process, it could simply be a
>> loadable shared library. The way another system deals with its c-
>> extensions is
>> a good example of what I mean by this.
>>
> A MahtLink program can run on a second computer, that will not work
> with dynamic librarys
>
>>
>> It seems like those official mathematica functions implemented in
>> native code do not have the above constraint and it is possible to
>> pass pointers without deep copying, is this correct?
>>
> What is a official Mathematica function ?
>
> Why do you assume that the kernel store your data in the same way as
> your code does ? Since Mathematica can store symbols, expressions,
> integers, multi-precision numbers in its lists your program would never
> know what is in the memory location. MathLink is the only way to
> unfold the internal structures and to send it to an external program.
>
>
> If so, how is it
>> possible to implement a function like this?
>>
>
> *IT IS NOT POSSIBLE*
>
> And with a little cleverness in cen C-code it is not necessary.
> I have C-programs that deal with several GByte of data for visualization
> and image processing tasks and I have no idea for what this should be
> good for.
>
> Regards
> Jens
>
>> On 18 Apr, 08:37, Jens-Peer Kuska <ku... at informatik.uni-leipzig.de>
>> wrote:
>>> Hi,
>>>
>>> a) MathLink is always buffered and
>>> b) it must copy the data into a memory area
>>> that bot programs can access
>>> c) your C-program can never access the memory space
>>> of an other program/process. The operating system is
>>> responsible to hinder/prevent such operations
>>>
>>> Regards
>>> Jens
>>>
>>> axnav... at googlemail.com wrote:
>>>> Hi,
>>>> When passing a list/array from mathematica to c using mathlink, eg
>>>> when calling MLReleaseInteger32List(), it seems mathlink allocates
>>>> some memory which later has to be freed, eg using
>>>> MLReleaseInteger32List(). The question is, how is it possible to
>>>> pass
>>>> a list from mathematica to c without this c memory allocation?
>>>> This example helps to make the question more clear: Assume
>>>> MyFunction
>>>> [BigList] to be a function implemented in c using mathlink, where
>>>> BigList is a list of integers. Since BigList has been already
>>>> created
>>>> in mathematica and its corresponding memory has been allocated, we
>>>> do
>>>> not wish to allocate memory once again in c. It looks like one
>>>> option
>>>> is to pass BigList by reference, that is:
>>>> SetProperties[MyFunction, HoldAll]
>>>> In this case, does mathlink still allocate memory and copy BigList
>>>> in
>>>> the c side?





  • Prev by Date: Re: Re: Looking for more Mathematica online user groups/forums
  • Next by Date: Re: Re: Looking for a scrolling list control, or how to create/simulate
  • Previous by thread: Re: Passing arrays to MathLink without extra memory allocation
  • Next by thread: running mathematica notebooks without interface