Read/Write streams in parallel

*To*: mathgroup at smc.vnet.net*Subject*: [mg109841] Read/Write streams in parallel*From*: ChrisL <chris.ladroue at gmail.com>*Date*: Wed, 19 May 2010 20:13:35 -0400 (EDT)

For a future project, I'll need to read and write some streams in parallel. I've started playing with a simple example and can't make it work. What I want to do is to read an entry from a stream ('testfile') and write its value to another stream ('testfile2'). This is straightforward with one processor but in parallel, I get either a wrong result or errors messages. For example (testfile contains the first 100 prime numbers): ParallelEvaluate[sr = OpenRead@FileNameJoin[{$TemporaryDirectory, "testfile"}]] ParallelEvaluate[wr = OpenAppend@FileNameJoin[{$TemporaryDirectory, "testfile2"}]] ParallelDo[ x = Read[sr]; Print[{x, $KernelID}]; Write[wr, {x, $KernelID}], {10}]; ParallelEvaluate[Close[sr]] ParallelEvaluate[Close[wr]] testfile2 then contains: {2, 2} {2, 1} {3, 2} {3, 1} {5, 2} {5, 1} {7, 2} {7, 1} {11, 2} {11, 1} This is not satisfactory: the entries are read twice, once per processor. Obviously, it's because the two streams are opened by both processors in parallel. So we want the streams to be opened only once, and their 'pointers' (location of last entry read) to be updated concurrently. I try this instead: sr = OpenRead@FileNameJoin[{$TemporaryDirectory, "testfile"}] wr = OpenAppend@FileNameJoin[{$TemporaryDirectory, "testfile2"}] SetSharedVariable[sr, wr] ParallelDo[x = Read[sr]; Print[{x, $KernelID}]; Write[wr, {x, $KernelID}], {4}]; Close[sr] Close[wr] This time the two streams should be open only once and sync-ed thanks to SetSharedVariable. But testfile2 ends up being empty*, and Mathematica sends some error messages: InputStream["/tmp/testfile", 33] (* Value of sr *) OutputStream["/tmp/testfile2", 34] (* Value of wr *) (kernel 2) Read::openx: InputStream[/tmp/testfile,33] is not open. (* yes it is! *) (kernel 1) Read::openx: InputStream[/tmp/testfile,33] is not open. (* yes it is! *) etc. Is there any way I can make it work? The correct content for testfile2 should be the list of the 10 first prime numbers, possibly in a different order, and the kernelID that wrote the entry. * if it starts from an empty file. Notice the OpenAppend (!=OpenWrite) Thank you very much in advance, Cheers, Chris