Read/Write streams in parallel
- To: mathgroup at smc.vnet.net
- Subject: [mg109841] Read/Write streams in parallel
- From: ChrisL <chris.ladroue at gmail.com>
- Date: Wed, 19 May 2010 20:13:35 -0400 (EDT)
For a future project, I'll need to read and write some streams in
parallel. I've started playing with a simple example and can't make it
work.
What I want to do is to read an entry from a stream ('testfile') and
write its value to another stream ('testfile2'). This is
straightforward with one processor but in parallel, I get either a
wrong result or errors messages.
For example (testfile contains the first 100 prime numbers):
ParallelEvaluate[sr = OpenRead@FileNameJoin[{$TemporaryDirectory,
"testfile"}]]
ParallelEvaluate[wr = OpenAppend@FileNameJoin[{$TemporaryDirectory,
"testfile2"}]]
ParallelDo[
x = Read[sr]; Print[{x, $KernelID}];
Write[wr, {x, $KernelID}], {10}];
ParallelEvaluate[Close[sr]]
ParallelEvaluate[Close[wr]]
testfile2 then contains:
{2, 2}
{2, 1}
{3, 2}
{3, 1}
{5, 2}
{5, 1}
{7, 2}
{7, 1}
{11, 2}
{11, 1}
This is not satisfactory: the entries are read twice, once per
processor. Obviously, it's because the two streams are opened by both
processors in parallel. So we want the streams to be opened only once,
and their 'pointers' (location of last entry read) to be updated
concurrently.
I try this instead:
sr = OpenRead@FileNameJoin[{$TemporaryDirectory, "testfile"}]
wr = OpenAppend@FileNameJoin[{$TemporaryDirectory, "testfile2"}]
SetSharedVariable[sr, wr]
ParallelDo[x = Read[sr]; Print[{x, $KernelID}];
Write[wr, {x, $KernelID}], {4}];
Close[sr]
Close[wr]
This time the two streams should be open only once and sync-ed thanks
to SetSharedVariable. But testfile2 ends up being empty*, and
Mathematica sends some error messages:
InputStream["/tmp/testfile", 33] (* Value of sr *)
OutputStream["/tmp/testfile2", 34] (* Value of wr *)
(kernel 2) Read::openx: InputStream[/tmp/testfile,33] is not open. (*
yes it is! *)
(kernel 1) Read::openx: InputStream[/tmp/testfile,33] is not open. (*
yes it is! *)
etc.
Is there any way I can make it work? The correct content for testfile2
should be the list of the 10 first prime numbers, possibly in a
different order, and the kernelID that wrote the entry.
* if it starts from an empty file. Notice the OpenAppend (!=OpenWrite)
Thank you very much in advance,
Cheers,
Chris