Import HTTP data in asynchronous/parallel way
- To: mathgroup at smc.vnet.net
- Subject: [mg125923] Import HTTP data in asynchronous/parallel way
- From: Rodrigo Murta <murta at rodrigomurta.com>
- Date: Sat, 7 Apr 2012 05:57:00 -0400 (EDT)
- Delivered-to: l-mathgroup@mail-archive0.wolfram.com
Hi All
I'm testing some web scraping using Mathematica and would like to
know how to work with Import in an asynchronous/parallel way.
Now I'm using Parallelize to do that. It works but it doesn't look
like the best way to do that, due to the kernel number limitation.
My code is like that:
result = Parallelize[Import/@urlList]
How can I do it in another asynchronous way? Something like
backgroud process using & in bash?
I can do it inside Mathematica? I know that I could speed up a lot
my scrap with that.
tks in advance
Murta