Re: Working with large data files (hundreds of megas)
- To: mathgroup at smc.vnet.net
- Subject: [mg9152] Re: [mg9112] Working with large data files (hundreds of megas)
- From: "Parker E.C. Bradley" <PJDRINKA at concentric.net>
- Date: Thu, 16 Oct 1997 03:37:53 -0400
- Sender: owner-wri-mathgroup at wolfram.com
Gilberto Espinoza wrote:
> I am trying to figure out the best way to to work with huge data
> files. For
> example, I have a compressed (gzip) file that is 13 mega and when I
> uncompress it it inflates to abut 90 mega and i have several of these.
>
> I would like to know if there is a way to work with compressed files
> within
> mathematica either 2.2 on a unix platform or 3.0.1 on a windows nt
> 4.0. my
> nt is hooked up to the network so i have acces to the unix drive where
> my
> file is located. I would like to read the data from the compressed
> file and
> perform my analysis and write my results into a compressed file to
> avoid
> giving my sytem administrator a fit as I would use up all the disk
> space
> space quite fast working with the uncompressed files.
>
> thanks a lot
>
> gilberto espinoza
>
> gespinoza at earthlink.net
This is a very interesting problem -- fortunately, I my situation, I
was able to just add more disk space and memory (though I may have to
utilize my own advice in the near future). In your case, the first
possiblility (for the Unix scenario) that comes to mind is to utilize
Mathlink with a unix shell script. When Mathematica needs the data,
MathLink would act as an intermediary between Mathematica and the Unix
shell script; passing data back and forth and giving appropriate
commands to the shell script. The shell script's job would be to handle
the compression/decompression with gzip/gunzip, and perform any other
needed file handling.
Hope this helps. If anyone has a more elegant/direct approach -- I
too am very interested.
PECB
--
******************************************************* Parker E.C.
Bradley
315-330-1893
Air Force Rome Labs / C3BB PJDRINKA at concentric.net 525
Brooks Road
bradley at rl.af.mil
Rome, NY 13441
*******************************************************
-----BEGIN PGP PUBLIC KEY BLOCK----- Version: PGP for Personal Privacy
5.0
mQGiBDOm5eIRBADp8SCxIot5L4/vRBlmpPfu0ilD1osbsfqhCARMFqRXEosnq1MA
D8P00xtlL9eo/ukq31PGnlOlrhIxe3KB4JEAejMC7dWnCgxzu5SIVtvv1HiiLcD0
Or/QzdlEdqtbFLrg1ckXrcjeqGZnSHT4PicpBO+CYDAcTddfOcCGkPkZvwCg/5QL
amgvH7S9db+1SYJiBovplGUD/ibk3Y9eWmpgy3B9TFOpG1Al0mgTEdEpfLTRiAY2
Mfbfxfa34kmKhYYM1gMrKFK0QxWNJ4Tx1iY64rGYD7laJQm0mPQqcVOB3mWqddGD
Xp2jyyj5NIZ/aBFGdMCcLHgZ12IjPU2n97fcKBhghmAMcs+3zMKOTWKoQMToQXHY
MIZUA/966kb02r6ehrdCsArouToWuCTC7NCX1ODPDJKwcBkS3UpSujeulOK2o+0+
PB3pMbFTW2L3QITkXEVe/S9EGTImKx6QewSjQCTBNy4aU3BxOhMeLDJdb4UEInHZ
PXOuyFOyuQfqxYVEL0RYD78gIeE7OnQbiimWRvQsNgn2fkGKd7QwUEVDQiA8YnJh
ZGxleUBybC5hZi5taWwsUEpEUklOS0FAY29uY2VudHJpYy5uZXQ+iQBLBBARAgAL
BQIzpuXiBAsDAQIACgkQ4woxdu/qIenODACg32FVMKY8AysyoGn+V2+YqWlAcBsA
mgLi3YiLanWsC6NFMIIcnH07AnEJuQGNBDOm5eIQBgD49Vlf3HZSTz09jdvOmeFX
klnN/biudE/F/Ha8g8VHMGHOfMlm/xX5u/2RXscBqtNbno2gpXI61Brwv0YAWCvl
9Ij9WE5J280gtJ3kkQc2azNsOA1FHQ98iLMcfFstjvbzySPAQ/ClWxiNjrtVjLhd
ONM0/XwXV0OjHRhs3jMhLLUq/zzhsSlAGBGNfISnCnLWhsQDGcgHKXrKlQzZlp+r
0ApQmwJG0wg9ZqRdQZ+cfL2JSyIZJrqrol7DVelYfEcAAgIGAIsB4lQUb3n1URwR
htUJjNkptPcN/xTSkV16VmmqX+oBFayo0VKS89N+l2pQ6+g3OCUOPEA5h20xeKIA
mN777oyHvhcbu6M7sJ1CDZiM7Yj7wbms2i6g1nux+dSAPNdlTunVy2E4/RUgbDAc
aQSrYEuX8dv5PJN8YZvO3pzExiVYpAnXE9iz+D8bQsSUYMRMYAIio2lbrWdCaiRM
BCYLBGN/x3CMhDQQwFWlVWIM52SX2K6LwaI8HvowtGSTtmkBuIkAPwMFGDOm5eLj
CjF27+oh6RECmLsAoMW/gZOiAy9fQVfjj4nQVKFGrd+rAJsHm+F+UcAIPJ+JnyZA
gngXM6WThQ==
=SqV5
-----END PGP PUBLIC KEY BLOCK-----