Re: Working with large data files (hundreds of megas)
- To: mathgroup at smc.vnet.net
- Subject: [mg9152] Re: [mg9112] Working with large data files (hundreds of megas)
- From: "Parker E.C. Bradley" <PJDRINKA at concentric.net>
- Date: Thu, 16 Oct 1997 03:37:53 -0400
- Sender: owner-wri-mathgroup at wolfram.com
Gilberto Espinoza wrote: > I am trying to figure out the best way to to work with huge data > files. For > example, I have a compressed (gzip) file that is 13 mega and when I > uncompress it it inflates to abut 90 mega and i have several of these. > > I would like to know if there is a way to work with compressed files > within > mathematica either 2.2 on a unix platform or 3.0.1 on a windows nt > 4.0. my > nt is hooked up to the network so i have acces to the unix drive where > my > file is located. I would like to read the data from the compressed > file and > perform my analysis and write my results into a compressed file to > avoid > giving my sytem administrator a fit as I would use up all the disk > space > space quite fast working with the uncompressed files. > > thanks a lot > > gilberto espinoza > > gespinoza at earthlink.net This is a very interesting problem -- fortunately, I my situation, I was able to just add more disk space and memory (though I may have to utilize my own advice in the near future). In your case, the first possiblility (for the Unix scenario) that comes to mind is to utilize Mathlink with a unix shell script. When Mathematica needs the data, MathLink would act as an intermediary between Mathematica and the Unix shell script; passing data back and forth and giving appropriate commands to the shell script. The shell script's job would be to handle the compression/decompression with gzip/gunzip, and perform any other needed file handling. Hope this helps. If anyone has a more elegant/direct approach -- I too am very interested. PECB -- ******************************************************* Parker E.C. Bradley 315-330-1893 Air Force Rome Labs / C3BB PJDRINKA at concentric.net 525 Brooks Road bradley at rl.af.mil Rome, NY 13441 ******************************************************* -----BEGIN PGP PUBLIC KEY BLOCK----- Version: PGP for Personal Privacy 5.0 mQGiBDOm5eIRBADp8SCxIot5L4/vRBlmpPfu0ilD1osbsfqhCARMFqRXEosnq1MA D8P00xtlL9eo/ukq31PGnlOlrhIxe3KB4JEAejMC7dWnCgxzu5SIVtvv1HiiLcD0 Or/QzdlEdqtbFLrg1ckXrcjeqGZnSHT4PicpBO+CYDAcTddfOcCGkPkZvwCg/5QL amgvH7S9db+1SYJiBovplGUD/ibk3Y9eWmpgy3B9TFOpG1Al0mgTEdEpfLTRiAY2 Mfbfxfa34kmKhYYM1gMrKFK0QxWNJ4Tx1iY64rGYD7laJQm0mPQqcVOB3mWqddGD Xp2jyyj5NIZ/aBFGdMCcLHgZ12IjPU2n97fcKBhghmAMcs+3zMKOTWKoQMToQXHY MIZUA/966kb02r6ehrdCsArouToWuCTC7NCX1ODPDJKwcBkS3UpSujeulOK2o+0+ PB3pMbFTW2L3QITkXEVe/S9EGTImKx6QewSjQCTBNy4aU3BxOhMeLDJdb4UEInHZ PXOuyFOyuQfqxYVEL0RYD78gIeE7OnQbiimWRvQsNgn2fkGKd7QwUEVDQiA8YnJh ZGxleUBybC5hZi5taWwsUEpEUklOS0FAY29uY2VudHJpYy5uZXQ+iQBLBBARAgAL BQIzpuXiBAsDAQIACgkQ4woxdu/qIenODACg32FVMKY8AysyoGn+V2+YqWlAcBsA mgLi3YiLanWsC6NFMIIcnH07AnEJuQGNBDOm5eIQBgD49Vlf3HZSTz09jdvOmeFX klnN/biudE/F/Ha8g8VHMGHOfMlm/xX5u/2RXscBqtNbno2gpXI61Brwv0YAWCvl 9Ij9WE5J280gtJ3kkQc2azNsOA1FHQ98iLMcfFstjvbzySPAQ/ClWxiNjrtVjLhd ONM0/XwXV0OjHRhs3jMhLLUq/zzhsSlAGBGNfISnCnLWhsQDGcgHKXrKlQzZlp+r 0ApQmwJG0wg9ZqRdQZ+cfL2JSyIZJrqrol7DVelYfEcAAgIGAIsB4lQUb3n1URwR htUJjNkptPcN/xTSkV16VmmqX+oBFayo0VKS89N+l2pQ6+g3OCUOPEA5h20xeKIA mN777oyHvhcbu6M7sJ1CDZiM7Yj7wbms2i6g1nux+dSAPNdlTunVy2E4/RUgbDAc aQSrYEuX8dv5PJN8YZvO3pzExiVYpAnXE9iz+D8bQsSUYMRMYAIio2lbrWdCaiRM BCYLBGN/x3CMhDQQwFWlVWIM52SX2K6LwaI8HvowtGSTtmkBuIkAPwMFGDOm5eLj CjF27+oh6RECmLsAoMW/gZOiAy9fQVfjj4nQVKFGrd+rAJsHm+F+UcAIPJ+JnyZA gngXM6WThQ== =SqV5 -----END PGP PUBLIC KEY BLOCK-----