Services & Resources / Wolfram Forums / MathGroup Archive
-----

MathGroup Archive 2010

[Date Index] [Thread Index] [Author Index]

Search the Archive

Re: convert txt to binary file

  • To: mathgroup at smc.vnet.net
  • Subject: [mg108841] Re: convert txt to binary file
  • From: Bill Rowe <readnews at sbcglobal.net>
  • Date: Sat, 3 Apr 2010 06:08:18 -0500 (EST)

On 4/2/10 at 5:22 AM, jihane.ajaja at mail.mcgill.ca (jihane) wrote:

>I have a 4Gb txt file and it is not possible to work with that
>because it is huge. Does someone knows how to convert an ascii file
>to a binary file using mathematica? I have a collegue that does it
>with another system but I can't find out how to do that. I searched
>a lot without sucess.

It is not difficult to convert an ASCII file to binary. There
are two ways that come readily to mind.

First, if the file contains just numeric data the following flow
would work

Open the file to be converted and open an output file to save
the binary representation. Once you have the data streams for
each of the above files, then doing

data=ReadList[inputStream, Number, n];
BinaryWrite[outputStream, data, format]

in a loop will do the trick. Here n is the number of items to
read at any one time.

If the contents of the file are not all numeric, then the loop
body would be something like

data = ReadList[inputStream, Byte, 4 n]
BinaryWrite[outputStream, FromDigits[#,
256]&/@Partition[data,4], "Integer32"]

But while the above will convert the ASCII file into a binary
representation, I strongly suspect this isn't going to be that
useful to you. The limitations on how large of a data set you
can work with in Mathematica are determined by the amount of RAM
you have and the way Mathematica represents the data internally.
They are independent of how the data is stored on disk.
Converting a large ASCII file to binary will reduce the file
size on disk and likely will reduce the time needed to read the
data back from disk. But it will not do anything to increase the
amount of data you can work with at one time in Mathematica.



  • Prev by Date: Re: trying to use InputField as a continually attentive string input
  • Next by Date: Re: 0/1 knapsack-like minimalization problem and file
  • Previous by thread: Re: convert txt to binary file
  • Next by thread: Re: convert txt to binary file