Re: convert txt to binary file
- To: mathgroup at smc.vnet.net
- Subject: [mg108841] Re: convert txt to binary file
- From: Bill Rowe <readnews at sbcglobal.net>
- Date: Sat, 3 Apr 2010 06:08:18 -0500 (EST)
On 4/2/10 at 5:22 AM, jihane.ajaja at mail.mcgill.ca (jihane) wrote: >I have a 4Gb txt file and it is not possible to work with that >because it is huge. Does someone knows how to convert an ascii file >to a binary file using mathematica? I have a collegue that does it >with another system but I can't find out how to do that. I searched >a lot without sucess. It is not difficult to convert an ASCII file to binary. There are two ways that come readily to mind. First, if the file contains just numeric data the following flow would work Open the file to be converted and open an output file to save the binary representation. Once you have the data streams for each of the above files, then doing data=ReadList[inputStream, Number, n]; BinaryWrite[outputStream, data, format] in a loop will do the trick. Here n is the number of items to read at any one time. If the contents of the file are not all numeric, then the loop body would be something like data = ReadList[inputStream, Byte, 4 n] BinaryWrite[outputStream, FromDigits[#, 256]&/@Partition[data,4], "Integer32"] But while the above will convert the ASCII file into a binary representation, I strongly suspect this isn't going to be that useful to you. The limitations on how large of a data set you can work with in Mathematica are determined by the amount of RAM you have and the way Mathematica represents the data internally. They are independent of how the data is stored on disk. Converting a large ASCII file to binary will reduce the file size on disk and likely will reduce the time needed to read the data back from disk. But it will not do anything to increase the amount of data you can work with at one time in Mathematica.