Err... When I try to Import a .dat file by
dat = Import["D:\\TIEGCM\\solar_input\\omni_m_all_years.dat",
"Table"];
or
dat = Import["D:\\TIEGCM\\solar_input\\omni_m_all_years.dat"];
mathematica run out of memory and crashed the kernel,I admitted that the dat file is a bit large ~38Mb, however, it shouldn't have been a problem considering my 2G RAM. And when I checked the progresses by Process Explorer, I found that the Private Bytes of one MathKernel.exe grows rapidly from 63Mb all the way to more than 700Mb when I evaluate the code dat = Import[...] listed above. How could this happen?Help!!!!!
the omni_m_all_years.dat file can be download under ftp://cdaweb.gsfc.nasa.gov/pub/data/omni/low_res_omni/ (in fact you can Import the dat file directly from website if your Mma has good connection with internet) and I use Mathematica9.0 under Windows7
Answer
M9 under Windows 7, Intel Core i7 970. 24 GB RAM, Import without any problem.
Alternatively you can read the file as a stream.
str = OpenRead["z:\\temp\\omni_m_all_years.dat"]
ReadList[str, Record] // StringSplit
Close[str]
You may add //ToExpression to get strings converted to numbers
Example for 3 records (lines in text file):
ReadList[str, Record, 3] // StringSplit //ToExpression
{{1963, 2, 21, -3.2, 26.1, 999.9, 999.9, 999.9, 999.9, 9999., 999.9,
999.9, 999.9, 1.*10^7}, {1963, 2, 22, -3.2, 26.2, 999.9, 999.9,
999.9, 999.9, 9999., 999.9, 999.9, 999.9, 1.*10^7}, {1963, 2,
23, -3.2, 26.2, 999.9, 999.9, 999.9, 999.9, 9999., 999.9, 999.9,
999.9, 1.*10^7}}
Hint: a) You may close all other applications running including browser and b) increase virtual memory cross-check available resources using Piriform Speccy (https://www.piriform.com/speccy).
EDIT: Memory requirements using Import :
mmu = MemoryInUse[];
Import["z:\\temp\\omni_m_all_years.dat", "List"] // Timing
(MemoryInUse[] - mmu )/2^10 // N
(*58075.6*)
Memory requirements processing stream with ReadList:
m = MemoryInUse[];
str = OpenRead["z:\\temp\\omni_m_all_years.dat"];
ReadList[str, Record]
(MemoryInUse[] - m )/2^10 // N
(*59178.1*)
Memory requirements processing stream with ReadList and splitting records into substrings and convert to expressions :
m = MemoryInUse[];
str = OpenRead["z:\\temp\\omni_m_all_years.dat"];
ReadList[str, Record] // StringSplit // ToExpression
(MemoryInUse[] - m )/2^10 // N
(*163 403*)
Comments
Post a Comment