Reformulating my question in simpler terms.
Take two lists
list1 = RandomInteger[{1, 100000000}, 25000000];
list2 = RandomReal[{1, 10}, 25000000];
Make an Association out of them
as = AssociationThread[list1, list2];
Export["somewhere", as, "WDX"];)
And I want to do this for even larger lists. At the moment I can't manage it - the reason being that Mathematica eats all my RAM. I have noticed that when I try creating the Association there is a spike in RAM usage (in steps of a few hundred Megabytes). If a make the lists a bit smaller, so that the command can be evaluated, after the spike is gone, I can go on using as in other calculations with far less RAM being taken. This spike in memory usage is also seen when I Import the exported file (the file is around 200mb).
My question is whether it is possible to avoid this somehow - maybe create (and later Import) the Association in pieces or something else? I also tried using a SparseArray instead, replacing the AssociationThread code by
sa = SparseArray[Table[list1[[i]] -> list2[[i]], {i, 1, Length[list1]}]];
The spkie is still there, even more prominent than with the Association. Not only that but another problem comes up, even if I reduce the number of elements significantly, but keep the large values in the first list, which means that the dimension of the SparseArray will be enormous. When I try using it in a calculation, my Mathematica just resets (that is - clears all definitions - like when it runs out of memory). Is this expected behaviour or am I doing something wrong.
Thank you
Comments
Post a Comment