Is it possible to split a large text file into smaller files by line numbers in Mathematica?
Answer
Yes, of course.
text = Import["http://www.norvig.com/big.txt"];
StringLength[text]
(* 6488665 *)
lines = StringSplit[text, "\n"];
Length[lines]
(* 128457 *)
breaks = Union[Append[Range[1, Length[lines], 20000], Length[lines]]];
MapIndexed[
Export["~/Downloads/textPart" <> ToString[#2[[1]]] <> ".txt",
Take[lines, #1], "Text"] &, Partition[breaks, 2, 1]]
(* {"~/Downloads/textPart1.txt", "~/Downloads/textPart2.txt", \
"~/Downloads/textPart3.txt", "~/Downloads/textPart4.txt", \
"~/Downloads/textPart5.txt", "~/Downloads/textPart6.txt", \
"~/Downloads/textPart7.txt"}*)
Verification with Unix / Mac OS commands:
[14:40:26]Downloads> ls | grep textPart | xargs -Iaaa wc aaa
20000 239243 1398793 textPart1.txt
20000 166040 1042501 textPart2.txt
20000 155655 948331 textPart3.txt
20000 166764 948771 textPart4.txt
20000 169422 964634 textPart5.txt
20000 167882 957716 textPart6.txt
8456 30737 228187 textPart7.txt
Comments
Post a Comment