string manipulation - A simple StringCases problem

Given this text:

text1 = "  A  Vitamin D Deficiency (ICD-9-CM 268.9) (ICD-9-CM 268.9) 09/11/2015  01 ";

Yes, the ICD code values are duplicates- possibly but not certainly in all the records.

What string expression will match only the code values - twice if duplicated? I've tried:

StringCases[text1, "(ICD-9-CM " ~~ code__ ~~ ")" .. :> code, 
 Overlaps -> False]

(* {"268.9) (ICD-9-CM 268.9"} *)

StringCases[text1, "(ICD-9-CM " ~~ code__ ~~ ")" .. :> code,   Overlaps -> True]

(* {"268.9) (ICD-9-CM 268.9", "268.9"} *)


StringCases[text1, "(ICD-9-CM " ~~ code__ ~~ ")" .. :> code, 
 Overlaps -> All]

(* {"268.9) (ICD-9-CM 268.9", "268.9", "268.9"} *)

Removing the .. repeat pattern doesn't alter the outputs. What's the explanation for this behavior?

Answer

You could simply find the shortest match:

StringCases[text1, "(ICD-9-CM " ~~ Shortest[code__] ~~ ")" :> code]

{"268.9", "268.9"}

If it is possible that there is additional space or other characters a combination may be more robust:

text2 = "  A  Vitamin D Deficiency (ICD-9-CM   268.9) (ICD-9-CM: 268.9) 09/11/2015  01 ";

StringCases[text2, Shortest["(ICD-9-CM" ~~ __ ~~ code : NumberString ~~ ")"] :> code]

{"268.9", "268.9"}

---

Albert Retey suggested a method scanning for Except[")"] to restrict the pattern to a single field. I believe this is equivalent to my use of Shortest, but I was curious to find out how the two compared. We can view the Regular Expression that Mathematica internally converts each StringExpression to with StringPattern`PatternConvert. I will include two placements of Shortest.

StringPattern`PatternConvert /@ {
   "(ICD-9-CM " ~~ Shortest[code__] ~~ ")",
   Shortest["(ICD-9-CM " ~~ code__ ~~ ")"],
   "(ICD-9-CM " ~~ code : (Except[")"] ..) ~~ ")"
 } // Column

{"(?ms)\\(ICD-9-CM (.+?)\\)", {{Hold[code], 1}}, {}, Hold[None]}
{"(?ms)\\(ICD-9-CM (.+?)\\)", {{Hold[code], 1}}, {}, Hold[None]}
{"(?ms)\\(ICD-9-CM ((?:[^)])+)\\)", {{Hold[code], 1}}, {}, Hold[None]}

We see that both Shortest placements produce the same RE. We also see that the method using Except is more complex, and would appear to be performing additional testing that is ultimately unnecessary. I would expect it to be slower, and it is, but only slightly:

textBig = StringJoin @ ConstantArray[text1, 1*^6];

StringCases[textBig, Shortest["(ICD-9-CM " ~~ code__ ~~ ")"] :> code] // Timing // First


StringCases[textBig, "(ICD-9-CM " ~~ code : (Except[")"] ..) ~~ ")" :> code] // 
  Timing // First

0.718

0.812

Interestingly the hand-written RE is quite a bit slower:

StringCases[textBig, RegularExpression["\\(ICD-9-CM\\s*([^\\)]*)\\s*\\)"] :> "$1"] // 
  Timing // First

1.872

I started a new question about this here: Why is StringExpression faster than RegularExpression?