dynamic - Using more than one set of Locators in a Manipulate

I need to add locators dynamically to different sets.

With one set this works OK:

Manipulate[
 Graphics[{Point@pts}, PlotRange -> {{0, 1}, {0, 1}}],
 Control[{{pts, {RandomReal[1, 2]}}, ControlType -> Locator}],
  Row[{
   Button["Add L1", pts = pts~Join~{RandomReal[1, 2]}]}],
 Dynamic@pts]

But when I try to use more than one set, it doesn't:

Manipulate[ (*fails*)

 Graphics[{Point@pts, Point@ptsNA}, PlotRange -> {{0, 1}, {0, 1}}],
 Control[{{pts, {RandomReal[1, 2]}}, ControlType -> Locator}],
 Control[{{ptsNA, {RandomReal[1, 2]}}, ControlType -> Locator}],

 Row[{
   Button["Add L1", pts = pts~Join~{RandomReal[1, 2]}], 
   Button["Add LNA", ptsNA = ptsNA~Join~{RandomReal[1, 2]}]}],

 Dynamic@pts, Dynamic@ptsNA]

I could work out my way by adding all locators to the same set and manually assigning them to one "set" or another, but in my real application I have to add a different number of locators to each set at a time and also be able to delete them, so that complicates the code.

Edit

Thanks to Sjoerd's comment in another question, I found this limitation is covered in the documentation:

Due to internal limitations, it is not possible to combine individual Locator variables with a variable that is a list of multiple Locator variables: you can have only one multipoint Locator variable in a Manipulate. However, in exchange, it is possible to add the option LocatorAutoCreate->True to that one Locator multivariable specification, and thereby allow you to create and destroy Locator points interactively (changing the length of the list of points stored in the variable).

So, no hope using Manipulate[], accepting jVincent's answer.

Answer

I know this doesn't answer the question of the buggy behavior, but I would suggest an implementation without using Manipulate:

DynamicModule[{pt = {}, pt2 = {}},
  Column[{

     Graphics[{
       Dynamic@MapIndexed[With[{i = #2[[1]]}, Locator[Dynamic[pt[[i]]], Appearance -> Small]] &, pt],
       Dynamic@MapIndexed[With[{i = #2[[1]]}, Locator[Dynamic[pt2[[i]]], Appearance -> Large]] &, pt2] 
     }, PlotRange -> {{0, 1}, {0, 1}}],

     Row[{

        Button["Add L1", AppendTo[pt, RandomReal[1, 2]]], 
        Button["Add LNA", AppendTo[pt2, RandomReal[1, 2]]]
     }]
  }]
]