list manipulation - How to merge Datasets representing matrices with named rows/columns?

I am working with square (non-symmetric) matrices with named rows and columns. The data I receive is sometimes messy and rows/columns may come in various permutations. I usually need to semi-manually put every matrix into a canonical permutation before I can work with it. I'd rather never have to think about permutations and not even define a canonical one, just index the matrices using the names of their rows/columns.

I was hoping that Dataset would be helpful here, and I made Dataset objects like these:

ds1 = Dataset[<|"a" -> <|"a" -> 1, "b" -> 2|>, "b" -> <|"a" -> 3, "b" -> 0|>|>]

ds2 = Dataset[<|"a" -> <|"b" -> 1, "a" -> 5|>, "b" -> <|"b" -> 1, "a" -> 2|>|>] (* I used a different permutation on purpose *)

What is the simplest way to merge these together to obtain the following?

Dataset[<|"a" -> <|"a" -> {1, 5}, "b" -> {2, 1}|>, "b" -> <|"a" -> {3, 2}, "b" -> {0, 1}|>|>]

Instead of merging with List I will usually need to merge with other operations, e.g. Plus. Once I've built a dataset with List it's easy to do that though.