matrix - How to accelerate combinations and sum calculations here?

I have a 4d matrix H defined as below (embedded with combinations and sums)

H[n1_, n2_, n3_, n4_] := 
(1/(4*Sqrt[n1*(n1 + 1)*n2*(n2 + 1)*n3*(n3 + 1)*n4*(n4 + 1)]))*
   Sum[(-1)^(m1 + m2 + m3 + m4)*Binomial[n1 + 1, m1 + 2]*Binomial[n2 + 1, m2 + 2]*Binomial[n3 + 1, m3 + 2]*Binomial[n4 + 1, m4 + 2]*
     Binomial[m1 + m3, m1]*Binomial[m2 + m4, m2]*(m1 + m3 + 1)*(m2 + m4 + 1)*
     (((m1 + m3 + 2)/2^(m1 + m3))*Sum[Binomial[m1 + m3 + k + 2, k]/2^k, {k, 0, m2 + m4 + 1}] + 
      ((m2 + m4 + 2)/2^(m2 + m4))*Sum[Binomial[m2 + m4 + l + 2, l]/2^l, {l, 0, m1 + m3 + 1}]), {m1, 0, n1 - 1}, {m2, 0, n2 - 1}, {m3, 0, n3 - 1}, 
    {m4, 0, n4 - 1}]

Here n1, n2, n3, n4 are indices that range from 1 to N.

When N = 6, it takes my Mac (2.7 GHz Intel Core i5) about 30s to figure out the whole 4d matrix. However, it takes 1.5hrs for N=11. The time scaling is highly nonlinear..

The problem is that when N is big (N>20), it takes forever to run. Anyone has some good suggestion to accelerate the calculation? Thanks.

UPDATE: Actually I do not need all the matrix entries, please see here:

Delete rows and columns in a matrix based on the element index

After getting and 4d matrix H and reshaping it into a 2d matrix, I will delete according to the rule:

If i>j OR k>l, then delete this element.

So it seems redundant to figure all the entries out in the first place?

Answer

You could introduce further conditional definitions for H which will prevent those computations whose results would end up being thrown away.

For instance, you could add:

H[i_, j_, k_, l_] /; (i > j || k > l) = Missing[];

As a toy example:

m = Table[H[n, 2, 3, 4], {n, 1, 10}]
(* Out: {(3 Sqrt[5])/128, (5 Sqrt[15])/256, Missing[], Missing[], Missing[]} *)

DeleteMissing[m]
(* Out: {(3 Sqrt[5])/128, (5 Sqrt[15])/256} *)