matrix - Cell-adjacency Graph of a Square Mesh

Consider the list of points

pts = {{1, 1}, {1, 2}, {2, 1}, {2, 2}}

I want to use them to define a 2x2 square mesh using VoronoiMesh, where each cell has two neighbours. Following the discussion in this question, consider the following code

mesh = VoronoiMesh[pts, ImageSize -> Medium];
conn = mesh["ConnectivityMatrix"[2, 1]];
adj = conn.Transpose[conn];
centers = PropertyValue[{mesh, 2}, MeshCellCentroid];
g = AdjacencyGraph[adj, PlotTheme -> "Scientific", 

   VertexCoordinates -> centers];
Show[mesh, g]

As one can see, unlike other meshes, this one does not seem to work exactly as I want, since the diagonal edge should not appear. Why is this happening? Any way of avoiding that edge and get

as one would expect from a square lattice?

Edit: As noticed in the comment section, some of the polygons seem to have sharing edges that are single points, which is enough for them to be considered neighbouring cells. This effect is unchanged with the size of the lattice. If I consider, for example, the points

pts = Flatten[Table[{i, j}, {i, 7}, {j, 5}], 1];

I get

Any ideas on how to solve this? Maybe omit the extra edge in a way that doesn't this or other non-square meshes. For example, considering a random VoronoiMesh, nothing seems to wrong, though it could, theoretically, go

Answer

We can delete the rows in our incidence matrix that correspond to these edges of length 0.

pts = Flatten[Table[{i, j}, {i, 7}, {j, 5}], 1];

mesh = VoronoiMesh[pts, ImageSize -> Medium];
conn = mesh["ConnectivityMatrix"[1, 2]];


lens = PropertyValue[{mesh, 1}, MeshCellMeasure];
$threshold = 0.;
keep = Pick[Range[MeshCellCount[mesh, 1]], UnitStep[Subtract[$threshold, lens]], 0];
conn = conn[[keep]];

adj = Transpose[conn].conn;
centers = PropertyValue[{mesh, 2}, MeshCellCentroid];
g = AdjacencyGraph[adj, PlotTheme -> "Scientific", VertexCoordinates -> centers];


Show[mesh, g]