What is the humour in the poem "The Mending Wall?"

The poem tells of the annual walk of the narrator and his neighbor, as they put back the rocks that have fallen from the wall that marks their property line. Some stones have been dislodged by gravity, some by weather, and some by hunters or other folks. The poet seems to find more humor in this activity than his neighbor does. The other owner merely keeps saying, “Good fences make good neighbors.” But the rocks are irregularly shaped, and sometimes it takes a bit of work to get them to stay put. The narrator jokes:

We have to use a spell to make them balance:

"Stay where you are until our backs are turned!" (lines 18-19)

He sees this rock-replacing task not as work, but as something interesting and fun, like a sport:

Oh, just another kind of out-door game,

One on a side. (lines 21-22)

At a point where the wall disappears, and where his apple trees and the neighbor’s pines make another kind of natural border, he says:

My apple trees will never get across

And eat the cones under his pines, I tell him. (lines 25-26)

He makes a bit of fun of his neighbor’s repetitive remark. Evidently neither one of them owns farm animals that need to be kept behind such a wall:

Spring is the mischief in me, and I wonder

If I could put a notion in his head:

"Why do they make good neighbors? Isn't it

Where there are cows? But here there are no cows. (lines 28-31)

As for how the rocks have fallen off in the first place, the poet has an additional theory he thinks he could pose to his friend, in jest:

I could say "Elves" to him,

But it's not elves exactly, and I'd rather

He said it for himself. (lines 36-38)

He would like his friend to be having as much fun as he is. When the poem is read with an eye to look for such humorous statements, it turns out that the verse is much funnier than you may expect it to be, at first. You get the impression that this annual walk means much more to the poet’s serious neighbor than to the poet himself.