What is the meaning of the phrase "talking overmuch is a sign of vanity"?

For modern speakers of English, this phrase may be somewhat difficult to understand because the word “overmuch” is not one that is commonly used today.  It is a much more archaic usage.  The phrase that you have given us means something like “if a person talks too much, it shows that they are vain.”

In English, we can use the word “over” in a variety of ways.  One way we can use it is as a prefix meaning something like “excessively.”  For example, if I “overbake” a cake, it means I baked the cake too long and it is excessively cooked.  If I say that someone “overstayed” their welcome, it means that they stayed too long.  In the phrase you have given us, “over” is being used in this way.  The word “overmuch” means “excessively much.”

Once we understand this, we can understand the phrase better.  It is saying that talking overmuch (talking too much or excessively much) is a sign of vanity.  Vanity is the quality of being excessively proud of oneself.  If you talk too much, it could very well mean that you are too proud of yourself.  You think that your own opinions are so great that everyone should hear them.  You think everyone will be fascinated by the stories you tell.  You are, in a word, vain.

Thus, this phrase means something like “if a person talks too much, it is evidence that they are vain.”