If you refuse to accept a statement, and justify your refusal by criticizing the person who made the statement, then you are guilty of abusive argumentum ad hominem — the truth of an assertion doesn't depend on the virtues of the person asserting it.
To reject a proposition based on the fact that it was also asserted by some other easily criticized person is another form of abusive argumentum ad hominem.
Try and persuade someone to accept a statement you make, by referring to that person's particular circumstances, and you are also an abuser.
It's not always invalid to refer to the circumstances of an individual who is making a claim. If someone is a known perjurer or liar, that fact will reduce their credibility as a witness.