Jeffrey Dean Morgan has been in movies like “Watchmen” and beloved TV shows including “Supernatural” and “The Good Wife,” but he’s now best known as “The Walking Dead”‘s barbed-wire bat-wielding baddie, Negan.

That has its disadvantages, he told ABC News.

“I get a lot of dudes, I’ll be … getting coffee or something, and they … sidle up next to me. … I can see them sizing themselves up next to me …[like] ‘I could take Negan. I could kick Negan’s a–!” Morgan told said. “And, I’m a 50-year-old dude, man. I don’t want to throw down, but I f—ing will! Don’t be sidling up next to me!”

At times, that attention can be overwhelming. Morgan, who is married to actress Hilarie Burton, with whom he has a son, said that fans can make it difficult for him to even walk down the street.

“I’m a very private dude,” he noted. “I live in the middle of nowhere, so it’s been an adjustment, and it’s an adjustment for my family as well.”

But bad encounters are rare.

“Ninety-nine percent of fans of ‘The Walking Dead’ are some of the coolest people I’ve ever met,” he said. “But there’s that 1 percent, man. … Lock ya doors!”

This Sunday’s season seven finale of “The Walking Dead” will have Rick Grimes’ gang teaming up with several other enclaves of survivors to take on Negan and his Saviors. The episode starts at 9 p.m. on AMC.