I said, 'I think you have the wrong person,'" Morgan tells PEOPLE exclusively. "I live on a farm, and I kill zombies, and I'm a dad, and I'm kind of a pain in the a--."Throughout his decades-long career as a successful actor,

Any Walking Dead fan will tell you, if there is one guy you don’t want to piss off, it is the dude known for carrying around the barbed wire-covered baseball bat. But that's exactly what the cast of Destination X did when they got on host Jeffrey Dean Morgan ’s bad side at one point during filming.

Destination X whisks contestants away on the journey of a lifetime as they travel through Europe and put their geography knowledge to the test in hopes of winning it all and taking home $250,000 – but where would the players be without their host?