I absolutely think there's an underlying angst to Zeb and Kallus' relationship at its core. Kallus experienced something traumatic that made him view all Lasat as violent monsters. Which is obviously a messed up viewpoint and one he's doing his best to consciously challenge... But there's definitely moments in the beginning where the subconscious fear gets the better of him. Waking up from a ptsd nightmare to a Lasat reaching for him is not something he handles with much grace or sensitivity and it's definitely a thing that complicates their relationship.
It's worse for Zeb, 'cause while Kallus is left with guilt and shame, Zeb has to deal with burning anger and resentment during similar episodes. When he has a flashback Kallus is just about the last person he wants to see the face of, and unlike Kallus he can't just reprimand himself for misplaced emotions 'cause Kallus carries more blame than Zeb likes to think of. It definitely takes him a long time to reconcile the Kallus he knows - loves - with the Kallus who made his world - his family - crumble to pieces, even if part of his people survived.
It's a lot - to Kallus, Zeb is worth it. He can't quite wrap his head around how, but somehow Zeb thinks he's worth it too - they're worth it.