Doubt
HEADCANON:
Despite how much of his heart and soul he puts into it, James has so many doubts about himself being a father. His own father had always been a great example, and that was never in question, but he's seen first had what shitty parenting can do to a child. The first time Sirius had openly talked about his parents, something about it, - not just a hatred for how they treated Sirius, their son, but a sense of unrest that came with knowing people could just be like that to their children and be perfectly content with it, - just never sat right with him.
When he and Lily found out she was pregnant, the first thing he felt, above all else, was euphoria. Pure, complete happiness. But the doubt crept in, and the fear, and even when Harry had arrived, and he was set in James' arms in a little bundle of soft blankets, James couldn't deny that haunting doubt that loomed in the back of his mind.
He always wanted to be a good father. The only thing standing in his way was his own insecurities, doubts and fears.