Just thought about why Five trusted Vanya the most even after so long.
Other than Dolores, his next most constant (companion) is Vanya's book. It's her thoughts and experiences he's reading and accompanied by. He may hear Dolores's voice in his head, but amongst his siblings, Vanya is the one who he feels is by his side, making him feel like he can trust the woman that wrote the book when he came back.
Just a thought though.