"To a stranger"

I think the poet is talking about a past lover, most likely a childhood friend or someone who he has known since he was young, who he developed feelings for. I think this because of the lines "You grew up with me, were a boy with me or a girl with me,
I ate with you and slept with you, your body has become not yours" I think this refers to someone he has become very close to on a friendship level but he might have wanted it to be more than friends. I also think the poet expressed his feelings for this person, boy or girl, through poetry. He couldn't speak to anyone about these feelings he was having, because it was unacceptable and looked as a sin and an act of the devil. So they only he would express these feelings was through literature. 

Honestly I can understand how someone could believe this was about a sibling, I just don't see that. He seems like a man of high morals because he had good jobs, an good education. So I do think it was about a lover who he had met, but something happened and then they became strangers.