I spoke to him in the Ruby Room, but didn't go with him. I said I would meet him. However, there is no marker for where I need to go.

I know it's the haunted house with the godling, but I have no idea where that is in Novigrad. I've done numerous searches and I've not seen this question before.

Can someone help me out, please? This is horribly frustrating.