I am about halfway through the game at this point, I was wandering from east to west past the settlement outside of Novigrad when a female elf came out of nowhere, said "remember me" and attacked. Problem is I don't remember her, anyone know who it was? I have come across several ungrateful elves that I tried to help out, it could even have been that elf that I let go in Flotsam in Witcher 2.

No, that's the elven commander from the quest "woodland beast" http://thewitcher3.wiki.fextralife.com/Contract+Woodland+Beast (look it up in your quest log). She attacks you if you let the elves in the woods go (or say that you do) and then betray their presence/position to the guards anyway. So she's basically right, you did betray her.

Also fun fact, there's another topic about the same quest on page 1: https://www.gog.com/forum/the_witcher_3_wild_hunt/spoilers_question_about_woodland_beast

That would make sense, thanks.