The story
Answer hides another word inside it: swear. Old English andswaru was built from and-, "against", and swaru, "an affirmation, an oath". Its original sense was probably legal: a sworn statement given in reply to a charge. To "answer" an accusation was to swear against it.
The W belongs to the swear half of the word, and in Old English it was pronounced: roughly AND-swah-roo. As the compound wore down in everyday speech, it lost pieces from the middle. The D of and- went, and so did the W, which sat in an unstressed syllable where sounds erode fastest.
Nothing happened to the spelling. That is why answer still looks like "an-swer" — and why it makes sense once you know the word is really "against-swear". The verb swear itself, stressed and on its own, kept its W.
