Albatross around your neck
From Idiom Wiki
This idiom originates from the poem, "The Rime of the Ancient Marniner." In it, the main character kills an albatross, which is usually regarded as a sign of good luck, and afterward is forced to wear the bird around his neck. The phrase has come to refer to a burden or stigma, often caused by some offense, which a person is forced to bear.  It appears in popular culture often and is the subject of songs by Bastille (The Weight of Living, Part 1) and the Foals.