Teeth have shifted after braces

I am a 32 year old woman. At age 14, 9th grade, my braces were put on. I wore them for one year, removed 10 grade. Shortly there after, I was fitted for my retainers. I was first told to wear my retainers for 24 hours of every day for months. After another check up, I was told to only wear them at night. After one year, my orthodontist told me I didn't have to wear my retainers ever again. When I got to college, I was told by my peers that this was incorrect. I've met several people that told me your retainers are always to be worn. Over the years, my teeth have shifted back and I am very displeased! I don't feel I need to pay for braces or Invisalign after being told the wrong information. What am I supposed to do?