In all the years I've been in this business as a voiceover artist and a VO coach, the number one concern/complaint I hear from people is "My friends say there is no way I can do this". It's a question about human nature I suppose, that it can be difficult to get support from another person for ones new endeavors. However, there are always a small number of people who are naturally supportive and those are the ones we should hang with.
I don't know of anyone who has tried something that didn't work out who wasn't at least glad they tried, but I know legions of people who are miserable because of the un lived life.
We have lots of cliches that remind us of this, Nothing ventured, nothing gained. Follow your dreams, etc etc.
The most important thing about choosing any career, or not, is to not make decisions based on what you're afraid of. Every profession is very competitive, Every career takes time and training and hard work. The question is always simple. What do you love to do. What do you want to do? There is a running joke that I tell everyone in voiceover class, "People dumber than you are making a fortune"
If you are willing to do the work, you can succeed. It's never too late to become who you might have been:)