I've done both, as well as had several mediocre jobs where the pay was alright but nothing special. Working at a place you hate is hell, and even if it pays you enough money to live well out of work you'll spend every working day wondering if it's worth throwing your life away just for nice things. Work in a job you like, and even more importantly with people who are friends as well as colleagues, and work becomes a pleasure. OK, it'll take a bit longer until you have enough cash for a new car, but who cares? You're happy.
I'm in my late 50's and have experienced both scenarios. I'd rather work someplace where I loved the job and the people vs. make a lot of money and be miserable. Of course, it's not as simplistic as that: at the lower pay job we still need to make enough to meet our financial obligations, so I'd have to qualify my answer by adding that I could not work a job that could not pay me enough to sustain my relatively spartan lifestyle, even if I would love the work and people.
I work a brow-beating mother of a job...But, I can retire in 4 years with a great pension and benefits...IT sucks...But I got kids in college and not being financially secure brings me anxiety...if I could live another life I would not have kids and work at a guitar shop...
There are 24 hours in each day that we exist on this planet. We sleep about 8 of those, so that leaves us 2/3rds of our life to enjoy being here. If we spend another 8 plus hours earning all that money while being miserable, that leaves, well you can do the math.
To me quality of life is uppermost in importance, and I mean all 24 hours.
If one has a scheme whereby you can sacrifice some period of time busting ass, and then retire for the rest, that might be a good investment. I spend some years in the Navy, now I get free health care.
I like the addage, "if your having fun while your wasting time, your not wasting time".
Find something you love to do that earns money, then figure out how to live on that budget is my advice.