We (wife & I) worked hard & borrowed when young and definitely missed out on travel, which is the biggest drawback imo, but we had a future plan...until the GFC. Our plan was solid as long as the economy kept climbing - which it hasn't. In saying that, our 3 young kids are the reason now not to travel far (cost too much), but we've found plenty of camping places within a day's drive and plan a big (2 month) Aussie trip in the next couple of years.
I think it can be a balance and if you feel comfortable borrowing (safely), do so - if not, scrimp & save - you'll get there in the end.
