QDM, you're on the right track already by asking these questions. I think the real answer depends on your goals. I own a farm, I own rental properties and I own stock market related investments. I sell real estate for a living, so I most definitely believe in owning it. Those who "lost" money in their 401k's during the big decline have gained that back by now. The stock market is risky, but over the long haul, it will likely provide you a good return. Rental properties will definitely give you a better dollar return than a farm, but nothing will likely bring you more pleasure than a farm.
There is nothing I love to do more than hunt big, old mature bucks...5 years or older. It's a really tough addiction to have because you need a huge piece, or one next to a large piece that is not hunted. My wife and I also want to pay for our children's college education. So there is a balance to be found. I would love to do nothing but buy farms, but that isn't going to provide an education for my kids.
I would not agree with some of the statements that there is no better investment than land, IF you're looking for a $ return. Yes, it will appreciate over the long-term. Yes, it is fairly safe. And yes, it provides recreational pleasure with family and friends...a very valuable and worthwhile "return" on your investment. And yes, it will appreciate over the long term. But again, it depends on your goals. For guys who say they are going to use their farms as "retirement,"they can only do so IF they sell them (unless there is consideration income producing acreage). I don't ever want to sell my farm(s). I want to buy more. My idea of retirement is enjoying the farm(s) I worked to buy.
Buying our farm is most definitely one of the best decisions we ever made, as we live on it. But Truth and Honest is right, it's not worth it if you can't afford it without making unreasonable sacrifices. It sounds like you have the discipline to make sacrifices in areas that won't create a huge amount of additional stress in your life. If you can, go for it! If not, I don't see recreational land prices increasing very much over the next few years, so waiting a bit is likely not going to hinder your ability to buy.
Lastly, here's the worst "problem" with buying land...no matter how much we have, we always want more!