Houkouonchi, sounds like you're a pretty sharp guy and think of things before you react, so I'll just give you some food for thought.
Regarding paying off your mortgage early, there are valid reasons for doing this, especially if your interest rates are on the high side. If not, consider investing the difference in a moderate portfolio (maybe an equal mix of bond/stocks, mutual funds are ok). But once again, you must invest the difference, not spend it.
The question we need to ask ourselves is who's better off? Someone who has paid off his mortgage but has no cash or someone who has a $100,000 mortgage but $150,000 in cash? I know what my answer is.
Once again, good luck to you.