First, learn about yourself. What do you like to do and can make money doing? What can you be the best in the world at? What are you already naturally good at? With whom do you like to associate? Whom would you want to serve?
Start building your "empire." Earn trust. Get attention. Let your network know what you're about. Certainly go through worse case scenarios, then ask yourself if it would be worth it. What is the worst that can happen? What are you most afraid of? What are some other obstacles in that stand in your way?
Now start testing the market. Have you niched down enough? Do you have potential customers? Do you have current customers? How will you find customers? Survey people. Ask people if they would pay x amount of dollars for x product or service. Create something people will actually want to buy.
Finally, the timing will never be right. Be realistic, but courageous. Be conservative, but aggressive. Work in the evenings and on the weekends, then, if you are ready and you just can't take it anymore, go for it. If you decide not to, that's perfectly fine, but you may be wondering the rest of your life what it would have been like if you did take the plunge.
So go, make the impact you want to make and prove to yourself you can do this, but only if you're into that sort of thing.