I would say go ahead with manual transmission because like others have said numerous times, you control the car, not your car controlling you.
My first stick shift car was 08 Scion tC and I learned it within an hour, but it took me like a month to get it smooth and driving well. You will have hard time starting on an incline, but with enough practice and learning to let go of your fear of rolling back, you'll be fine. Also, during the time when you drive manual, you will be frustrated because it's not smooth and not going the way you wanted to, but don't get frustrated because it's all learning process, I have face this many times in my tC. Have fun with your manual! You can't go wrong!