Java is open source, even C# is open source now (announced by Microsoft Nov, 2014). I agree C# is less verbose than Java, C# has more support and backing from Microsoft. You are targeting Microsoft platforms only (it's worth remembering that Microsoft effectively cloned Java to create C# because they wanted a Java-like language that would lock people into Windows. They could have used Java, but that would have enabled people to easily run applications on other platforms....)
On the other hand, Java has some big advantages as well:
Huge open source ecosystem - the libraries for Java that you can get for free are by far the best of any language. It is hard to overstate the importance of this - from the point of getting things done, Java is very effective. Tools - the Java tools are in my view better than what you can get the the .Net world. e.g. Maven (once you have mastered it!) is particularly impressive. Maintainability - Java has been around a while and has been successful in big companies precisely because it is relatively stable and there has been a lot of effort put into backwards compatibility. The simple and slightly verbose syntax also helps Java here - it's easier to read and maintain code if the code is very clear and explicit. New languages - The JVM has some amazing new languages (Scala, Clojure, Groovy etc.) that are the future of the Java platform. This is where much of the language innovation is happening, and it's happening much faster than in either Java or C#.
So Java vs. C# is a pretty close call and it really comes down to whether you want to be in the Microsoft camp or the Open Source / cross-platform camp. Personally, I prefer Java because:
The library ecosystem is in my view much more important that the fact that C# has nicer syntax than Java In the long run, I want all my code to be properly cross-platform and able to run on big clusters of cheap linux machines on the cloud. Clojure is IMHO the most promising language in the world right now, and if I stick with the JVM platform I'll be able to transition my code and skills easily into Clojure over time.
Before, 2014, most of the start-ups, built their stuff using Java because it was open source since 1995. I don't see much difference in Java & C# in terms of concepts of programing or understanding programming fundamentally. So, it depends on you, I would go with Java. All the best!!