To be a success in book publishing, you must get stocked in brick-and-mortar bookstores, who in turn require a 40% discount shipped.
Taking the latter part of the question first, there's nothing holy about the 40% discount, some publishers (particularly academic presses) sell at a shorter discount, some of the chains get a larger discount for books going to their warehouse. Accepting returns is far more important than the exact discount when it comes to making it feasible for bookstores to order your titles for stock. But if you want, you can achieve the 40% discount and accept returns using Lightning Source on demand printing for a virtual publishing solution.
But the real meat of the question is what constitutes success as a book publisher. Perhaps the best answer would be achieving the goals you set for your company when you start, and having the flexibility to build from there. If somebody's idea of success in book publishing is to publish the next Harry Potter series or compete with Random House and McGraw-Hill for gross sales, that's just not my idea of a business plan. I'm not saying it's impossible, I'm just saying it makes less sense for a new publisher without extensive industry connections than, say, starting to learn guitar after college with the goal of becoming a rock star. It happens, just not very often.
Small publishers often throw around numbers that they've seen in Publishers Weekly for the print runs of new titles from big trades, and they associate those numbers with success. Not only are print runs of tens of thousands of books impractical for a small publisher, they don't guarantee success for a large publisher either. A large percentage of those books fail to sell out the initial print run, go into the remainder channel, or get pulped. The large print run isn't just an attempt on the part of the publisher to lower the cost per book, it's the publishing industry equivalent of gorilla chest thumping.
"Thump, thump, thump. Look here, bookstore buyers. I'm printing a lot of copies of this book. I'm confident that it will sell. You can be confident I'm going to spend a lot of money promoting the book to try to get my investment back. I'm even willing to pay you some co-op money to pile them up in your window or front and center in the store. Thump, thump, thump."
Most small publishers don't have the muscle to get their books front and center in a couple thousand bookstores on launch. Even if a small publisher achieves a real publicity break through, they won't see the same sales as a big trade would with the same publicity, because they haven't managed to fill the channel with books before the publicity hit. If a small publisher wants to take a big risk on a title, print a lot of books, spend a lot of money and time on pre-launch publicity, scores some top reviews and has an aggressive distributor, it is possible to emulate the big trade model to an extent, but it's a big financial gamble and the odds of success are low. Printing all of those books makes a small publisher into a paper tiger, or a paper gorilla, but I'm convinced that virtual publishing is a much more logical business model for most publishers starting out today.
I don't doubt that the big print run, along with the frantic publicity efforts and expenses that commitment entails, give the paper gorilla a leg up if the goal is to sell 100,000 books. I don't know of any print on demand titles (at the moment) that have sold 100,000 copies without being transitioned to offset. But for small publishers whose goal is to earn a living while doing what they love, rather than mortgaging the house and getting ulcers chasing a long-shot, virtual publishing is the way to go. And for every new publisher who sells 100,000 copies of a title, I'll bet you can find 1,000 new paper gorilla publishers who spent $100,000 chasing that dream and lost their shirts. You just don't hear them bragging about it.