Since most engineering decisions are a matter of compromises - a billion user requirement results in a loss somewhere else. Maybe data quality, maybe software maturity, maybe functionality, maybe hosting costs. And some of these compromises may interfere with the project getting to 10000 users.
Probably a few factors involved:
- Programmer optimism - most innovative people I know are inherently optimistic. If they weren't optimistic then they often wouldn't bother. And perhaps after spending x number of hours creating it naturally increases ones optimism.
- VC funding, goals and rewards - ok, just a guess, but I'm thinking that many startups and VCs are more interested in gambling on sites that can go very large, rather than ones that might just be moderately successful. I've also heard that VCs are pushing start-ups away from relational databases.
- The hype-cycle - the fashionable trend is definitely in support of very large big-data applications.
- False sense of scalability limits in relational databases - the partitioning, optimization and parallelism limitations of MySQL have convinced quite a few that relational databases are very slow at processing large volumes of data. Though MySQL hardly defines the limits of the technology.
- Career building - ok, there's clearly a personal advantage to be had in working on the newest and most exciting. No good developers are looking forward to working with Visual Basic, COBOL, etc any longer, and I don't think we should be surprised that we follow the incentives.
Of course, there are other reasons besides scalability that may warrant going for the big-data architecture right out the gate. Testing a potential future architecture, PR, establishing an iron-clad reputation for scalability, etc all have some value. But I think it's best if decisions like this are very deliberate and conscious - more of a calculated risk than an unnecessary one.
But at the end of the day, when practical competes with sexy the outcome is predictable. But not always pretty.