The following are what I believe to be the three most important areas of radical (vs. evolutionary) innovation in web application server components over the coming 6 - 18 months:
- Non-relational databases. SQL has overstayed its welcome: it is a fundamentally non-scalable solution, and has been overapplied to too many different types of data storage problems. There are already a handful of extremely viable next-gen datastores including CouchDB, Tokyo Cabinet/Tyrant, Redis, MongoDB, and SimpleDB. If you’re starting a new, non-time-critical project today, you should try using one of these instead of defaulting to a SQL database.
- Recipe-based server provisioning. Forget about the “cloud” hype and zoom in on a real technique: how to construct a virtualized server instance. Recipe-based configuration systems like Chef, Puppet, and CFEngine are one part of this. Another is building and managing of machine images (what on EC2 are called AMIs); Xen Tools is an early example. The final piece is a management interface for provisioning, monitoring, and deprovisioning of servers and related resources like IP addresses; commercial products (RightScale, Engine Yard Solo/Flex, Cloudkick) and open source (Pool Party) are tackling this area.
- Work queues. aka workers, background jobs, background processes. Workers are processes that run asynchronously to any web request, doing heavy lifting and communicating to the front-end web processes via a message bus like RabbitMQ or SQS. It also encompasses the realm of scheduled jobs (aka cron), although how these two interact exactly is still very much up in the air.
The field is wide open for innovation, discovery of best practices, and standardization to happen in all of these realms. It’s an exciting time to be a dynamist working in web software.