Anatomy of a Modern Website

Why building a "simple" site is more complex than you think

Say you're a software engineer. You come up with a great idea for a product: something novel, innovative, maybe even revolutionary. You know that once it's released, customers will flock to buy it, log in, or download it. You've done plenty of market research, you've identified your target audience, and you've got all your requirements, specifications, and (most importantly) financing nailed down. Everything's going smoothly, and you're just about ready to release it into the world.

Oh, and you'll probably need a website. Something simple, though — how hard could that be? You're a programmer, after all.

How Times Have Changed

If it were 1996, you'd probably be right. Find a cheap webhost, fire up Notepad++, write some HTML, and maybe put together a few images to spruce it up a bit.

It goes without saying, though, that in 2015 the face of the Web is markedly different from what it once was. Users have become accustomed to a certain amount of professional polish in the websites they use, and the technological landscape has become complex enough that performance, security, and scalability are mandatory considerations. Having a website that's "good enough" simply no longer is.

Building a modern website from scratch can be a daunting challenge to even the most seasoned software veteran. For those less experienced, however, the process is WordPress, with the promise of many PaaS providers (such as Elastic Beanstalk, Heroku, and Google App Engine) sometimes giving a false sense of the number of considerations needed for even a basic company site.

And once you move beyond a static, purely informational site and add even the smallest bit of interactivity or server-side code, you have new issues to face: You're going to want to keep tabs on how users are interacting with your site, be aware of how your server infrastructure is holding up, get notifications when things go wrong, create automated and repeatable deployments, and ensure that scalability and security considerations have been taken into account. And that's all post-launch, without even taking into account the process and tools used in developing the actual site.

The Paradox of Choice

In start contrast to the early days of the web, one of the toughest challenges faced by developers in the modern era is sifting through and understanding the incredible number of architectures, platforms, frameworks, and tools that are available for building and managing websites.

Part of the problem is that it's difficult to organize these tools into distinct categories because so many of them deal with overlapping concerns: Something like Ansible, for example, can be an amazing tool for configuration management, but you could also use it as a provisioning tool as a substitute for CloudFormation. An ELK stack could handle your log collection and analysis, but you'd need a separate alerting system. Graylog could handle log analysis and alerting, but you'd need a log collection system. It's easy to get turned around with so many tools doing so many things.

The 12 Factor App does a great job of outlining how a modern, robust web application should be written and structured, but where do you find something similar that describes, rather than software architecture, the critical aspects of web design, infrastructure, and analytics?

Well, as it happens, we at Touchstone recently went through this very process ourselves, putting together our corporate site and blog, and we've come up with a convenient checklist of key decisions you'll need to make if you're going to develop one of your own.

8 Pillars of Modern Web Development

Catchy, huh? Well, you know what they say.

Intended Audience

This list is intended for developers and dev-ops engineers who are facing the task of building a new website, whether as one-man job in a startup, or as a project for a team that's part of a larger organization. For some, particularly those working for bigger companies, many of the decisions will have already been made for you. This can be a blessing! You have more time to focus on the choices you do have control of. For those on their own, don't be afraid to just pick something and go with it; find out quickly whether it's working for you and re-evaluate early on.

How to Use the List

The checklist is designed to guide your research, but is by no means exhaustive. For each decision on the list, you'll want to decide based on the scope of your site, requirements, and timelines whether to:

  • Build - Do it yourself or write an extension to an existing tool you use
  • Buy - Use a paid, free, or open-source product, but install and configure it yourself
  • Rent - Use a paid or free SaaS provider to handle it entirely

The best way to use the list is to treat it as a kind of traceability matrix. You definitely won't need separate solutions for every decision on the list, so try to check off as many items as you can for each tool you use, see how they overlap, and (more importantly) see what aspects you are missing.

Pillar 1: Fundamentals

task and issue management system

Track goals, progress, and bugs consistently.

Yep, you're going to need it. A programmer's best tool is going into a project with direction and focus, setting boundaries and staying organized. Make a plan, try to stick to it, and document issues you find along the way so they get addressed. It can be as simple as emailing tasks and keeping them in a folder, or using an in-house or hosted tracking system like Jira or Mantis, or a project management system like Wrike.

version control system

Use version control to track changes to code.

This should go without saying, but regardless of how small a project you're about to embark on, you need to version control your work. Now that decentralized VCS systems are ubiquitous, it's almost trivial to get started (although a long road to mastery awaits). Pick one: Git, Mercurial, or Bazaar, and move on to more pressing matters.

source code repository host

Share the code among all contributors.

Distributed version control is great, but unless you and your team are sharing code via SSH, you'll need a hub (pardon the pun) to use for collaborating. The big players are GitHub, BitBucket, and GitLab, with CodePlex being fairly popular on the .NET side. Newer players include Stash and CodeCommit. Consider your needs around private repositories and whether you will take advantage of supplementary features like issue tracking and wikis.

programming language

Pick the right language for the job.

Surprisingly, not as important a choice as most people would think, and unfortunately this might be the one newer engineers spend the most time on. When deciding, keep in mind that nearly any language will be satisfactory, whether it's something traditional like PHP, C#, Python, or Ruby, or spicing it up by with the likes of Go, Node.js, Scala, or Rust. You'll also want to judge whether to choose something familiar and comfortable, or take the risk of trying something new.

integrated development environment

Develop the app within a rich editing environment.

Your choices here are somewhat dictated by the language you choose to use, but there are still plenty of options. Full-featured IDEs with integrated tools, like Eclipse, NetBeans, the IntelliJ family, and Visual Studio provide built-in debugging, code analysis, and plugin support. However, many developers instead choose "just the code" editors like Sublime, Komodo, Emacs, or the recently released VS Code, preferring to use the terminal directly for other operations like version control and build processes. You'll want to choose one that fits your particular style, depending on whether you feel more comfortable with an 'everything in the IDE' workspace.

Pillar 2: Design

frontend package manager

Pull in frontend code dependencies using a package manager.

While most developers are familiar with using package managers for their backend dependencies, many still rely on checking in third-party Javascript or CSS directly into their code base. Bower brings the power of package and dependency management to the front-end as well. Use it, and never worry about finding and installing the latest versions of UI libraries again.

base ui framework

Build the UI on top of a responsive, grid-based framework.

Let's face it, the content flow + box model of plain old HTML/CSS is a bit clunky. There's no reason for a developer to spend loads of time attempting to position or align that one pesky element that won't do what you want. Use a higher level abstraction like Bootstrap, Foundation, or Cascade, or create your own customized grid with a tool like Modest Grid. Get familiar with the Flexbox model supported by modern browsers, and save yourself some headache!

ui theme

Keep a clean and consistent style throughout the site.

One of the most important ways of ensuring users have a good experience on your site is to maintain a clean and consistent style across all pages. Keep common sections like nav bars, footers, and sidebars the same as users navigate around, and maintain base styles for common elements like form inputs, lists, and headers. You can create your own set of base styles from scratch, or get inspiration from sites like WrapBootstrap and the Foundation Resources page.

site url structure

Provide a URL structure that is consistent and user-friendly.

One of the most visible aspects of your website is your URL - it's right up there in the top of the address bar the whole time a user is on your site. Keeping it clean and readable helps your users out, and can also be helpful for SEO. Structure your URLs in a logical, hierarchical manner, try to eliminate unnecessary information (such as file extensions), and be consistent about whether or not you use trailing slashes.

multi-browser and multi-device testing

Design with mobile devices and multiple browsers in mind.

While we're long past the days of having to write separate stylesheets for each different browser, don't be fooled by near-universal CSS3 support. You're going to find small quirks in a variety of different browsers and operating systems, so it's important to test your site in as many browsers and devices that you can. Don't just rely on a single browser's Developer Tools; while they will show you how your site looks in different resolutions, they won't change the underyling rendering engine. Browsersync is an incredible tool for local development, and post-deployment tools like BrowserStack and Browserling can be helpful too.

custom error pages

Provide friendly error pages that use the global UI theme.

Errors happen, and when they do you don't want it to be a jarring experience for your users. Create custom error pages: either a single generic page, or specific pages for error codes like 404, 500, etc. Make sure the page conforms to the general theme of your site, and give users a way out via navigation or some default content. Read what Google has to say on the matter.

Pillar 3: Development

backend package manager

Pull in backend code dependencies using a package manager.

You won't have much of a choice for your package manager since it's almost entirely based on your choice of programming language. For PHP, it's Composer; for Python, pip; for Node.js, npm, for Ruby, RubyGems…you get the idea. But consider that a good thing: A unified package ecosystem makes discovering and installing dependencies much easier, and keeps everything in one place. Go with the one your language specifies.

web application framework

Leverage web frameworks to avoid writing boilerplate code.

Good programmers know to use their coding time efficiently, and one of the best ways to do that is to stand on the shoulders of others. Take advantage of the abundance of existing web application frameworks available for your language. Libraries like Laravel for PHP, Flask for Python, Ruby on Rails, ASP.NET MVC, and Play for Java and Scala let you spend less time worrying about how to route and serve web requests and more time on developing the core content for your website.

html templating framework

Maintain a clear separation between presentation and logic.

Most web frameworks come with built-in support for html templating of some kind, but if not, you still want to ensure your HTML remains as separate as possible from your back-end code. Use an MVC-style approach, where you feed models to a templating framework like Handlebars or Jinja2, which then replaces values in your HTML templates appropriately. Be vigilant about handling calculations and logic only in the back-end code, and transferring the results via model objects into your templated HTML.

application settings management

Keep internal application settings together, in the code.

Application settings are things that do not vary in between deploys: things like route configurations, code module wiring, format strings, etc. These types of settings should be placed in a well-defined, common location so they are easy to view and change, but try to keep them separate from environment settings (things that vary between deploys) to protect against misconfiguration. Most opinionated web frameworks will ask you to place these settings in specific locations by default.

content management system

Source dynamic content externally using a CMS.

If your site features dynamic or frequently updated content like articles, blog posts, testimonials, or any content that non-developers will be changing, you should consider managing that content within a CMS. This way, your team won't be dependent on redeploying the application just to add new content, or to fix a typo. Depending on your needs, you can use an integrated solution like Drupal, WordPress, or Joomla!, a lightweight system like TextPattern, or a hosted service.

persistent storage

Configure backing services as third-party resources.

Almost any website is going to need some kind of persistent storage, and there are all kinds of options depending on your needs. There are traditional relational stores like MySQL and PostgreSQL, document stores like MongoDB and CouchDB, or even hybrid solutions like Cassandra. You might want to consider hosted solutions like Amazon RDS or DynamoDB, too. But whatever you choose, treat it as you would any other third-party resource, using the environment's config to connect it to your app.

Pillar 4: Packaging

build tool

Automate the build and packaging process.

A lot of developers shy away from using a build tool for websites, especially if they are using a purely interpreted language. But there's a lot more to a build than just compiling your code. You need to prep your web assets, perform static analysis, create deployment packages, and manage your local development environment. While it's possible to do these things manually, it can be error-prone and frustrating. Use an automated build tool, whether it's Make, Rake, Grunt, Gulp, Gradle, Maven, SBT, or even a bunch of custom shell scripts — you'll save time and energy.

css preprocessor

Create modular, reusable styles using CSS metalanguage.

Even if your site is simple, with only a smattering of custom styles, you'd do well to familiarize yourself with Sass or Less. Powerful features like variables, mixins, conditionals, and expressions will allow you to write better-organized and more compact stylesheet code. Additionally, since most UI packages use one or both of these, you'll be able to customize them easily without having to edit third-party source files.

static code analysis

Lint your code to quickly discover common errors.

As part of your build, you'll want to run basic analysis on your source code, javascript, and CSS. Linting might not always discover critical bugs, but tools like JSHint, CSS Lint, pylint, or even just padding additional check flags to your compiler will help identify code that violates standards or that might not do what you think it does. Don't get too carried away though - many linting tools are ridiculously opinionated, so you may need to dial down the settings a bit rather than blindly follow their advice.

asset revisioning

Append content hashes to UI asset filenames.

Long-duration content caching helps significantly reduce bandwidth and speeds up response time, but in order to safely take advantage of that, you're going to want to rev your assets. This process involves adding a unique, content-based hash to things like .css, .js, and images, so that the file name changes when the contents do. gulp-rev does this for you as part of your gulp build, so you can safely set year-long cache times for these files.

asset bundling and minification

Combine and minify UI assets.

Another way to improve performance and responsiveness is to combine all your CSS and all your JS together, respectively, and minimize or compress the output. The easiest way to do this is as part of a build pipelines using tools like gulp-useref, gulp-cssnano, and gulp-uglify. This way, you can use the separate, uncompressed files during development, and only compress and minify during the packaging step.

application bundle

Build a complete, universally deployable application bundle.

The application bundle should be contain only the code necessary to run the application, and should not know or care about the deployment mechanism or the target environment. Whether it will be run via Apache, Docker, Heroku, or Elastic Beanstalk, the bundle itself should remain the same. The details of integrating a bundle with its deployment configuration should instead be handled by the deployment and release management process. Although for small apps these might be the same tool, keeping the steps conceptually separated will help you scale out later.

Pillar 5: Deployment

build management / continuous integration

Automate application builds, test execution, and packaging.

If you want to adopt a more continuous deployment process, or if you have a large number of tests you want to run before passing a build, consider setting up a CI server like Jenkins, Hudson, or TeamCity. You won't even need to manage it yourself with a hosted solution like Travis CI. Using a build system will give you much more control of your packaging and release process, and allow you to release confidently at a faster pace.

deployment process

Deploy using a repeatable process with frictionless rollback.

You can use your build tool, a separate system like Ansible or Capistrano, a containerized solution like Docker, rkt, or Elastic Beanstalk, or an OS package manager like apt or yum. Use these tools, custom written scripts, or plugins for your build / CI server to initiate deployments and rollbacks of your application, and avoid manual steps if at all possible.

environment provisioning

Provision resources using a template, under version control.

Treat servers like cattle, not pets. Don't have standing test environments, carefully tinkered with to work properly. Spin them up as needed and terminate them when you're done. Instead of relying on slow, manual setup, automate this process with a tool like Ansible, CloudFormation, or Terraform. This way you can maintain dev-prod parity and save money on infrastructure costs.

environment configuration management

Configure environments identically, in an automated fashion.

Dev-prod parity is a critical part of the testing process. You want to ensure the software you test will behave exactly the same in production, and avoid bugs due to version incompatibility or missing configuration. To that end, you can use tools like Puppet, Chef, and Ansible, or even simple cloud-init scripts, to manage the configuration of your resources identically across environments.

application configuration management

Store deployment-specific settings in environment variables.

For those settings that vary between deploys, like database credentials, network paths, hostnames, etc., manage them outside of code and version control and instead use environment variables. Environment variables are one of the few things that enjoys near-universal support across software packages, and are usually trivial to set and retrieve. Treat your app as though it were a function having these config settings as arguments and you'll be able to run it anywhere, even locally, configured exactly as needed.

Pillar 6: Infrastructure

web server

Use a properly configured, production-class web server.

While many web application frameworks come with built-in web servers, you should avoid using them in production settings. Instead, choose reliable and scalable technologies like httpd and nginx, or web application servers like Jetty or Tomcat for Java and IIS for .NET. Additionally, be sure to configure the server with proper MIME types, output compression, and cache control headers for maximum performance.

application host

Choose a robust and fault-tolerant application host runtime.

Usually this would simply be the OS distribution or JVM of your choosing, configured to specifications by your chosen configuration management tool. But these days, you might want to look at containerized solutions like Docker and rkt. Software containers are an even easier way to ensure parity, avoid misconfiguration, and deploy quickly, but you might want to limit production use to small or non-critical systems for now.

platform host

Deploy to a scalable, stateless platform host.

Your choice here is whether to use an all-in-one platform-as-a-service, or to manage the underlying infrastructure on your own. For simple applications, using a service like Heroku, Google App Engine, AppScale, or Elastic Beanstalk can save you a lot of time and effort, but you'll need to choose a compatible platform and possible tailor your app to it a bit. Either way, your app should run statelessly to allow it to scale outward if needed.

infrastructure host

Build out and host infrastructure with a modern cloud provider.

If you're not using a service-based platform, you'll have to choose where to host your servers. These days, cloud providers are your best bet in both price and flexibility, and there are plenty to choose from. From the heavyweight Amazon Web Services, to DigitalOcean, Microsoft's Azure, Google's Compute Engine, and even Rackspace, each offers a wide variety of virtual components to choose from.

pre-production environment

Configure a staging environment identical to production.

Whatever your target infrastructure or platform, you'll want to ensure that you use an identically-configured pre-production environment for testing and reviewing changes before they go live. Many platform providers faciliate this, like Elastic Beanstalk, which allows you to swap two environments at the click of a button. But if you've automated your provisioning and configuration, you'll be able to spin one up on-demand whenever you want.

Pillar 7: Services

domain registrar

Register domains with reliable, privacy-oriented registrars.

When it comes to your domain provider, be sure to do your homework. Go with reputable companies that provide good service at good prices, and make sure they support privacy controls. Some examples are Namecheap and Gandi, or if you use Route53, Amazon can register domains for you (via Gandi). If you're looking for a more comprehensive comparison, check out domcomp for a great way to find one that fits your needs.

dns service

Manage DNS as a part of environment configuration

Few people these days want to manage their own DNS, so you'll probably stick with letting your domain registrar handle it (as most provide basic DNS services). But you might consider using a service like Route53, especially if you're on AWS, or other managed DNS like UltraDNS or DynDNS, since they provide APIs for programmatic access to DNS management. You can use this to automatically update DNS records as you provision new resources or environments.

smtp service

Use a trusted, reliable SMTP service.

Like most things, you can roll your own if you're really ambitious, but it's much more sensible to delegate to a third-party smtp provider (especially if you'll be sending bulk messages like alerts or newsletters). SES makes the most sense if you're using an AWS stack, but there are many other options if you need templating for bulk or transactional email.

content distribution network

Serve static content through a global distribution network.

The biggest chunk of data served up by your website generally consists of static assets like javascript, css, and images. Storing them on a global CDN like CloudFront, Akamai, or CloudFlare can dramatically reduce page speed and server load in high-traffic sites.

search engine optimization

Actively manage search engine indexing using webmaster tools.

Set up and use webmaster tools accounts at various search engines. Google, for example, provides the Search Console, while Bing provides their Webmaster Tools for more visibility into how your data is being indexed.

web analytics service

Track user behavior toward measurable goals with analytics.

Without analytics, judging the success of a website is almost entirely guesswork. By building in an analytics component, you can identify precisely how users are finding and interacting with your site, and whether these vists are meeting the goals of the website itself. Google Analytics is the biggest player in town, and combined with Google's Tag Manager, you can dynamically reconfigure all your analytics without touching your website's code again.

Pillar 8: Operations

log management

Stream interesting logs into a centralized aggregation platform.

The logs produced by your applications and servers let you know what's going right, and what's going wrong. Putting together a clear picture means you're going to need to parse, store, and query these logs. There are a wealth of options including service-based approaches like Splunk, Loggly, Papertrail, and Cloudwatch Logs, or DIY setups like Fluentd or logstash on top of Elasticsearch and Kibana, or all-in-one tools like Graylog. Pick the approach that makes the most sense, financially and in terms of setup and maintenance time.

application and infrastructure monitoring

Monitor the health of the entire stack to a reasonable degree.

Use automated tools to monitor the health of your resources, and where possible automatically take corrective action (replacing servers, restarting services, etc.) AWS provides CloudWatch for monitoring, but more popular are systems like Nagios and sensu. Be sure to monitor not just the servers, but the applications running as well.

notification and alerting

Send notifications only on actionable alert conditions.

While more information can sometimes be better, your monitoring software should only send alerts, whether by email or text, on actionable alert conditions. This means that unless you can take action, or plan to take action, to correct the problem specified, it doesn't need to be an alert.

load balancing and scaling

Plan for the unexpected, scale automatically whenever possible.

Design your applications to be stateless and resilient to interruption and failure. Doing so allows your to build your infrastructure in a way that scales easily (e.g., AWS Auto Scaling) and lets you respond to demand via load-balancing (e.g., Elastic Load Balancer or nginx). This way, even if your site accidentally goes viral, you'll be able to handle the traffic.