Author: jedreynolds

About jedreynolds

A programmer that has adopted an all-weather bicycle commuting lifestyle.

Quick Essay on HA Web Applications

answered this reddit post and decided I should also post the same answer to the blog.

HA design for web sites is a battle I’ve fought. As with any HA system, you are first looking for any single point of failure. In this case, having a single NFS server behind web nodes would be a failure waiting to happen.

The second thing about running web applications, is that you need to engineer cache-ability into your content. This way assets like pictures, css files, and videos can be reliably delivered from your reverse proxies in front of your web heads.

You only need as many web heads (web application servers) as you need to interact with live client sessions (un-cacheable content)…and then add as many extra nodes as you need to account for maintenance and load spikes as determine necessary.

To distribute your application (like php files) and your assets, I would start with source control. Best is to have a pair of repos in your application network, and scripts that push to the web heads, or trigger pulls from the web heads.

Roll-back is also a HA topic: some designs encourage twice the number of web heads and twice the number of proxies (depending on load) and astute DNS control in order to do a quick roll backs. The deployment in those cases is to only half the nodes (writing over the old pool) and then if your deploy tilts, a DNS change will hop you over to the previous pool.

Deploying database changes is more arduous, especially for small shops, because you have to keep in mind that if you are rolling out to a subset of your servers (like the two pools mentioned above) you also have to partition your database resources as equitably.

A schema roll-out is often the first step before an asset roll-out, because you do not want a flood of errors from new application code looking for things like missing columns and tables for the few minutes or hours your asset roll-out takes. This takes shrewd release planning because you must not roll out a schema that breaks the currently running application code before the upcoming application code is rolled out.

Doing HA for your databases is a good argument for hiring a dedicated DBA. Doing HA-*staffing* for your HA application implies hiring *two* so that one can take a vacation or go home during a roll out to spell the other when he’s done with his 18 hour day.

You can cluster databases, and you can do trees of replication, and you can round-robin your masters. The simplest thing to do for an HA setup is to dedicate two database nodes to be a master and a next-master. The next-master can be used as a RO slave most of the time. Then you can chain your RO slaves off both of those. When you need to do a fail-over of the write master, you would switch the replication path of the first-tier children from your master to your next-master and de-pool your master.

The notion of database pools is very application specific, mostly because you need to explicitly run multiple database connections inside your application: one for transactions (writes and updates) and another for simple queries (ro activity). Your application needs to know when a connection has gone stale (like a pool change) and to re-open a new connection. And ideally your pool needs to explicitly check for failures to connect to the wo connection and throw itself into a “sorry, try again later mode.”

Consistency between all nodes in the db pool is a regular challenge. The first in the line of that challenge is: how far behind in replication are each of my nodes? This, and other criteria, as known as ‘fitness criteria’ and your application, or your pool monitoring agents (which might be on each pool node) need to kick nodes out of a pool as soon as they fall behind in replication, show errors, are at high load or in maintenance mode. Ideally you would have something like an ESB (enterprise service bus) or a DNS service that populated hostname-ip mappings very quickly (<= 1Hz) to keep unfit pool members out of service until they catch up.

You should read up on mysql-proxy. That was the last promising project I read about that seemed to encapsulate much of this logic. Otherwise, as of five years ago, there was no good encapsulation for this kind of solution for MySQL. MySQL-Cluster was not the solution I was looking for (and I suggest you read the short book on it) because it is architected for OLTP in-memory transactions. My data-set was highly relational and document oriented.

Net Neutrality | Electronic Frontier Foundation


The FCC has a poor track record of getting net neutrality right. In January 2014, a federal court rejected the bulk of the FCC’s 2010 Open Internet order. The rules that the court threw out, however, were deeply flawed. Protecting net neutrality is a hard problem, with no easy solutions. … [W]e are asking folks to contact both the FCC and Congress and send a clear message: It’s our internet, we won’t let you damage it, and we won’t let you help others damage it.

via Net Neutrality | Electronic Frontier Foundation.

Timeline of NSA Domestic Spying | Electronic Frontier Foundation

Quick summary:

It is intended to recall all the credible accounts and information of the NSA’s domestic spying program found in the media, congressional testimony, books, and court actions. The timeline also includes documents leaked by the Guardian in June 2013 that confirmed the domestic spying by the NSA. The documents range from a Top Secret Court Order by the secret court overseeing the spying, the Foreign Intelligence Surveillance Court (FISA Court), to a working draft of an NSA Inspector General report detailing the history of the program.

Timeline of NSA Domestic Spying | Electronic Frontier Foundation.

How Low-Income Commuters View Cycling – CityLab

This is where our society needs to seriously loose the brainwashing:

Even increasingly popular car-sharing was not satisfying for respondents; when asked if they would “rather share a car through a program like Zipcar or Car2Go than own my own vehicle,” 35 percent strongly disagreed. Also, 32 percent strongly disagreed with “I want a lifestyle where I don’t need to own a car.”

How Low-Income Commuters View Cycling – CityLab.

When will we get true civilian Internet? — “Internet’s Own Boy” Briefly Knocked Off YouTube With Bogus DMCA Claim – Slashdot

This is why we need copyright reform, and we need to invest in truly civilian Internet spaces. Things like youtube are entirely taken for granted but they live at nothing more than the whim of corporations, with no actual rights of free speech on them.

The whole Google and “right to be forgotten” drama is another case where civilian government is unable to form a basis for a search engine for a search engine ruled by civilian law. We act as if Google is a utility…and while it might be a de-facto utility, it is not, and it is ephemeral and as temporary as its stock price lives well.

“Internet’s Own Boy” Briefly Knocked Off YouTube With Bogus DMCA Claim – Slashdot.

The Great Artisanal Cheese Panic of 2014: A Postmortem | Cornucopia Institute

This is a good punch thrown in the battle. However,

One such critic, Food & Water Watch, used the incident to argue, somewhat incongruously, that the FDA is spending its time hassling small-businesspeople when it should be going after Big Food. “Any rule that promotes processed, industrial food like Velveeta over handcrafted foods is not something we should support,” the group’s statement declared, while also making sure to mention that a former Monsanto executive, Michael Taylor, heads the FDA.

And really the confusion is over why the enforcement over something that on-one has gotten sick over:

Which, by all accounts, is nonsense. It’s true that wood, being porous, provides a friendlier environment for bacteria. But there is nothing to indicate that wood shouldn’t be used to store aging cheese, as long as it is properly sanitized. What few studies have been done have tended to conclude that the practice is essentially safe. Nobody, including the FDA, can think of a single incidence of listeriosis caused by wood-aged cheese, much less an outbreak.

via The Great Artisanal Cheese Panic of 2014: A Postmortem | Cornucopia Institute.