Social is big, and getting bigger…

Last year, there were 1.4 billion social network users; by the end of this year, 60% of Internet users are expected to be on Facebook alone, which represents an increase of nearly 20% this year.

source: FT

This growth rate and the size of this opportunity are staggering: within four years, brands’ social marketing spending will pass $11 billion per year, $14 billion will be spent in social commerce, and 50% of e-commerce will be referred by social and mobile posts. In the face of year-over-year declines in traditional media advertising, it isn’t an exaggeration to describe social as the future of advertising.

…but it needs to be measured

What’s possibly most amazing about this growth is that it comes even though companies’ social investments have basically been blind gambles. According to a recent survey, 47% of marketers do not monitor social communities at all. Of those who do, 40% only measure superficial metrics such as “likes” and comment volume; and fewer than 25% measure the ROI of their campaigns.

Infosys

Demand clearly exists for measurement and accountability in this space, and the market has responded — there is growth in the category of sales and measurement tools. But awe.sm is the only technology that measures social performance, and performance — customers acquired, visitors converted, dollars and cents — is the only measure that matters.

Brands and agencies are next

awe.sm has spent the past four years developing extensible and powerful technology for measuring the outcomes of every single social post. Since the company took its first round of funding, we rolled out a toolkit of powerful developer APIs, and developed innovative reporting tools used by some of the industry’s most interesting social-enabled software.

These developments make it possible to visualize the success of earned media — that is, how does sharing lead to business? Now, it’s time to bring this visibility to owned media — the work brands and agencies do to start conversations and spread the word on social networks.

In coming weeks, awe.sm will be rolling out a tool to view, on one screen, the entire social funnel: how many retweets, likes, and engagements did each social post receive?; how many clicks back to a site does each drive?; and what business impact — in registrations, pageviews, and transactions — resulted?

Let’s do this

I will be applying my two decades of experience in the media industry to introduce this technology to the brands and agencies on the leading edge of innovative social marketing. Meanwhile, our talented — and expanding — team continues to innovate and improve our powerful performance measurement tools. The ultimate goal is no less than revolutionizing social marketing by bringing full visibility and performance optimization to every social action.

This is ambitious, exhilarating, and … totally awe.sm.

In Bennett’s blog post he talked about the process of selecting java as the right solution for our core APIs.

When we started discussing platforms for writing  products, Rails, Express, and Zend were all considered. We discussed many aspects, but when we got to performance I realized I didn’t really know how they compared. I mean, I had rough guesses, but no actual data. So I decided to run a quick experiment to give myself a bit more perspective on how these frameworks compare.

At this point I should include a disclaimer: this is not intended to be a rigorous scientific benchmark of these platforms. This is more along the lines of anecdotal evidence to satisfy personal curiosity. It was my intent to keep the playing field as level as I reasonably could. I would definitely encourage you to run your own tests and benchmarks before making any decisions.

Test setup

I built a simple application in each framework, limited in scope to one model and controller. The versions used as follows: Rails: 3.2.12, Zend: 1.12.1, Express: 3.0.5.  You can check out the source here. Obviously, a feature identical, complete application written in all three frameworks would be best, but writing three complete applications in different frameworks is a fairly big time commitment.

I made an effort to make these applications reflect real applications. They use sessions, cookies, and CSRF middleware. Additionally, I didn’t take shortcuts on the models: this isn’t about streaming raw query results to the client, objects were instantiated to encapsulate each record returned to get a real object oriented model (Except for Zend, where I followed the normal awesm convention of keeping database results in associative arrays).

Tests were run in an Ubuntu 12.04 virtual machine on a 4 core i7 laptop. The database was postgres 9.2 and the web server was Nginx 1.2.7. Requests were made and data was collected using apache benchmark. Each data point is an average of 1024 requests.

I wanted to put each platform’s best foot forward, so for Zend I used FastCGI, for Rails I used Unicorn with unix sockets, and for Express I used an http proxy. You can view the configuration files for FastCGI, Unicorn, and Nginx.

Test results

Concurrency vs Request Time

The first thing I wanted to see was how well each of the frameworks handled load. As the number of concurrent requests increased, how did the applications perform? The results show very similar trends for each of the frameworks, roughly linear increases in response time as the number of concurrent requests increase.

The first thing that stood out when looking at this data was how much faster Express was. Since the data is plotted on a log-log scale, the difference between Express and Rails is nearly an order of magnitude. Additionally, I was surprised at how similar Rails and Zend were. I would have expected Zend to be substantially faster than Rails.

Records Returned vs Request Time

The next aspect I was interested in was the impact of the number of records returned on the response time. The results show that Rails has the highest impact, which makes sense given that a new Active Record object is created for each record returned. Express is less impacted by the number of records, which makes sense given that the objects created for each record are lighter weight than Active Record. Surprisingly, Zend seems to not be affected by the number of records it is processing at all. This makes sense as the records returned in Zend are simple associative arrays, and new class instances are not being created to wrap each result. I chose to use that style in Zend as it matches how our existing Zend applications are written.

I wouldn’t read too far into this particular test, as it’s uncommon to render a large number of result to the client. Both Rails and Express have ways of dealing with large number of records without instantiating objects, if the situation calls for it.

Conclusion

The goal of these tests was to satisfy my personal curiosity about the performance characteristics of these three platforms. My expectations were that Express or Zend would be the fastest, and Rails would be the slowest. The results were surprising. I wasn’t sure how much difference to expect between Express and Rails, but I didn’t think it would be as large as an order of magnitude. Additionally, I expected Zend to be substantially faster than Rails, but it appears to only be slightly faster. Compared to my initial guesses, this takes a bit of wind out of Zend’s sails, and makes me look at Express more seriously than I initially had.

These tests have helped make the performance differences between the three frameworks a little less murky for me. When we talk about tradeoffs such as development speed vs framework performance, I now have a bit more perspective. If you are considering some of these frameworks, I encourage you to run your own set of tests to evaluate them in the context you care about. If you have run similar benchmarks, let me know as I’d love to look at the results.

In my last post, I showed you how to categorize and organize shares for a given project, in the context of promoting a movie. Organizing your own shares isn’t much different — let’s revisit that data.

Start with the report I made last time, Godzilla Movie Shares. Now, I want to know which of my shares are generating the most traffic, so I’ll create a new Group Table, and name it “Godzilla Shares by Beth.” (It’s always good to use descriptive names; this provides some context if I share this report with my team.) For this table, I’ll set up a Filter: filter by Username: Beth, and group by awe.sm ID, to see only the links that I shared.

I can see my San Francisco premiere is still beating my Austin premiere in terms of traffic — but how much of that was from my social posts? To find out, let’s make a new graph, named “Clicks from Beth’s Shares.” I want to understand my shares in the context of my San Francisco and Austin premiere campaigns, so I’ll group by campaign. Now, I can see that on March 14th, 945 clicks for the San Francisco campaign were from my shares.

If I’ve fully integrated with awe.sm’s Conversion Tracking, then I can even understand how my posts drive revenue.

The Godzilla Movie project is tracking several goals, including box set purchases, app downloads, VIP ticket sales and iTunes streams. I want to know how much revenue in VIP ticket sales my sharing drove, so I’ll create a new Spark Graph named “VIP Ticket Sales.” Once again, I’ll filter my visualization by Username, to see only my own shares, and I’ll select that I wish to view the count and value for VIP Tickets in Conversions.

Now I can see that my sharing resulted in 64 VIP ticket sales, for a total of $9,800. I can instead choose total conversions to see the total value of my contribution to all tracked revenue generated on this campaign.

COOL, HUH? But of course this is just the tip of the … giant radioactive tail. among the countless features of the new awe.sm is the ability to slice and dice across literally any dimension of your data.

And don’t forget: with great power comes great user support. We’re always here to help, by live chat, email, or with free custom training for you and your team. Don’t be a stranger!

Format

The format has evolved over time, and this is what’s working for us right now.  We have one speaker present every week.  The presentations last between 15 and 30 minutes.  Every team member takes a turn as a presenter.  The topics are usually on technical introductions/explanations, product ideas/demos, or we have external speakers.  Although they are presentations we encourage everyone to interrupt with questions and comments.  Finally, we keep a calendar of past and upcoming speakers.

Why do it?

Improve your skills

If the presenter doesn’t have something to talk about, they can learn about something new and show it off.

Get better at communicating.  It’s really difficult to talk about how code functions.  Getting time to focus on a single topic and explain it really well has helped me learn how to simplify what I want to communicate.

Spend time on presentation skills.  Crafting your message is a hard task, so the more time you spend practicing doing that helps you figure out what works and what doesn’t.

Share your knowledge

We are constantly teaching one another about new technologies, libraries, and ideas, but usually this sharing just involves pointing someone to a blog post or giving a brief explanation.  Presentations allow you to explain a topic in depth to many people and provide a forum to ask questions.  We’ve learned a lot about each other’s skills and interests through these presentations.

Our team is 13 people, and even with this few people not everyone is aware of all the projects happening.  Sometimes for our presentations, we give an in-depth look at a product we are working on to make sure everyone on the team is aware of it and can offer their insight.

Making it part of your routine

We could always schedule a meeting when someone wants to present a topic, but by making this a weekly event it becomes part of our weekly routine.  This has lead to good attendance and made learning and communication part of our culture.  Also, it is easier to find a designated spot when everyone can be available week after week, than to find a time that works for everyone when someone wants to make a presentation.

What we’ve got to show for it

• 2 years of Show & Tells
• 15 different speakers
• 32 presentations
• Some particularly great presentations
  • Intro to Angular.js (disclaimer: we swear a lot)
  • Intro to Ember.js (out of date with the current Ember.js)
  • Distributed Systems and Consistency

We hire people who want to give talks.  People with a diverse set of interests who are interested in learning and want to help the rest of the company learn as well.  A great example of this was after we hired @agrieser.  He gave a great presentation about Ember.js and raised the bar for all of us. I personally have learned a lot about many company projects and new technologies.  I’ve also gotten much better at presenting.

How you can do it as well

• Pick a day
• Works best over lunch
• Presentation length should be 15-30 minutes
• Pick topics that will appeal to your audience
• Keep a calendar of presenters, topic, and presentation deck
• At the end of a presentation mention who is up next and their topic
• Stop thinking and get started!

Tools and tips

Google Docs presentations are easy to get setup and use.  Another simple yet powerful tool aimed at developers is Reveal.js.

Know your audience.  Our team is small so we try to make sure any topic chosen will appeal to at least a few people.

Presentations take time to make.  A presentation isn’t an excuse to neglect your existing work.  It takes time to make polished presentation, but our Show & Tells are meant to convey ideas more than be perfect presentations, so we try to not take too much time to prepare.  Also, over time you will learn how to quickly gather materials and effectively use the presentation tools so you preparation time will be reduced.

And more

We are still getting better at this, creating better presentations and adding more guest speakers to come in and talk.  So if you’d like to attend or present give us a shout.  And if you’d like to be part of the fun take a look at our jobs page

First, my thesis: a lot of less-experienced developers are using big data^[1] and NoSQL technologies because they are new and cool, and because SQL is old and hard. A lot of these people would save themselves time and effort by learning more about SQL and tuning their databases and hardware just a little bit. Rather than just gripe about it, I’m going to suggest a grab-bag of basic techniques of varying complexity that can help a lot (especially if you are using MySQL), and later I’m going to follow up with another post about more complex strategies for more experienced devs.

My task has been made easier by some other recent posts by other people that made most of my argument for me. Namely:

1. Pick your battles by Zef Hemel

   The money quote is this:

   go and build amazing applications. Build them with the most boring technology you can find. The stuff that has been in use for years and years. Where every edge case has been covered. Where every library you will ever need has been in production for years. Where every part of the release cycle has been ironed out. Where the best practices on how to do testing are known.

   In essence: trying to solve a problem by using a technology stack you’ve never tried before gives you, at minimum, two problems. Big data tool chains are very different from more general purpose data-stores, and newer data stores are less reliable at scale, simply because fewer of their edge-cases have been met and managed.

2. SQL is agile by Armin Ronacher

   Armin’s post has fewer pithy quotes, but the key point is that early in the life of your project, when your data is small^[1] but your ideas and use-cases can change dramatically, the structure and flexibility of an RDBMS trumps the single-case-optimized performance of a document store.

3. MySQL: choose something else by Owen Jacobson

   Opinionated and inflammatory as this post is, what I took away from it is: don’t judge all RDBMS by the failures of MySQL. MySQL is not the best database in the world, it’s just the easiest^[2]. It’s getting better all the time, but Postgres is really good right now^[3]. Check to make sure the problem you have with “SQL” is really just a problem with MySQL. Further disclaimer: I still mostly use MySQL myself, out of familiarity and habit, and all of my tips are from that perspective.

So if you’re not on board with the premise that new technologies require substantial up-front investment that will delay your actually building anything useful, that old and tested technologies can be great, and that SQL in particular is cruelly maligned, take it up with those guys. What I’m going to be sharing are some strategies for making MySQL work better in a lot of common use-cases.

Don’t use vanilla MySQL

A while back, Sun Microsystems bought MySQL AB, the MySQL company. Then Oracle, maker of a major competitor to MySQL, bought Sun Microsystems. Conspiracy theories abound, but I don’t believe they really bought the whole company just to kill MySQL.^[4] Nevertheless, Oracle suffers from a clear conflict of interest as the steward of MySQL, and the company who in my opinion have taken up the torch as the true home of MySQL are Percona.

Percona Server is MySQL, just fully patched and cared for, and you can install it with apt-get just like Oracle’s version. I think switching to Percona MySQL has the highest ratio of performance improvement to effort of any of these strategies. If you try nothing else, try using Percona.

Use a great in-memory database: MySQL

If your data is small enough that it can fit in memory, then the sadly under-appreciated innodb_buffer_pool configuration settings can be trivially modified to make MySQL behave like an in-memory database, like MongoDB. Don’t be put off by all the caveats in the documentation: if you have less than 50GB of data, buy a server with 64GB of RAM, set innodb_buffer_pool_size to 50GB, and watch it fly^[5]. Of course, if your data all fits in memory, almost any database will fly, because you don’t have big data. You don’t even have medium data. You have no problems at all: memory is 100,000 times faster than disk access, so you can do really, really inefficient things without issue.

Cache it if you can

Caching is a solution so obvious I hesitate to put it in, and yet also such a big, varied solution that I could never cover it in one post. At its most basic: if some subset of your data is more frequently accessed than other parts, for instance because it’s popular, or because it’s recent, and if that data changes less than about once per second, you can put that data in an in-memory cache, and speed up reads by an order of magnitude or more (remember: 100,000 times faster).

For a huge variety of applications, caching can be enough to take you a long way: for example, if you have a list of your top articles, or recent posts, and you are serving 100 requests per second, implementing a cache that lasts only 1 second will reduce the number of reads to your database from 100 to 1, i.e. two orders of magnitude (but beware of thundering herds when that cache expires).

You will also need to solve the problem of cache invalidation, i.e. knowing when to use the cache and not. That’s a case with endless nuance, one of the two hard problems of computer science, but frequently solvable for simpler use-cases simply by automatically expiring cache entries after a certain amount of time (a feature built into memcached, the most popular in-memory cache).

Replicate to handle more reads^[6]

If your data-access pattern is not (or not entirely) amenable to caching, then a complementary solution for scaling reads may be replication. RDBMS like MySQL and Postgres have replication built-in, and while the details vary, the central idea is simple: instead of having one database that all your reads come from, have many “slaves”, each a continuously-updated copy of the primary, “master” database.

Replication works well if your read-queries are competing with each other for resources on your master database, by providing more memory, disk time and processing power for each query. It can also help if you simply have too many incoming connections to your database^[7].

Replication’s major complication is “replication delay”, i.e. the time it takes for a write on the master to turn up on one of the slaves. This means you can read an out-of-date value from a slave while the new value is en route from the master — particularly a problem if your app writes a record to the master and then immediately reads it back for confirmation, and that read goes to a slave.

Replication is also frequently used as a backup/high-availability solution. As the ill-fated Ma.gnolia famously learned, a replicated database is in no way a backup (though it can help in creating them, by making snapshots and dumps more convenient). However, it can help to some degree with availability, especially if your application can tolerate a period of read-only activity while the master is upgraded/repaired or a slave is promoted.

Use indexes, but carefully

This is another one that falls into the “duh” category for the experienced, but indexing is very poorly understood by new developers. In essence, an index is a sub-set of your data, held in memory like a cache, but ordered in such a way as to make a particular type of lookup quicker, and unlike a cache guaranteed to always be accurate. That sounds basic but can be incredibly powerful:

• If your data is on a disk, locating the record via an in-memory index means that instead of hitting the disk dozens of times, your database needs to hit the disk exactly once: to retrieve the record itself. Once again: memory is 100,000 times faster than disk for this kind of access. Cutting out even a single disk read helps enormously.
• If your query is only on a subset of your data, MySQL has the concept of a “covered query”: for instance, if you just need to look up a username and password, put those together as an index and queries that need only those two values will be pulled directly from that index in memory, without hitting disk at all.

The most common pitfall of indexing is to index every column individually. It is seldom the case that you do equally frequent lookups on all your database columns, and each index comes with a cost — it has to be held in memory. If you index too aggressively, some of your indexes will be written to disk instead of held in memory, greatly slowing access. Also remember that every time a column is changed, the related index must be updated, so indexing a frequently-updated value that is seldom used as a primary lookup (such as a user’s score, or last login date) will leave your database doing a lot of extra work for no reason.

The simplest, first pass rule for indexing is: find the columns you use in your WHERE clause, and put them into a single index. But with the right indexing strategy you can go a lot further: Keith Murphy of MySQL magazine has a pretty good slide deck on index tuning and query optimization.

Shard your data, but only after you’ve tried everything else

The principle of data sharding is simple: instead of having one big table, split it into many tables across multiple boxes. This allows you to store more data than can fit on a single box, and can also speed up queries by limiting the size the index needs to be on any given box.

Sharding is a powerful, potentially very useful design pattern. It’s also difficult to get right, with lots of potential pitfalls. Unlike lots of the other patterns mentioned in this post, it’s also been quite extensively discussed and explained recently — almost everybody I’ve interviewed in the last three years has mentioned sharding to solve database problems, even people who hadn’t heard of caching or replication. So I won’t be covering it in this post. Eric Ries wrote a pretty good practical introduction to sharding over at Startup Lessons Learned; they also rely on sharding at Instagram.

Shard your queries

The name of this pattern is not, as far as I’m aware, an industry term — it’s just what we call it internally at awe.sm. In data sharding, you split your data across multiple boxes, and run all types of queries on these subsets of your data. In query sharding, you instead copy all of your data (for instance, via replication) to multiple boxes, and run subsets of your queries on each box.

This allows you to optimize each box for a given type of query: for instance, one could be optimized for quick key-value lookups (large numbers of connections, a small amount of memory for each one) and another for large aggregations (few connections, lots of memory). It also allows for more effective indexing: put exactly one index on each shard, tailored to the requirements of that query. Your box will be able to fit more or even all of that single index in memory, and it will also spend less time updating unused indexes.^[8]

Coming up later: more!

Because this post was already hella long, I decided to split the next batch of patterns into a second blog post, to follow a week or two after this one. More experienced users should find that post of more interest.

[1] In an early version of this post, I spent a long time defining big data. Suffice to say that it’s a tricky thing to nail down, but a good rule of thumb is that if you have less than 100GB of data, you definitely don’t have big data, and if you have less than a terabyte of actively-used data (not just logs) you probably still don’t. These claims are horribly unsubstantiated, but spending a lot of time on them distracts from my point.

[2] You read my disclaimer about subjective calls, right? But what I mean is: MySQL is a classic disruptive technology. When it first became popular 15 years ago, it’s because it was cheaper than all commercial solutions, and a hell of a lot easier to get running than Postgres was (especially on Windows, where a lot of junior developers like myself got started, and in multi-tenant environments like shared hosting). In the subsequent decade and a half, hundreds of competitors have arisen and Postgres has done much to close the newbie-friendliness gap, but MySQL has remained “good enough” to stay the most popular open-source database.

[3] And if you can afford Oracle, then by all means do. Evil they may be, expensive as all hell they certainly are, but Oracle does everything you’ve ever dreamed of in a relational datastore and is unbelievably performant. If you’ve got the budget, it’s a magic wand of awesome.

[4] Apart from being a very expensive way to force startups using MySQL to upgrade to Oracle even though they can’t afford it, MySQL is open source, so buying the parent company doesn’t give you control. It seems Oracle wanted Java, and Sun’s hardware business, and MySQL just came along as a freebie, but that’s pure speculation, so let’s please not have a bunch of comments about it.

[5] Percona’s performance blog has a realy useful post on tuning innodb_buffer_pool_size.

[6] I would have listed replication as “too obvious” as well, but after three years of interviewing junior developers, I’ve concluded that replication is neither widely known nor well-understood; it apparently falls into the “too scary/too hard” feature category.

[7] If too many connections is your issue and you’re using PHP, you may get further faster by looking into using persistent connections.

[8] This is one of those times when I wish I’d had time to gather some quantitative data on the effectiveness of query sharding. Instead, all I can say is that it works for us.

To clarify: while we do not allow EBS in our critical path (as bootable/attached instance volumes[1] or from EBS backed services[2]), we do make significant use of EBS for backups. Over our 3+ years using AWS we have learned a lot about building a highly available service using virtual hardware, and have found EBS to be too much of a risk for the reward.

In this post we are going to discuss our usage of EBS, and specifically how we manage backups and recovery procedures to get the best mix of durability and speed.

Recap: EBS vs Instance storage

For those just tuning in: EBS (Elastic Block Store) is a service that provides block device storage to instances running in EC2. At the OS level, EBS looks like a block device (just like a real, physical hard drive), so EBS volumes can be used as bootable storage for EC2 instances. This allows portability between instances and simplifies backups and restores. Because of this it is considered an EC2 best practice to use EBS backed instances.

Due to concerns of performance and reliability of the EBS service (covered in our previous post) we have migrated to ephemeral storage. Amazon recommends against using ephemeral storage for one critical reason: as the name implies, it’s not permanent. If the instance shuts down for any reason other than a warm reboot the data cannot be recovered[3]. Ephemeral instances also cannot be paused, since unused capacity is quickly recycled in EC2.

On the upside, ephemeral storage:

• has relatively consistent I/O
• has isolation from EBS availability and performance issues
• is included in the base instance cost

For awe.sm, consistent I/O is the most important benefit of ephemeral storage. Before migrating off EBS our database machines suffered from noisy neighbors, network turbulence, and eventually the workload exceeded the I/O performance that EBS was delivering. It was for these reasons that we moved to instance storage. As our data and workload has grown over the past year we have also begun RAIDing the ephemeral storage[4] of our database machines for increased I/O throughput and storage capacity.

awe.sm’s backup strategy

Now we’re going to look at some backup and recovery strategies. The most important factor in selecting a backup method is the time it takes for recovery. In order to minimize downtime we use a tiered backup plan involving redundancy, recycling (cannibalizing) backup hardware, and progressively slower layers of persistent storage. The first layer, redundancy, allows for seamless operation in the face of failure at the cost of keeping additional capacity on hand. The next layer, recycling, comes into play when capacity dips too low. Hardware may be promoted to fill gaps, or backups may be suspended while the EBS volumes are moved around to avoid copying large data over the public EC2 network. Lastly, all backups are persisted to EBS or S3, both of which are extremely durable, though slow.

Redundancy in EC2

The first line of defence in backups is redundancy. awe.sm is mirrored into 2 logical groups, each group covering 2 zones within us-east-1. We distribute traffic and manage configuration using HAproxy and DNS. This grouping helps us maintain availability during the loss of a single instance or an entire zone. While it is not ideal to run with only one logical group, it gives us enough capacity to focus on a proper recovery.

Backing up MySQL databases

Percona Xtrabackup

For MySQL databases over 1GB, backing up and restoring from raw SQL is impractical. Our MySQL backup tool of choice is Percona Xtrabackup. Xtrabackup allows non-blocking incremental backups of InnoDB databases and produces a usable MySQL data directory as its output.

Backing up small MySQL databases

For our smaller databases we use the streaming mode of Xtrabackup to create a compressed tarball of the MySQL data directory. These are stored on S3 for portability and the convenience of auto-expiration.

innobackupex --stream=tar /backups/tmp --slave-info \
     --user=backups --password=xxxxxxxx | gzip - > /backups/2013/02/hostname.tgz

Restoring from a streaming backup

To restore a streaming backup: uncompress the backup and prepare it with the apply-log command. Note: Be sure to use the -i option to tar (ignore EOFs) when unpacking the tarball[5].

mkdir /var/lib/MySQL.new
tar zxvif /backups/2013/02/hostname.tgz -C /var/lib/MySQL.new/
innobackupex --apply-log /var/lib/MySQL.new/

Once prepared: move the new directory into place, change ownership to the MySQL user, and start MySQL. To reconfigure replication use the replication values found in the xtrabackup_slave_info file in the restored MySQL directory.

Backing up large MySQL databases

Our main dataset is a group of MySQL databases weighing in around 500GB. A script performs incremental backups to EBS volumes and inintiates snapshots of the volumes at appropriate intervals.

First a base backup is made:

/usr/bin/innobackupex --user=backups --password=xxxxxxxx --slave-info \
     /backups/groupset-2013-01-20/base

Subsequent backups are made incrementally, and applied against the base backup.

/usr/bin/innobackupex --user=backups --password=xxxxxxxx --incremental --slave-info \
     --safe-slave-backup --incremental-basedir=/backups/groupset-2013-01-20/base \
     /backups/groupset-2013-01-20/incremental
/usr/bin/innobackupex --apply-log --redo-only /backups/groupset-2013-01-20/base \
     --incremental-dir=/backups/groupset-2013-01-20/incremental/2013-02-01_07-15-05

Restoring from an incremental backup

After copying the data to the host (from the backup host or a new snapshot clone) a final apply log step needs to be performed on the base backup to create the InnoDB log files:

innobackupex --apply-log /restore_mount/groupset-2013-01-20/base

Then move the base directory to the MySQL data directory location, change ownership of the directory to the MySQL user, and start MySQL. To reconfigure replication use the values from the xtrabackup_slave_info file in the latest incremental backup directory within the backup set.

A word about EBS snapshots

Snapshots of EBS drives are stored on S3, which has slower transfer speeds than EBS. When a volume is created from a snapshot EBS mocks the volume, attaches it to the instance and begins a background operation to sync the snapshot from S3. During this sync period, read operations on un-synced sectors will be proxied to S3 directly[6]. For this reason restoring from an EBS snapshot is the slowest method of recovery, and avoided where possible.

Other data

Riak

We’re beginning to use Riak here at awe.sm, and once again we find ourselves ignoring best practices[7] and running with our Riak data stored on ephemeral drives instead of EBS. Having our Riak data on EBS would allow portability and simplify recovery from lost instances. However, performance is more valuable to us than the flexibility EBS provides.

Our backup strategy is new and subject to change, but so far it looks like this: an exclusive lock is coordinated using a monitoring server. The host with the lock confirms the ring is healthy with the riak-admin ringready command to ensure the ring is ready for the loss of a node. The Riak service is stopped on the backup host and the ring, data, and config directories are copied to an EBS drive, which is snapshotted at regular intervals.

Restoring Riak requires copying the data back into place, confirming the node config (especially node name), configuring the ring for a node replacement, and letting Riak’s read-repair handle the rest. Note: When replacing a crashed riak node, make sure to explicitly ‘down’ the missing node before running the force-replace command. Otherwise the missing node will cause the riak cluster to be unable to converge ring changes.

Non database data

For small non-database data we primarily use S3. The tools s3cmd and jets3t have been especially useful for synchronizing files and directories to S3.

Another note on redundancy

While rare, there are times when being available in multiple zones within one region is not enough. For that contingency we maintain a warm backup cluster in us-west-1.

If your service has a critical need for uptime you should at minimum have an executable disaster recovery plan. Sync custom AMIs when they’re updated, have AWS launch configurations or scripts for spinning up instances and make sure your puppetmaster/chef server/ansible repository has its own backup and restore plan. The only thing worse than losing critical instances is the hopeless feeling of starting from scratch when you realize your build/deploy server went down with the fleet.

Final words

As a web service that connects our clients’ customers to the content they want, we aim to never get in the way, so we take availability extraordinarily seriously at awe.sm. Through the use of tiered redundancy and backups, and careful use of AWS services we have found a balance between convenience and stability. We enjoy the benefits of the flexibility of AWS while protecting ourselves from some of the risks of running in a cloud environment. For any service having a layered, tested backup strategy is paramount to preserving continuity. Remember that no system is perfect, but with proper preparation you can greatly reduce your risk in the chaos of the cloud.

Notes:
[1] The linux kernel can hang if it attempts to poll a mounted block device which is completely unresponsive. This can be dimished with mount options, but is still a pain.
[2] An unofficial list of EBS backed AWS services.
[3] While decreeing that ephemeral drives can never be recovered is a layer 8 decision, it is understandable in terms of efficiently utilizing resources in EC2. However not having a way to recover data off of *crashed* ephemeral instances is one hell of a stick for leading customers to pick EBS-backed instances.
[4] There is no guarantee that the instance storage drives of an instance are on different physical spindles of the host machine. Use a tool to test the performance of the disks separately and in parallel when building new ephemeral storage RAID arrays.
[5] From the Xtrabackup documentation on on streaming backups.
[6] From the section on persisting snapshots from the EC2 EBS Doc
[7] This time from our friends at Basho, great folk who were nice enough to come to our office recently and attempt to dissuade us from our crazy notions.

That being said, sometimes you need the raw numbers, so we’ve got your back. In addition to awe.sm’s live Google Spreadsheet tool, starting today you can export the data from any awe.sm module directly into a CSV for slicing and dicing in a spreadsheet application.

After you’ve used our query builder and explorer to save a module displaying the data you need, export in two easy steps.

1. Mouse over the module’s title to reveal a gear icon, and select Export:
   
2. In the Export dialog that appears, ensure the option for Download is selected, then click the Get CSV icon.
   

Your browser will begin downloading a CSV file. Save it to your local drive, and open it into your favorite spreadsheet app.

A Few Notes

• This feature is available to every awe.sm user once you switch over to our new version. (There are so many reasons to make the switch. Try it now!)
• INTERNET EXPLORER USERS: If you’re having trouble getting the file you’ve downloaded to open into Excel, try manually adding “.csv” to the end of its filename.
• If you really do have a “favorite” spreadsheet application, you almost certainly need a vacation.

Happy crunching! And never forget: if you have any questions or feedback, we’re just a live chat or email away. Or sign up for our free training webinars — they’re a blast.

As part of the filmmakers’ presentation of their efforts at this year’s Sundance Film Festival, our team paired awe.sm sharing data & attributed signups and purchases with IP-based geolocation click data to create a beautiful 3D visualization of how word spread throughout the campaign.

We’ve been playing with a working interactive demo of the project on an internal server, and our aspiration is someday to package displays like this as an off-the-shelf product for our marketer customers. To provide a taste of the kind of tracking that’s possible, here’s a quick video showing the campaign’s kickoff:

It’s striking to watch how quickly a single Facebook post in Los Angeles spawned clicks and child-shares elsewhere around the globe (the outgoing arrows), which in turn led to an impressive volume of mailing-list registrations and purchases (the inbound arrows). Any marketer curious to know the viral dynamics of a social post can instrument her marketing campaigns with awe.sm, track multiple generations of sharing and their outcomes, and understand exactly how posts lead to business results.

• Learn more about the Bones Brigade campaign
• Understand how awe.sm tracks multiple generations of sharing
• Ask us how to use awe.sm data to power similar visualizations of your own
• Come work here, and build more cool stuff like this!

We asked our investors and other startups for referrals. We also looked on GitHub to find contributors to the open source tools we were planning to use. After narrowing the list to three companies, we reached out and interviewed each company.

Our selection criteria:

• Expertise: We had decided to use Backbone.js. The framework has few opinions on The Right Way of doing things, so we wanted a team that had opinions about using it in a large application and could justify them with their experiences.
• Cost: We expected a rate of $100–$150 / hour and hoped to negotiate down by agreeing to a monthly minimum and a 3–6 month engagement.
• Similar engineering ideologies: No CoffeeScript. Appreciation of Douglas Crockford‘s opinions. Use of native DOM methods over jQuery when possible. Only performant uses of LESS.
• Size of team: We wanted a team, not a single developer. This is the “hit by a bus” factor: would we have to start over if the lead developer suddenly became unavailable? We also liked the option to have additional engineers working on self-contained features in parallel with primary development.
• Pair programming, if any, only when necessary: In our opinion, the primary beneficiaries of pair programming are the apprentice, who’s gaining experience, and the company employing the engineers, which is creating a sustainable workforce. The benefits of pair programming to awe.sm would be minimal. That said, pair programming is useful when beginning large features, as it allows engineers to have a common understanding of integration considerations when working independently on features. We wanted a team that had this pragmatic approach.
• Direct access to engineers who were fluent English speakers within 3 timezones of San Francisco: We wanted a company that felt like an extension of our team, not a middle manager that proxied information asynchronously. We knew that we were going to build a product with complex data handling and GUI interactions. The ability to communicate by text and video chat throughout the workday would allow us to most effectively discuss features, bugs, and details we hadn’t considered.
• Previous customers we could interview candidly about their experiences: We were making a big investment and needed to move quickly. We wanted to know the team’s strengths and weaknesses upfront.

After the interviews and contract negotiation, awe.sm chose Quick Left. I traveled to Boulder, CO to kick off the project with them. We discussed the features, interactions, and motivations for every detail in the spec and wireframes. They broke the user stories down in Pivotal Tracker, which awe.sm also used. With the stories agreed upon, I returned to San Francisco.

We used Campfire for text chat and Skype for video chat daily. I was able to review their commits in GitHub. We prioritized stories and bugs in Pivotal Tracker weekly. Six months and 70,000 lines of JavaScript later, the new awe.sm was ready for beta testers.

I learned a few things along the way and I’d do some things differently in the future:

• Text chat is great, but be quick to take conversations to video chat. We seem to express ourselves better with speech than text. Screen sharing is also useful.
• Gigantic specs with every minor detail in a single document don’t get used. Break each section of the application into a separate document. Kick off each section with a video chat. The spec should contain wireframes with relevant interactions, features within the context of user flows, and any forgettable specific technical details. If developers understand why features are valuable to the user and how the user will interact with them, many of the details don’t need to be documented. You spend less time writing specs and developers can more easily find the details they need to reference. Help the team understand your goals and they’ll make smart decisions.
• Expect fluctuations in workload when dealing with multiple parties. awe.sm was responsible for delivering additional server-side capabilities for the front-end application Quick Left was building. We also used another contractor for the visual design. Every interdependency increases the probability of lulls and spikes in workload.
• Managing external teams takes more time than you think. Coordinating assets, testing, and dealing with unknown unknowns is a full time job.

By the time our application launched, awe.sm had hired a front-end team. Quick Left trained our engineers on Backbone.js and helped us transition to internal development. Quick Left gave us a solid technical foundation for the new product they helped launch that we’re now rapidly iterating upon. Their quality of code matched awe.sm’s own high standard and their expertise helped us make the right engineering decisions for this product.

By working with Quick Left, awe.sm was able to take the time it needed to hire the right people and launch a new product without hampering the development of its existing products. When time-to-market is important and you have the cash available, using an outside development team can be a useful strategy for startups.

Also: We’re still hiring!

Thank you, Quick Left (Ingrid Alongi, Sam Breed, Alex McPherson, David Aragon, Jessica Dillon, Nico Valencia, Rachel Ryle)!

Personal satisfaction aside, this project and the data that the filmmakers and Topspin will be sharing at Sundance this week are nothing short of a revolution in film distribution. If you’re into the business of content, you should read more about the overall release on Topspin’s blog. The quick summary is that through an innovative combination of viral marketing, well-designed windowing, and creatively tiered pricing the producers were able to build an email list from 0 to more than 46,000 fans in just 2 months and ultimately make nearly 4 times the money they were offered in a conventional distribution deal.

Sharing drove 10% of the total revenue from the release. And thanks to deep awe.sm integration excellently implemented by the experts at The Uprising Creative, we tracked the value generated by each individual Tweet and Facebook post. Measuring the ROI of sharing is what awe.sm was built to do, and we have fascinating data on conversions from sharing across our hundreds of customers. But the unprecedented transparency of the Bones Brigade filmmakers is allowing us for the first time to publicly share some of the incredible insights we deliver our customers everyday.

Topspin’s strategy was to use the existing fan-bases of the Bones Brigade members, including Tony Hawk’s 4.2m Facebook fans and 3.3m Twitter followers, to get the word out to the core most passionate fans first and then encourage those fans to spread the message to their friends. This approach was extremely successful in driving a significant volume of high-value traffic with an average value of $1.59 per visit ($2.11 per visit from sharing by the cast and $1.21 per visit from sharing by ordinary fans). Being able to track the value of each share and the relationships between shares gives you visibility into the mechanics of how and why sharing is effective and not just the end results. This is essential to understanding the patterns of success and how to replicate them.

Here are some of the most interesting patterns we observed in the Bones Brigade sharing data:

• Fans were more than 7 times as likely to share to Facebook than to Twitter (87.8% of fans shared to Facebook vs 12.2% to Twitter);
• The value per visit from a fan’s Facebook share was more than 2 times higher than that from a fan’s Twitter share ($1.29/visit from Facebook vs $0.60 from Twitter);
• Sharing by the cast (i.e. “Celebrities”) was roughly even to both Facebook and Twitter but drove 82.0% of revenues from shares to Twitter and 18.0% from Facebook;
• However, the value per visit from the cast’s shares to Facebook and Twitter was roughly equal ($1.93/visit from Facebook vs $2.07/visit from Twitter);
• Sharing by the cast to Twitter drove more than twice as many visits per share as Facebook and more than 2.5 times the visits per fan/follower (1.10% eCTR for Facebook vs 2.82% eCTR for Twitter).



This stark contrast between the performance of sharing by celebrities vs ordinary fans across Facebook and Twitter demonstrates several underlying (and somewhat related) phenomena:

1. Normal people primarily use Twitter to discover content and Facebook to share it with friends – The overwhelming volume of fan sharing to Facebook (87.8%) clearly demonstrates that most people do not view Twitter as an important personal sharing channel, but the fact that Twitter was still able to drive 40.3% of the traffic and 47.3% of the revenue (primarily from celebrity sharing) shows that there is still a significant audience discovering and consuming content through Twitter despite their lack of sharing;
2. Twitter is a network of loose ties and Facebook is a network of strong ties – Visits driven by celebrities had roughly equivalent value across Facebook and Twitter while visits driven by regular fans were more than twice as valuable on Facebook than on Twitter, which demonstrates that consumers have tighter connections with and are more swayed by the recommendations of their friends on Facebook than by the non-celebrities they follow on Twitter;
3. Twitter followers are more likely to click than Facebook fans – Whether it’s an impact of NewsFeed Optimization or just a side-effect of the fact that normal people are primarily using Facebook to share with and consume from their friends and family, the numbers show that celebrity sharing to Twitter is driving over 2.5 times more visits per follower than sharing to Facebook is driving visits per fan with a roughly equal value per visit.

We’ll be taking a deeper look at each of these topics in upcoming blog posts. So follow @awesm on Twitter to learn more about this amazing campaign.

]]> http://blog.awe.sm/2013/01/18/how-the-bones-brigade-tweeted-their-way-to-6-figure-sales/feed/ 2