Let's scale twitter 6

Posted by pratik
on Friday, August 24

The common wisdom in the Rails community at this time is that scaling Rails is a matter of cost: just throw more CPUs at it.

- Taken from a fuddy interview

Now, lets

From asset_tag_helper :

Caching multiple javascripts into one

You can also cache multiple javascripts into one file, which requires less HTTP connections to download and can better be compressed by gzip (leading to faster transfers). Caching will only happen if ActionController::Base.perform_caching is set to true (which is the case by default for the Rails production environment, but not for the development environment).

Now lets look at some of the headers returned by twitter :

1
2
3

GET /javascripts/prototype.js?1187986190
GET /javascripts/effects.js?1187986190
GET /javascripts/application.js?1187986190

And oh yeah, they return 200 instead of 304 almost always!

What more can I say ? Please scale ;-)

Comments

Leave a response

  1. ORLY? – August 24, 2007 @ 10:35 PM

    Um, I really doubt Twitter’s problems can be boiled down to not making their static files cacheable.

  2. Pratik – August 24, 2007 @ 10:49 PM

    Hah. I didn’t say anything about boiling down all twitter problems to this issue. But solving small things like this definitely help at the end when you’re running something as large as twitter.

  3. RSL – August 28, 2007 @ 12:57 PM

    What was the point of Twitter again? To tell everyone on the Internets what you’re doing all day? And to read what everyone else is doing? How fascinatingly mundane!

  4. Piyush – September 14, 2007 @ 10:05 PM

    GET /javascripts/prototype.js?1187986190 GET /javascripts/effects.js?1187986190 GET /javascripts/application.js?1187986190

    Well it doesn’t really matter because these files are being served by apache and not mongrel/<rails>. Also they are using gzip compression sending these files. Also the combining js files into one is not always a good idea because apache can serve multiple requests at a time so all these files can be theoretically served simultaneously. Anyways rails scaling “problem” has more to do with developers not understanding what they are doing.

  5. Pratik – September 17, 2007 @ 10:41 AM

    Piyush,

    1. It’s not apache, it’s litespeed.
    2. It’s not apache serving mutilple requests in parallel. It’s about combining all your JS shit in one file and making it cachable by Browser. And then there is no question of serving them in parallel as there should be just one request for which server should be retuning 304 response.
    3. Rails scaling was used here as just humor bit. It’s not as simple as whatever you said.
  6. Dan – December 10, 2007 @ 07:00 PM

    Of course scalability and performance are not the same thing, but they are related. Often improving one will improve the other, but not always. In this case I think Pratik’s suggestion will improve both.

    A while ago Yahoo did some testing and they found that from the user’s view over 80-90% of the time spent waiting was for javascript, stylesheets and images to download. Only 10-20% of that time was waiting for the HTML to be generated and sent down the pipe.

    That means increasing the speed of your web app won’t usually increase your application’s speed anywhere near as much as optimizing the front-end. What’s the point in bringing an application’s average request time down from 3 seconds to 2.5 if it takes the end-user 30 seconds to download all the images, stylesheets and javascript? I say focus on the user’s experience first and you’ll get the most for your time.

    I use tools like YSlow and webpageanalyzer.com to check sites I work on, and have found that most apps are extremely inefficient in terms of what’s sent down to the end user. Most of the points YSlow measures can be addressed in a few minutes to a few hours (at most) and will easily result in a 100-200% speed increase for unoptimized apps.

Comment