Scalable PHP with phpa, memcached and LVS (Part 1)


One of the major pitfalls of the PHP LAMP stack is that run-time compilation does not scale very well. An experienced developer would lament over PHP bloat and the other downfalls that come with the ease and convenience of using a scripting language as the foundation for your application.

Over the last month or so, I have been working on creating a better version of (AMO). It is a task that I enjoy, mostly because it isn’t everyday you get the great opportunity to work on a project that touches so many people and means so much.

My journey is similar to the journey taken by Drupal, PHPBB or MediaWiki developers. Given a great PHP product, you not only have to fulfill usability requirements, you also have to make the thing work for an insanely large user base. In my journey I have found an appreciation for the delicate balance between usability/features/design and the pain of having to maintain them. It’s the line between being a designer or marketing droid and being a core developer and a sysadmin.

I’ve learned that, in any successful project, you need to have someone who understands both sides. If you don’t, you run the risk of having one of two things:

  • A great looking website that meets your users’ needs but doesn’t run and is not secure
  • A fast and secure web app that nobody uses

A successful project usually lands somewhere inbetween. It’s a compromise between the best performance and the usability and flexibility most people demand — albeit users or administrators.

So when you’re faced with circumventing PHP bloat, you go through predictable steps:

  • Try to run your site as-is, but then realize it fails miserably because of the database bottleneck
  • Duplicate your database, splitting it between read/write and read-only — add a slave or multiple slaves via RRDNS to serve the bits you need
  • When this fails, you start looking for output caching mechanisms that give your database a break
  • You then realize that full output caching is unrealistic because your cache hits are not high enough
  • You find yourself stuck, looking for answers — and you find ways to deliver features with the hardware you have, without making any large-scale app-level changes

When you’re at this point, there are two solutions:

  • Optimize PHP somehow to reduce redundant compilation between requests
  • Use some sort of granular caching to give your DB a break

In the next part of this post, I’ll talk about implementation of phpa/memcached across an LVS cluster and what a difference it made compared to altnernative solutions. I will post graphs explaining perf gains for AMO, and explain why we used phpa and memcached as the final solution.

Building scalable applications is challenging, but you’re not alone.

The Danger of Mediocrity


a bobombAddons make the difference between a regular web browser and a browser that just got “pimped” (you can also pimp your email client, too :D). Having a wide variety of extensions and themes helps developers and product managers focus on core features the majority of users will need without having to simply say, “sorry, nope, can’t do that”.

Okay, so maybe your average extension isn’t going to turn your world upside-down the way West Coast Customs spending a week with your Honda would… fine. But addons for and by the people are a vital and important bridge between a non-bloated application core and end users who want personalized functionality.

Once we understand the importance of addons, and the role (AMO) plays for Mozilla products, we can start to focus on improving it.

So it’s great to see healthly discussions about the future of the service, and how it should be modified to ensure we ship addons with a level of quality that is comparable with Firefox or Thunderbird.

I recently read David Baron‘s excellent post about the danger of extensions and how an undersupported and relatively unregulated addon service could mean disaster for Mozilla products and Mozilla’s reputation in the community.

To say the least, I am now filled with uncertainty and doubt.

Though it’s not about the service itself, or whether or not the community will be able to recover from the first security hole found in an extension, or how extensions are external to the normal Mozilla review standards.

I’ve got FUD about how these products will thrive if they cannot change and adapt themselves quickly to new trends, web services or client-side tools that are available in other browsers.

Despite the theoretical holes in AMO, it’s there, it’s important and it’s popular — for better or worse. It has many great extensions, some good ones, and many poor extensions as well. It’s a distribution as diverse as the community; filled with the good, bad, and the ugly.

And the dangerous? Maybe. David’s points are valid — I share his concerns as well — but assuming problems with extensions will translate into plumetting consumer confidence in an application isn’t necessarily such a straight line. The risks we take with AMO would also need to be compared with consumer confidence in a product that didn’t adapt and offer unique features not found anywhere else.

It’s clear — we’ll have to find the solution halfway. We need to improve the overall quality of addons by improving the review process, and making moves to encourage the same openness and exposure Firefox and Thunderbird get. Most of these changes start with an improved (and solidified) policy, which is currently a major focus.

The technical solution could be CVS integration, stricter review requirements, Bugzilla integration. Ideally, we would have all of those, and everybody would develop, test and verify quality for their own extensions the way Mozilla does for Firefox and Thunderbird.

That all sounds great. Now, who’s going to do it? What would the practical schedule be for a new addon? A new extension version? When does the process of controlling the processes become simply overwhelming?

While I wish it were, I have to say the complete Mozilla review process isn’t a perfect fit for addons. It would delay addon releases, create a barrier between developers and the community, and create a lot of additional work for addon developers — who would most likely be satisfied with keeping their extension simply on their own site, in their own repository, with their own bug reporting software.

So how do we properly review this stuff, then? Myk Melez brought up some good ideas about a modified rating system to guage the overall “trustability” of a given extension. I thought that his approach would be a good one given the unique nature of the addon life cycle:

(Our current system) has a number of problems:

  1. it ignores other trust metrics that would refine our sense of each extension’s trustworthiness;
  2. there have never been enough reviewers, so extensions often wait days or weeks to get reviewed;
  3. generally only one reviewer reviews each extension, increasing the risk of human error;
  4. reviewers review extensions they aren’t personally interested in, so they exercise them less thoroughly than the ordinary users they approve them for;
  5. there’s little reward to being a reviewer, and the downside (being held responsible for approving a harmful extension) is significant.

An alternative approach is to publish each extension to the site the moment it is submitted but hide it from the majority of users until it reaches a trustworthiness threshold which takes into account all available trust metrics, including user feedback, code review, whether it is open-source, whether it is signed with a valid code-signing certificate, the trust score for other extensions by the same author, etc.

Until then, only specially designated “tester” users who have been apprised of the dangers and are willing to face them can access the extension. These testers, a much larger group than the current pool of reviewers, will provide feedback on such extensions like ordinary users, and that feedback will serve as one of the trust metrics through which an extension can reach the trustworthiness threshold required to make it available to the rest of the users.

This approach has several benefits over the current approach:

  1. testers try out extensions they’re interested in using, so they exercise them more thoroughly;
  2. multiple testers must look at and provide generally positive feedback for each extension before it satisfies its “user feedback” metric, so no one tester is solely responsible for deciding if an extension is good;
  3. each extension’s trust score comprises multiple trust metrics, increasing the quality of our decisions about each extension’s trustworthiness.

And, I strongly suspect, the mean time to publication will decrease significantly.

Myk’s idea is basically to properly weight the different metrics unique to the AMO process to determine whether or not an extension is trustworthy. It’s like an improved Digg mentality with a bit of moderation. While there is definitely more discussion needed in the next month, this would be a great place to start.

Sometimes pushing the envelope can give you a paper cut.

SQL Weirdness


Maybe I was just really tired from putting together my toolbox, but I couldn’t figure out why my initial query didn’t work last night when I was debugging the SQL for the Extend Firefox Finalists Page. The original SQL query would not sort the result set based on v.vid DESC and I would receive the same v.version everytime (the smallest one).

I fixed it with a subselect to ensure we were pulling the max v.vid per, but the original query still bugs me. So if anybody has any comments or ideas about why the initial one was borked, I’m curious to find out.

If you stare at something long enough, the solution disappears and instead you find yourself staring at a pretty unicorn.

AMO Movin’ On Up


movin' on up!!!

2006 brings some awesome news for AMO. There have been many meetings and discussions with MoCo about the progress of AMO, and it has all been very encouraging.

We have taken steps to speed up development by contracting with programmers, and there is more of a project management presence. Overall, things are progressing very well since the Firefox Summit gave us a chance to get together and talk about everything.

In December, mconnor and I discussed a schedule and what we really needed to get done. Getting everything out on the table was important, and it made the rest of December more productive. We accomplished the following since then:

  • Reconfigured and cleaned up addons CVS
    • Split public and developer tools into two separate applications
    • Removed Smarty from CVS, since it should be managed on the server level
    • General house cleaning on unnecessary files or directories
  • Migrated new template
  • Migrated search-engines page, started on recommended (getting started content)
  • Implemented Smarty’s caching capabilities to improve LVS node performance and noticed a great perf gain
  • Fixed all the stuff that Smarty caching broke 🙂

The upcoming month will be a time to focus on a re-release of the public-facing side of AMO. See the TODO list and the tentative schedule for more on what’s going on.

There are also some important things to think about as we approach this next version:

  • How can we better utilize the community to moderate comments and addons?
  • How can we make the site simpler for the average user?
  • How can we make site content more community-driven?
  • What can we do to make the review queue smaller without sacrificing QA?

There has been a lot of valuable discussion about these newer features/needs. Please chime in — join #umo and discuss it with me or anyone, or add your thoughts to the wiki.

There have been great thoughts about comment rating and user moderation, using “trust metrics” in order to determine the overall quality of a given extension, and making a Digg-like interface for deciding “what’s hot” on AMO to encourage user participation in the site.

2006 is going to be a better year for AMO, Mozilla and Firefox.

AMO Year-end Summary


During the Firefox Summit, there was a lot of healthy discussion regarding the immediate future of (AMO). We came up with a solid plan and timeline for top-priority items. The general idea is:

  • Split public and admin pages into two separate appliations.
  • Re-use core libraries, store them in a separate place in CVS.
  • After the rewritten public site is re-launched in January, start development on admin CRUD pages and inremental rewrite of all developer tools.
  • Profit! (just kidding)

random picture

The topics covered in the AMO breakout session ranged from standing problems, project overview, and additional resources needed to improve progress. I think it was a valuable discussion. Mostly I was happy that people recognized the importance of AMO. I left the room with a much better feeling about the future of the project (and also the knowledge that I need to work on my public speaking skills).

That said, I’d like to talk about why AMO is actually so important, because I don’t think everyone truly understands how vital it is to Firefox and Thunderbird, and consequently the entire Mozilla Foundation and Corporation.

It comes down to feature coverage, really. Most projects can hit 70-80% of the core feature requests. Firefox or Thunderbird don’t differ from this norm — Mozilla as an organization can only get so much done until they have to make the decision on where to draw the line for the next release. At some point project managers have to say, “Ok, these features are the most important and they will be what we focus on. These other ones will be slated for the next release, or whenever we get to it.”

And it’s not something unique to Mozilla — it happens everyday in projects across the world. The key, though, is managing what happens to the other 20-30% of the features. You might want to read a Wired article passed to me by Scott Kveton called The Long Tail — it explains the concept in more detail.

Successful projects today realize the value in developing for the developer — empowering the community to improve the application on their own. And I think that so far we have hit the tip of the iceberg in terms of getting a return on the ingenuity and open-mindedness of the general community.

We are already seeing a trend in more popular projects like Flickr or Google Maps where these services kick ass and deliver some awesome features just out-of-the-box, but what makes these apps _really_ special is their APIs. It’s also what makes Firefox and Thunderbird so special.

Coming down the pipe we are starting to see uses of these APIs that are mind blowing. With Katrina we saw an interesting use of the Google Maps API to report area status updates in Louisiana. On AMO we have seen some awesome extensions that have actually been integrated into 1.5 because they were so, “Duh, we should have done that from the beginning”.

For Mozilla, AMO is the playground for the determined user to do their thing and get exactly what they want to get out of their web browser or email client. It covers the extra 20-30% and best of all it benefits the product, the community, and the world (yeah, melodramatic, I know) by improving an already sound base of features.

Over the past year I’ve developed an appreciation for the importance of AMO. I’ve learned about its challenges, all the players involved, and hopefully we’ve come up with a solid plan for 2006. I’d like to see us provide a better tool for the community to develop, submit and distrubute great ideas. With that in place, Firefox and Thunderbird will continue to have a community-centric family of extensions that sets them apart from all competitors.

Oooh. Shiny.

Web 2.0 Has its Consequences


When people want to find information, they don’t care how well it’s packaged. They care more about:

  • Is it easy to find?
  • Is it accurate?
  • Is it simple to read?

Web 2.0 is a buzzword that almost makes me want to vomit. What gets me is the complete lack of understanding and responsibility when it comes to reinventing the usability model of a web interface. By “usability” I mean “things that people expect from a web site”.

Technologies such as Flash or JavaScript can compeltely alter web site behavior through manipulation of the DOM. I typically approach these technologies as only necessary when you have extra time to create value-added user experiences. This means that if you stripped these items out, the site should still be understandable and would work for anybody using a modern browser that supported HTML standards.

Web 2.0 is completely misunderstood on two fronts:

  • It’s not just AJAX. Get over AJAX.
  • It’s more than Flock, Flickr, Gmail or Google maps.

Myth #1 is that if you plug AJAX into your site you are Web 2.0 and you will be greeted in outer space by 72 virgins, etc. Please give me a break already. Most people misuse AJAX, because they don’t take responsibility for recreating the usability model of Web 1.0 (or whatever you want to call it) and create asynchronous pages that have no context.

Case in point — say you are on your homepage and you use an AJAX operation to request data for the current page. You decide suddenly that you messed up your page and you want to go back. Well, most of the time, there is no back. Because your browser isn’t aware of the change in state.

To be fair, it could be, but most browsers aren’t. I’m not sure if they should be. The fact of the matter is that people became used to a standard for viewing web pages and that is endangered by the media buzz surrounding Web 2.0.

And shame on the media — and their followers. Again, they have blown things out of proportion and have left caution to the wind. The Web 2.0 ideology truly surrounds a user-centric approach to information sharing versus a server-centric or website-centric approach. It’s a new way of thinking that puts the user at the center of the internet universe.

This doesn’t doesn’t equate to riddling web pages with unnecessary scripting that takes advantage of The Next Best Thing just because Slashdot had some bullshit article about why it was cool.

Myth #2 is that using AJAX can easily make your site this wonderfully dynamic web application. People don’t realize that Gmail and Google maps are very complex, and they go through painstaking lengths to reinvent the common web interface using JavaScript. All of their efforts are put into preserving familiarity and client state. Normal operations and functions of a web browser are reinvented. Imagine having to reimplement:

  • A back button.
  • The address bar.
  • A forward button.
  • A refresh button.

Arguably, you may not successfully be able to reimplement all of these with today’s browsers in a “Web 2.0” application. Still worth it? Maybe. It depends on what your users need. Oh yes — the users.

Overall, Web 2.0 is mostly about making better use of standards that do exist (which, ironically, is almost what AJAX is — although it’s not treated as such). It’s about creating sites that make sense, use less words, and utilize existing technologies to their fullest potential.

In the end, I’m not saying AJAX is bad — it has great uses. Flash and other plug-ins also have their place amont the web elite. But as far as Web 2.0 goes, the best thing you could probably do is assume responsibility for what the hell you are doing and truly consider the best experience for your users.

Oddly enough, it doesn’t always mean overcomplicating things. Most of the time it means cutting down on the fluff and just delivering the goods. If anything, it’s shortening the path between any user and what they need.

All the fancy Flash and JavaScript in the world can’t make up for shitty content.