AMO v2

Standard

The public rewrite of AMO was released today.

Fixed in this release:

  • Stuff
  • More stuff
  • Scalability
  • Other stuff

No, but seriously, you might find that your bookmarks are a little off, or _____. If so, find us on irc in #umo@irc.mozilla.org and let us know — we aren’t at the “file a bug if it’s broke” stage yet.

Thanks to Wil Clouser, who worked on a large portion of the updates. Thanks to everyone who pointed out bugs and helped us fix stuff, and thanks most importantly to the reviewers who help us keep this ship afloat.

Speaking of which, please take some time to review our draft policy that we have been working on!

Better late than never, and better late than totally crappy.

Scalable PHP with phpa, memcached and LVS (Part 1)

Standard

One of the major pitfalls of the PHP LAMP stack is that run-time compilation does not scale very well. An experienced developer would lament over PHP bloat and the other downfalls that come with the ease and convenience of using a scripting language as the foundation for your application.

Over the last month or so, I have been working on creating a better version of addons.mozilla.org (AMO). It is a task that I enjoy, mostly because it isn’t everyday you get the great opportunity to work on a project that touches so many people and means so much.

My journey is similar to the journey taken by Drupal, PHPBB or MediaWiki developers. Given a great PHP product, you not only have to fulfill usability requirements, you also have to make the thing work for an insanely large user base. In my journey I have found an appreciation for the delicate balance between usability/features/design and the pain of having to maintain them. It’s the line between being a designer or marketing droid and being a core developer and a sysadmin.

I’ve learned that, in any successful project, you need to have someone who understands both sides. If you don’t, you run the risk of having one of two things:

  • A great looking website that meets your users’ needs but doesn’t run and is not secure
  • A fast and secure web app that nobody uses

A successful project usually lands somewhere inbetween. It’s a compromise between the best performance and the usability and flexibility most people demand — albeit users or administrators.

So when you’re faced with circumventing PHP bloat, you go through predictable steps:

  • Try to run your site as-is, but then realize it fails miserably because of the database bottleneck
  • Duplicate your database, splitting it between read/write and read-only — add a slave or multiple slaves via RRDNS to serve the bits you need
  • When this fails, you start looking for output caching mechanisms that give your database a break
  • You then realize that full output caching is unrealistic because your cache hits are not high enough
  • You find yourself stuck, looking for answers — and you find ways to deliver features with the hardware you have, without making any large-scale app-level changes

When you’re at this point, there are two solutions:

  • Optimize PHP somehow to reduce redundant compilation between requests
  • Use some sort of granular caching to give your DB a break

In the next part of this post, I’ll talk about implementation of phpa/memcached across an LVS cluster and what a difference it made compared to altnernative solutions. I will post graphs explaining perf gains for AMO, and explain why we used phpa and memcached as the final solution.

Building scalable applications is challenging, but you’re not alone.

The Danger of Mediocrity

Standard

a bobombAddons make the difference between a regular web browser and a browser that just got “pimped” (you can also pimp your email client, too :D). Having a wide variety of extensions and themes helps developers and product managers focus on core features the majority of users will need without having to simply say, “sorry, nope, can’t do that”.

Okay, so maybe your average extension isn’t going to turn your world upside-down the way West Coast Customs spending a week with your Honda would… fine. But addons for and by the people are a vital and important bridge between a non-bloated application core and end users who want personalized functionality.

Once we understand the importance of addons, and the role addons.mozilla.org (AMO) plays for Mozilla products, we can start to focus on improving it.

So it’s great to see healthly discussions about the future of the service, and how it should be modified to ensure we ship addons with a level of quality that is comparable with Firefox or Thunderbird.

I recently read David Baron‘s excellent post about the danger of extensions and how an undersupported and relatively unregulated addon service could mean disaster for Mozilla products and Mozilla’s reputation in the community.

To say the least, I am now filled with uncertainty and doubt.

Though it’s not about the service itself, or whether or not the community will be able to recover from the first security hole found in an extension, or how extensions are external to the normal Mozilla review standards.

I’ve got FUD about how these products will thrive if they cannot change and adapt themselves quickly to new trends, web services or client-side tools that are available in other browsers.

Despite the theoretical holes in AMO, it’s there, it’s important and it’s popular — for better or worse. It has many great extensions, some good ones, and many poor extensions as well. It’s a distribution as diverse as the community; filled with the good, bad, and the ugly.

And the dangerous? Maybe. David’s points are valid — I share his concerns as well — but assuming problems with extensions will translate into plumetting consumer confidence in an application isn’t necessarily such a straight line. The risks we take with AMO would also need to be compared with consumer confidence in a product that didn’t adapt and offer unique features not found anywhere else.

It’s clear — we’ll have to find the solution halfway. We need to improve the overall quality of addons by improving the review process, and making moves to encourage the same openness and exposure Firefox and Thunderbird get. Most of these changes start with an improved (and solidified) policy, which is currently a major focus.

The technical solution could be CVS integration, stricter review requirements, Bugzilla integration. Ideally, we would have all of those, and everybody would develop, test and verify quality for their own extensions the way Mozilla does for Firefox and Thunderbird.

That all sounds great. Now, who’s going to do it? What would the practical schedule be for a new addon? A new extension version? When does the process of controlling the processes become simply overwhelming?

While I wish it were, I have to say the complete Mozilla review process isn’t a perfect fit for addons. It would delay addon releases, create a barrier between developers and the community, and create a lot of additional work for addon developers — who would most likely be satisfied with keeping their extension simply on their own site, in their own repository, with their own bug reporting software.

So how do we properly review this stuff, then? Myk Melez brought up some good ideas about a modified rating system to guage the overall “trustability” of a given extension. I thought that his approach would be a good one given the unique nature of the addon life cycle:

(Our current system) has a number of problems:

  1. it ignores other trust metrics that would refine our sense of each extension’s trustworthiness;
  2. there have never been enough reviewers, so extensions often wait days or weeks to get reviewed;
  3. generally only one reviewer reviews each extension, increasing the risk of human error;
  4. reviewers review extensions they aren’t personally interested in, so they exercise them less thoroughly than the ordinary users they approve them for;
  5. there’s little reward to being a reviewer, and the downside (being held responsible for approving a harmful extension) is significant.

An alternative approach is to publish each extension to the site the moment it is submitted but hide it from the majority of users until it reaches a trustworthiness threshold which takes into account all available trust metrics, including user feedback, code review, whether it is open-source, whether it is signed with a valid code-signing certificate, the trust score for other extensions by the same author, etc.

Until then, only specially designated “tester” users who have been apprised of the dangers and are willing to face them can access the extension. These testers, a much larger group than the current pool of reviewers, will provide feedback on such extensions like ordinary users, and that feedback will serve as one of the trust metrics through which an extension can reach the trustworthiness threshold required to make it available to the rest of the users.

This approach has several benefits over the current approach:

  1. testers try out extensions they’re interested in using, so they exercise them more thoroughly;
  2. multiple testers must look at and provide generally positive feedback for each extension before it satisfies its “user feedback” metric, so no one tester is solely responsible for deciding if an extension is good;
  3. each extension’s trust score comprises multiple trust metrics, increasing the quality of our decisions about each extension’s trustworthiness.

And, I strongly suspect, the mean time to publication will decrease significantly.

Myk’s idea is basically to properly weight the different metrics unique to the AMO process to determine whether or not an extension is trustworthy. It’s like an improved Digg mentality with a bit of moderation. While there is definitely more discussion needed in the next month, this would be a great place to start.

Sometimes pushing the envelope can give you a paper cut.

SQL Weirdness

Standard

Maybe I was just really tired from putting together my toolbox, but I couldn’t figure out why my initial query didn’t work last night when I was debugging the SQL for the Extend Firefox Finalists Page. The original SQL query would not sort the result set based on v.vid DESC and I would receive the same v.version everytime (the smallest one).

I fixed it with a subselect to ensure we were pulling the max v.vid per main.id, but the original query still bugs me. So if anybody has any comments or ideas about why the initial one was borked, I’m curious to find out.

If you stare at something long enough, the solution disappears and instead you find yourself staring at a pretty unicorn.

AMO Movin’ On Up

Standard

movin' on up!!!

2006 brings some awesome news for AMO. There have been many meetings and discussions with MoCo about the progress of AMO, and it has all been very encouraging.

We have taken steps to speed up development by contracting with programmers, and there is more of a project management presence. Overall, things are progressing very well since the Firefox Summit gave us a chance to get together and talk about everything.

In December, mconnor and I discussed a schedule and what we really needed to get done. Getting everything out on the table was important, and it made the rest of December more productive. We accomplished the following since then:

  • Reconfigured and cleaned up addons CVS
    • Split public and developer tools into two separate applications
    • Removed Smarty from CVS, since it should be managed on the server level
    • General house cleaning on unnecessary files or directories
  • Migrated new template
  • Migrated search-engines page, started on recommended (getting started content)
  • Implemented Smarty’s caching capabilities to improve LVS node performance and noticed a great perf gain
  • Fixed all the stuff that Smarty caching broke :)

The upcoming month will be a time to focus on a re-release of the public-facing side of AMO. See the TODO list and the tentative schedule for more on what’s going on.

There are also some important things to think about as we approach this next version:

  • How can we better utilize the community to moderate comments and addons?
  • How can we make the site simpler for the average user?
  • How can we make site content more community-driven?
  • What can we do to make the review queue smaller without sacrificing QA?

There has been a lot of valuable discussion about these newer features/needs. Please chime in — join #umo and discuss it with me or anyone, or add your thoughts to the wiki.

There have been great thoughts about comment rating and user moderation, using “trust metrics” in order to determine the overall quality of a given extension, and making a Digg-like interface for deciding “what’s hot” on AMO to encourage user participation in the site.

2006 is going to be a better year for AMO, Mozilla and Firefox.

AMO Year-end Summary

Standard

During the Firefox Summit, there was a lot of healthy discussion regarding the immediate future of addons.mozilla.org (AMO). We came up with a solid plan and timeline for top-priority items. The general idea is:

  • Split public and admin pages into two separate appliations.
  • Re-use core libraries, store them in a separate place in CVS.
  • After the rewritten public site is re-launched in January, start development on admin CRUD pages and inremental rewrite of all developer tools.
  • Profit! (just kidding)

random picture

The topics covered in the AMO breakout session ranged from standing problems, project overview, and additional resources needed to improve progress. I think it was a valuable discussion. Mostly I was happy that people recognized the importance of AMO. I left the room with a much better feeling about the future of the project (and also the knowledge that I need to work on my public speaking skills).

That said, I’d like to talk about why AMO is actually so important, because I don’t think everyone truly understands how vital it is to Firefox and Thunderbird, and consequently the entire Mozilla Foundation and Corporation.

It comes down to feature coverage, really. Most projects can hit 70-80% of the core feature requests. Firefox or Thunderbird don’t differ from this norm — Mozilla as an organization can only get so much done until they have to make the decision on where to draw the line for the next release. At some point project managers have to say, “Ok, these features are the most important and they will be what we focus on. These other ones will be slated for the next release, or whenever we get to it.”

And it’s not something unique to Mozilla — it happens everyday in projects across the world. The key, though, is managing what happens to the other 20-30% of the features. You might want to read a Wired article passed to me by Scott Kveton called The Long Tail — it explains the concept in more detail.

Successful projects today realize the value in developing for the developer — empowering the community to improve the application on their own. And I think that so far we have hit the tip of the iceberg in terms of getting a return on the ingenuity and open-mindedness of the general community.

We are already seeing a trend in more popular projects like Flickr or Google Maps where these services kick ass and deliver some awesome features just out-of-the-box, but what makes these apps _really_ special is their APIs. It’s also what makes Firefox and Thunderbird so special.

Coming down the pipe we are starting to see uses of these APIs that are mind blowing. With Katrina we saw an interesting use of the Google Maps API to report area status updates in Louisiana. On AMO we have seen some awesome extensions that have actually been integrated into 1.5 because they were so, “Duh, we should have done that from the beginning”.

For Mozilla, AMO is the playground for the determined user to do their thing and get exactly what they want to get out of their web browser or email client. It covers the extra 20-30% and best of all it benefits the product, the community, and the world (yeah, melodramatic, I know) by improving an already sound base of features.

Over the past year I’ve developed an appreciation for the importance of AMO. I’ve learned about its challenges, all the players involved, and hopefully we’ve come up with a solid plan for 2006. I’d like to see us provide a better tool for the community to develop, submit and distrubute great ideas. With that in place, Firefox and Thunderbird will continue to have a community-centric family of extensions that sets them apart from all competitors.

Oooh. Shiny.

Web 2.0 Has its Consequences

Standard

When people want to find information, they don’t care how well it’s packaged. They care more about:

  • Is it easy to find?
  • Is it accurate?
  • Is it simple to read?

Web 2.0 is a buzzword that almost makes me want to vomit. What gets me is the complete lack of understanding and responsibility when it comes to reinventing the usability model of a web interface. By “usability” I mean “things that people expect from a web site”.

Technologies such as Flash or JavaScript can compeltely alter web site behavior through manipulation of the DOM. I typically approach these technologies as only necessary when you have extra time to create value-added user experiences. This means that if you stripped these items out, the site should still be understandable and would work for anybody using a modern browser that supported HTML standards.

Web 2.0 is completely misunderstood on two fronts:

  • It’s not just AJAX. Get over AJAX.
  • It’s more than Flock, Flickr, Gmail or Google maps.

Myth #1 is that if you plug AJAX into your site you are Web 2.0 and you will be greeted in outer space by 72 virgins, etc. Please give me a break already. Most people misuse AJAX, because they don’t take responsibility for recreating the usability model of Web 1.0 (or whatever you want to call it) and create asynchronous pages that have no context.

Case in point — say you are on your homepage and you use an AJAX operation to request data for the current page. You decide suddenly that you messed up your page and you want to go back. Well, most of the time, there is no back. Because your browser isn’t aware of the change in state.

To be fair, it could be, but most browsers aren’t. I’m not sure if they should be. The fact of the matter is that people became used to a standard for viewing web pages and that is endangered by the media buzz surrounding Web 2.0.

And shame on the media — and their followers. Again, they have blown things out of proportion and have left caution to the wind. The Web 2.0 ideology truly surrounds a user-centric approach to information sharing versus a server-centric or website-centric approach. It’s a new way of thinking that puts the user at the center of the internet universe.

This doesn’t doesn’t equate to riddling web pages with unnecessary scripting that takes advantage of The Next Best Thing just because Slashdot had some bullshit article about why it was cool.

Myth #2 is that using AJAX can easily make your site this wonderfully dynamic web application. People don’t realize that Gmail and Google maps are very complex, and they go through painstaking lengths to reinvent the common web interface using JavaScript. All of their efforts are put into preserving familiarity and client state. Normal operations and functions of a web browser are reinvented. Imagine having to reimplement:

  • A back button.
  • The address bar.
  • A forward button.
  • A refresh button.

Arguably, you may not successfully be able to reimplement all of these with today’s browsers in a “Web 2.0″ application. Still worth it? Maybe. It depends on what your users need. Oh yes — the users.

Overall, Web 2.0 is mostly about making better use of standards that do exist (which, ironically, is almost what AJAX is — although it’s not treated as such). It’s about creating sites that make sense, use less words, and utilize existing technologies to their fullest potential.

In the end, I’m not saying AJAX is bad — it has great uses. Flash and other plug-ins also have their place amont the web elite. But as far as Web 2.0 goes, the best thing you could probably do is assume responsibility for what the hell you are doing and truly consider the best experience for your users.

Oddly enough, it doesn’t always mean overcomplicating things. Most of the time it means cutting down on the fluff and just delivering the goods. If anything, it’s shortening the path between any user and what they need.

All the fancy Flash and JavaScript in the world can’t make up for shitty content.

The Value of Face-time

Standard

At some point I inevitably hit a wall and need to bounce ideas off someone. IRC and other forms of chat work great. I can ping someone and get a quick answer, or at least get a RTFM or RTFW.

blank face with a question mark

Often times, though, the quick answer or linkification doesn’t make things abundantly clear, and the solution to my problem is still rather fuzzy — and not the good fuzzy — the bad fuzzy, like hangover fuzzy.

So Google doesn’t hold the answer, or at least not in a form I can understand. The server guys who know everything have no idea wtf I’m talking about, so I’m stuck. What I need is a whiteboard, pens of different colors, and a 12-step program from someone in the project who knows what the hell is going on.

It’s a part of working with a team. You learn and build off of your teammates for certain things, and you help them grow in other areas as well. It’s complimenting strengths and weaknesses that can make the whole greater than its parts. The whiteboard discussions, and the shorter pen-and-paper talks can remove the bad fuzziness and replace it with clarity.

In addition to providing a solution, these discussions (usually, anyway) form great relationships between team members and can help open new paths of communication. They can also lead to healthy tangets and the inevitable brainstorming that follows.

In a way, all this sounds like what friends might do. Sounds great, right? Well, hold on. I have a point here somewhere…

Whiteboard or pen-and-paper discussions don’t always happen for people who work together on open source projects. They take place in cyberspace, and assume many forms, but don’t ever match up to face-time. As a result, I believe projects can really suffer without face-to-face interaction between contributors.

So we are left with technology to fill the gaps between us. The first problem with this is the waiting — five, ten minutes — maybe hours, maybe days. In the meantime our problem ferments and makes us dream code or leads us to beer as an escape (yay!). The problem isn’t solved right away, if ever, and progress hiccups.

The second problem is more about human interaction. You simply can’t build rich and complex relationships between people through IRC, newsgroups, AIM or emails. Sooner or later you have to take the extra step to meet these IRC handles, emails or nicknames and place a face on them. Why? Because great projects are fueled by great face-to-face relationships.

This brings us to what I’m doing right now. All of my current projects are done remotely in collaboration with people who are mostly in a different time zone and sometimes on a different continent. December has been an eye-opening month for me because I’ve gained an appreciation for how many people are actually out there.

Over time you start to get used to the email or IRC buzz. Names are all over the place, and sometimes you lose touch with the “who” and concentrate more on the “what” and “when”. So having the chance to visit Mountain View and meet almost everyone I’ve talked to about anything regarding Mozilla and heading over to Tucson to meet the Kuali team has meant a lot to me.

It’s more than the warm fuzzy (the good fuzzy) feeling I get from meeting everyone in person and hang out for a bit after work. The positive vibe is also reflected in terms of productivity. We get that chance to voice our concerns and know that everyone is listening. We get time to sit in front of the whiteboard and hammer things out. We take wild tangents, have great discussions, get pissed, laugh it off, come up with a plan or solution, and feel great. And oh yeah – we all get a chance to listen a lot, too. :)

My second week on the road is quickly coming to an end. When I get back home I will have driven a total of 2200 miles and flown for 600 miles. Getting to meet everyone, listen and learn from them, and work on building a plan for the future of our projects was well worth the trip.

You can’t send a human being through a wire.

Mozilla in Da House

Standard

mozilla on the billboard

At the Firefox Summit, most of us headed out for a San Jose Sharks hockey game that Thursday night. Just so happened we saw a Mozilla welcome message on the billboard. The game was against the Florida Panthers, and it must have been a sign — the Sharks won 6-2.