That’s not defense, Bruce

Standard

Dear Bruce Bowen,

I know you tried really hard to foul the crap out of Dirk and get away with it. So sorry you and your team of muggers couldn’t hack your way to another championship at the expense of a better team.

I read your website, which is almost entirely dedicated to the “art of defense”. You spent all that money making a pointless flash site that didn’t have anything about defense except, “eat your greenies” and “train hard!”.

I was disappointed to find out that even on your website, when you’ve had time to think about it all, you don’t seem to understand anything about defense.

You have career averages of 7.5 points, 3.9 rebounds and 1.5 assists per game. You average 0.9 steals a game, shoot a horrible percentage on offense and are pretty much a non-factor in every statistical category.

In 10 years, you have not learned how to shoot free throws. You’re a 6’7″ small forward/shooting guard who can’t shoot free-throws or score effectively from outside or inside. You don’t steal the ball and don’t pass very much.

So you did what you had to do, I can dig it. You made a living off of frustrating the hell out of the other team’s best offensive player. It’s honest work.

But as I watched you tonight in game 7, shamelessly holding and slapping at Dirk, I just felt sorry for you. You have been framed as a defensive specialist but when your team needed real defense all you had was a 6’7″ frame and hope that Dirk would miss.

Better luck next season, Bruce. Let’s hope the refs don’t catch on to your strategy. I’d hate to see you try to earn a starting spot on any team based on any other part of your game.

<3,
Mike

PS – The Spurs sure cry a lot for a team who won championships on blown calls (also, I’m a jackass for writing this post).

Americano Bizarro

Standard

<rant>

Bizarre. That’s about the only way I can explain things. Baseball players are required to testify under oath in front of senate judiciary committees, but oil executives and district attorneys aren’t.

who cares about bonds?
Sure enough, Mr. Specter, maybe you should have rethought how much you can trust people nowdays — especially people involved with the white house — and especially the ones who deny they were.

Although, I don’t think you thought very much at all about it. You probably just did what you were told. Either way — shame on you for not holding people accountable for telling the truth.

This crap has been building up for a while — so let me get some stuff off my chest. Let’s see — things our congress has focused on:

  • Steriods in baseball (who gives a shit)
  • Building a WALL — A FRICKING WALL — between the US and Mexico (didn’t work for Germany or China; way to regress 200 years guys; done for political gain in November)
  • Discussion about gay marriage (robbing life, liberty and the pursuit of happiness from US citizens based on an uncontrollable part of their nature; again, way to regress 200 years; didn’t work for slave owners or restricting women’s voting rights; didn’t work for Hitler either)
  • Janet Jackon’s nipple (not something to ban from TV — embrace it!)

Things that were pretty much ignored:

  • Domestic wire-tapping (national security, shh, it’s so secret nobody is accountable for it)
  • Constitutional checks and balances (wtf?! there are limits to presidential power?)
  • Safety regulation in mining operations (oh, people can die from underregulation that benefits rich business owners?; no lobbyists for saving lives?)
  • Health care reform
  • Social security reform
  • Budget deficit
  • Lobbying reform
  • Election reform
  • Shit, well, pretty much everything they should have been instead of building walls and working on conservative (and pointless) legislation

So my response to this stuff — including Barry Bonds hitting seven-hundred-something home-runs? Get a grip, people. The country is going down the tubes and you’re worried about baseball records and steroids? Wake up.

Congress does a lot of great things, but lately I just don’t know what those things are.

</rant>

Open Source Undefined

Standard
Most persons do not see the sun. At least they have a very superficial seeing. The sun illuminates only the eye of the man, but shines into the eye and heart of the child.
– Nature, Emerson

Open source can be any number of things, depending on who you’re asking. To the hacker it’s about teamwork, comradery, IRC, CVS or SVN and great flamewars. To the CEO it’s risky, and often times very dependent on highly-skilled staff who may or may not be here tomorrow. To Wall Street it’s a non-commodity with high risk and low returns. To mom and dad it’s a mysterious concept you’ve asked about many times but never quite understood. To my cats, it’s what makes me sit at the glowing rectangles for hours on end.

To all of us, though, it’s the future. Most of us agree on that much.

In my time here at OSU in Central Web Services and the Open Source Lab I’ve spent a lot of time thinking about open source, because I’ve been in the middle of it. Working with the Mozilla community has also taught me a lot about the masses and community development. I learned over the past three years that it’s less about the masses and more about individuals who care a great deal about where we’re all going.

So the #1 question people ask me is, “What is open source?” To be honest, I’ve stopped worrying about defining open source. Instead I’ve tried to appreciate it.

Does it really matter what open source is? If it’s an ideal, a business model, communication style, natural progression, another step in our evolution — I’ve stopped caring about trying to outline it and present it in a bulleted list.

To me it’s just chaos, and it’s wondrous. I’m so caught up in it. It’s intoxicating and pulls me in some undefined direction. Like riding in an airplane with an unknown destination. You don’t know the pilots or where you’re going but you’re still excited about the possibilities. Like stepping forward onto a stage, lights blinding, and having an act but still not being entirely sure about how the show is going to turn out.

Or maybe it’s an emotion. You can’t quantify sadness or jubilation. How do you explain the color yellow to a 5-year-old? You can’t. How do you explain happiness to a rock? You can’t.

So I’m done trying to draw blueprints for the stage, or profiling the pilots and passengers. I’ve given up and put down the pen.

Because no matter how many times I fill a whiteboard or flowchart, I leave the audience thirsty. They are given a little sip of an idea — ephemeral and slight as something like the American Dream. It’s something we all whisper about and pull for but can’t really define. It’s something driving us that seems so strong and powerful yet at times, in its tiny fissures and failures, it all seems so fragile and brittle.

It is what it is — whitewash pitted with valleys and riddled with amazing plateaus. It’s us reflected in code which is scary and unpredictable but it’s alright if we remember we’re in it together.

It’s life, and you can’t define life — you just live it.

Something doesn’t have to be intricately defined to have meaning.

Mark Cuban Kicks Ass

Standard

mark cuban rocks chuck norris Okay, I’ll admit it. I initially didn’t like the guy. He sat in the front row at Mavericks games, all riled up, fists pumping and emotions swaying with every call. He just seemed pretty obnoxious, which makes me hastily throw you in my “oh no, not him again” category.

Be careful about who you dismiss.

I spent a few hours last night replacing much-needed sleep with a complete tour of Mark Cuban’s life, courtesy of Google, Wikipedia and blogmaverick.com.

Sure enough — I was wrong. Somewhere behind the fist pumping, shouting at refs and NBA fines there lies a fricking brilliant man who I find terribly interesting and inspiring.

Inspired by Fountainhead, driven by something that is a lot like the open source “itch” we all know in the programming world, Cuban has gone from successful enterprise to successful enterprise by fixing things that piss him off.

It takes a tremendous amount of courage to reject the norm, and even more to recreate it. Don’t like what you’re doing today? Change it. Dante and Randall would have agreed. If you want to blame somebody for your situation, blame yourself.

Mark Cuban never let his situation control him. He wasn’t the guy at the kwik-e-mart saying, “…but I’m not even supposed to be here today!!”

WWMCD? He would have started up his own kwik-e-mart that put the other one out of business. Or, he would have moved on to something completely tangential that sparked his interest. He would have succeeded there, too.

Why? He’s a rare combination of grit, determination, preparation, research and luck. Obviously nowdays he is a bit blessed with a large bankroll, but before it took some cajones to step up to the plate and pitch the ideas he had. It took a lot of faith and determination in himself to make them succeed. That is what separates him from the rest of the rich slugs. He has great ideas, sure, but he also follows through on them.

And he eats his own words gracefully, too. Ask the people at Dairy Queen.

In a time when we don’t have leaders or good examples of Americans, I think Mark Cuban serves as a good icon. He’s worlds better than crooked senators or anybody in the White House simply because he’s a multi-billionaire and he still takes the time to keep up with blogs, speak his mind freely and uses his money for good causes. He is also not stopping his pursuit of what’s right, or at least what’s cool.

You don’t see Mark Cuban crying for the people he might hurt or upset because what he is saying may not necessarily be something they agree with. He isn’t constantly weighing media reaction to the truth before he speaks it. That makes him genuine. It makes him the most human billionaire I’ve seen or read about. It makes him someone I can believe and relate to regardless of how much money he has or what he’s done. He’s a guy with ideas and feelings, not some larger-than-life self-absorbed asshole. Well, a normal guy with a $40M jet he bought online.

He’s a guy who:

  • Created a fallen soldier’s fund (Fallen Patriot Fund) to help injured or fallen soldiers’ families cope with their loss
  • Supported P2P networking in the Groklaw v. MGM case
  • Created an internet radio broadcast site because he thought not being able to listen to college basketball games over the internet was annoying
  • Created an HDTV network because HD programming from regular cable companies was crappy
  • Brought the Mavs back from extinction because his friends dared him to
  • Founded a company that helps people sift through the blogosphere because he recognized how important blogs are long before the rest of us
  • Started up 2929 entertainment because he thinks waiting for movies to come out on DVD is bullshit

So, thanks Mark, for bringing back the Mavs (though I’m a Suns fan, I appreciate the Mavs too), and showing that being yourself and believing in what’s right (for everyone, not just yourself) can take you as far as you’re willing to go. Keep it up, dude.

Mark Cuban kicks ass. Chuck Norris ain’t got shit on this guy.

How Much For a Life?

Standard

junior a year ago My little buddy Junior is developing some problems with his hind legs. He is no longer jumping or able to push off with his feet, so he’s vertically challenged — unable to jump on to the bed, or other simple things he used to be able to do with ease.

For most of his life, he was never as agile or good at jumping as Rio. Junior was always a little reluctant to jump to high places, but could make it when he wanted to. So now that he has complications, there might be a deeper reason why Junior wasn’t as acrobatic as bengals usually are.

This weekend, though, he showed the first signs of real difficulty with his legs. Saturday morning we noticed him walking funny, and realized that he could no longer jump or even stand on his hind legs to grab a treat (something he loved to do). Since then he has been getting a little better, but he still can’t jump or stand, and spends an abnormal amount of time sleeping in one spot.

Vet said his xrays were fine, and he is otherwise normal — proper reactions in his legs, etc. I paid for a full bloodwork including pathogen testing so we’ll see how that comes out, but I suspect it’ll be pretty much all clear. The next steps will be trying some general antibiotics, doing some blood titres to check for toxoplasma (I think I got that right) but other than that we’d start getting into some pretty expensive testing — MRIs, neurological examinations, etc.

So one thing I’ve been struggling with is how much to spend on the tests, and how much I am willing to pay before it’s too much. It’s not an easy decision.

My left brain tells me that spending thousands of dollars on a cat is pretty ridiculous. My heart tells me that Junior is basically a kid, and I signed an invisible contract when I adopted him that I’d take care of him for better or for worse.

So as easy as it would be to pretend not to care and not spend the money, I don’t think I could ever do that with a clear conscience. I mean, there may come a time when his quality of life is so poor that he’d need to be put down, but we’re not there yet and until then I’ll fight for the little guy.

But for animals and pets in general, it’s a tough question — how much is the life of a pet worth? Why isn’t it worth as much as a human life?

And I ask myself, why is it that we even have to equate the two in the first place? They seem to be disjoint on the grander scale in life. Comparing life and dollars doesn’t seem to add up.

So Junior is fine for now — I hope this is the worst of it, but I fear he will be a crawling kitty someday in the future and I will have to make some tough decisions. Until then, I will just do what I can and appreciate the time I still have with him.

Lots of things in life have a price tag, yet there’s so much you can’t sell or buy back.

Scalable PHP with APC, memcached and LVS (Part 2)

Standard

In part 1 of this post I talked about some of the challenges we encountered when trying to scale a LAMP application. It’s pretty much what you’d read on danga’s memcached site, just dumbed down.

So after some discussion, caffeine and Googling, you’ll probably end up knowing you’ll need:

  • Memcached!
  • An internal cache to speed up and optimize PHP across requests.
  • To continue to find ways to slim down your application.
  • Get more caffeine.

I had originally intended this post to be a summary of test results, but I am beginning to realize that what you get out of apache bench or httperf isn’t really as important as how much thought you put into your application. If you think about it, all of these perf tests are just trying to quantify something qualitative, and the tests themselves are nowhere near as important as how you get there.

So instead of showing a lot of Excel or Omnigraffle graphs that won’t help you very much, I’d much rather spend this time talking about the process. That way, you might be able to learn from our mistakes, and not make them yourself.

Together with the Mozilla infra team we worked together to put a lot of thought into this application, and that is what really made the biggest impact. In the end, the big win is just snappy pages in production — and we’ve achieved that. And since I’m a massive tool, I’ll draw a comparison between scalability and basketball.

For one, it takes teamwork and unselfishness to succeed. You need the sysadmins to be involved with the application developers from an early point, because they always ask the right questions — and often times the obvious ones developers miss. You need good coaches who know the game, and can direct on both sides of the coin. And after all is said and done on the performance side of things, you need your fans — the community — to gauge your overall success.

You hope along the way that when the game’s over, the score is in your team’s favor and the fans are cheering.

So when you’re planning your app, the best thing you can do is minimize your code by not including massive libraries or classes. Not to knock PEAR or overgeneralize things, but anytime you include a PEAR class, you have to be very careful. PEAR classes are often times bloated and written to “just make it work”. They work well for your blog, or some weekend project, but if you need some serious performance, including a PEAR class is typically a bad decision.

Includes in PHP are a bit like interest rates — it may seem like a small sacrifice to just include something, but over time and over a lot of requests, it can amount to a huge loss. Imagine if you had a 1% fee every time you hit the ATM. Seems like a minor sacrifice, it’s just 1%, but everybody knows that you’d lose a lot over time. So why would you give up 1% over millions of PHP transactions? You should follow some simple rules when dealing with PHP includes in your application:

  • Make your includes modular. You should allow yourself the ability to mix-and-match includes or class definitions. Some may have dependencies, that’s fine, but you shouldn’t limit yourself by making everything dependent on your DBI, for example. You should think about what you’d do if you had a page that didn’t pull from the DB, and how your app would serve it up.
  • Use only what you need. It’s easy to throw everything into one init script, but you should only include what your page actually needs to compile. It’s like importing java.util.* instead of just java.util.List. Doesn’t make sense.
  • Make the most use of what PHP has to offer built-in, and when that fails, write your own wrappers if PECL doesn’t already have a solution. If you’re adventurous and have C experience, you could write your own PHP extension to avoid run-time compilation of common functions. We didn’t necessarily need to do this, but you might consider it if you have a specific need that isn’t addressed with any available PECL extension.
  • Ask yourself if you really need DB layer abstraction. DBI’s are great, but hey are also huge. PEAR::DB is massive, and if your app isn’t going to be ported to other RDBMS’s, then you should really consider using your own wrapper for the mysql/mysqli functions built-in to PHP. In my experience, people hardly ever switch their DB layer over, and even if they did, if you write a clear and concise DB class, it is easy to switch out anyway. Abstraction here isn’t worth the overhead.
  • Ask yourself if you really need a template language with more rules and more syntax to mess up. PHP itself is a scripting language made to write web pages — so how much sense does Smarty make? Having been down the Smarty path, I’ve given it a shot, and I don’t think it’s worth it to replicate everything PHP does. If you’re doing it for design purposes, PHP syntax is already pretty simple, and most WYSIWYG editors have built-in escaping/tokenization for PHP’s syntax. If you’re using Smarty for separation of your view component, you can do the same thing in just PHP using app logic. And if you’re doing it so you can cache or precompile output, you’re just duplicating what memcached and APC would already offer you. If we could do it again, Smarty would not be worth the performance loss. So be wary of templating languages in such a high-level language. It’s usually a lose-lose.

At the app level, before you even get into server configuration or caching, you need to avoid violating the rules above. In our journey with addons.mozilla.org (AMO) we made some interesting group decisions a year ago that we regretted later:

PEAR::DB was unnecessarily large, and Smarty is just not worth it — it confuses the issue and redoes things PHP is already good at using arbitrarily complicated syntax. Any quick run through with something like the Zend Profiler or APD will tell you how much of a dog these things can be. If you haven’t already, I highly recommend profiling your app to see where you’re losing performance — I bet it’s mostly in includes.

For caching, we looked at:

  • Page/output caching
  • Internal caching / optimization
    • phpa (meh, and turning into a proprietary solution — double meh)
    • APC 3.0.10 (w00t)
    • A handful of other outdated and lesser internal PHP caches

For external caching, the clear choice was memcached. Used and designed for LiveJournal.com, it is a pretty standard way to provide key-based caching of any serialized data. It has APIs for almost every language used in web development, so it was an easy choice. It gave the other caching methods an ass whooping.

Based on user comments in my previous post, we punted phpa and went for APC 3.0.x and we liked the results. Initially, using the default settings in APC.ini, we faced some performance losses. After some tweaking, though, APC showed about a 40% increase over the antiquated phpa. Just make sure the read the INSTALL doc. 🙂

AMO currently runs on a handful of memcached instances, feeding multiple LVS nodes configured to use APC 3.0.10. We can now easily handle release traffic and during peak hours the web app doesn’t even break a sweat. The database bottleneck is history.

So we are happy with the results, but they were achieved using methods that are still less than ideal. There are so many more things we can do to improve performance:

  • Remove SSL for non-sensitive pages
  • Remove PEAR::DB and Smarty so when pages are compiled and set in the cache it is less expensive
  • Move away from page-level caching and get into object-level caching to replace DB calls with queries against memached.
  • Improve the memache implementation in the app to be truly decentralized with fallback. Currently it does not map a set key with a particular server. We still need to add a key->server hash so the app knows which server to try first per key. The trick there then becomes failover combined with the hash — so the app could learn which server to hit if the first choice wasn’t available and remember that choice. That is an interesting challenge in a stateless environment.
  • Make certain high-load pages purely static and avoid PHP altogether.
  • Additional tweaks and Apache config changes to improve performance.

Overall, I have to say it was a great ride and a good learning experience playing with these tools. Working with everyone on this was an exercise in open source development, and it showed us that with the right open source tools you can make some pretty decent enterprise-level web apps performance-wise. I hope that in reading this, you pick up a few things you can use in your next project. If you have any comments or suggestions I’d like to hear them.

Don’t just learn as much as you can from what others have tried — write and talk about it too.

AMO v2

Standard

The public rewrite of AMO was released today.

Fixed in this release:

  • Stuff
  • More stuff
  • Scalability
  • Other stuff

No, but seriously, you might find that your bookmarks are a little off, or _____. If so, find us on irc in #umo@irc.mozilla.org and let us know — we aren’t at the “file a bug if it’s broke” stage yet.

Thanks to Wil Clouser, who worked on a large portion of the updates. Thanks to everyone who pointed out bugs and helped us fix stuff, and thanks most importantly to the reviewers who help us keep this ship afloat.

Speaking of which, please take some time to review our draft policy that we have been working on!

Better late than never, and better late than totally crappy.

Scalable PHP with phpa, memcached and LVS (Part 1)

Standard

One of the major pitfalls of the PHP LAMP stack is that run-time compilation does not scale very well. An experienced developer would lament over PHP bloat and the other downfalls that come with the ease and convenience of using a scripting language as the foundation for your application.

Over the last month or so, I have been working on creating a better version of addons.mozilla.org (AMO). It is a task that I enjoy, mostly because it isn’t everyday you get the great opportunity to work on a project that touches so many people and means so much.

My journey is similar to the journey taken by Drupal, PHPBB or MediaWiki developers. Given a great PHP product, you not only have to fulfill usability requirements, you also have to make the thing work for an insanely large user base. In my journey I have found an appreciation for the delicate balance between usability/features/design and the pain of having to maintain them. It’s the line between being a designer or marketing droid and being a core developer and a sysadmin.

I’ve learned that, in any successful project, you need to have someone who understands both sides. If you don’t, you run the risk of having one of two things:

  • A great looking website that meets your users’ needs but doesn’t run and is not secure
  • A fast and secure web app that nobody uses

A successful project usually lands somewhere inbetween. It’s a compromise between the best performance and the usability and flexibility most people demand — albeit users or administrators.

So when you’re faced with circumventing PHP bloat, you go through predictable steps:

  • Try to run your site as-is, but then realize it fails miserably because of the database bottleneck
  • Duplicate your database, splitting it between read/write and read-only — add a slave or multiple slaves via RRDNS to serve the bits you need
  • When this fails, you start looking for output caching mechanisms that give your database a break
  • You then realize that full output caching is unrealistic because your cache hits are not high enough
  • You find yourself stuck, looking for answers — and you find ways to deliver features with the hardware you have, without making any large-scale app-level changes

When you’re at this point, there are two solutions:

  • Optimize PHP somehow to reduce redundant compilation between requests
  • Use some sort of granular caching to give your DB a break

In the next part of this post, I’ll talk about implementation of phpa/memcached across an LVS cluster and what a difference it made compared to altnernative solutions. I will post graphs explaining perf gains for AMO, and explain why we used phpa and memcached as the final solution.

Building scalable applications is challenging, but you’re not alone.

OSU Athletes Rule (in intramurals)

Standard

After stewing on this for a while, I realized my “beef” ranks down there with steroids in baseball, underwater basketweaving and having people swear an oath when they testify in front of the senate judiciary committee (sarcasm). I know — these aren’t the best wars to fight — but we should at least be aware of issues like these, because they are symptoms of a slightly more serious affliction.

When I question why varsity players are allowed to play in intramural games, it’s not a matter of me being upset about losing to a team full of varsity athletes (which I am, but that’s beside the point). I am more concerned with issues of accountibility, duty and fairness. These are issues that plague professional sports, our government, the media and our society in general — we have a problem with integrity. We don’t respect the rules, the law, or our duties. We’re teenagers away from home, and we’re flirting with danger.

When a varsity athlete is chosen to play for a collegiate team, they are binded in an agreement with the university — the donors, the fans, the coaches. They enter an agreement with all of us at OSU that covers things like drugs, skydiving, or riding motorcycles without a helmet. Altogether the idea is simple: don’t put yourself in a position to get hurt because you’re supposed to be playing for OSU, and OSU is relying on you.

It’s important for athletes to accept responsibility for this general rule. It is their duty to uphold their part of the bargain. It means being mature, responsible and held accountable for their safety. It means staying true to the mission a true NCAA athlete should have — to be the best student athletes they can be and represent their university.

I think it is obvious that playing in intramural sports is an unnecessary risk, and it crosses the line of accountibility and duty for any student athlete — particularly those on scholarships. They made a deal with the university to do a job, including taking care of their body and preparing themselves properly for upcoming seasons.

At OSU, intramural rules dictate that:

Varsity Athletes in Related Sports: Varsity athletes may participate in other intramural activities outside of their varsity sport. However, they may not participate in their varsity sport or its related activities at the intramural level. (i.e. football/flag football; baseball/softball; swimming/water polo).

I just don’t buy it. You have to take into account that NCAA athletes train at a much higher level than the weekend athlete, and their physical condition in terms of speed, quickness, endurance and strength are all much better than the average OSU student.

Plus, what happens when any varsity athlete breaks a leg or strains a muscle during an intramural game? I think they should get thrown off the team and lose their scholarship. This isn’t intermediate 7th grade sports, kids. These athletes get compensated tens of thousands for tuition, books, room and board, the works — they are compensated for the work they put into being great athletes.

I’m not sure what happens with their medical bills in case of in injury, either. If they aren’t already I think they should be held accountable for these costs, too.

On the other side of the coin, we have paying students, normal kids that don’t train at the level NCAA athletes train — mostly because they are busy working jobs to pay for tuition that can support an environment where student athletes get great scholarships. We have staff or faculty who work hard to contribute to the OSU community day in and day out and don’t have three hours a day to spend in the gym.

So what do we do to thank them?

We let NCAA athletes play in their intramural leagues and dominate them. We allow these same athletes the right to decide to put themselves in danger and jeopardize their ability to represent OSU. We pretty much let them do whatever they want.

Haven’t we had enough? At what point will OSU athletes start being held accountable to their obligations? Are we to the point where our OSU athletes are such a joke they can waste time playing in intramural leagues with people who are nowhere near as fast, big or strong as them?

To all the varsity athletes who played in intramurals this year — go train seriously, stop picking on the little people, and stand up and accept responsibility for your role in the OSU community. Go be the best athlete you can be and leave the intramural leagues alone.

Your place is in Gil or Reser, proudly wearing an OSU uniform — definitely not in a sweaty used jersey in Dixon playing against scrubs like me.

In most highschools, varsity players can’t play in intramural leagues. Sadly, OSU has a lot to learn from high schools.