Pagerduty’s Fantastic Zookeeper Bug

zookeeperOk, I don’t particularly like calling a bug fantastic, in this case, it is more of a fantastic troubleshooting of a bug. What I found interesting was the layers that were unpeeled one by one to reach the probable region of the root cause. (Yeah, the root cause is probably so esoteric and confined to a specific combination of version, that it is unlikely to be looked at by anybody).

Here is Pagerduty’s summary of the bug.

After more than a month of tireless research and testing, we have finally got to the bottom of our ZooKeeper mystery.

Corruption during AES encryption in Xen v4.1 or v3.4 paravirtual guests running a Linux 3.0+ kernel, combined with the lack of TCP checksum validation in IPSec Transport mode, which leads to the admission of corrupted TCP data on a ZooKeeper node, resulting in an unhandled exception from which ZooKeeper is unable to recover.

Jeez. Talk about a needle in a haystack… Even after all this, we are still unsure where precisely the bug lies. Despite that fact, we’re still pretty satisfied with the outcome of the investigation. Now all we need to do is work around it.

The Story Is More Important Than the Tool

Pixar Toys

I have a confession to make. Hollywood has always fascinated me. Not because of the larger-than-life stories they come up with. But because of the enormous machinery that churns out a movie. To the utter frustration of my family, I always stay back at the end of a movie, looking at all the credits which flash by – to see the rest of the iceberg under the tip. The thousands of people who made this movie happen, out of which only a fraction gets the world wide adulation, but all of them were needed to make it happen.

The actors are important, yes, but I am generally more impressed with the production values of a movie. My handful of attempts to even cut a 2 minute home video, made me aware of how difficult editing really is.And how editing, just one of the functions in making a movie, can make or break a movie – make a difference about whether it kept you glued to the story, or made your attention waver. And every body knows about sound effects – how the closer the sound effects are to your assumption of reality, the more seriously you take the movie. Seriously, I lose interest in a movie as soon as I hear a single “dhishum” – too many years of crappily produced Indian movies will do that to you.

But nothing is more important to the movie that the story. I am a bit aware of the role of the screenplay. I can sort of imagine the effort it takes to translate even a well written book to a detailed screenplay. But I certainly have no idea of the scale of the effort it takes.

story-board-flickr-topgold-6207448053

So when I read this article (No amount of technology will make a bad story good), I was fascinated.

According the article, Job once relates the effort it takes to make an animated movie:

In the clip Jobs tells how the typical live action film will shoot between 10 to 100-times more footage than will actually appear in the film. After shooting, the film is assembled in editing, which leaves most of what was shot on the cutting room floor. But animation is much too expensive to create a film in this way. Because animation is so expensive you have to edit the film before you actually make it, Jobs explains. This is where storyboarding comes in. The story team has sketches of each scene which follow the progression of the script. A film could have thousands of these.

“Basically we build our movie before we make it out of these story sketches, and we video them, put scratch music and scratch voices so that we can watch our movie. And invariably what you think is going to work crashes and burns when you see it in the reels.”

It is this dedication to achieve perfection in the fundamental purpose of a movie – story telling, that floors me. It makes the difference between a movie that I will remember forever and a movie, where by not paying attention to the story, you feel you have been robbed of a couple of hours of your natural life.

In Job’s words:

Pixar has a story crisis on every movie they make, says Jobs. When the story is not working, Jobs says, “we stop, we stop and we fix the story. Because John Lasseter really instilled a culture of story, story, story. Even though Pixar is the most technologically advanced studio in the world, John has a saying which has really stuck: No amount of technology will turn a bad story into a good story. That’s one of the reasons why we have been so fortunate is that we get to look at our stories before we really make them and perfect them
in reels, and then go make them.”

Image credits: Pixar toys by Thanh Nguyen, Storyboarding by Bernard Goldbach.

Apple Patents Tech to Allow Govt to Block Recording on Mobile Devices

 

A troubling development:

apple-logo-100
Apple has patented a piece of technology which would allow government and police to block transmission of information, including video and photographs, from any public gathering or venue they deem “sensitive”, and “protected from externalities.”

In other words, these powers will have control over what can and cannot be documented on wireless devices during any public event.

And while the company says the affected sites are to be mostly cinemas, theaters, concert grounds and similar locations, Apple Inc. also says “covert police or government operations may require complete ‘blackout’ conditions.”

And those who think that this is not coming for Android in the future are deluded. If Apple managed to get this technology into the field, it is only a matter of time that Android handset manufacturers are forced to incorporate this as well. If the technology exists, in today’s post 9/11 world, it is difficult to resist government pressure on such matters.

Of course, it would be interesting to see the security features for this tech, as this is very likely to be abused – by repressive governments (read, every one) as well as criminal enterprise (recording-free drug zones everybody?)

A Security Vulnerability in the Toilet

toto-washlet

Who said the field of security cannot have humour! An Android app to control the commode in Japan (you know the land of fully programmable toilets, I kid you not) has announced a vulnerability because the bluetooth pairing code is hardcoded.

The alert goes:

An attacker could simply download the “My Satis” application and use it to cause the toilet to repeatedly flush, raising the water usage and therefore utility cost to its owner. Attackers could cause the unit to unexpectedly open/close the lid, activate bidet or air-dry functions, causing discomfort or distress to user.

(via Sophos Security)

Apple-touch-icon 404 Errors in Logs

apple-logo-100

Curious about several peculiar Apple related 404 errors for images in my web server logs, I decided to find what is going on, and became knowledgeable about yet another nugget that I really didn’t want to know. (sigh)

The puzzle was these log lines frequently coming up in my web server access logs.

...
GET /apple-touch-icon-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon.png HTTP/1.1" 404
GET /apple-touch-icon-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon.png HTTP/1.1" 404
GET /apple-touch-icon-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon.png HTTP/1.1" 404
GET /apple-touch-icon-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon.png HTTP/1.1" 404
GET /apple-touch-icon-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon-114x114-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon-114x114.png HTTP/1.1" 404
GET /apple-touch-icon-precomposed.png HTTP/1.1" 404
GET /apple-touch-icon.png HTTP/1.1" 404
...

I first thought that I got something in my web site theme wrong, and grepped everywhere but could not spot this URL anywhere. When it didn’t turn up anything, I searched on the web for a bit and this is what I found out.

These queries are sent out by mostly Apple devices, and maybe some Android ones too, just to be compatible.

These URLs are the equivalent of favicons in Apple’s world. So instead of implementing the standard favicon definition using “link” tags, Apple apparently implemented their own convention.

So, if you have OCD about these matters (404s in logs) like me, and really lazy on the other hand (the Apple standards will require you to create several versions of these icons) you should create a 144x144 PNG file with your favicon and place it in the root of your website with both these names – apple-touch-icon.png and apple-touch-icon-precomposed.png. That will show them apples. Heh. Umm. I will end here.

References:

  1. Stackoverflow: Why am I getting error for apple-touch-icon-precomposed.png
  2. Ghacks: Why Webmasters Should Analyze Their 404 Error Log
  3. Mathias Bynens: Everything you always wanted to know about touch icons
  4. Apple’s guidelines on creating “web clips”

Law Enforcement Was Not Supposed to Be Easy

5548817119_032118d7b9
Photo from Flickr by Craig Duffy

In this day and age of the surveillance state, a quotation worth remembering from the legendary Orson Welles over 50 years back.

A policeman’s job is only easy in a police state.

— Charlton Heston as Mike Vargas in the movie “Touch of Evil” (1958), Orson Welles (screenwriter and director)

Curiously, a similar statement was made over a decade back, in fact a couple of years before 9/11, before the world changed, or actually before the United States’ war on terror changed the world.

We should not be building surveillance technology into standards. Law enforcement was not supposed to be easy. Where it is easy, it’s called a police state.

— Jeff Schiller, an IESG member and MIT network manager, Wired Magazine, 1999

(via Answer Girl and Steve Worona)