saladwithsteve

John McCarthy has a good sense of humor

2008-05-11T10:32:00.000-07:00

From an informal talk he gave at Stanford recently that was written up in Hacker News:

Q. Can computers know?

A. This is largely a question of definition. If a camera looked at a table, we could
say it "knows" that there are four containers of liquid on the table (which was true).

Q. Is there any definition of "know" in which computers cannot succeed?

A. Well, I suppose the biblical sense.

Q. Ha, well, what makes you think that?

A. They don't satisfy the necessary axioms (laughter)

What are you doing?

2008-04-21T20:08:00.000-07:00

reading @biz out me as a Twitter employee.

curious what delicious is saying about something?

2008-04-15T09:36:00.000-07:00

Here's a bookmarklet that checks the current URL you're visiting with del.icio.us.

My first Thrift app

2008-04-13T11:41:00.000-07:00

When you find yourself working on big systems, a useful technique is to decompose it into services. Moving from a big monolithic server to a bunch of separate services can be a big challenge but if you had foresight, many of your services were already decoupled in your system from day 1 even though you were deploying it monolithicly.

A common technique for decomposing services is using RPC. At Google, we used protocol buffers, which were briefly descibed in the Sawzall paper.

Basically, you describe your data and the interface that process the data in a language-independent format (a DDL, essentially) and use code generators to turn that DDL into set of objects in your target langauge that can create and send those structures over the wire. This makes it easy to write servers in one language and clients in another and the generated code deals with serialization.

I found that using a DDL to describe your code and services was really nice. When building a new service, you could simply reference your DDL in the design doc and have a meanginful discussion about the service without getting into the details of how it would be written until you had the semantics nailed down.

Facebook, as they were growing, decided to move to a homegrown binary RPC mechanism similar to protocol buffers called Thrift.

Let's say I wanted to write a simple service that would tell the client what time it was on the server. Here would be the DDL file describing both the data and the service plus a little extra to help out the generated code files.

# time.thrift
namespace java tserver.gen
namespace ruby TServer.Gen

typedef i64 Timestamp

service TimeServer {
  // Simply returns the current time.
  Timestamp time()
}

After running thrift --gen java --gen rb time.thrift on the file, I'd have an interface and server that I could implement in Java and a client that I could use in Ruby.

Based on the generated java code, I could write a short server in Scala:


package tserver

import tserver.gen._
import com.facebook.thrift.TException
import com.facebook.thrift.TProcessor
import com.facebook.thrift.TProcessorFactory
import com.facebook.thrift.protocol.TProtocol
import com.facebook.thrift.protocol.TProtocolFactory
import com.facebook.thrift.transport.TServerTransport
import com.facebook.thrift.transport.TServerSocket
import com.facebook.thrift.transport.TTransport
import com.facebook.thrift.transport.TTransportFactory
import com.facebook.thrift.transport.TTransportException
import com.facebook.thrift.server.TServer
import com.facebook.thrift.server.TThreadPoolServer
import com.facebook.thrift.protocol.TBinaryProtocol

/**
 * TimeServer.time returns the current time according to the server.
 */
class TimeServer extends TimeServer.Iface {
  override def time: Long = {
    val now = System.currentTimeMillis
    println("somebody just asked me what time it is: " + now)
    now
  }
}

object SimpleServer extends Application {
  try {
    val serverTransport = new TServerSocket(7911)
    val processor = new TimeServer.Processor(new TimeServer())
    val protFactory = new TBinaryProtocol.Factory(true, true)
    val server = new TThreadPoolServer(processor, serverTransport,
      protFactory)
    
    println("starting server")
    server.serve();     
  } catch { 
    case x: Exception => x.printStackTrace();
  }
}

(Geez, most of that space was taken up in my obsessive need to separate out all my imports. You can thank Google for that bit of OCD.)

The client is even shorter:


#!/usr/bin/ruby
$:.push('~/thrift/lib/rb/lib')
$:.push('../gen-rb')

require 'thrift/transport/tsocket'
require 'thrift/protocol/tbinaryprotocol'
require 'TimeServer'

transport = TBufferedTransport.new(TSocket.new("localhost", 7911))
protocol = TBinaryProtocol.new(transport)
client = TimeServer::Client.new(protocol)

transport.open()

puts "I wonder what time it is. Let's ask!"
puts client.time()

The ruby client took about 20ms to get an answer from the Scala server.

Thrift advantages:

Pipelined connections means you spend less time in connection setup/teardown and TCP likes longer-lived connections.
Asynchronous requests. Asynchronous replies would be nice too but would be trickier to use.
Binary representation is much more efficient to transmit and process than, say, XML.

Thrift drawbacks:

Integrating generated source into your build system can be tricky. Typically, you rarely have to regenerate your stubs but debugging generated code can be a huge pain.
It's Java server should move away from ServerSocket to NIO for increased throughput. That's probably not more than a week's work as long as the existing code isn't too tightly coupled.
Currently it doesn't build cleanly on the Mac. I did some work and got it working but I don't think it's used extensively on the Mac so if that's your primary platform, you should be prepared to send them patches from time to time.

If you're looking to move towards decoupled services, Thrift is worth a hard look.

Here's a tarball with my time server. It contains all the generated code as well as libthrift.jar and a Makefile to run the example server.

GVN and gold

2008-04-06T18:25:00.000-07:00

Two things popped up on my radar recently:

gvn, Google's wrappers around Subversion to help them work in their code-review heavy workflow. Even if you're not into code reviews, tkdiff integration is a nice improvement over colordiff or FileMerge.

gold, a new ELF linker built with giant binaries in mind. When you're building 900MB+ static binaries routinely, linking speed matters. gold claims to be at least 5x faster currently. Even if you have a massive distcc cluster, linking is still serial. One of gold's future design goals is to be concurrent and that would be pretty awesome. Imagine how fast I could link with a concurrent linker on my 8-core Mac Pro! Not that using an ELF linker under Leopard helps much since OS X uses Mach-O binaries but hey, there's always cross-compiling.

BTW, Ian Lance Taylor, the author of gold, has an excellent series of blog articles on linkers.

It's not just you

2008-03-14T09:45:00.000-07:00

Since it's back now, check out al3x's neat new site.

White People think Indian Food is Hot.

2008-03-01T21:20:00.000-08:00

I just finished reading a secular biography of the Buddha recently and Stacy and I were chatting about the Buddha's magic Iddhi powers which he acquired as a highly skilled yogin. She said her first impressions of yoga came from watching Dhalsim's stretchy limbs in Street Fighter II. As a kid, she figured there must be some small kernel of truth to that. Equally stupid is that, as a kid, that's exactly where I got my first impressions of Indian food from. The manual says that Dhalsim got his "Yoga Flame" from the hot curry he ate. I couldn't imagine a food so hot that you felt like you were breathing fire. Wasn't that a bad meal? Wouldn't you ask for your money back? How could a place like that stay in business? These types of comics didn't help, either:

This clichéd comedic target was common in the magazines that adults read when I was a kid, the particular one I've always remembered had a guy running to the bathroom with flames coming out of his pants in a restaurant with a clearly Indian name. At 10, I recognized the dangers that the comic bravely warned me of. I knew that often political comics took aim at targets that the media didn't dare attack and figured this must be the same.

The first time somebody actually offered me Indian food, I was very nervous. It had a dangerously bright orange color but smelled and tasted great. Thankfully fire did not come out of my ass and set my pants on fire.

JavaScript undefined vs. null

2008-02-17T09:58:00.000-08:00

I was reading a modern, popular book on JavaScript last night and was disappointed by the handling of null. The author started out doing a lot of checking like:


if (foo == null) {
  alert('foo is not set.');
}

Then told the reader that they could just remove the == null because javascript knows you mean "== null"

What?! This isn't why you don't check for equality with null. It's because foo == null doesn't even remotely do what most people think it does in this context.

It's a commonly held belief that uninitialized properties in JavaScript are set to null as default values. People believe this mostly for 2 reasons: 1) foo == null returns true if foo is undefined and 2) authors don't teach JavaScript properly.

A property, when it has no definition, is undefined. Put that way, it's pretty obvious.

null is an object. It's type is null. undefined is not an object, it's type is undefined. That part is less obvious.

The real trouble is that == does type coercion. === checks for both type and value and is the most intuitive form of equality in JavaScript, in my opinion.

I fired up a Jash console to hopefully clear things up for you.

>> window.hello
null
>> window.hello.something
window.hello has no properties
>> window.hello == null
true
>> window.hello === null
false
>> window.hello === undefined
true
>> if (window.hello) { alert('truthy'); } else { alert('falsy'); } // will print falsy.
null
>> window.hello == undefined
true
>> null == undefined
true // there's the rub, sir.
>> null
null
>> undefined
null
>> typeof null
object
>> typeof undefined
undefined

So people write

if (foo == null) {
   foo = "Joe";
}

When what they really mean is

if (!foo) {
   foo = "Joe";
}

If you find yourself with a lot of null checks in your JavaScript, set aside some time and watch Douglas Crockford's "The JavaScript Programming Language" talk on Yahoo Video. It's part 1 of a 3-part series of excellent and enlightening talks.

Ubuntu on the Mac Pro

2008-02-10T20:55:00.000-08:00

Last year I bought a Mac Pro to replace my aging Linux PC. It was a pretty significant upgrade, from a HyperThreaded 2.5Ghz Pentium 4 to a dual-processor, quad-core 3Ghz Xeon with 8G of RAM. Leopard is nice but I wanted to try and get Ubuntu running on it.

Here's the setup, I have 4 drives:

750G - Leopard
200G - Windows
750G - Backups
750G - To be Ubuntu

and wanted to install Ubuntu on the 4th drive.

Here are the basic steps I followed:

Install refit
Install from LiveCD (7.10)
Select manual Manual Partition and from the advanced options, choose hd3,0. I setup my disk with one ext3 partition and 1 swap partition.
After install, boot the linux disk from the refit menu
Grub will get to stage 1.5, you will see the menu but be unable to start the system, it will complain about unknown partition type
hit 'c'

Here's a transcript of my tinkering with grub to find the right drive with my initrd and kernel on it so I could boot my fancy Apple-branded Linux machine.

grub> root (hd<TAB>

showed a bunch of drives with various partitions but most importantly showed that hd0,0 had the Linux partition I was looking for. Even though with the ubuntu installer, we told it to use hd3, here we use hd0. the boot manager has shuffled the order of the drives. Telling the ubuntu installer to use hd0,0 will only result in an unbootable disk.

grub> root (hd0,0)
grub> kernel /boot/vmlinuz<TAB>

will show you some vmlinuz files. typically just the one ubuntu installed. pick it.

grub> kernel /boot/vmlinuz-2.6.22-14-generic root=/dev/sdd1 ro
grub> initrd /boot/initrd.img-2.6.22-14-generic
grub> boot

Now your system will boot. Edit your /boot/grub/menu.lst to point the root (hd0,0)

Now reboot and you will be in Ubuntu. Your wifi adapter won't work. Follow these steps to enable ndiswrapper to use the windows drivers.

I used a usb thumb drive to move the windows drivers zip file between my laptop and desktop.

Once you have networking up, don't forget to enable ssh so you can login in case you screw up your X config.

Edit /etc/apt/sources.list to include what you might want.

I have an ATI Radeon X1950 Pro video card. To get it working, I simply installed Envy which downloaded the right drivers and installed them for me.

But it didn't span monitors, they were just clones so I ran aticonfig (which Envy installed for me)

aticonfig --initial=dual-head --overlay-on=1

And we were good to go.

Banshee imported all my music from my iPod after converting it's database from an iTunes format it didn't recognize.

Using hibernate left the Mac's wifi adapter in a weird state. Rebooting it didn't help but booting into OS X and back into Linux corrected the wifi card.

The fan speed still isn't quite right, various fans will become louder and softer at odd intervals. I will try monkeying with lm-sensors later.

I have nice smooth fonts by running Emacs 23 from Alexandre Vassalotti's repo and the Inconsolata font for programming.

Overall, it's a great Linux machine and maybe one day it won't be so much trouble to get running. Hopefully my instructions will help you get it up and running.

Lift Tutorial

2007-11-19T22:32:00.000-08:00

I've been working on lift quite a bit lately, fitting it in with some other projects and my obsessive need for football. (Both NFL and Madden)

If you're interested in trying it out, give the lift tutorial I wrote a try. Please drop us a note if you have any problems.

Tomato Wifi!

2007-10-21T16:57:00.000-07:00

While doing that podcast with Daniel, I noticed I was seeing 30% or so packet loss on my wifi network. After rebooting some equipment and keeping an eye on things, I decided to scrap my faulty Airport Express base stations and replace them with Linksys WRT54GLs running a custom opensource firmware called Tomato.

While the instructions are good, they assume you already have the router up and running. The first thing I realized was to throw the windows install cd away and just plug my mac into the switch on the back. The router will be listening on 192.168.1.1:80 with a username and password of admin/admin. The instructions will carry you from that point.

I have a 3 story house so I bought two routers and bridged them together with WDS. My only complaint is that Tomato's WDS is difficult to setup. The Airport Express admin utility did a wonderful job of easily bridging the two aiport express base stations together.

Except for one of the routers dying after only a few days, which Amazon was nice enough to replace for free with next-day shipping, things have been great on Tomato. 4 bars everywhere I go in the house.

[Hat tip to Nelson for the recommendation.]

My interview with Daniel Brusilovsky

2007-10-06T21:33:00.001-07:00

Daniel runs the Apple Universe podcast and interviewed me today. We had a fun chat, check it out!

What's next?

2007-09-28T23:21:00.000-07:00

Werner Vogels on Michael Stonebraker's 50X challenge:

I like this challenge, given that 50X is likely to be able to make impact, where 2-4X in general can be easily compensated for by the next generation hardware. But something bugs me about the challenge and also about some of the demonstrations in the papers; 50X is still focused on scaling-up, just as many of the current database systems do, instead of scaling out, which is what the world really needs. The evidence in the paper is indeed about single box performance. This continuing N=1 thinking will never yield systems that can break through the current scalability limitations of enterprise software, regardless whether it runs 50 times faster or not.

I couldn't agree more. It's exciting to see so many people arguing for new kinds of databases, especially distributed databases. Why do I hate on relational databases? It's not the relational part, it's the part where they fail at 3am and wake me up. Joins are nice but I need my beauty sleep.

43 Folders makes the switch...

2007-09-21T22:39:00.001-07:00

that I expect a number of others to follow. They just switched from a blogging system to a content management system. Lots of sites that have daily articles aren't blogs but until recently it was easier to have separate blogs and auxiliary sites than to setup a full-blown CMS. These sites are easy to distinguish: they'll have forums, wikis, or other miscellany that aren't simply chronologically ordered notes of interest. For instance, 43 Folders has a Forum, Job Board, and sub-sites like gtd and Inbox Zero.

I always assumed CMS's would get easier to setup and use and some of the more popular sites would switch to them. I fully expect this to become a growing trend and am glad to see traditional blogging tools start to get some competition for serious users.

Update: Earlier, I mentioned in this post that Matt pointed to a site that switched from WordPress to a CMS. I misread it. It was the other way around. Thanks, Jeremy. I hate it when you're right.

Putting my money where my mouth is

2007-09-04T19:55:00.000-07:00

I've been going on and on about Scala and lift these past few months to pretty much anybody who would listen. Lift, an alpha web framework for the object-functional language Scala, takes advantage of Scala's numerous features to provide a surprisingly powerful web framework in less than 10k lines of code. Take a look at the lift site to get an idea of what it can do. (Hint: David wrote a wiki in 45 lines of code)

I can see it doing for Scala was Rails did for Ruby.

So when David asked me if I wanted to be a committer, I jumped at the opportunity. I have a couple of patches going in my local SVN working copy mostly related to proper serving of MIME types and caching that I hope to get in soon.

Hacking the 46.

2007-08-31T12:57:00.000-07:00

I love this story where Matt Hasselbeck, based on what he's seen and heard on his home TiVo, "hacks" the Panther's defense.

The future is messaging

2007-08-19T21:38:00.000-07:00

Sam Ruby and Tim Bray have each picked messaging systems as a trend to watch in the future. In particular, they are naming Jabber/XMPP as something to watch. While I think XMPP is a great way to connect people to systems or other people, I think there are better ways to connect systems to systems.

Enter AMQP, a protocol designed by the financial industry to replace their existing proprietary message queues with something built on open standards. The working group is made up of seasoned messaging profesionals who are used to having billions of dollars worth of transactions moving through their systems without mysterious delays or drops.

There are two things that stand out in my mind about AMQP.

Payload Fidelity: Any stream of bytes you place in the payload will come out the other end the same way. At the end of this post, I show an example of this. If you can do this with Jabber, please say so. I certainly haven't seen a way to do it.
Queues instead of Buddy Lists: Instead of having human-centric buddy lists like with Jabber, you have accounts who have access to a namespace called a 'virtual host'. Inside of the 'virtual host' are named 'queue's and 'exchange's. Shove messages in one end and read them from the other.

RabbitMQ is an AMQP implementation built in Erlang and using Mnesia. Building clustered, high performance messaging systems in Erlang really plays to that language's strengths, I feel. Erlang seems great at shoveling bits from one network interface, doing some pattern matching on them, changing some bits, and blasting them out another interface.

In my experience, you can have a clustered rabbitmq setup running at home in under 20 minutes. It's all in the admin guide.

Plus that question about transporting binary data over your messaging layer? Here's a hack I wrote serializing Scala objects into RabbitMQ and operating on them on the other side. It will work with your default RabbitMQ installation:

Server
Client
Multiplication class definition

Anecdotally, I've heard that RabbitMQ can saturate a 100Mbps ethernet connection with a single 3Ghz Pentium 4. I have a couple of quad core opteron's (2x2 core) with 16G of RAM and multiple gigabit ethernet interfaces in my basement; Seeing how much processing power is needed to saturate a gigabit ethernet interface is definitely on my ToDo list!

A real-life cold case.

2007-08-17T19:37:00.000-07:00

Back in July, I received a jury duty summons. I didn't end up serving but it took 3 long days for the lawyers to finish selecting it. It was for a 20 year old rape and murder case considered San Francisco's first real Cold Case. It looks like that jury just entered deliberation.

I'm just BlueFlashin' my Apple ][, yo.

2007-08-15T10:57:00.000-07:00

BlueFlash is a "Bluetooth/Disk Controller/Disk Image Archive Card for Apple ][" that allows you to download files for the apple ][ onto your PC and transfer them to your Apple ][ via bluetooth.

Weighted Slope One in Scala

2007-08-14T21:14:00.000-07:00

Here's a basic procedural version of Weighted Slope One in Scala. It's pretty much a direct port of the Python version. It's about 15 lines longer due to curly braces and Map syntax in Scala not having a few niceties that Python dictionaries have.

Next I'm going to convert it to take advantage of Scala's powerful functional abilities. I'm going to take a stab to see if it lends itself to being a map and reduce. If that fails, I will fall back to good 'ol fold. The goal will be for those for-comprehensions to vanish and reveal an algorithm that's more parallelizable and then to run it on multiple cpus.

noatime

2007-08-10T21:48:00.000-07:00

Linus says: "'noatime,data=writeback' will quite likely be *quite* noticeable (with different effects for different loads), but almost nobody actually runs that way."

Well, if you need it, you really need it. Otherwise, why would you ever care? I also don't believe in this 10% number people are spouting. How much it effects you will be determined by how many files you access. I think that should be fairly obvious. I've worked on systems that would crash after just a few minutes without noatime. Yikes!

Google meets with the Five Families for Linux patent protection.

2007-08-10T21:26:00.000-07:00

All OIN members - including big names such as IBM, NEC, Novell, Philips, Red Hat, and Sony, as well as Google - agree not to use their Linux-related patents against each other, and all have free access to a collection of additional open-source-related patents purchased by the consortium as a whole.

Seems like a good idea. Somebody will eventually have to take out Barzini, though.

2007-08-09T16:43:00.000-07:00

"50 Really Good Indie Games" Unfortunately, all of them look windows only. But there's a lot of fun, innovative games in here. At least, they look fun and innovative from the screenshots.

Time is an invention of man.

2007-08-05T17:08:00.000-07:00

It's exceedingly unclear to pretty much everyone what I do with my time. This was pointed out by my cousin who just called and signed off with a sincere "Ok, have fun with.... well, whatever it is that you do."

Let's take today as an example of what I do with my time.

Stinky the cat woke me up at 6am meowing about being hungry. As I am apprently elderly, I fell asleep at 10pm last night so it wasn't a big deal.
I poured over an old wiki and imported the notes to sidenote, an "outboard brain" application for OS X.
Wrote new notes after being inspired by these existing notes.
Farted around with YUI! for a frontend to my lift app.
Started writing a Weighted Slope One algorithm in Scala. I was interrupted by lunch so I put that in OmniFocus' NextAction queue.
Went to our nephew Kyo's Red egg and ginger party followed by a trip to Target.
Played Burnout Revenge for the Xbox 360. I am currently 25% through the game, I should be done by the end of next week.
Found several sets of instructions for installing Trac on dreamhost. I made a note in OmniFocus.
Stacy programmed the universal remote we picked up at Target (more on that later)
Investigated using an in-memory lucene index instead of something like Splunk. Serialized that into sidenote.
My cousin called and signed off with the above quote.
I wrote this stupid blog post.
Will pick up work on the weighted slope one algorithm.
Stacy and I will decide what's for dinner.
Feed Stinky and Pudding and then retire for the evening.
I'll pick one of the 3 books I'm currently reading and work through it.
Fall asleep in a stupor complaining about something or other.

Most days are a little different. Replace 'red egg and ginger party' with 'exercising' or 'loafing'. Also, I spend too much time using Google Reader, Twitter, and del.icio.us. At least I broke my programming.reddit habit.

NFL in HD

2007-08-05T07:26:00.000-07:00

HD Sports Guide has the low down on NFL games in HD this year. I only saw Monday Night Football in HD last year and so I'm excited to start seeing local games. DirecTV tries to squeeze nearly $400 out of you to see other teams in HD.

For those of you 49er's fans not local, here's the 4 games you can expect to see:

8/13/2007  Denver Broncos vs. San Francisco 49ers (Pre)  
8:00 PM ESPNHD (720p)

9/10/2007  Arizona Cardinals vs. San Francisco 49ers  
10:15 PM ESPNHD (720p)

11/12/2007  San Francisco 49ers vs. Seattle Seahawks  
8:30 PM ESPNHD (720p)

12/15/2007  Cincinatti Bengals vs. San Francisco 49ers  
8:15 PM      NFL Network (1080i)

If I remember last year, NFL Network HD runs on channel 96. It might be different this year.