saladwithsteve

More Varnish: Thoughts from the Author

2008-12-08T21:02:00.000-08:00

Poul-Henning Kamp, the lead author of Varnish, was kind enough to respond to my blog post. Hi Steve, I read your comments about Varnish and thought I'd better explain myself, feel free to post this if you want. The reason why we don't multiplex threads in Varnish is that it has not become a priority for us yet, and I am not convinced that we will ever see it become a priority at the cost it bears. A common misconception is that varnish parks a thread waiting for the client to send the next request. We do not, that would be horribly stupid. Sessions are multiplexed when idle. The Varnish worker thread takes over when the request is completely read and leaves the session again when the answer is delivered to the network stack (and no further input is pending). For a cache hit, this takes less than 10 microseconds, and we are still shaving that number. The problem about multiplexing is that it is horribly expensive, both in system calls, cache pressure and complexity. If we had a system call that was called "write these bytes, but don't block longer than N seconds", multiplexing could be sensibly implemented. But with POSIX mostly being a historical society, it will cost you around 7 system calls, some surprisingly expensive, to even find out if multiplexing is necessary in the first place. In comparison, today Varnish delivers a cache-hit in 7 system calls *total* of which a few are technically only there for debugging. The next good reason is that we have no really fiddled with the sendbuffer sizes yet, but obviously, if you sendbuffer can swallow the read, the thread does not wait. And if that fails, a thread blocked in a write(2) system call is quite cheap to have around. It uses a little RAM, but it's not that much, and we can probably tune it down quite a bit. The scheduler does not bother more with the thread than it has to, and when it does, the VM hardware system is not kickstarted every time we cross the user/kernel barrier. Without getting into spectroscoping comparisons between apples and oranges, a thread just lounging around, waiting for the network stack to do its thing, is much, much cheaper than a thread which does a lot of system calls and fd-table walking, only to perform a few and small(-ish) writes every time the network stack wakes up. And the final reason why it may never become necessary to multiplex threads, is that servers are cheap. But if we get to the point where we need multiplexing, we will do it. But I like the old design principles from the X11 project: we will not do it, until we have a server that doesn't work without it. But if you are in the business of delivering ISOs to 56k modems then yes, Varnish is probably not for you. Poul-Henning

Thoughts on Varnish

2008-12-06T17:15:00.000-08:00

Varnish is getting a lot of attention these days around the internet, and with good reason, it’s a nicely written and speedy cache, and has a nice DSL for caching. It has great features like hot reloading of cache rules and ESI.

One thing that’s really surprised me, though, is that Varnish uses one thread per connection. Most network programs designed for high number of connection don’t use one thread per connection anymore as it has serious drawbacks.

With slow clients, many of your threads are spending a lot of time doing nothing but blocking in write(). In all internet consumer apps, I believe, slow clients make up the majority of your connections. But even though the threads are doing nothing, the OS still has memory and scheduling overhead in dealing with them. You find yourself with an artificially low ceiling on the amount of users you can service with a single machine.

What makes a client slow, though? Both speed and latency. Cell phones, 56k modems, and users on high speed links but not geographically close to your data center can all be classified as ‘slow’.

One design that is more appropriate for dealing with the slow client problem uses a pool of worker threads or processes behind the scene and epoll / kqueue / event ports handling slow clients and telling the pool of workers that a socket is ready with a change notification. Your cost is still correlated with growth but at a much lower rate and the number of users you can service will dramatically increase.

So why does Varnish use this older, more troublesome model? Probably because most services aren’t going to notice the bottleneck; They simply don’t have enough concurrent connections to worry about using a few extra machines. If you’re never saturated a load balancer or firewall, you’ve probably never had to seriously consider the C10k issues involved.

Also, unfortunately, the way most people write load tests is that they are only testing the All Fast Clients scenario and not a mix of fast clients and slow clients. I’m guilty of this, too.

My executive summary: Varnish is a nice piece of software, and I hope they spend the time to make it useful for larger sites as well as smaller ones.

More RPMs means faster access times. No news there.

2008-11-17T16:07:00.000-08:00

When I upgraded my home laptop from a 2-year old MacBook Pro to one of the newly released unibody models, I decided to upgrade from a 5400 RPM drive to a 7200 RPM drive. I ran some bonnie-64 benchmarks and noticed a 40% improvement in random seeks/sec and some other impressive numbers. It's helped make my weekend hacking much more pleasant.

Here are the old numbers:

and the new numbers:

Bottom line: Recommended

Amazon as Political Pulpit

2008-09-13T14:56:00.001-07:00

Check out the Amazon reviews for Spore, the new Will Wright game. 2,016 of the 2216 reviews gave it 1 star for the heavy-handed DRM. I use Amazon reviews heavily and I've never seen protest brought to the reviews at this scale before. Let's see if people take notice.

TorrentFreak points out the following:

DRM doesn’t stop people from pirating a game, on the contrary. It only hurts legitimate customers since the DRM is removed from the pirate version."

There are lots of philosophical reasons not to buy things with DRM. For me, the practical reason wins out. I'd rather deal with having to store music CDs and not play some potentially awesome games than deal with losing my data due to short-sighted DRM.

Alex's Scala Talk at C4[2]

2008-09-12T15:54:00.000-07:00

al3x gave a talk on Scala recently at C4[2] and it hits a lot of the high points as to why we're using Scala at Twitter to build back-end services. I've been programming in Scala seriously for about a year and a half now and it's positively ruined me for plain Java.

So I don't forget it.

2008-07-30T13:33:00.000-07:00

A cheesy generic tcp proxy I found cruising the webs built out of netcat, fifos, and tee:

$ mkfifo backpipe $ sudo nc -l 80 0<backpipe | tee -a inflow | nc localhost 8080| tee -a outflow 1>backpipe

This way you can also look at inflow and outflow to see what the actual contents were of the transaction.

Simulating Byzantine failure with SIGSTOP

2008-06-29T18:30:00.000-07:00

If your service relies on connecting to an internal network server and that server isn't accepting connections, your client will obviously throw an error. This happens often enough that you probably already check for this and do the right thing in your various projects. But what if the server is accepting connections but never returning any data? This failure case is rare but very deadly. Chet mentioned that you could simulate this using SIGSTOP so I decided to whip up an experiment with memcached as my victim.

stevej@t42p:~$ ps auxww |grep memcache
stevej    3451  0.0  0.0   2928  1872 pts/0    T    01:21   0:00 memcached -vv -p 11211
stevej@t42p:~$ kill -stop 3451

In another terminal:

stevej@t42p:~$ irb
irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'memcache'
=> true
irb(main):003:0> CACHE = MemCache.new "localhost:11211"
=> <MemCache: 1 servers, 1 buckets, ns: nil, ro: false>
irb(main):004:0> CACHE.get("foo")

The client library happily hung for several hours while I did other things. How can a process that's suspended not timeout incoming connections? Well, it's the kernel that services network requests and the process itself is only reading the buffers. If you want proof, look at this tcpdump output. Remember, the process has already been suspended by the time I ran tcpdump here.

stevej@t42p:~$ sudo tcpdump -i lo port 11211
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 96 bytes
18:02:40.576255 IP localhost.48124 > localhost.11211: F 2018798159:2018798159(0) ack 2012359105 win 257 <nop,nop,timestamp 15455978 14281170>
18:02:40.577037 IP localhost.11211 > localhost.48124: . ack 1 win 256 <nop,nop,timestamp 15455979 15455978>
18:03:19.037410 IP localhost.35662 > localhost.11211: S 2731273926:2731273926(0) win 32792 <mss 16396,sackOK,timestamp 15465593 0,nop,wscale 7>
18:03:19.037435 IP localhost.11211 > localhost.35662: S 2723119696:2723119696(0) ack 2731273927 win 32768 <mss 16396,sackOK,timestamp 15465593 15465593,nop,wscale 7>
18:03:19.037449 IP localhost.35662 > localhost.11211: . ack 1 win 257 <nop,nop,timestamp 15465593 15465593>
18:03:19.037768 IP localhost.35662 > localhost.11211: P 1:10(9) ack 1 win 257 <nop,nop,timestamp 15465593 15465593>
18:03:19.037776 IP localhost.11211 > localhost.35662: . ack 10 win 256 <nop,nop,timestamp 15465593 15465593>

So a connect timeout wouldn't help here, you need a recv timeout or something else. Restarting your client process won't help at all, it'll simply get stuck in the same place. In Ruby, the easiest thing to do is to use the Timeout module. Sadly, it only has second granularity but that's a lot better than hanging for several hours. You can also set use Socket#setsockopt with a recv timeout if you need finer grained timeout resolution.

stevej@t42p:~$ irb
irb(main):001:0> require 'rubygems'
=> true
irb(main):002:0> require 'memcache'
=> true
irb(main):003:0> CACHE = MemCache.new "localhost:11211"
=> <MemCache: 1 servers, 1 buckets, ns: nil, ro: false>
irb(main):004:0> require 'timeout'
=> false
irb(main):005:0> foo = Timeout::timeout(1) do
irb(main):006:1*   CACHE.get("foo")
irb(main):007:1> end
/usr/lib/ruby/1.8/timeout.rb:54:in `cache_get': execution expired (Timeout::Error)
        from /usr/lib/ruby/gems/1.8/gems/memcache-client-1.5.0/lib/memcache.rb:209:in `get'
        from (irb):6:in `irb_binding'
        from /usr/lib/ruby/1.8/timeout.rb:56:in `timeout'
        from (irb):5:in `irb_binding'
        from /usr/lib/ruby/1.8/irb/workspace.rb:52:in `irb_binding'
        from /usr/lib/ruby/1.8/irb/workspace.rb:52

Do you want to guess what happened when I sent SIGCONT to memcache? My client processes, even the ones that had been hanging for hours, immediately returned with the expected data.

The obvious thing to do is to write a new MemCache subclass decorating all the calls to get, put, get_multi, etc with safer versions. Don't naively trust that the expected data made it to the cache.

More on Twitter!

2008-05-22T12:55:00.000-07:00

Al3x wrote up a nice blog post talking about the future of twitter.

How work's going?

2008-05-21T20:44:00.000-07:00

Since I started working at Twitter last month, I put up a standard work disclaimer along the side. It always applies.

Jack posted on the company blog: I have this graph up on my screen all the time. It should be flat. This week has been rough.

So we have open job postings for something called a Systems Engineer, which is what I do at Twitter. Systems Engineering means building systems where graphs like that stay flat and where downtime means it was either planned or making sure that particular problem won't happen again (if it can be avoided: typical engineering trade-offs apply).

Our problems are really interesting, I think. Lots of users, lots of connections, lots of messages flowing through the system, lots of endpoints, and lots of details to keep straight. All of this needs to be turned into a cohesive system that's simple to reason about and to run in order for me to consider my job a success. It's a tall order but it's what I signed up to do. I've been watching Twitter for a long time (I'm user #150) so I walked into things with my eyes wide open.

If you've been reading this blog for a while, you know that I'm more interested in engineering than hacking together a site. Thinking and then doing. Measuring and then reasoning. Making guesses and then testing them. There's a natural tension between cowboying around and Analysis Paralysis and you have to learn to walk that tightrope if you want to succeed and I think at Twitter, we work pretty hard to Do the Right Thing.

I'm writing this quick post because we're looking for great people who are interested in engineering big systems and in helping to make Twitter the utility-class company we see ourselves as needing to be. If you think you either have the skills or can learn them, please send us your resume to jobs@twitter.com.

Fun DTrace script

2008-05-19T21:26:00.001-07:00


#!/usr/sbin/dtrace -s

syscall:::entry
/pid == $1/
{
 @sys[probefunc, ustack()] = count();
}

END {
    trunc(@sys, 2);
}

Tells you the 2 most often called system call/stack trace pair. Running it against firefox 3 beta while using Google Reader shows:


$ sudo ./syscalldist.d 240
dtrace: script './syscalldist.d' matched 428 probes
^C
CPU     ID                    FUNCTION:NAME
  1      2                             :END 

  munmap                                            
              libSystem.B.dylib`munmap$UNIX2003+0xa
              libSystem.B.dylib`free+0x6a
              CoreGraphics`CGEventCreateFromDataAndSource+0xbce
              CoreGraphics`CGSDecodeEventRecord+0x6a
              CoreGraphics`CGSDispatchDatagramsFromStream+0x28f
              CoreGraphics`snarfEvents+0x12a
              CoreGraphics`CGSGetNextEventRecordInternal+0x9f
              CoreGraphics`CGEventCreateNextEvent+0x2c
              HIToolbox`PullEventsFromWindowServerOnConnection(unsigned int, unsigned char)+0x58
              CoreFoundation`__CFMachPortPerform+0x75
              CoreFoundation`CFRunLoopRunSpecific+0xf51
              CoreFoundation`CFRunLoopRunInMode+0x58
              HIToolbox`RunCurrentEventLoopInMode+0x11b
              HIToolbox`ReceiveNextEventCommon+0x176
              HIToolbox`BlockUntilNextEventMatchingListInMode+0x6a
              AppKit`_DPSNextEvent+0x291
              AppKit`-[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]+0x80
              AppKit`-[NSApplication run]+0x31b
              XUL`JSD_GetValueForObject+0xad6ce
              XUL`XRE_GetFileFromPath+0x61c563
              961
  mmap                                              
              libSystem.B.dylib`mmap+0xa
              libSystem.B.dylib`large_and_huge_malloc+0xcb
              libSystem.B.dylib`szone_malloc+0x1cf
              libSystem.B.dylib`malloc_zone_malloc+0x51
              libSystem.B.dylib`malloc+0x37
              CoreGraphics`CGEventCreateFromDataAndSource+0x15e
              CoreGraphics`CGSDecodeEventRecord+0x6a
              CoreGraphics`CGSDispatchDatagramsFromStream+0x28f
              CoreGraphics`snarfEvents+0x12a
              CoreGraphics`CGSGetNextEventRecordInternal+0x9f
              CoreGraphics`CGEventCreateNextEvent+0x2c
              HIToolbox`PullEventsFromWindowServerOnConnection(unsigned int, unsigned char)+0x58
              CoreFoundation`__CFMachPortPerform+0x75
              CoreFoundation`CFRunLoopRunSpecific+0xf51
              CoreFoundation`CFRunLoopRunInMode+0x58
              HIToolbox`RunCurrentEventLoopInMode+0x11b
              HIToolbox`ReceiveNextEventCommon+0x176
              HIToolbox`BlockUntilNextEventMatchingListInMode+0x6a
              AppKit`_DPSNextEvent+0x291
              AppKit`-[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:]+0x80
              997

Thrilling, I know!

DTrace for Java 6 on Leopard

2008-05-18T20:33:00.000-07:00

When Java 6 for Leopard was released a few weeks ago, one thing that nobody seemed to notice was that Java now had DTrace probes on par with Java on Solaris.

What you expect is there:

With one exception: jstack doesn't appear to work. ustack works fine.


$ sudo dtrace -x jstackstrsize=2048 -n 'syscall::read:entry /execname == "java"/ { jstack(); }' 

dtrace: description 'syscall::read:entry ' matched 1 probe
CPU     ID                    FUNCTION:NAME
  3  17600                       read:entry 

  2  17600                       read:entry 

  3  17600                       read:entry 

  3  17600                       read:entry 

  2  17600                       read:entry 

  2  17600                       read:entry 

  2  17600                       read:entry 

  2  17600                       read:entry

There should be java stack traces under each read:entry line. (This is true even with -XX:+ExtendedDTraceProbes enabled)

I used robey's scarling for my guinea pig and had a lot of fun poking around at it with dtrace.

John McCarthy has a good sense of humor

2008-05-11T10:32:00.000-07:00

From an informal talk he gave at Stanford recently that was written up in Hacker News:

Q. Can computers know?

A. This is largely a question of definition. If a camera looked at a table, we could
say it "knows" that there are four containers of liquid on the table (which was true).

Q. Is there any definition of "know" in which computers cannot succeed?

A. Well, I suppose the biblical sense.

Q. Ha, well, what makes you think that?

A. They don't satisfy the necessary axioms (laughter)

What are you doing?

2008-04-21T20:08:00.000-07:00

reading @biz out me as a Twitter employee.

curious what delicious is saying about something?

2008-04-15T09:36:00.000-07:00

Here's a bookmarklet that checks the current URL you're visiting with del.icio.us.

My first Thrift app

2008-04-13T11:41:00.000-07:00

When you find yourself working on big systems, a useful technique is to decompose it into services. Moving from a big monolithic server to a bunch of separate services can be a big challenge but if you had foresight, many of your services were already decoupled in your system from day 1 even though you were deploying it monolithicly.

A common technique for decomposing services is using RPC. At Google, we used protocol buffers, which were briefly descibed in the Sawzall paper.

Basically, you describe your data and the interface that process the data in a language-independent format (a DDL, essentially) and use code generators to turn that DDL into set of objects in your target langauge that can create and send those structures over the wire. This makes it easy to write servers in one language and clients in another and the generated code deals with serialization.

I found that using a DDL to describe your code and services was really nice. When building a new service, you could simply reference your DDL in the design doc and have a meanginful discussion about the service without getting into the details of how it would be written until you had the semantics nailed down.

Facebook, as they were growing, decided to move to a homegrown binary RPC mechanism similar to protocol buffers called Thrift.

Let's say I wanted to write a simple service that would tell the client what time it was on the server. Here would be the DDL file describing both the data and the service plus a little extra to help out the generated code files.

# time.thrift
namespace java tserver.gen
namespace ruby TServer.Gen

typedef i64 Timestamp

service TimeServer {
  // Simply returns the current time.
  Timestamp time()
}

After running thrift --gen java --gen rb time.thrift on the file, I'd have an interface and server that I could implement in Java and a client that I could use in Ruby.

Based on the generated java code, I could write a short server in Scala:


package tserver

import tserver.gen._
import com.facebook.thrift.TException
import com.facebook.thrift.TProcessor
import com.facebook.thrift.TProcessorFactory
import com.facebook.thrift.protocol.TProtocol
import com.facebook.thrift.protocol.TProtocolFactory
import com.facebook.thrift.transport.TServerTransport
import com.facebook.thrift.transport.TServerSocket
import com.facebook.thrift.transport.TTransport
import com.facebook.thrift.transport.TTransportFactory
import com.facebook.thrift.transport.TTransportException
import com.facebook.thrift.server.TServer
import com.facebook.thrift.server.TThreadPoolServer
import com.facebook.thrift.protocol.TBinaryProtocol

/**
 * TimeServer.time returns the current time according to the server.
 */
class TimeServer extends TimeServer.Iface {
  override def time: Long = {
    val now = System.currentTimeMillis
    println("somebody just asked me what time it is: " + now)
    now
  }
}

object SimpleServer extends Application {
  try {
    val serverTransport = new TServerSocket(7911)
    val processor = new TimeServer.Processor(new TimeServer())
    val protFactory = new TBinaryProtocol.Factory(true, true)
    val server = new TThreadPoolServer(processor, serverTransport,
      protFactory)
    
    println("starting server")
    server.serve();     
  } catch { 
    case x: Exception => x.printStackTrace();
  }
}

(Geez, most of that space was taken up in my obsessive need to separate out all my imports. You can thank Google for that bit of OCD.)

The client is even shorter:


#!/usr/bin/ruby
$:.push('~/thrift/lib/rb/lib')
$:.push('../gen-rb')

require 'thrift/transport/tsocket'
require 'thrift/protocol/tbinaryprotocol'
require 'TimeServer'

transport = TBufferedTransport.new(TSocket.new("localhost", 7911))
protocol = TBinaryProtocol.new(transport)
client = TimeServer::Client.new(protocol)

transport.open()

puts "I wonder what time it is. Let's ask!"
puts client.time()

The ruby client took about 20ms to get an answer from the Scala server.

Thrift advantages:

Pipelined connections means you spend less time in connection setup/teardown and TCP likes longer-lived connections.
Asynchronous requests. Asynchronous replies would be nice too but would be trickier to use.
Binary representation is much more efficient to transmit and process than, say, XML.

Thrift drawbacks:

Integrating generated source into your build system can be tricky. Typically, you rarely have to regenerate your stubs but debugging generated code can be a huge pain.
It's Java server should move away from ServerSocket to NIO for increased throughput. That's probably not more than a week's work as long as the existing code isn't too tightly coupled.
Currently it doesn't build cleanly on the Mac. I did some work and got it working but I don't think it's used extensively on the Mac so if that's your primary platform, you should be prepared to send them patches from time to time.

If you're looking to move towards decoupled services, Thrift is worth a hard look.

Here's a tarball with my time server. It contains all the generated code as well as libthrift.jar and a Makefile to run the example server.

GVN and gold

2008-04-06T18:25:00.000-07:00

Two things popped up on my radar recently:

gvn, Google's wrappers around Subversion to help them work in their code-review heavy workflow. Even if you're not into code reviews, tkdiff integration is a nice improvement over colordiff or FileMerge.

gold, a new ELF linker built with giant binaries in mind. When you're building 900MB+ static binaries routinely, linking speed matters. gold claims to be at least 5x faster currently. Even if you have a massive distcc cluster, linking is still serial. One of gold's future design goals is to be concurrent and that would be pretty awesome. Imagine how fast I could link with a concurrent linker on my 8-core Mac Pro! Not that using an ELF linker under Leopard helps much since OS X uses Mach-O binaries but hey, there's always cross-compiling.

BTW, Ian Lance Taylor, the author of gold, has an excellent series of blog articles on linkers.

It's not just you

2008-03-14T09:45:00.000-07:00

Since it's back now, check out al3x's neat new site.

White People think Indian Food is Hot.

2008-03-01T21:20:00.000-08:00

I just finished reading a secular biography of the Buddha recently and Stacy and I were chatting about the Buddha's magic Iddhi powers which he acquired as a highly skilled yogin. She said her first impressions of yoga came from watching Dhalsim's stretchy limbs in Street Fighter II. As a kid, she figured there must be some small kernel of truth to that. Equally stupid is that, as a kid, that's exactly where I got my first impressions of Indian food from. The manual says that Dhalsim got his "Yoga Flame" from the hot curry he ate. I couldn't imagine a food so hot that you felt like you were breathing fire. Wasn't that a bad meal? Wouldn't you ask for your money back? How could a place like that stay in business? These types of comics didn't help, either:

This clichéd comedic target was common in the magazines that adults read when I was a kid, the particular one I've always remembered had a guy running to the bathroom with flames coming out of his pants in a restaurant with a clearly Indian name. At 10, I recognized the dangers that the comic bravely warned me of. I knew that often political comics took aim at targets that the media didn't dare attack and figured this must be the same.

The first time somebody actually offered me Indian food, I was very nervous. It had a dangerously bright orange color but smelled and tasted great. Thankfully fire did not come out of my ass and set my pants on fire.

JavaScript undefined vs. null

2008-02-17T09:58:00.000-08:00

I was reading a modern, popular book on JavaScript last night and was disappointed by the handling of null. The author started out doing a lot of checking like:


if (foo == null) {
  alert('foo is not set.');
}

Then told the reader that they could just remove the == null because javascript knows you mean "== null"

What?! This isn't why you don't check for equality with null. It's because foo == null doesn't even remotely do what most people think it does in this context.

It's a commonly held belief that uninitialized properties in JavaScript are set to null as default values. People believe this mostly for 2 reasons: 1) foo == null returns true if foo is undefined and 2) authors don't teach JavaScript properly.

A property, when it has no definition, is undefined. Put that way, it's pretty obvious.

null is an object. It's type is null. undefined is not an object, it's type is undefined. That part is less obvious.

The real trouble is that == does type coercion. === checks for both type and value and is the most intuitive form of equality in JavaScript, in my opinion.

I fired up a Jash console to hopefully clear things up for you.

>> window.hello
null
>> window.hello.something
window.hello has no properties
>> window.hello == null
true
>> window.hello === null
false
>> window.hello === undefined
true
>> if (window.hello) { alert('truthy'); } else { alert('falsy'); } // will print falsy.
null
>> window.hello == undefined
true
>> null == undefined
true // there's the rub, sir.
>> null
null
>> undefined
null
>> typeof null
object
>> typeof undefined
undefined

So people write

if (foo == null) {
   foo = "Joe";
}

When what they really mean is

if (!foo) {
   foo = "Joe";
}

If you find yourself with a lot of null checks in your JavaScript, set aside some time and watch Douglas Crockford's "The JavaScript Programming Language" talk on Yahoo Video. It's part 1 of a 3-part series of excellent and enlightening talks.

Ubuntu on the Mac Pro

2008-02-10T20:55:00.000-08:00

Last year I bought a Mac Pro to replace my aging Linux PC. It was a pretty significant upgrade, from a HyperThreaded 2.5Ghz Pentium 4 to a dual-processor, quad-core 3Ghz Xeon with 8G of RAM. Leopard is nice but I wanted to try and get Ubuntu running on it.

Here's the setup, I have 4 drives:

750G - Leopard
200G - Windows
750G - Backups
750G - To be Ubuntu

and wanted to install Ubuntu on the 4th drive.

Here are the basic steps I followed:

Install refit
Install from LiveCD (7.10)
Select manual Manual Partition and from the advanced options, choose hd3,0. I setup my disk with one ext3 partition and 1 swap partition.
After install, boot the linux disk from the refit menu
Grub will get to stage 1.5, you will see the menu but be unable to start the system, it will complain about unknown partition type
hit 'c'

Here's a transcript of my tinkering with grub to find the right drive with my initrd and kernel on it so I could boot my fancy Apple-branded Linux machine.

grub> root (hd<TAB>

showed a bunch of drives with various partitions but most importantly showed that hd0,0 had the Linux partition I was looking for. Even though with the ubuntu installer, we told it to use hd3, here we use hd0. the boot manager has shuffled the order of the drives. Telling the ubuntu installer to use hd0,0 will only result in an unbootable disk.

grub> root (hd0,0)
grub> kernel /boot/vmlinuz<TAB>

will show you some vmlinuz files. typically just the one ubuntu installed. pick it.

grub> kernel /boot/vmlinuz-2.6.22-14-generic root=/dev/sdd1 ro
grub> initrd /boot/initrd.img-2.6.22-14-generic
grub> boot

Now your system will boot. Edit your /boot/grub/menu.lst to point the root (hd0,0)

Now reboot and you will be in Ubuntu. Your wifi adapter won't work. Follow these steps to enable ndiswrapper to use the windows drivers.

I used a usb thumb drive to move the windows drivers zip file between my laptop and desktop.

Once you have networking up, don't forget to enable ssh so you can login in case you screw up your X config.

Edit /etc/apt/sources.list to include what you might want.

I have an ATI Radeon X1950 Pro video card. To get it working, I simply installed Envy which downloaded the right drivers and installed them for me.

But it didn't span monitors, they were just clones so I ran aticonfig (which Envy installed for me)

aticonfig --initial=dual-head --overlay-on=1

And we were good to go.

Banshee imported all my music from my iPod after converting it's database from an iTunes format it didn't recognize.

Using hibernate left the Mac's wifi adapter in a weird state. Rebooting it didn't help but booting into OS X and back into Linux corrected the wifi card.

The fan speed still isn't quite right, various fans will become louder and softer at odd intervals. I will try monkeying with lm-sensors later.

I have nice smooth fonts by running Emacs 23 from Alexandre Vassalotti's repo and the Inconsolata font for programming.

Overall, it's a great Linux machine and maybe one day it won't be so much trouble to get running. Hopefully my instructions will help you get it up and running.

Lift Tutorial

2007-11-19T22:32:00.000-08:00

I've been working on lift quite a bit lately, fitting it in with some other projects and my obsessive need for football. (Both NFL and Madden)

If you're interested in trying it out, give the lift tutorial I wrote a try. Please drop us a note if you have any problems.

Tomato Wifi!

2007-10-21T16:57:00.000-07:00

While doing that podcast with Daniel, I noticed I was seeing 30% or so packet loss on my wifi network. After rebooting some equipment and keeping an eye on things, I decided to scrap my faulty Airport Express base stations and replace them with Linksys WRT54GLs running a custom opensource firmware called Tomato.

While the instructions are good, they assume you already have the router up and running. The first thing I realized was to throw the windows install cd away and just plug my mac into the switch on the back. The router will be listening on 192.168.1.1:80 with a username and password of admin/admin. The instructions will carry you from that point.

I have a 3 story house so I bought two routers and bridged them together with WDS. My only complaint is that Tomato's WDS is difficult to setup. The Airport Express admin utility did a wonderful job of easily bridging the two aiport express base stations together.

Except for one of the routers dying after only a few days, which Amazon was nice enough to replace for free with next-day shipping, things have been great on Tomato. 4 bars everywhere I go in the house.

[Hat tip to Nelson for the recommendation.]

My interview with Daniel Brusilovsky

2007-10-06T21:33:00.001-07:00

Daniel runs the Apple Universe podcast and interviewed me today. We had a fun chat, check it out!

What's next?

2007-09-28T23:21:00.000-07:00

Werner Vogels on Michael Stonebraker's 50X challenge:

I like this challenge, given that 50X is likely to be able to make impact, where 2-4X in general can be easily compensated for by the next generation hardware. But something bugs me about the challenge and also about some of the demonstrations in the papers; 50X is still focused on scaling-up, just as many of the current database systems do, instead of scaling out, which is what the world really needs. The evidence in the paper is indeed about single box performance. This continuing N=1 thinking will never yield systems that can break through the current scalability limitations of enterprise software, regardless whether it runs 50 times faster or not.

I couldn't agree more. It's exciting to see so many people arguing for new kinds of databases, especially distributed databases. Why do I hate on relational databases? It's not the relational part, it's the part where they fail at 3am and wake me up. Joins are nice but I need my beauty sleep.

43 Folders makes the switch...

2007-09-21T22:39:00.001-07:00

that I expect a number of others to follow. They just switched from a blogging system to a content management system. Lots of sites that have daily articles aren't blogs but until recently it was easier to have separate blogs and auxiliary sites than to setup a full-blown CMS. These sites are easy to distinguish: they'll have forums, wikis, or other miscellany that aren't simply chronologically ordered notes of interest. For instance, 43 Folders has a Forum, Job Board, and sub-sites like gtd and Inbox Zero.

I always assumed CMS's would get easier to setup and use and some of the more popular sites would switch to them. I fully expect this to become a growing trend and am glad to see traditional blogging tools start to get some competition for serious users.

Update: Earlier, I mentioned in this post that Matt pointed to a site that switched from WordPress to a CMS. I misread it. It was the other way around. Thanks, Jeremy. I hate it when you're right.