Mnesia
This weekend I read a short paper on Mnesia; a relational database designed for telecom applications with soft realtime requirements. It features election-free replication (!!) and is linked into the application's address space for efficiency.
Mnesia is written entirely in Erlang and is only 20,000 lines of code. It's query and operational language are also Erlang meaning that there's no potentially absurd impedance mismatch like you see with various ORM tools such as Torque or Ruby on Rails. You won't be mysteriously executing 500 queries to route a single phone call.
You can reconfigure it dynamically. Since it's meant for a highly available telecom environment, the idea is that you never turn it off. Erlang's Actors model helps it acheive fault tolerance.
Queries are built using list comprehensions and a more powerful form of Views are built out of logic variables.
You are able to bypass the transaction manager and access any data using only a lightweight dirty lock interface to make sure your reads aren't garbled. The authors recognize that sometimes it just doesn't matter if a little of your data is wrong and that you should be able to control when that might happen.
In practice, data in Mnesia is not usually normalized as many of the current applications that use it want to be able to get as much data as needed in a single query. Even with it's query optimizer and ability to sidestep the lock manager, many telecom apps still need to be able to get all of their data in a single query. How long do you usually wait for a telephone call to start ringing? 100 milliseconds? That's as long as some RDBMS take to optimize a single query let alone execute it and return the results.
Apparently quite a few applications within Ericsson and Nortel use Mnesia. It's open source along with the rest of Erlang.
If I have time to play with it, I'll post some example code. I'd like to see how well this election-free replication works.
posted by Steve Jenson at 10:24 PM
![[Atom Enabled]](http://saladwithsteve.com/valid-atom.png)
12 Comments:
Locking your development language and your data store together is a very big design decision. A decision I don’t think I’d ever be comfortable making. I could see the justification in a hypercritical environment like corporate real-time telecommunications, but in few others.
I’m very much of the school that programming languages and data are two separate things that need a clear dividing line. Granted the move from procedural to OO has created some real strains on the relationship, but I think that that is more a problem with our dependence on out of date open-source RDBMS’s.
I think that it’s interesting that companies like IBM are throwing to the open-source wolves various relational database systems... the commons as corporate dumpster. Are they moving on? I wonder where it’s all going. Everything seems to be pushing towards a great convergence. Personally, I’m almost at the point in my thinking where objects are data and everything else is function. I just need to get my skills up to that level.
9:21 AM
"Locking your development language and your data store together is a very big design decision. A decision I don’t think I’d ever be comfortable making."That's fine, then write a thin wrapper around it and talk to it via sockets. That's actually what I'm thinking of doing.
9:42 AM
"I’m very much of the school that programming languages and data are two separate things that need a clear dividing line."And yet you program in objects which are fundamentally data combined with behaviour all wrapped tightly up in one package.
Why do you believe there needs to be separation? What are you arguing that it gains you?
9:50 AM
"I think that it’s interesting that companies like IBM are throwing to the open-source wolves various relational database systems... the commons as corporate dumpster."But they're not just throwing RDBMS into the opensource ether, they're throwing everything! It fits in with their business model: they give away software hoping to sell hardware and services.
9:57 AM
"And yet you program in objects which are fundamentally data combined with behavior all wrapped tightly up in one package."
While true… at the end of my post I was trying to point out the acknowledged flaws of developing that way.
The main thing that I’d be careful to avoid is that by trying to solve the problems posed by traditional SQL your solution goes back to the way things were before SQL. Heck, adding an ODBC driver to the thing would probably shut me up. Looks like people tried in the past, but didn’t get far.
It does look very impressive. I just don’t like being locked into any one thing when it comes to data. I want to get at it via OBDC, JDO, SQL, or whatever if I have to. There can be an optimal way of doing it, but the optimal way shouldn’t be the only way.
1:51 PM
Wikipedia is down currently because MySQL isn't robust in ways that Mnesia is. This coming on the heels of LiveJournal being down for the same reason (somebody tripped the power _inside_ the datacenter, where all the redundancy in the world can't help you) is very apropos to our discussion.
Maybe having real fault tolerance _does_ trump RPM installers, GUI Schema builders, and being able to access your data from Excel or a bash script?
Now, to answer you honestly, being locked into Erlang doesn't bother me for a few reasons: 1) I prefer it to be locked into SQL, 2) I can at least understand the computational model of Erlang, and 3) I can export my data with as little (or as much) custom scripting as I would need to get my data from one twisty form of SQL to another. (or whatever twisty XML vocab your RDBMS supports exporting to).
If I had a working Mnesia DB up that spit out JSON, how would you feel about that? Seriously.
8:51 PM
That would be very sweet, actually. I like the format and a JSON to XML dumper would be very easy to create, if someone hasn't already.
The only thing left is how to query the data? Right now the landscape for OO queries is pretty chaotic.
Maybe you just create something on your own with a flexible enough architecture so that it would be possible to plug in other methods such as XQuery or JDO as things solidify.
Ya got all that and I'm happy. Abandoning the constraints of SQL may be a big plus. I haven't heard anyone sing the praises of the Object/Relational hack, but perhaps I need to get out more.
5:11 AM
So this is the trade-off for language independance. Mnesia already provided a nice query language: Erlang. I figured that even if I were to write a socket layer over an Mnesia database, I would probably still export Erlang as the query language. It just fits.
I'm using ZODB a lot lately and it'd probably raise your hackles in the same way: no obvious query language. How I structure my data is how I query it. I stick my stuff into persistent BTrees and request objects by name. For more complex needs (I really should say: for things that are outside the scope of a datastore), I also index my data in PyLucene and one of the keyword fields is it's id in the BTree.
It's working out way better than I imagined. I haven't missed SQL.
11:18 AM
The problem is that it doesn't pass the bus test. (The bus test is how well you are prepared for the ramifications of not looking both ways when crossing the street.)
It also doesn't deal with the ramifications of success. Sure, you can do it. The problem is that if you don't want to design standard methods to query your data you'd better make sure that your software is so crappy that no one else wants to use it and that the data contained there in is so trivial that no one else cares what it is. Cause if they do you're creating a very pretty headache for the poor soul that has to patch a query abstraction layer onto your product later on.
That is, unless your product is a part of an organization that creates its own gravity... that thing that I was talking about before. If the implementor is so powerful (telecom) it can dictate terms because it controls its destiny. (Yahoo Stores worked cause Yahoo! bought it. As an open source product for resale it would have been a total failure cause without Yahoo!'s gravity it would be virtually unusable for most developers.) Thus PHP and MySQL.
For me all this seems backwards. How is secondary to what. The fact that everything is driven by how feels off.
12:06 PM
We're looking at this from two different angles. I don't think I"m looking at it from 'how', I'm looking at it from the requirement that people's data never gets lost or goes offline for days at a time. MySQL clearly doesn't cut it. I'm looking at it from the requirement that my programs operate correctly in the face of it's components sometimes failing. PHP clearly doesn't cut it.
Honestly, users don't care if your stuff is written in Java, or Fortran, or k, or brainfuck. They don't care that it's written as some beautfully abstract gem that fits nicely into a UML diagram. They don't care that you've got your Enterprise Certification from Sun. They care that 2 hours before their deadline, your software is working while your competitors are recovering from InnoDB corruption.
Having a "standards-based" access method is no guarantee of anything, not even of maintainability. Not even of interoperability. Not even of usability. What's it actually a guarantee of?
Yes, prepare for the future. But be mindful of what really matters.
I hope we're not talking past each other. Let me know if I'm not paying attention to a certain part of your argument.
9:48 PM
The type of users I'm talking about are more developers that need to update and extend your code. It's my opinion that SQL was one of the most powerful revolutions in computers. Getting rid of it is a big move.
It sounds like an interesting trade off. 100% uptime is a pretty high bar. I could see it trumping certain things. The main thing is that you know what you're doing it and why you're doing it.
My bet is that a lot of my objections will be solved by the community on their own as interest grows in Erlang.
I wouldn't just slam MySQL. Mom could tell you a thing or two about the global Oracle systems that she designs. You get big enough and wierd shit happens.
6:45 AM
Mnesia is absolutely wonderful,
3:19 AM
Post a Comment
Links to this post:
Create a Link
<< Home