“The Apache Cassandra Project develops a highly scalable second-generation distributed database, bringing together Dynamo’s fully distributed design and Bigtable’s ColumnFamily-based data model.”

So, from what I’ve experienced so far this is either a distributed Berkely DB, or a persistent Memcached, depending on which concept you are more familiar with (I know, that’s incredibly over simplified).

Cassandra was developed by Facebook and was made open source several years ago. It uses the Thrift API to communicate, which was also developed by Facebook. Supposedly there are some steps being taken to move Cassandra on to a different API, but Thrift is the defacto for now.

There are two modules on CPAN that deal with Cassandra. Net::Cassandra is the original, and in my opinion, the superior/simpler/easier module. Net::Cassandra::Easy is a later implementation that really isn’t any simpler from what I can tell and neither seems any more or less tied to the intricacies of the Thrift API despite Net::Cassandra::Easy’s accusations of Net::Cassandra. Both implement the Thrift API under the scenes in their own way, and both are lacking in features.

Now, the Thrift API itself is distributed with a perl library called simply “Thrift”. This library is very new and it shows. It needs a lot of work, a lot of automated tests, and IMO should be on CPAN. I consider it a fail when a project decides to distribute Perl libraries independently from CPAN. This leads to very little exposure – I was lucky I found the perl library in the Thrift tarball, I doubt others have been as lucky.

So, I’d like to make a Thrift driver library and put it on CPAN. Then, a truly Thrift independent Cassandra client library can be written that uses the Thrift library behind the scenes, but is not directly tied in with it. Later, when Cassandra decides to change their API library, the Cassandra library can be changed to use a new driver.

An additional benefit to this is if any Cassandra features are not yet available via the Cassandra module, then the perl developer can easily drop down to the Thrift driver and directly write Thrift queries.

Has anyone thought of this, or perhaps begun working on something like this? If not, I’d like to do it and I’d love to hear any suggestions and thoughts people have.

Edit: It turns out there is already a module on CPAN that does exactly what I talked about here. Its called DBIx::Connector. I haven’t tried it yet, but it looks like the guy that wrote it designed it with input from the DBIC guys.

I’m a big fan of DBIx::Class. Among DBIC’s many great features is its superb transaction and connection handling via DBIx::Class::Storage::DBI. When I’m using raw DBI I feel like I’m missing these core components. I should always be able to expect robust transaction and connection handling.

I started playing around with creating a new CPAN module (something like DBIx::Robust, DBIx::Transaction, etc…) but each time I dove in to it I kept coming to the conclusion that DBIC’s DBI storage drivers provide everything I need and I would just be shooting myself in the foot by either porting DBIx::Class::Storage::DBI to a DBIC independent API or by writing it from scratch.

Before I go any further, let me show you how attractive and awesome DBIC’s storage layer is:

• Transactions may be nested, even without savepoints.
• Transaction savepoints can automatically be tracked allowing for reliable and incremental rollbacks.
• The appropriate DateTime::Format class is automatically determined based on the type of database.
• All database calls efficiently pass through a layer that detects stale connections that will attempt to re-connect to a database.

“There is no charge for awesomeness, or attractiveness.”

The DBIx::Class::Storage::DBI pod illustrates all this in a round-about way: “If you set AutoCommit => 0 in your connect info, then you are always in an assumed transaction between commits, and you’re telling us you’d like to manage that manually. A lot of the magic protections offered by this module will go away. We can’t protect you from exceptions due to database disconnects because we don’t know anything about how to restart your transactions. You’re on your own for handling all sorts of exceptional cases if you choose the AutoCommit => 0 path, just as you would be with raw DBI.”

This stuff is really powerful, and we should never have to work with databases in Perl without it. But, currently, it is only meant to work under DBIC, which is a shame. I’m betting that DBIx::Class::Storage::DBI could be used directly, bypassing DBIC completely. There is some DBIC stuff layered in there, but that should be ignorable.

What do the rest of you think? Has anyone considered moving this logic out in to a generic distribution that isn’t DBIC specific? Is there merit in what I’m talking about? Does anyone use DBIx::Class::Storage::DBI directly, bypassing DBIC?