STF - A Distributed Object Store

My employer decided to opensource one of their internal tools: STF. STF is a distributed object store, much like MogileFS. The main difference is that STF uses HTTP (as opposed to a custom protocol) as its main protocol, uses components such as Apache/nginx, MySQL, memcached, Q4M/TheSchwartz. It's also new and therefore natively speaks PSGI internally, so you can use all your PSGI aware goodies. We also have an proof of concept setup to deploy STF on dotCloud platform.

Livedoor serves about 300 million objects (which means about 1.4 billion physical entities) using commodity hardware, running Apache 2.2 (+mod_reproxy), Memcached, MySQL 5.1, Q4M. Dispatchers are deployed on Plack + Starlet. We're also wrapping the application processes with Server::Starter for zero-downtime deployment (that means I can deploy to these heavily bombarded servers pretty much any time I want!)

STF can survive backend storage servers outages gracefully. If one storage fails, you can simply flip a flag in the database to stop access to it. Meanwhile you can rebuild it and replace it back with another server. Adding a node is just inserting another row in the database, and we're back in business.

I was initially replacing another application, and then I realized that the old code used DAV to store some data. Wait, DAV? How was I going to test DAV (and I wasn't going to put up a new DAV server just for this) -- then the original author of STF told me: "Use STF!". Well, I did, but it turned out to be an Apache module written in C. Ugh, same problem here... but by this time STF had been accepted as a defacto data storage within in the company, so it was better to lean over to using STF then.

Well, my initial problem was still there: "How am I going to test this?" So I started porting it to Perl. It took me a good two weeks to get a rough version out, and a total of two months until I was able to deploy one setup for our (internally) shared STF deployment. That was back in beginning of June. Since then we have replaced the image storage for one of the largest blog services in Japan, our photo sharing service, our geo-social service, among others.

Remember, the old code was in C, but we're scaling without any problems with this Perl setup.

Interested? Please checkout our repos. Please check it out and perhaps try it in your system. Let me know!

4 Comments

Hi Is there any special reason to use Class::Accessor::Lite?

In my own test, Mouse vs Class::Accessor::Lite

  • Object creation
    666666.67/s : 714285.71/s
  • Accessor
    5000000.00/s : 2000000.00/s
  • Memory usage(100000 Object)
    17312KB : 16004KB

Mouse is more faster at accessor and not so much behind on object creation or memory efficiency.

I remember seeing this post published a couple of weeks ago, then it quickly vanished. Any insider scoop? :)

Leave a comment

About lestrrat

user-pic Japan Perl Association director; Livedoor, Inc; Tokyo, Japan