Thursday, December 13, 2007

Flickr Stats how is it built

    Flickr Stats how is it built.
  • All Collection is done realtime
  • MYISAM and INNODB is used
  • The data is spread across 6 clusters (12 servers-6 used, 6 for fail over) mainly for data storage requirements
  • Memcache is not used at all in the core of the product.

In summary this was the longest project that I worked on, other then rebuilding the backend for Flickr when I first came on. The inner workings are very complex to achieve real-time collection-while not affecting page load times of a photo page. Most of my time was spent on creating a distributed lock once my DB design was solid.

Things that would really make life easier:

MYSQL AB gets rid of MYISAM and makes PBXT its replacement. I don't need all the great features of INNODB but I would like some. I'll go more into this later.

Additionally INSERT DELAYED worked with ON DUPLICATE UPDATE. Currently it does not.

Finally cross Engine Transactions would be cool but really not required.


Olly said...

That sounds quite impressive, would be really nice if you could post some details.
Don't know how much you can/want tell here but maybe some information about the general setup, data volumes etc.?

Anonymous said...

So, logging into MyISAM with insert delayed for non-blocking INSERTs and scripts to summarise/transfer the data to InnoDB?

Adrenalin said...

Isn't PBXT more InnoDB like ?
Or it is easier to setup than InnoDB, just as easy as MyISAM ?

Dathan said...

@adrenalin PBXT is like INNODB but it's not ACID complient. It is also like MYISAM but recovery from crashes is much faster. It's INSERT speed at high concurrency is much better then INNODB.

Anonymous said...



A片,色情,成人,做愛,情色文學,A片下載,色情遊戲,色情影片,色情聊天室,情色電影,免費視訊,免費視訊聊天,免費視訊聊天室,一葉情貼圖片區,情色,情色視訊,免費成人影片,視訊交友,視訊聊天,視訊聊天室,言情小說,愛情小說,AIO,AV片,A漫,av dvd,聊天室,自拍,情色論壇,視訊美女,AV成人網,色情A片,SEX