Tuesday, April 19, 2011

Scalability Stories

'It is not the strongest of species that survives, nor the most intelligent. It is the one that is most adaptable to change' -- This statement is so emphatic in every slide !

http://www.slideshare.net/tackers/why-we-chose-mongodb-for-guardiancouk

In a nutshell, the N-tier System swiftly evolved from traditional View-Service-DB architecture to View-Service-Memcached-DB.

Memcached is a very good candidate for distributed indexing. Extract main id from Web Request and then find related associated index and making fast lookup from mysql storage.

Hibernate can be effectively tuned to handle large data set specially by leveraging terracotta big data.

But even if ORM hides complexities, DB has strong influence on domain model.
Still one needs to implement Complex Caching and Optimizations.
Memcached still need lots of writes to DB and Ehcache needs to write-behind the DB.

Facebook heavily relies upon customized php-thrift-memcached-mysql-cassandra power-packed combo !

Guardian Architecture highlighted a very simplistic idea ... carrying the JSON from backend to frontend and vice-versa. Json/Rest is the smooth replacement for heavyweight soap/Axis2 in middle-layer. Json such a nice forwardly-extensible structure naturally fitting into to javascript and python. Here comes MongoDB - the Document-oriented Database storing parsed Json documents. The best part is schema can be changed in runtime without any down-time.

JSON can define different classes as tagged documents, multiple docs part of same collection.

So where has the DB gone ? Vanished !

And there is Apache SOLR, Hosted in EC2 [Document Oriented Search Engine, (NO DB .. huh) ..] .. a super fast Read API.

Its a huge paradigm shift.

http://www.slideshare.net/tackers/why-we-chose-mongodb-for-guardiancouk

So far the simplest explanation of CAP theory .. http://www.slideshare.net/jboner/scalability-availability-stability-patterns

This is the best Applicability-Chart of CAP theory - slide 9 of http://www.slideshare.net/pacoid/hidden-gems-found-with-hadoop

Upcoming Event : Register at http://www.10gen.com/conferences/mongosf2011

1 comment:

Kaniska said...

Another interesting story : http://www.dzone.com/links/r/linkedin_architecture.html

Its bit outdated ! But still offers very good insights into Scalable Architecture.