Setup Hadoop and Hbase Environment

Setup Apache BigTop

http://apachebigtop.pbworks.com/w/page/48434924/FrontPage


How to develop Big Data Pipelines for Hadoop
I Didn't Know You Could Do All that with Hadoop
Storm: distributed and fault-tolerant realtime computation
Hadoop Plugin for MongoDB: The Elephant in the Room


StrataConf 2012 Hive Tutorial
http://assets.en.oreilly.com/1/event/75/Hadoop%20Data%20Warehousing%20with%20Hive%20Paper.pdf
Hadoop Data Warehousing with Hive

Cloudera Hbase Training
http://university.cloudera.com/training/apache_hbase/hbase.html

Hadoop on Amazon AWS

http://cs.smith.edu/dftwiki/index.php/Hadoop_Tutorial_3_--_Hadoop_on_Amazon_AWS

Build and Run Hadoop Java program with Ant and Maven

http://stones333.blogspot.com/2012/02/build-and-run-hadoop-java-program-with.html

Machine Learning on Big Data with MapReduce
http://machinelearningbigdata.pbworks.com/w/page/37651454/FrontPage

Hbase & NoSQL Comparison :
http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis
http://ria101.wordpress.com/2010/02/24/hbase-vs-cassandra-why-we-moved/
http://natishalom.typepad.com/nati_shaloms_blog/hbase/
http://natishalom.typepad.com/nati_shaloms_blog/2011/07/real-time-analytics-for-big-data-an-alternative-approach.html

Beyond Hadoop : Stream based Messaging (Storm / Kafka )
http://natishalom.typepad.com/nati_shaloms_blog/2011/07/real-time-analytics-for-big-data-an-alternative-approach-to-facebooks-new-realtime-analytics-system.html

Building Multi-Tenant Platform for Data Acquisition, Analysis Visualization

Multi-Tenant Data Access Security :
Cross-Organization and Cross-Application SSO

Single sign-on for the cloud and SaaS - OneLogin

Multi-Tenant Data Compliance :

 Satisfy Data Compliance and Regulations and Data Encryption

Reference : http://www.sesamesoftware.com/pdf-docs/Satisfying_Compliance_Regulations_for_CRM_Data.pdf 

Multi-Tenant Data Acquisition :

Multi-Tenant Data Acquisition , MDM, Data Cleansing, Data Mapping - Component :

Capture data from Multiple Apps for Multiple Tenants through the same 'Integration Platform'

"If it's Not Multitenant, it's Not Really SaaS." --

http://www.ebizq.net/blogs/connectedweb/2010/02/why_your_cloud_applications_sh.php

http://www.boomi.com/products/technical/platform_architecture

http://blogs.boomi.com/bod/2010/06/boomi-suggest-the-power-of-multitenancy.html

Mule+ion , Boomi+atom, Cast~Iron

Multi-Tenant Data Storage :

http://msdn.microsoft.com/en-us/library/aa479086.aspx

Scalable Elastic NoSQL or Partitioned RDBMS

Multi-Tenant Metadata Manager and Application Provisioning :

Multi-Tenant Data Analysis :

Extensible Rules and Queries - configurable across tenants.

Tenant-Aware Resource Access Rules

Multi-Tenant Data Service API:

Tenant-aware Resource Access Rules.

Tenant-aware Server, Sessions and Cache.
Tenant-aware Rest Services , Work-flow and Rules-Engine

Multi-Tenant Data Apps Server : (Cloud-enabled and Tenant-aware )

 Application Resource Allocation (App Server / DB Connection / Fail-over ) for all the tenants without stepping into each other's resources.

POD System is a popular recommended Multi-Tenant Model

--  http://connectedco.com/2011/04/18/the-future-is-podular/ 

-- PaaS script will automatically roll out new resources ...  its a mini-combo of (app-server, db, vm for a logical group of tenants)

-- http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/VMDC/2.0/large_pod_design_guide/Large_Pod_Design_Guide.pdf

Multi-Tenant Data Viz :

Multi-Tenant Charts/Graphs and Configurable Dashboards.

Pentaho , YellowFin

Securing Enterprise Application Access : Learning from Salesforce

Securing Enterprise Application Access :  Learning from Salesforce 

1 :  SFDC Tab invokes the following url to access the application :

(a)  https://mysite.com/secure_access/validate.php?location={!API.Partner_Server_URL_220}&session_id={!API.Session_ID}

2 :  Web Request Verification : Load Balancer Level :

(a)  - We need to perform referral check and IP range check

Vulnerability Protection : (http://wiki.developerforce.com/index.php/Secure_Coding_Single_Sign_On )

Below is the regular expression to validate legitimate API Partner Server URL servers:

·         https://[^/?]+\\.(sales|visual\\.)force\\.com/services/(S|s)(O|o)(A|a)(P|p)/(u|c)/.*

To summarize the above regular expression, it ensures that the URL starts with ‘https://’, followed by a character other than ‘/ ‘ or '?' for 1 or more times, followed by a ‘.’, followed by ‘sales’ or ‘visual.’ followed by ‘force.com/services/SOAP/’, followed by ‘u’ or ‘c’, followed by ‘/’. This will allow:

·         https://*.salesforce.com/services/SOAP/u/

·         https://*.salesforce.com/services/SOAP/c/

·         https://*.visual.force.com/services/SOAP/u/

·         https://*.visual.force.com/services/SOAP/c/

3 : Delegate the request to validator  (php / javascript / servlet filter ) 

             (a)  we can double-check if request coming from salesforce

           (b) validate the session-id by making a reverse call to SFDC.

                        i. - simply retrieve the UserInfo for that connection and extract the user-id.

                           if there is error condition, then forward to trial page or whatever relevant.verify if the (c) User Authorization : Verify if user-id is registered. (highly recommended by salesforce)

                        iii - another layer of security – verify call is initiated by salesforce by using private security certificate from SalesForce

                           Include this certificate in every http / soap call to SFDC from validate.php.

            http://wiki.developerforce.com/index.php/Single_Sign_On_for_Composite_Apps   

4. Enable SSL in uat-apache

5. Use SalesForce Certificate :

This certificate is meant to identify that the request is coming from salesforce.com, not a specific user.

http://www.salesforce.com/us/developer/docs/api/Content/sforce_api_om_outboundmessaging_setting_up.htm#om_user_profile

Your application (endpoint) server's SSL/TLS may be configured to require client certificates (two-way SSL/TLS), in order to validate the identity of the Salesforce server when it takes the role of client to your server. If this is the case, you can download the Salesforce client certificate from the Salesforce application user interface. 

6. Enable SSO (in future) :

As per SalesForce : -  https://na7.salesforce.com/help/doc/en/sso_delauthentication_configuring.htm

Use - Delegated Authentication WSDL

If Single Sign-On (SSO) is enabled for your organization, users who access the API or a desktop client cannot log in to Salesforce unless their IP address is included on your organization's list of trusted IP addresses or on their profile, if their profile has IP address restrictions set. Futhermore, the delegated authentication authority usually handles login lockout policies for users with the “Uses Single Sign-On” permission. However, if the security token is enabled for your organization, then your organization's login lockout settings determine the number of times a user can attempt to log in with an invalid security token before being locked out of Salesforce. For more information, see “Setting Login Restrictions” and “Setting Password Policies” in the online help.

More details : http://www.salesforce.com/docs/developer/cookbook/Content/sso_delegated.htm

Broad Topics on application security : http://wiki.developerforce.com/index.php/Apex_and_Visualforce_Security_Tips

1. Cross-Site Scripting
2. S(O)QL Injection
3. Cross Site Request Forgery
4. Secure Communications and Cookies
5. Storing Secrets
6. Arbitrary Redirects
7. Access Control
8. Enforcing CRUD and FLS (Force.com)
9. SSO for Composite Apps 

 Just like PHP Toolkit / Java Client API for SalesForce, the ajax toolkit is also very simple to use  http://www.salesforce.com/us/developer/docs/ajax/apex_ajax.pdf

Online CS Courses

http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-00-introduction-to-computer-science-and-programming-fall-2008/
Introduction to Computer Science and Programming


http://web.mit.edu/newsoffice/2011/mitx-faq-1219.html

What is MITx?

http://academicearth.org/courses/programming-methodology

Computer Science I: Programming Methodology

Lightweight Persistence Service for Dynamic Entities

Goal :

• Persist incoming stream of Objects of into the Database.
• Parse the incoming XML/Json data to extract "Object of similar type" (i.e. a particular Business Entity) .
• The Business Entity may have a fixed set of attributes or a dynamically changing set of attributes.
• We should not create/maintain ORM layer and no need for POJO conversions from/to wsdl/http streams.
• We should keep the  persistence service layer very very light-weight.

Setup :

• mongodb2.0.1 (A database named "staging" needs to be created in MongoDB)
• spring-data-mongodb 1.0.0.M4, mongo-java-driver 2.5.2 (and all other dependencies specified in pom.xml)

Development Environment : SpringSource Tool Suite - 2.8.1.RELEASE

Implementation Approach :

• SimpleJdbcInsert Command make things really simple when Objects of same type has a fixed set of attributes.

Create the singleton instance of the SimpleJdbcInsert Command for the corresponding table.Prepare the Map of key-value  pairs by parsing the incoming data and simply push the data into the tables against the corresponding columns (keys in the map).

Number num = insertCommand.executeAndReturnKey(parameters);
Or insertCommand.executeBatch(…)  

• It may be tweaked to leverage DatabaseMetadata and perform DDL on RDBMS programmatically.
• But its very costly to Create/Update/Delete columns in relational data-store on-the-fly.
• So finally we take resort to MongoDB.
• If the Business Entity has a pre-defined set of attributes we should leverage the well-known annotated document object.
• Otherwise if Entity structure is changing constantly to accommodate newer attributes, then just create new DBOnjects and DBRefs.

MongoConfig creates a MongoTemplate against the database "staging" MongoDataAccessService provides all CRUD operations.MongoRestController specifies all the Rest operations.

Sneak Peak into the code .. org.jdom.Element rootElement = jdomDocument.getRootElement(); String rootElemName = rootElement.getName();

    if (!mongoDataAccessService.collectionExists(rootElemName)) {
        mongoDataAccessService.createCollection(rootElemName);
    }
    // we assume the element contains all the values for the table-columns
    // as the attributes.
    List list = rootElement.getAttributes();
    DBObject parentDocument = new BasicDBObject();
    // suppress _ClassName in the DbObject
    for (Object attr : list) {
        org.jdom.Attribute source = (org.jdom.Attribute) attr;
        if (source.getName().equals("Id")) {
            parentDocument.put("_id", source.getValue()); // use the original incoming Id
            // candidate for shard key -- TODO
        } else {
            parentDocument.put(source.getName(), source.getValue());
        }
    }
    // Now iterate through the Child Elements and store them as Database
    // References
    DBRef childDocumentRef = new DBRef(
            mongoDataAccessService.getDB("staging"), elementName, elementId);
    parentDocument.put(elementName + "_ref", childDocumentRef);

Runtime Steps :

• target folder already contains a pre-built war file.
• just drop the war inside an app server context root.
• http://localhost:8080/dynamic-data-collection/mongo/entities/add
• Run the MongoRestClient to dynamically extract an entity from XML fragment and add it to the corresponding Collection.
• Simple MongoClient just creates instance of a predefined Mongo Document. This comes handy when Entity has a fixed set of attributes.
• 
  

Code :  https://github.com/kaniska/Dynamic-Object-Persistence 

Take Away :

• Analyze the Business requirements and accordingly categorize the types of Entities.
• If all Entities can have pre-defined schema then use SimpleJdbcInsert Command along with MySQL / Oracle for best performance.
• But in case the Business Process consists of both fixed and dynamic entities, then consider the following approach.
• Define annotated POJO (Mongo Document) for the Entities having a fixed structure.
• If same type of Entity has different set of attributes for different entities then consider creating them dynamically.
• So the choice of database (MongoDB Vs RDBMS) - depends upon how much dynamism one needs to accommodate in the application.

Future Improvements

1. Externalize the MongoDB configuration in external properties file. (Tried but somehow it did not work.)
2. Use the latest Spring-MongoDB release
3. Work with XPathOperations instead of parsing xml fragments manually.
4. Figure out if Spring-Data MongoDB API leverages latest Morphia annotations for journal-sync, suppressing _ClassName in objects etc.
   Reference : http://www.springsource.org/spring-data/mongodb

DreamForce Presentations

Designing for the Cloud: The 12 Factor App — Valerie Ringer The slides are on Ryan's web page ryandotsmith.heroku.com Saturday, September 17, 2011 4:46 PM   Comment

Using Node.js on Heroku — Pat Patterson Session video on YouTube Using Node.js on Heroku http://www.youtube.com/watch?v=PACwQ632jIE Saturday, September 17, 2011 10:52 AM   Comment

The Next Step of Evolution in the Force.com REST API — Pat Patterson Session video on YouTube The Next Step of Evolution in the Force.com REST API http://www.youtube.com/watch?v=9ur5D1Q-oCQ Saturday, September 17, 2011 10:51 AM   Comment

Security and Authentication for Mobile Apps — Pat Patterson Session video on YouTube Security and Authentication for Mobile Apps http://www.youtube.com/watch?v=ia8yaTtnK0w Saturday, September 17, 2011 10:32 AM   Comment

Mashing Up the Cloud with Force.com Toolkits — Pat Patterson Session video on YouTube Mashing Up the Cloud with Force.com Toolkits http://www.youtube.com/watch?v=pe2Ke4a9zas Saturday, September 17, 2011 10:27 AM   Comment

HTML 5 and Geo-Location Using Visualforce and Apex — Pat Patterson Session video on YouTube HTML 5 and Geo-Location Using Visualforce and Apex http://www.youtube.com/watch?v=wwTdGqOygBQ Saturday, September 17, 2011 10:26 AM   Comment

Getting Started with the Force.com Streaming API — Pat Patterson Session video on YouTube Getting Started with the Force.com Streaming API http://www.youtube.com/watch?v=8UqeVRiv6-s Saturday, September 17, 2011 10:25 AM   Comment

Event-Driven Programming with Node.js and the Force.com Streaming API — Pat Patterson Session video on YouTube Event-Driven Programming with Node.js and the Force.com Streaming API http://www.youtube.com/watch?v=_Vt0TPS5HLw Saturday, September 17, 2011 10:23 AM   Comment

Authentication: A Practical Guide — Pat Patterson Session video on YouTube Authentication: A Practical Guide http://www.youtube.com/watch?v=9McaWgMLprs Saturday, September 17, 2011 10:02 AM   Comment

Challenge the Conventional Wisdom : MongoSF 2011

||| MongoSF , May 25, 2011  |||

* shamelessly borrowed from Kyle's great slides ...  I consider this as the best example to drive home the simple yet strong message from MongoSF Seminar !


Professor Calculus trying to convince Captain Haddock that he should embark upon the Shuttle to Planet MONGO !


Capt. H : My data is not super-transactional , but time-critical and move very fast .. So are you saying I don't need Normalization ?
Prof. C :  Exactly ! Design your database as per your usecase. In absence of a schema, no need to build/maintain the lifecycle of a model (conceptual [ERD]-> logical [UML] -> physical [DB] -> sql -> xsd ..)
Captain! just spell-out your thoughts as..{vendor_id : "apple", products : ["iphone","ipod", "ipad"] }
Embedded Array is a powerful concept to get a list of associated entities for a specific entity at one go ! 


With full respect to RDBMS world (which will always be relevant in the Operational Domain like sayManufacturing ...) few imp points to remember :
>>  Tradional DB - Schema has no bias for a specific usecase. It encompasses all possible relationships between the entities (stakeholders) of a domain !
>> So its great for ad-hoc queries and data-rigid ness ....  but very very slow for data manipulation and for hopping around !! This trade-off can be equated with the famous trade-offs between 'bloated Wsdl and simplistic Rest' or 'Schematic XML and Schema-less Json' !

Capt. H :  Traditionally, first I normalize the data for OLTP and then de-normalize the data for BI ..  ?
Prof. C :  I do understand your pain points ! Welcome to MongoDB  ! Just fire few aggregation queries on the same document that stores your business model and you extract all the intelligence at no extra cost !


Capt. H : So far so good ! Do I still need a Cache or ORM layer ?
Prof. C : Well.. Captain .. Caching was a Work-around for slow lookups from over-burdened database !
Now the db itself is extremely light weight engine - fast lock-free lookup, aycnchonous parrallel writes, distributed loads on shards etc. etc.
- ORM is another work-around to abstract rigid SQL and convert pojo from specific framework to db entities ! .. Well .. in most of cases .. no need for any sort of mappng ideally ! MongoDB manadates that Code and Data should reside in seperate layers and each understand one standard i.e. Json! 
It flows so smoothly from front-end  (Php/pearl/python/gwt/js .. all understand json) to the json-friendly Rest layer and finally lands on the lap of MongoDB as Bson ! 
Rememeber ... we now have a database which itself follows object structure that your app layer is based on ...  so no need of any ORM layer !
Now for the sake of backward compatibility and smooth integration with existing pojo-based apps (which were talking to rdbms), Spring-data provides nice wrappers to NoSqls !


Capt. H :Well I love breaking down a large Entity like Product into multiple components like - { line_item, payment, order, etc.. } .. but I pay the heavy price of slow joins ?
Prof. C : Hmm .. how about creating a Rich Document - Product and then embed sub-documents {line_ites, payment, order..} inside the parent..
Then index on sub documents and perform super-fast asynchrnous atomic updates on a single document


Capt. H : Great ! You know .. I love storing the parent-child relationship .. i.e. the Inheritence or.. a Tree-Hierarchy in a single table ....  but as a result in RDBMS .. you know .. I have redundant columns !!
Prof. C : No worries ! in MongoDB, data itself defines its schema ! (metadata free modelling) 
So you can specify ...
db.shapes.find()
  {_id:"1",type:"circle",area: 3.14, radius: 1 }
  {_id:"2",type:"square",area: 4, d: 2 }
  {_id:"3",type:"rect",area: 10, length: 5 , width: 2}
-- And wait .. you can apply the beautiful 'Sparse Index' to diverse data-set stored inside same collection !
Did you catch the sweet spot 'radius' - just by its sheer presence as an unique attribute an 'inherent restriction' is applied on the collection.


Capt H: Professor .. You simply Rock ! .. I kinda learnt how to partition my data in rdbms ! But so difficult to maintain when data outgrows the chunks ...
Prof C. .. Sorry to hear that captain ! ... data-partitioning is a no brainer !
db.runCommand( { addshard : "localhost:10000" } )

{ "shardadded" : "shard0000", "ok" : 1 }
db.runCommand( { addshard : "localhost:10001" } )
{ "shardadded" : "shard0001", "ok" : 1 }

All the goodness of sophisticated master-slave nodes allocation, replication, internal node state management, auto-recovery are provided out-of-the box ! 
ReplicaSet Mode detects all the hosts every 5 sec.
Replica1 : {chunk1(user1 - user100) => shard1, chunk2(user101 - user 200) => shard1}
Replica2 : {chunk1(user1 - user100) => shard1, chunk2(user101 - user 200) => shard1}


MongoS process is the load-balancer. Upon startup it finds out which servers are up and based on the latency picks the shard. It routes the chunks between shards in replica sets.
If new hot chunk is created, MongoS can dynamically allocate the chunk to an available shard - shard2.
Otherwise it will spawn new shard by dividing existing shard at its median !


Capt H: Very Interesting ! So I can choose whether my query will work on primary or secondary.
Prof C: You got it right ! For immediate consistency, query from / write into the primary master. For eventual consistency query the slaves ! Reads may go out-of-date on slaves. Its just a tip of iceberg ! So refer to the document for more details !


Capt H: Great ! Now that my app working fine ... need to do some analysis .. So coming to data-crunching in batch ! I am a startup guy and have no time to setup hadoop-hive-mahout on a seperate cluster , neither have the luxury to build the infrastructure of Amazon Elastic Map-Reduce ....  so .. I guess .. MongoDB ..


Prof C:  Oh ! Yah ! Absolutely ! Right here ... Why as a developer in a fast-paced-env you need to worry about the gory details of data crunching !! Just call map-reduce api ! Stay tuned ! Out-of-the box Aggregation API coming soon ! Remember the key point ...  you can now enjoy a real-time streaming analytics with negligible latency !


Capt H:  I am ecstatic to hear this ! On a different note I love indexed-based searching using lucene-solr .. but just checking in if ..
Prof C:  Sure ! MongoDB hears you ...  full-fledged text-searching .. will be released by end of 2011 !


Capt H: I love Scala and Node.js .. a big fan of executing requests parallely ..  but it gets very tricky to solve table-level and row-level locking issues in Database ....
Prof C: Absolutely ! Your writes per document are atomic and reads are parallel in nature (cpu cores scaling out not speeding up ... so mongodb native threads just love asynchronous parallel processing) ! But if writes fail .. there is no auto-rollback machanism .. your app should be responsible for data consistency !  No support for distributed transactions !  So use a language of your choice and no need to shard in the app layer !


Capt H: A compelling reason for adopting AWS was SimpleDB .. I mean instead of moving my DB to Cloud I tried to  access a DB on Cloud .. but sacrificed many features of RDBMS ....  So .. is it a Big Ask from Mongo ... ?
Prof C:  Not at all !  MongoDB is Cloud-ready by virtue of its horizontal scalability, multi-tenancy , fail-safe nature, statelessness and atomic transactions. Scale linearly, increase capacity with no down-time.


So Captain Welcome Aboard ! Enjoy your journey into Mongo !


With freedom comes responsibility :


1. MongoDB does not encourage storing reverse links in referred documents for implementing many-2-many relationship.
Captain I know you love your language .. you wont mind couple of extra lines of query in app layer :-)
//All categories for a given product
product = db.products.find(_id, prod_id);
db.categories.find({_id:  {$in : product.category_ids} })


2. Its not a full-fledged caching solution ... so there are pending issues with data-eviction ! Expired objects reportedly sit back in memory ! MongoDB believes in reusing memory rather than suffering from reclaiming-reallocating overhead every now-n -then ! But the garbage collection algo definitely get better and better with community support !


3. The thrust is on 'eventual consistency' for writes. there is no 'Retries' for writes ! So we need to judiciously handle the write-exceptions in app layer. We need to remember - if we explicitly mark a 'transaction request' as 'done' then only mongoDB driver will clean-up the connection from the thread-local !


4. While designing the data model, we need to understand where to 'embed' and where to 'link.
Definitely 'Linking' has a higher 'cost of relocation' than 'Embedding'. That implies in a Tree we better store the list of descendants directly.


5. The workingSet size should be kept minimal and need to be distributed evenly.
Pre-splitting for initial bulk loading is always advised. 


6. Queries on sharded key are fastest.


7. For pagination we should depend upon stateful cursor_index instead of depending on stateless sharded key.


Tips, tricks, references :


Thought-provoking :  http://www.slideshare.net/tackers/moving-from-relational-to-document-store


Great summary on app-design principles :
http://speakerdeck.com/u/kbanker/p/the-mongodb-gamut-four-app-designs


50 Tips and Tricks for MongoDB Developers :           http://my.safaribooksonline.com/book/databases/9781449306779/firstchapter


Schema Design Basics : http://speakerdeck.com/u/kbanker/p/mongodb-schema-design-mongosf-2011
http://www.slideshare.net/mongodb/mongodb-schema-design-richard-kreuters-mongo-berlin-preso

New Aggregation Features : http://www.slideshare.net/cwestin63/mongodb-aggregation-mongosf-may-2011

Performance Considerations : http://speakerdeck.com/u/kbanker/p/10-key-mongodb-performance-indicators

http://blog.zawodny.com/2011/03/06/mongodb-pre-splitting-for-faster-data-loading-and-importing/

Great Teachings from MongDB Mentor : http://www.scribd.com/alvin_richards

MongoSF Streaming : http://www.justin.tv/mongodb/b/286578814
Webinars : http://www.10gen.com/webinars
Original Docs : http://www.mongodb.org/display/DOCS/Home
The little mongo book and Interactive Tutorial :  http://mongly.com/
Tools to play with MongoDB :  mongolab.com ,  http://mongo.cloudfoundry.com/


CaseStudy :  Grant Goodale has the best collection of MongoDB Slides 
http://www.slideshare.net/ggoodale/geospatial-indexing-and-querying-with-mongodb