Lightweight Persistence Service for Dynamic Entities

Goal :

• Persist incoming stream of Objects of into the Database.
• Parse the incoming XML/Json data to extract "Object of similar type" (i.e. a particular Business Entity) .
• The Business Entity may have a fixed set of attributes or a dynamically changing set of attributes.
• We should not create/maintain ORM layer and no need for POJO conversions from/to wsdl/http streams.
• We should keep the  persistence service layer very very light-weight.

Setup :

• mongodb2.0.1 (A database named "staging" needs to be created in MongoDB)
• spring-data-mongodb 1.0.0.M4, mongo-java-driver 2.5.2 (and all other dependencies specified in pom.xml)

Development Environment : SpringSource Tool Suite - 2.8.1.RELEASE

Implementation Approach :

• SimpleJdbcInsert Command make things really simple when Objects of same type has a fixed set of attributes.

Create the singleton instance of the SimpleJdbcInsert Command for the corresponding table.Prepare the Map of key-value  pairs by parsing the incoming data and simply push the data into the tables against the corresponding columns (keys in the map).

Number num = insertCommand.executeAndReturnKey(parameters);
Or insertCommand.executeBatch(…)  

• It may be tweaked to leverage DatabaseMetadata and perform DDL on RDBMS programmatically.
• But its very costly to Create/Update/Delete columns in relational data-store on-the-fly.
• So finally we take resort to MongoDB.
• If the Business Entity has a pre-defined set of attributes we should leverage the well-known annotated document object.
• Otherwise if Entity structure is changing constantly to accommodate newer attributes, then just create new DBOnjects and DBRefs.

MongoConfig creates a MongoTemplate against the database "staging" MongoDataAccessService provides all CRUD operations.MongoRestController specifies all the Rest operations.

Sneak Peak into the code .. org.jdom.Element rootElement = jdomDocument.getRootElement(); String rootElemName = rootElement.getName();

    if (!mongoDataAccessService.collectionExists(rootElemName)) {
        mongoDataAccessService.createCollection(rootElemName);
    }
    // we assume the element contains all the values for the table-columns
    // as the attributes.
    List list = rootElement.getAttributes();
    DBObject parentDocument = new BasicDBObject();
    // suppress _ClassName in the DbObject
    for (Object attr : list) {
        org.jdom.Attribute source = (org.jdom.Attribute) attr;
        if (source.getName().equals("Id")) {
            parentDocument.put("_id", source.getValue()); // use the original incoming Id
            // candidate for shard key -- TODO
        } else {
            parentDocument.put(source.getName(), source.getValue());
        }
    }
    // Now iterate through the Child Elements and store them as Database
    // References
    DBRef childDocumentRef = new DBRef(
            mongoDataAccessService.getDB("staging"), elementName, elementId);
    parentDocument.put(elementName + "_ref", childDocumentRef);

Runtime Steps :

• target folder already contains a pre-built war file.
• just drop the war inside an app server context root.
• http://localhost:8080/dynamic-data-collection/mongo/entities/add
• Run the MongoRestClient to dynamically extract an entity from XML fragment and add it to the corresponding Collection.
• Simple MongoClient just creates instance of a predefined Mongo Document. This comes handy when Entity has a fixed set of attributes.
• 
  

Code :  https://github.com/kaniska/Dynamic-Object-Persistence 

Take Away :

• Analyze the Business requirements and accordingly categorize the types of Entities.
• If all Entities can have pre-defined schema then use SimpleJdbcInsert Command along with MySQL / Oracle for best performance.
• But in case the Business Process consists of both fixed and dynamic entities, then consider the following approach.
• Define annotated POJO (Mongo Document) for the Entities having a fixed structure.
• If same type of Entity has different set of attributes for different entities then consider creating them dynamically.
• So the choice of database (MongoDB Vs RDBMS) - depends upon how much dynamism one needs to accommodate in the application.

Future Improvements

1. Externalize the MongoDB configuration in external properties file. (Tried but somehow it did not work.)
2. Use the latest Spring-MongoDB release
3. Work with XPathOperations instead of parsing xml fragments manually.
4. Figure out if Spring-Data MongoDB API leverages latest Morphia annotations for journal-sync, suppressing _ClassName in objects etc.
   Reference : http://www.springsource.org/spring-data/mongodb