Storage is one part of the puzzle. The other is querying the stored data. Easily and effectively querying data is almost mandatory for any database to be considered seriously. It can be especially important when building the operational data store for applications with which people are interacting. An RDBMS thrives on SQL support, which makes accessing and querying data easy. Standardized syntax and semantics make it an attractive choice. The fi rst chapter in this book talks about the quest for a SQL-like query language in the world of NoSQL.

Among document databases, MongoDB provides the best querying capabilities. Best is a relative term, and developers argue about what they consider superior to alternatives, but I base my judgment on three factors: similarity to SQL, an easy syntax, and an easy learning curve. CouchDB’s querying capabilities are equally powerful and rather more straightforward once you understand the concepts of views and design documents. However, the concept of views as CouchDB defines it is new and can pose initial challenges to developers.

For key/value pairs and in-memory stores, nothing is more feature-rich than Redis as far as querying capabilities go. Redis has one of the most exhaustive sets of methods available for querying the data structures it stores. To add icing to the cake, it is all nicely documented. Read about the access methods at http://redis.io/commands.

Column-family stores like HBase have little to offer as far as rich querying capabilities go. However, an associated project called Hive makes it possible to query HBase using SQL-like syntax and semantics. Hypertable defines a query language called HQL and also supports Hive.

Bringing Hive into the mix raises the question of manipulating data for operational usage versus accessing it for batch processing and business intelligence. Hive is not an interactive tool in a way SQL is to RDBMS. Hive resembles SQL in syntax but is really a way to abstract MapReduce-style manipulations. Hive allows you to use SQL like predicate-driven syntax instead of map and reduce function definitions to carry batch data manipulation operations on the data set.

Source of Information : NoSQL

• 0 comments »

RDBMS offers a consistent way of modeling data. Relational algebra underlies the data model. The theory is well established and implementation is standardized. Therefore, consistent ways of modeling and normalizing data is well understood and documented. In the NoSQL world there is no such standardized and well-defined data model. This is because all NoSQL products do not intend to solve the same problem or have the same architecture. If you need an RDBMS-centric data model for storage and querying and cannot under any circumstances step outside those definitions, just don’t use NoSQL. If, however, you are happy with SQL-type querying but can accommodate non-relational storage models, you have a few NoSQL options to choose from.

Document databases, like MongoDB, provide a gradual adoption path from formal RDBMS models to lose document-centric models. MongoDB supports SQL-like querying, rudimentary relational references, and database objects that draw a lot of inspiration from the standard table- and column based model. If relaxed schema is your primary reason for using NoSQL, then MongoDB is a great option for getting started with NoSQL.

MongoDB is used by many web-centric businesses. Foursquare is perhaps its most celebrated user. Shutterfl y, bit.ly, etsy, and sourceforge are a few other users that add feathers to MongoDB’s cap. In many of these use cases MongoDB is preferred because it supports a flexible data model and offers speedy reads and writes. Web applications often evolve rapidly and it often gets cumbersome for developers to continuously change underlying RDBMS models, especially when the changes are frequent and at times drastic. Added to the schema change challenges are the issues relating to data migration. MongoDB has good support for web framework integration. Rails, one of the most popular web application frameworks, can be used effectively with MongoDB. The data from Rails applications can be persisted via an object mapper. Therefore, MongoDB can easily be used in place of an RDBMS. Read about Rails 3 integration at www.mongodb.org/display/DOCS/Rails+3+-+Getting+Started.

For Java web developers, Spring offers fi rst-class support for MongoDB via its Spring Data project. Read more about the Spring Data Document release that supports MongoDB at www.springsource.org/node/3032. Spring Data project, in fact, adds support for a number of NoSQL products, and not just MongoDB. It integrates Spring with Redis, Riak, CouchDB, Neo4j, and Hadoop. Get more details online at the Spring Data project homepage, which is www.springsource.org/spring-data. MongoDB acts like a persistent cache, where data is kept in memory and flushed to disk as required. Therefore, MongoDB could also be thought of as an intermediate option between an RDBMS and an in-memory store or a flat file structure. Many web applications like real-time analytics, comments system, ratings storage, content management software, user data system, and event logging applications benefit from the fluid schema that MongoDB offers. Added to that, such applications enjoy MongoDB’s RDBMS-like querying capabilities and its ability to segregate data into collections that resemble tables.

Apache CouchDB is a document database alternative to MongoDB. Apache CouchDB is now available as Couchbase server, with the primary creators of CouchDB having recently merged their company, CouchOne, with Membase, Inc. Couchbase offers a packaged version of Apache CouchDB with GeoCouch and support in the form of Couchbase Server.

Couchbase Server epitomizes adherence to web standards. Couchbase’s primary interface to the data store is through RESTful HTTP interactions and is more web-centric than any database has ever been. Couchbase includes a web server as a part of the data store. It is built on top of Erlang OTP. This means you could effectively create an entire application using Couchbase. Future versions of Couchbase will be adding access to the data store through the Memcached protocol, gaining from Membase’s ability to manage speed and throughput with a working set. Couchbase also plans to scale up, growing from Membase’s elastic capabilities to seamlessly span across more nodes. Although Couchbase is very powerful and feature-rich, it has a very small footprint. Its nimble nature makes it appropriate for installation on a smartphone or an embedded device. Read more about mobile Couchbase at www.couchbase.com/products-and-services/mobile couchbase.

Couchbase models support REST-style data management. A database in CouchDB can contain JSON format documents, with additional metadata or supporting artifacts as attachments. All operations on data — create, retrieve, update, and delete — are performed via RESTful HTTP requests. Long-running complex queries across replicated Couchbase servers leverage MapReduce.


Not Just a Map
In typical in-memory databases and caches, the most well-known data structure is a map or a hash. A map stores key/value pairs and allows for fast and easy access to data. In-memory NoSQL stores provide filesystem-backed persistence of in-memory data. This means that stored data survives a system reboot. Many NoSQL in-memory databases support data structures beyond just maps, making using them for more than simple cache data extremely attractive. At the most basic level, Berkeley DB stores pairs of binary key/value pairs. The underlying store itself does not attach any metadata to the stored key/value pairs. Layers on top of basic storage, like the persistence API or the object wrappers, allow persistence of higher-level abstractions to a Berkeley DB store.

Membase, on the other hand, supports the Memcached protocol, both text and binary, and adds features around distributed replica management and consistent hashing on top of the basic key/value store. Membase also adds the ability to grow and shrink the number of servers as part of a cluster without interrupting client access. Redis takes a slightly different approach. It supports most popular data structures out of the box. In fact, it is defined as a “data structure” server. Redis supports lists, sets, sorted sets, and strings in addition to maps. Redis has even added transactionlike capabilities to specify atomicity across a number of discrete operations.

If your use case gains from using a file-backed in-memory NoSQL product, consider the supported data models to make a choice on the best fi t. In many cases, a key/value storage is enough, but if you need more than that look at Berkeley DB, Membase, and Redis. If you need a powerful and stable distributed key/value store to support large user and activity load, you are not likely to go wrong with Membase.


What about HBase and Hypertable?
In the previous section on scalability, I gave my entire vote in favor of the column-family stores. When it comes to supporting the rich data models, though, these are generally not the most favorable choices. The upfront choice of row-keys for lookup and only column-family-centric model metadata support is usually considered inadequate. With a powerful abstraction layer on top of column-family stores, a lot becomes possible.

Google started the column-family store revolution. Google also created the data modeling
abstraction on top of its column-family store for its very popular app engine. The GAE data modeling support provides rich data modeling using Python and Java. With the DataNucleus JDO and JPA support, you can use the popular object modeling abstractions in Java to persist data to HBase and Hypertable. You can also draw inspiration from the non-relational support in Django that works well on the app engine.

Source of Information : NoSQL

• 0 comments »

Transactional integrity is relevant only when data is modified, updated, created, and deleted. Therefore, the question of transactional integrity is not pertinent in pure data warehousing and mining contexts. This means that batch-centric Hadoop-based analytics on warehoused data is also not subject to transactional requirements.

Many data sets like web traffic log files, social networking status updates (including tweets or buzz), advertisement click-through imprints, road-traffic data, stock market tick data, and game scores are primarily, if not completely, written once and read multiple times. Data sets that are written once and read multiple times have limited or no transactional requirements.

Some data sets are updated and deleted, but often these modifications are limited to a single item and not a range within the data set. Sometimes, updates are frequent and involve a range operation. If range operations are common and integrity of updates is required, an RDBMS is the best choice. If atomicity at an individual item level is sufficient, then column-family databases, document databases, and a few distributed key/value stores can guarantee that. If a system needs transactional integrity but could accommodate a window of inconsistency, eventual consistency is a possibility. HBase and Hypertable offer row-level atomic updates and consistent state with the help of Paxos. MongoDB offers document-level atomic updates. All NoSQL databases that follow a master-slave replication model implicitly support transactional integrity.

Source of Information : NoSQL

• 0 comments »

Although all NoSQL databases promise horizontal scalability they don’t rise up equally to the challenge. The Bigtable clones — HBase and Hypertable — stand in front and in-memory stores, like Membase or Redis, and document databases, like MongoDB and Couchbase Server, lag behind. This difference is amplified as the data size becomes very large, especially if it grows over a few petabytes.

Bigtable and its clones promote the storage of large individual data points and large collections of data. The Bigtable model supports a large number of columns and an immensely large number of rows. The data can be sparse where many columns have no value. The Bigtable model, of course, does not waste space and simply doesn’t store cells that have no value.

Google led the column-family-centric data store revolution to store the large and ever growing web index its crawlers brought home. The Web has been growing in unbounded ways for the past several years. Google needed a store to grow with the expanding index. Therefore, Bigtable and its clones were built to scale out, limited only by the hardware available to spin off newer nodes in the cluster. Over the past few years, Google has successfully used the Bigtable model to store and retrieve a variety of data that is also very large in volume.

The HBase wiki lists a number of users on its Powered By page (http://wiki.apache.org/hadoop/Hbase/PoweredBy). Some users listed clearly testify to HBase’s capability to scale. Meetup (www.meetup.com) is a popular site that facilitates user groups and interest groups to organize local events and meetings. Meetup has grown from a small, unknown site in 2001 to 8 million members in 100 countries, 65,000+ organizers, 80,000+ meetup groups, and 50,000 meetups each week (http://online.wsj.com/article/SB10001424052748704170404575624733792905708.html). Meetup is an HBase user. All group activity is directly written to HBase and is indexed per member. A member’s custom feed is directly served from HBase.

Facebook is another big user of HBase. Facebook messaging is built on HBase. Facebook was the number one destination site on the Internet in 2010. It has grown to more than 500 million active users (www.facebook.com/press/info.php?statistics) and is the largest software application in terms of the number of users. Facebook messaging is a robust infrastructure that integrates chat, SMS, and e-mail. Hundreds of billions of messages are sent every month through this messaging infrastructure. The engineering team at Facebook shared a few notes on using HBase for their messaging infrastructure. Read the notes online at www.facebook.com/notes/facebook-engineering/the-underlyingtechnology-of-messages/454991608919.

HBase has some inherent advantages when it comes to scaling systems. HBase supports auto load balancing, failover, compression, and multiple shards per server. HBase works well with the Hadoop distributed fi lesystem (a.k.a. HDFS, which is a massively scalable distributed fi lesystem). You know from earlier chapters that HDFS replicates and automatically re-balances to easily accommodate large files that span multiple servers. Facebook chose HBase to leverage many of these features. HBase is a necessity for handling the number of messages and users they serve. The Facebook engineering notes also mention that the messages in their infrastructure are short, volatile, and temporal and are rarely accessed later. HBase, and in general Bigtable clones, are particularly suitable when ad-hoc querying of data is not important. You know that HBase supports the querying of data sets but is a weak replacement to an RBDMS as far as its querying capabilities are concerned. Infrastructures like Google App Engine (GAE) successfully expose a data modeling API, with advanced querying capabilities, on top of the Bigtable.

So it seems clear that column-family-centric NoSQL databases are a good choice if extreme scalability is a requirement. However, such databases may not be the best choice for all types of systems, especially those that involve real-time transaction processing. An RDBMS often makes a better choice than any NoSQL flavor if transactional integrity is very important. Eventually consistent NoSQL options, like Cassandra or Riak, may be workable if weaker consistency is acceptable. Amazon has demonstrated that massively scalable e-commerce operations may be a use case for eventually consistent data stores, but examples beyond Amazon where such models apply well are hard to find. Databases like Cassandra follow the Amazon Dynamo paradigm and support eventual consistency. Cassandra promises incredibly fast read and write speeds. Cassandra also supports Bigtable-like column-family-centric data modeling. Amazon Dynamo also inspired Riak. Riak supports a document store abstraction in addition to being an eventually consistent store. Both Cassandra and Riak scale well in horizontal clusters but if scalability is of paramount importance, my vote goes in favor of HBase or Hypertable over the eventually consistent stores. Perhaps places where eventually consistent stores fare better than sorted ordered column-family stores is where write throughput and latency is important. Therefore, if both horizontal scalability and high write throughput are required, possibly consider Cassandra or Riak. Even in these cases, consider a hybrid approach where you can logically partition the data write process from the access and analytics and use two separate databases for each of the tasks.

If scalability implies large data becoming available at an incredibly fast pace, for example stock market tick data or advertisement click tracking data, then column-family stores alone may not provide a complete solution. It’s prudent to store the massively growing data in these stores and manipulate them using MapReduce operations for batch querying and data mining, but you may need something more nimble for fast writes and real-time manipulation. Nothing is faster than manipulating the data in memory and so leveraging NoSQL options that keep data in memory and flush it to disk when it fi lls the available capacity are probably good choices. Both MongoDB and Redis follow this strategy. Currently, MongoDB uses mmap and Redis implements a custom mapping from memory to disk. However, both MongoDB and Redis, have actively been re-engineering their memory mapping feature and things will continue to evolve. Using MongoDB or Redis with HBase or Hypertable makes a good choice for a system that needs fast real-time data manipulation and a store for extensive data mining. Memcached and Membase can be used in place of MongoDB or Redis. Memcached and Membase act as a layer of fast and efficient cache, and therefore supplement well on top of column-family stores. Membase has been used effectively with Hadoop-based systems for such
use cases. With the merger of Membase and CouchDB, a well integrated NoSQL product with both fast cache-centric features and distributed scalable storage-centric features is likely to emerge.

Although scalability is very important if your data requirements grow to the size of Google’s or Facebook’s, not all applications become that large. Scalable systems are probably relevant for cases much smaller than these widespread systems but sometimes an attempt to make things scalable can become an exercise in over-engineering. You certainly want to avoid unnecessary complexity.

In many systems, data integrity and transactional consistency are more important than any other requirements. Is NoSQL an option for such systems?

Source of Information : NoSQL

• 0 comments »

SEO, paid search, and site search each offer a unique opportunity to understand the mind-set and the interconnected thoughts of the people who are looking for your content. The one downside to this data is that it is drawn from people who have already found you, not from people who have not found you yet. That is, in some way or another, these people have had at least one interaction with your brand.

SEO provides insights into the relevancy of your content to a specific set of terms. There really shouldn’t be many surprises when looking at what words drive people to your website if you are in a clear and defined field or if your site has a clear and defined focus. If your website is about cars, you should expect to see a great deal of car-related terms driving traffic to the site. In this case, what you are looking for is not so much themes as language and use of words, and pairings between search terms. Looking at two- or three-word search terms can give you a better idea of associations people make when looking for your type of content. This can give you insight into language use, and you may discover geographical variations as well. If your site is an exception to the rule of one thematic element—for example, if your business is involved in many disciplines or your site hosts blogs that cover a variety of topics—there may not be a simple theme that you can expect to see driving traffic to your site. In this case, you will need to look at theme clustering, or thematic grouping. You can figure out how to do this through your analytics program.

Paid search provides a testing ground for terms whose relevancy to your market or segment you are uncertain of. Perhaps they are terms that you think are thematically relevant, but you don’t have any traffic to validate this. You can use paid search to test terms and thematic elements and to validate the quality and type of traffic you get. You can also get a sampling of interaction data based on the words and terms that drive people to your site. Looking at this, you will be able to formulate more informed strategies for both your SEO and site search campaigns.

Site search allows you to see what people are looking for on your site. They know your brand, but do they have their own terms for describing your products? Looking at the terms people search for is a good way of determining whether your product naming conventions are in sync with how your customers think about your products. Site search
is a rich source of data to gather and analyze in order to further improve your SEO and paid search campaigns. Conversely, SEO can give you some ideas of where your Site Search is lacking, based on empty site search results. The three search sources can all provide feedback to each other, in both directions.

Tools you will need in this chapter:
• Clickstream tracking package (Google Analytics, Adobe SiteCatalyst, etc.)
• Spreadsheet program (Excel or something similar)
• AdWords or some other paid search tool
• A/B or multivariate testing tool (Google Website Optimizer, Test & Target, etc.)

Source of Information : MASTERING SEARCH ANALYTICS MEASURING SEO, SEM AND SITE SEARCH

• 0 comments »

ROI is a key measurement and the first thing you should look at, but how do you measure ease of use? How do you know if your online experience is a positive one for most of your customers? ROI won’t tell you these things. If you plan on growing ROI, you need to understand the factors that will help you move the ROI needle. ROI is a dollar value; further analytics are needed to understand how to optimize your ROI values. These typically come in the form of measuring the user experience. Further, not all sites derive their revenue directly from sales. Facebook, for example, generates revenue through advertising, as does Google, and both have worked to create a great user experience, striving to maximize this user experience and increase profit. There is a reason why Google is the most-used engine while Overture is a thing of the past. It’s why Facebook now rules, while MySpace is circling the drain. Connecting with users, and building experiences and communities, has helped these sites grow bigger than any competitor around them.

The goal of your analytics should be to improve the overall experience for your customers. A happy customer is a good customer. In some cases, a happy customer may simply be one who finds support quickly. There isn’t direct ROI in this case; instead, you’re looking at cost savings or customer satisfaction, both of which are part of the lifetime value of a customer. If you can show that improving self-support services on your website reduces costs, you have an ROI figure you can use. And even if it doesn’t reduce costs on the support side, it may improve long-term customer loyalty. In these cases, you may need to estimate what the long-term value of a customer is. You may not be able to get an exact ROI, but you will be able to estimate the value of improving your customer experience.

Beyond ROI, you should also be looking to use your website to foster communication and dialog with your customers. People who find your site may have come with a specific intent and found that you don’t offer what they are looking for. Running voiceof-the-customer surveys on your site can help you get feedback on what you can do to improve the site experience. Dialog may not have any direct ROI attached to it in this case, but you will get an idea of what people want, and that can drive improved ROI. You may get some ideas you can use to improve your customers’ experience. You may also find that what your customers want is not what you think they need.

ROI is a value and it helps you to know if you will be profitable or not, but it can’t tell you if customers are satisfied with the service you are providing. Ideally, customers will provide feedback to help you improve the overall experience.

As we look at ROI, keep in mind that to improve it you will need to look at many other metrics and analytics, related not only to increasing traffic to your site but also to improving the user experience once people have arrived. You need to start thinking beyond delivering users, and start to think about outcomes for users. Your job is to look at this data to figure out how to maximize profit. I see this being achieved by delivering more of the right people to the right web pages, improving what people currently do when they get to your pages, and making sure your site is as findable as possible by as many people as possible.

Source of Information : MASTERING SEARCH ANALYTICS MEASURING SEO, SEM AND SITE SEARCH

• 0 comments »