Thalasar Ventures

As data has grown exponentially in the past few years, companies have been forced to horizontally scale their data. Some have turned to the sharding of databases, while others have switched to new NoSQL data systems. There have been many debates in the last few years about SQL vs. NoSQL data management systems and which is better. However, what many have failed to grasp is how similar these systems are and how complex they can both be to run in production at a large scale.

Both of these systems represent what I call the data cloud. This data cloud is a logical data set spread across many nodes. While developers have heated debates about which system is better and how to design code around it, those in Dev/Ops usually struggle with very similar issues because the two systems are mostly the same. Both systems:

-Run across many nodes with large amounts of data flowing between them and to/from the application

-Strain the hardware of all nodes, and the network connecting them

-Maintain duplicate data across nodes for fault tolerance, and must have failover ability

-Must allow for growth by adding additional nodes

Running this data cloud in production presents a new set of challenges for DevOps, many of which are not well understood or addressed. One of the main challenges is the management and monitoring of these systems – traditional IT infrastructure monitoring or server monitoring solutions are unsuited for monitoring a complex data cloud.

When systems were smaller and you ran a single database in production, you probably had all the necessary systems in place. With a plethora of products for management, monitoring, visualizing data, and backups, it was hard not to be successful and meet your SLAs.

But now all this is much more complex once you move into the world of the data cloud. Now you have a large number of nodes, all representing the same system and still needing to meet the same SLAs as the old simple database did before. Let’s look at the challenges of running a successful data cloud:

1. Capacity planning – How many nodes do you need? How many nodes should go into each replica set? How much latency and throughput do you need in your network for the nodes to communicate fast enough?

2. Monitoring – How do you monitor dozens, hundreds or thousands of nodes all at once? How do you get a unified view of your data cloud, and then drill down to the problem nodes? Are there even any off-the-shelf monitoring tools that can help?

3. Alerting – How do you set up a common set of alerts across all nodes? And how do you keep your alert thresholds in sync as you add nodes and remove them?

4. Data visualization – How does your staff view data when it is distributed? In case of data inaccuracy, how can they quickly identify the faulty nodes and fix up the data?

5. Performance tuning – As performance degrades, how do you troubleshoot and identify the bottlenecks? How do you find which nodes are causing problems?

6. Data cloud management – How do you back up all the data? How do you make schema changes across all the nodes in one consistent step without breaking your app?

By now it should be clear that there is a lot to think about before endeavoring to launch a data cloud. Too many companies focus their energies on trying to figure out which database or NoSQL system to use and developing their applications for it. Furthermore, most vendors focus exclusively on helping scale the application – there are dozens of cloud application services , for example – but include no solutions for scaling the database. The application may turn out to be the smallest of your problems as you put a data cloud into production. Be sure you can answer all the above questions listed above before you launch.

More NoSQL Articles