Today we shipped a highly anticipated new capability with a novel approach, novel not only to Sumo Logic, but also novel within our space: Real-time Enterprise Dashboards. Dashboard technologies have been around for many years, but not all dashboard technologies are created equal. Most existing technologies leverage either precomputed summary data sets or recompute the entire data set every time a dashboard is viewed. As such, they suffer from long load times, stale information, an inability to handle the data volume.
Our customers faced a specific challenge: how to take terabytes of machine data per day, crunch it, transform it into information, and render that information in a way that supports making business and IT decisions in real time. Now they can.
When machine data is used to troubleshoot and monitor today’s production applications or infrastructure, data volume is the enemy. Large farms of Apache or IIS servers, SaaS and other applications, or data center infrastructure like VMware farms, Cisco networking gear, or Linux or Microsoft Windows server farms generate volumes of data that obey Moore’s Law: the data volume doubles every two years. It only makes sense that the volume of machine data would follow Moore’s Law – if machine computing capacity doubles, those machines do twice the work, as a result they generate twice the amount of machine data that describes that work.
This exponential growth has put existing dashboarding technologies under an insurmountable strain. Some of us here at Sumo Logic built previous-generation dashboards in our past lives. From our experience we realized that an entirely new approach is required to enable real-time monitoring and dashboarding and that realization drove development of a new architecture.
First, we adopted the cloud computing paradigm. That turned a data center into an API with lim(capacity)=∞. This enabled us to spin up and spin down additional capacity truly on demand with a single API call. Then we built our Streaming Query Engine that leverages that capacity in an elastic manner. It continuously takes data off the wire and computes results before the data ever hits its permanent resting place. This “one-time” computing is more efficient and less costly than traditional recompute methods. When you view a Sumo Logic Dashboard, you simply attach to the existing state, which is continuously computed by our Stream Query Engine in the background. What you get is freshest data available instantly enabling real-time visibility into your infrastructure or applications. And they are beautiful to boot.