How many people are using your website right now? Which features are their favorites? Are they experiencing errors or getting stuck? How are your servers performing? Is your code easy to work with? Are you making money? Create the right dashboards and you'll have the answers to these questions all around you, all the time.
Getting and responding to feedback is one of the key principles of agile software development. It’s what ensures you’re grounded in reality and building the right thing. The latest State of DevOps Report showed that organizations that employ visual displays for monitoring and use the “monitoring data to help inform decisions” perform better and have a better culture. I’m going to show you how you can set up effective dashboards so you can easily figure out what’s really happening with your apps and users.
What to dashboard
I’m going to use the example of a fictional online book retailer, called Sue’s Books, to give you some ideas for what you could dashboard (yes, it’s a verb now) and how to do it. Here are some dashboards Sue’s Books might use.
Example 1: System load and error rates
This dashboard is similar to many production monitoring dashboards you may have seen before. It shows real-time data about the load the system is under and provides some information on failure trends and reasons. In this example, the team at Sue’s Books might be concerned that more than 10 percent of transactions are errors. Maybe the problem is related to current memory usage?
Example 2: Usage information and business metrics
This dashboard shows how you can harness data for useful business metrics. Sue always shows some standard statistics on sales and customer acquisition. She is also monitoring how two new features are doing in production: the ability to redeem vouchers (Sue’s Books sent out a bunch of them to attract new customers) and a suggestions engine, which suggests additional books to be added to a customer’s cart. From this dashboard you can see that the vouchers are bringing in more than half of new business, while the suggestions engine has a low success rate.
Example 3: Build health
Sue has a team of great developers who are constantly adding new features. At any point in time, she’d like to know whether it’s possible to ship new features to production. This dashboard gives her a view of whether any regression bugs have been introduced and how healthy her applications are overall.
The benefits of dashboards
It’s a well-worn saying, but knowledge is power. Unfortunately, many leaders make decisions in a vacuum, based purely on their own ideas. The biggest benefit of effective dashboarding is that you’re able to make decisions that are informed by data that reflects reality. From the example usage data for Sue’s Books, Sue might decide that sending out more vouchers is a good business move, because they’re bringing in more business. She might also set up some further monitoring to see whether customers who sign up because they have a voucher come back to purchase again. Sue may also decide to dedicate resources to improving the suggestions engine, or perhaps scrap it altogether.
You’ll sometimes be surprised by what you learn about your users’ behavior. You may see them using the site in a way you don’t expect and think it’s a good idea to offer more of that kind of functionality. You may see them getting stuck on a particular screen and decide to improve the form layout. These are called observed requirements, because the ideas arise from what you see happening in production and have the benefit of arising from actual usage.
Another great benefit from effective dashboarding is the ability to see problems as they’re happening, not when your customers phone to complain. Imagine if Sue’s Books had a dashboard that showed repeat errors for a single customer in a short period of time. Sue could actually phone the customer having trouble entering their payment information and offer them some assistance, before the customer is forced to phone in or give up and shop elsewhere. This kind of proactive customer support is rare and can give your business a real edge. Dashboards can also help you get into the habit of practicing early warning and recovery in production. When you see a spike in CPU usage or a sudden drop in transaction rates, you have the opportunity to investigate and fix an issue before any customers are affected.
Getting the right data
Dashboards are a mechanism you can use to display data more effectively, but they’re pretty useless if you don’t have the right data. Luckily, sourcing the data you need might be easier than you think.
Many of the applications you’re using will already expose an API that you can mine for data. If you’re using a production monitoring solution like New Relic, you can make API requests to obtain information like how hard your servers are working. All the popular continuous integration servers have APIs you can use to retrieve information about the status and health of your builds. Popular web analytics solutions, like Google Analytics, can also be queried to display metrics like page views per hour.
Another great technique is to extract data from your application logs or database. To get an idea of which errors are most common, you can make sure your application logs all errors and exceptions as they happen. You can then use a combination of Elasticsearch, Logstash and Kibana (also known as the ELK stack) to make your log data available in a highly searchable database. You can use Kibana to create a dashboard that interrogates this data, or create your own solution by calling the Elasticsearch API. You can even pull data straight from your application database, but remember to keep performance in mind, since your queries will add additional load.
These techniques aren’t restricted to technical information like exception rates—you can also get valuable data on your business. If you logged every payment you received, you would be able to query that data to obtain statistics on payment trends and answer questions like, “Are we more profitable this month than last month?” It’s important to remember and respect your users’ privacy when logging information about their behavior, though. Use the principle of Datensparsamkeit: Keep only the information you really need. Don’t log any sensitive or personal information—all you need is an anonymous, aggregated view.
How to create a dashboard
The good news is that you’ve already done the hard part. Once you have the data, there are some useful tools you can leverage to display it. Many applications come with built-in dashboards. Kibana is a great tool for visualizing information in Elasticsearch. Splunk is a structured logging tool that also comes with some useful ways to view your data. Many web analytics and monitoring products come with built-in dashboards that are ready to use.
My personal preference is to compose dashboards with data from multiple sources. This allows you to group data by its importance to you, rather than by its origin. The example dashboards in this article have been put together using an open source tool called Dashing, which provides some useful visualizations out of the box and allows you the freedom to collect data from multiple sources using plain Ruby code. The build health example in this article uses Build Window, which is built with Dashing.
Now that you have created beautiful dashboards that display useful data, the next step is to surround yourself with them. Set up big monitors or even TV screens around your workspace, enabling everyone to get feedback from your users and make decisions based on what’s really happening.
Keep learning
Take a deep dive into the state of quality with TechBeacon's Guide. Plus: Download the free World Quality Report 2022-23.
Put performance engineering into practice with these top 10 performance engineering techniques that work.
Find to tools you need with TechBeacon's Buyer's Guide for Selecting Software Test Automation Tools.
Discover best practices for reducing software defects with TechBeacon's Guide.
- Take your testing career to the next level. TechBeacon's Careers Topic Center provides expert advice to prepare you for your next move.
