AppD Custom Metrics

In this tutorial, I will introduce how to publish custom metrics from a server into the AppDynamics controller. Let's start with what exactly AppDynamics is why we may use it. AppDynamics (or AppD for short) is an agent-based (i.e. node-local process that ships metrics to a centralized controller) enterprise monitoring tool that specializes in Application Performance Monitoring (or APM). It does this by converting various different application aspects into what AppD calls "business transactions". These business transactions can be API calls, database queries, logins, server restarts, auto-scaling invocations, and much more. It then compares these events against a baseline of events that it measures and compares against over the lifetime of monitoring the application. Therefore, the longer you have AppD installed, the smarter it becomes about your application and the more refined its alerting and remediation capabilities can be.

AppD can typically determine business transactions if you just give it a few hours to observe the application - or even better, a full business week to cover all load differentials that could arise throughout the week. However, sometimes what the tool determines as "important" is actually not, or it misses some key aspects of your application you want to capture. In these cases, we can actually use AppDynamic's custom metrics feature to write scripts locally on our nodes and export the metrics we want to the AppD contoller. Even if your metrics aren't initially found by AppD, it's still important to ship your custom metrics to the controller to live with all of your other measurements. With all of the metrics in one place, we can still trend against baselines and alert on abnormal values whether the metric was picked up by AppD initially or had been pushed through a custom script.

AppD has great documentation for custom metrics, but I hope to simplify and cut down some of the noise in the following tutorial. In order for AppD to monitor our application or servers, we need to install at least the machine agent and if we want even more application monitoring, we can install the process-specific agent provided by AppD (i.e. the Java agent for monitoring Java processes).

The installation of these monitoring agents is outside the scope of this tutorial, but the documentation I linked should be sufficient to get them up and running. Our custom metrics will ship through the machine agent, so it's important to have that installed. Another vital piece to shipping our metrics is the directory structure of our scripts and configurations. Within the same directory that the AppD machine agent is installed in will be a /monitors sub-directory. If this does not exist, feel free to create it. In the /monitors directory, there will need to be another sub-directory for each custom metric we want to ship. As an example, we will set up all of the files required to create a custom metric that returns a count of the number of JVMs currently running on the system. So, we will have a directory structure of:

<appd_machine_agent_directory>/monitors/jvm-count

Inside our custom directory, we will have three files (again, three files per metric we want to ship): config.yml, monitor.xml, and our main script, which we will call jvm-count.sh. Typically it's best practice to name this script the same as the directory it is located in, and that will be the same name we will find in the AppDynamics controller. monitor.xml will tell the machine agent how to call our script and what metadata to associate with it. Theconfig.yml file will hold very specific configurations for how the script runs such as number of threads to execute, script execution delays, etc. Finally, the jvm-count.sh will be our script that echoes our metric to STDOUT for the AppD machine agent to pick up and ship to the controller.

The configuration script located at <appd_machine_agent_directory>/monitors/jvm-count/config.yml:

metricPrefix: "Custom Metrics|"
numberOfThreads: 1
taskDelaySeconds: 300

The monitoring metadata script located at <appd_machine_agent_directory>/monitors/jvm-count/monitor.xml:

<monitor>
  <name>JVMCount</name>
    <type>managed</type>
      <description>Count of JVMs running on system</description>
      <monitor-configuration></monitor-configuration>
      <monitor-run-task>
        <execution-style>periodic</execution-style>
        <execution-frequency-in-seconds>60</execution-frequency-in-seconds>
        <name>Run</name>
        <type>executable</name>
        <task-arguments>
          <!-- Point this to your config.yml file -->
          <argument name="config-file" is-required="true" default-value="monitors/jvm-count/config.yml"/>
        </task-arguments>
        <executable-task>
          <type>file</type>
          <!-- Use only one file element per os-type. Use "windows" or "mac" if those hosts apply -->
            <file os-type="linux">jvm-count.sh</file>
        </executable-task>
      </monitor-run-task>
</monitor>

The custom metric capturing script located at <appd_machine_agent_directory>/monitors/jvm-count/jvm-count.sh:

# Node name grabbed dynamically from the node's machine agent config
NODE_NAME=$(grep -oP '(?<=<tier-name>).*?(?=</tier-name>)' ../../conf/controller-info.xml)

# Name of metric to show in AppDynamics, grabbed from script name
METRIC_NAME=$(basename "$0" .sh)

# Metric value that script is capturing, placed in "$metric" variable, here is the count of JVMs running on the system
metric=$(ps aux | grep -e java | wc -l)

# Echo metric to STDOUT for AppDynamics machine agent to pick up
echo "name=Custom Metrics|${METRIC_NAME}|${NODE_NAME}|count,value=$metric,aggregator=OBSERVATION,time-rollup=CURRENT,cluster-rollup=INDIVIDUAL"

Now, those final parameters I appended to the end of my echo to STDOUT are outlined in the AppDynamics custom metric documentation, but these are probably the best settings for metrics that are reported on in real-time, which is most custom monitoring.

After a quick restart of the machine agent, we will find our new custom metrics in the AppDynamics metric browser, ready to be analyzed and alerted on.

Thanks for reading! If you have any questions, feel free to reach out to me on Twitter

Pushing AppD Custom Metrics