Chapter 14: Performance, Monitoring, and Troubleshooting

Cohesity Certified Architect Expert (CCAE) — Interactive Study Guide

If the previous chapters explained how to design and operate a Cohesity estate when everything is healthy, this chapter is about what to do when it is not. The CCAE-level architect is expected to localize bottlenecks, generate the right diagnostic artifacts, integrate alerts into operational tooling, and engage Cohesity Support effectively when escalation is warranted. The discipline that ties all of this together is structured triage: walk the data path end to end, measure each segment, and find the lowest sustained throughput point.

Section 1: Performance Bottleneck Analysis

The Five-Stage Data Path

A Cohesity backup is a multi-stage assembly line. Bytes are read from a source, pushed through the network, ingested by the cluster, journaled into NVRAM, and finally destaged onto SSD or HDD by the writer service. Each stage has a maximum sustainable throughput, and the slowest stage sets the throughput of the whole line.

[Source App/VM] -> [Network/Proxy] -> [Cluster Ingest] -> [NVRAM Journal] -> [Writer -> SSD/HDD]
     Stage 1            Stage 2            Stage 3              Stage 4            Stage 5

Symptom Patterns by Stage

Bottleneck Triage Decision Tree

The SOBR Performance Trap (140 vs 20-30 MB/s)

When third-party tools like Veeam present Cohesity as a Scale-Out Backup Repository, throughput can collapse from ~140 MB/s on a single-node Cohesity repository to 20-30 MB/s once SOBR is enabled with the default placement policy. The remediation is to set SOBR placement to Performance rather than Data Locality, restoring parallel ingest across nodes.

Worked Example: Backup Running at 20 MB/s

1. Network baseline. iperf3 from vSphere proxy to three node primary IPs: 9.3 Gb/s. Network healthy. Stage 2 ruled out.
2. Job stats. Source read 22 MB/s, writer 21 MB/s, writer latency normal, NVRAM not saturated. Reads and writes balanced and low — cluster is not being fed.
3. Source check. vCenter shows backing array queue depth 95%, read latency 40 ms.
4. Conclusion. A SQL re-index is contending with the source array. Reschedule the protection group outside the re-index window. Do not open a Cohesity Support case — Cohesity is not the limiter.

Section 2: Monitoring and Alerting

Helios is Cohesity's primary alerting plane. It aggregates events from every registered cluster and fans them out via filterable notification rules to email, SNMP, syslog, and webhook channels. Alerts are categorized (cluster health, protection, replication, capacity, security, hardware) and tagged by severity (Critical, Warning, Info).

Configuring Alert Notification Rules

1. Navigate to Health > Notification in Helios/DataProtect.
2. Click Create > New Alert Notification Rule.
3. Set rule name and filters (category, severity, alert name, cluster scope).
4. Choose delivery method: email, SNMP, syslog, or webhook.
5. For email: specify To, Cc, Subject. Save.

Programmatically the endpoint is createAlertNotificationRule. The same alert can fan to multiple channels — create one rule per channel, or different filters per channel for tiered routing.

Email/SMTP Configuration

Validation Pattern (Memorize)

1. Configure SMTP/SNMP/syslog/webhook targets.
2. Create an alert notification rule with tight filters.
3. Trigger or simulate a matching event.
4. Confirm receipt at the email inbox / NMS / syslog / webhook endpoint.
5. Run POST /validate on SMTP to catch silent relay breakage.

SLA Reports

Beyond alerts, Helios provides SLA reports showing protection compliance per protection group, source, and cluster — the percentage of protected objects that met RPO over a reporting window. These surface chronic drift no individual alert reveals: a group sliding from 99.5% to 96% over six weeks is missing windows even though every job "succeeded" eventually.

Section 3: Logs and Diagnostics

iris_cli: The Supported CLI

iris_cli -server <cluster-IP> -username=admin -password=<pwd>

For the exam, iris_cli is the CLI to name when a question asks about cluster-side actions. Common groups: cluster status, cluster nodes list, protection-runs list, protection-jobs list, plus stats queries useful during triage.

Service Logs — Pick the Right Service

Generating a Log Bundle via Siren

Reach Siren at https://<cluster-VIP>/siren and click Cluster Support Bundle. The dialog scopes:

• Nodes — uncheck "Select all" to scope to specific node IPs
• Services — pick only relevant services (iris, Bridge, Magneto, etc.)
• Log level — verbosity threshold
• Time range — defaults to last 24 hours
• Include hardware logs — firmware, IPMI/BMC, SMART, chassis events

Bundles land at /home/cohesity/data/timecapsules. Sizes range from a few MB up to 2-3 GB. Upload via uploadFilePackage API or the Support case URL.

Heartbeat: The Continuous Diagnostic

Cohesity clusters emit a Heartbeat stream — a continuous, lightweight diagnostic feed reporting cluster health, version, configuration, and key metrics back to Cohesity. Heartbeat enables Proactive Support to spot brewing issues before they cause outages. Architects must keep Heartbeat egress (HTTPS/443) open or proactive support is blind.

Practical Bundle Hygiene

• Scope tightly. Minimum services, narrowest time window.
• Capture both healthy and unhealthy nodes for cluster-wide issues.
• Record context separately — UTC timestamps, job IDs, change events.
• Audit logs capture admin actions; forward to syslog/SIEM for HIPAA/PCI/FedRAMP.

Section 4: Common Failure Modes

Failure Modes Taxonomy

Backup Job Failures

Replication Lag

Replication lag is the canonical "silent" DR failure: protection succeeds locally but replication cannot keep up. Causes include WAN bandwidth insufficient for daily change rate, throttling windows misaligned with change rates, target ingest saturated, or compressed/encrypted replication competing for CPU on undersized clusters. Recovery requires a window where ingest exceeds change rate — not instant.

Disk and Node Failures

Cluster Network Partition

The most dangerous failure mode. SpanFS uses Paxos-based metadata with strict consistency — the side without quorum cannot serve writes. Detection: Heartbeat alerts, node-up/node-down events, intra-cluster latency spikes, NTP drift warnings (often the first symptom).

1. Identify partition boundary using iris_cli cluster status from multiple nodes.
2. Check physical network — switch, uplink, VLAN.
3. If healed quickly, cluster auto-recovers; otherwise generate Siren bundle scoped to Bridge, Apollo, Gandalf and engage Support before any manual remediation.
4. Document and review fault domain design.

Chapter Summary & Three Drills

For the exam, internalize three drills:

1. Bottleneck triage drill — given a slow-backup symptom, walk Source → Network → Ingest → NVRAM → Writer and name the metric that proves the verdict.
2. Log bundle drill — name Siren, the /home/cohesity/data/timecapsules path, and the four scoping inputs (nodes, services, time range, hardware logs).
3. Helios alerting drill — name the four channels (email, SNMP, syslog, webhook), the SMTP API PUT /v2/clusters/smtp, the validation step POST /validate, and the rule creator createAlertNotificationRule.