HackTricks Cloud
HackTricks Training Twitter Linkedin Sponsor

AWS - CloudWatch Enum

Support HackTricks

CloudWatch

CloudWatch collects monitoring and operational data in the form of logs/metrics/events providing a unified view of AWS resources, applications and services. CloudWatch Log Event have a size limitation of 256KB on each log line. It can set high resolution alarms, visualize logs and metrics side by side, take automated actions, troubleshoot issues, and discover insights to optimize applications.

You can monitor for example logs from CloudTrail. Events that are monitored:

Key concepts

Namespaces

A namespace is a container for CloudWatch metrics. It helps to categorize and isolate metrics, making it easier to manage and analyze them.

  • Examples: AWS/EC2 for EC2-related metrics, AWS/RDS for RDS metrics.

Metrics

Metrics are data points collected over time that represent the performance or utilization of AWS resources. Metrics can be collected from AWS services, custom applications, or third-party integrations.

  • Example: CPUUtilization, NetworkIn, DiskReadOps.

Dimensions

Dimensions are key-value pairs that are part of metrics. They help to uniquely identify a metric and provide additional context, being 30 the most number of dimensions that can be associated with a metric. Dimensions also allow to filter and aggregate metrics based on specific attributes.

  • Example: For EC2 instances, dimensions might include InstanceId, InstanceType, and AvailabilityZone.

Statistics

Statistics are mathematical calculations performed on metric data to summarize it over time. Common statistics include Average, Sum, Minimum, Maximum, and SampleCount.

  • Example: Calculating the average CPU utilization over a period of one hour.

Units

Units are the measurement type associated with a metric. Units help to provide context and meaning to the metric data. Common units include Percent, Bytes, Seconds, Count.

  • Example: CPUUtilization might be measured in Percent, while NetworkIn might be measured in Bytes.

CloudWatch Features

Dashboard

CloudWatch Dashboards provide customizable views of your AWS CloudWatch metrics. It is possible to create and configure dashboards to visualize data and monitor resources in a single view, combining different metrics from various AWS services.

Key Features:

  • Widgets: Building blocks of dashboards, including graphs, text, alarms, and more.

  • Customization: Layout and content can be customized to fit specific monitoring needs.

Example Use Case:

  • A single dashboard showing key metrics for your entire AWS environment, including EC2 instances, RDS databases, and S3 buckets.

Metric Stream and Metric Data

Metric Streams in AWS CloudWatch enable you to continuously stream CloudWatch metrics to a destination of your choice in near real-time. This is particularly useful for advanced monitoring, analytics, and custom dashboards using tools outside of AWS.

Metric Data inside Metric Streams refers to the actual measurements or data points that are being streamed. These data points represent various metrics like CPU utilization, memory usage, etc., for AWS resources.

Example Use Case:

  • Sending real-time metrics to a third-party monitoring service for advanced analysis.

  • Archiving metrics in an Amazon S3 bucket for long-term storage and compliance.

Alarm

CloudWatch Alarms monitor your metrics and perform actions based on predefined thresholds. When a metric breaches a threshold, the alarm can perform one or more actions such as sending notifications via SNS, triggering an auto-scaling policy, or running an AWS Lambda function.

Key Components:

  • Threshold: The value at which the alarm triggers.

  • Evaluation Periods: The number of periods over which data is evaluated.

  • Datapoints to Alarm: The number of periods with a reached threshold needed to trigger the alarm

  • Actions: What happens when an alarm state is triggered (e.g., notify via SNS).

Example Use Case:

  • Monitoring EC2 instance CPU utilization and sending a notification via SNS if it exceeds 80% for 5 consecutive minutes.

Anomaly Detectors

Anomaly Detectors use machine learning to automatically detect anomalies in your metrics. You can apply anomaly detection to any CloudWatch metric to identify deviations from normal patterns that might indicate issues.

Key Components:

  • Model Training: CloudWatch uses historical data to train a model and establish what normal behavior looks like.

  • Anomaly Detection Band: A visual representation of the expected range of values for a metric.

Example Use Case:

  • Detecting unusual CPU utilization patterns in an EC2 instance that might indicate a security breach or application issue.

Insight Rules and Managed Insight Rules

Insight Rules allow you to identify trends, detect spikes, or other patterns of interest in your metric data using powerful mathematical expressions to define the conditions under which actions should be taken. These rules can help you identify anomalies or unusual behaviors in your resource performance and utilization.

Managed Insight Rules are pre-configured insight rules provided by AWS. They are designed to monitor specific AWS services or common use cases and can be enabled without needing detailed configuration.

Example Use Case:

  • Monitoring RDS Performance: Enable a managed insight rule for Amazon RDS that monitors key performance indicators such as CPU utilization, memory usage, and disk I/O. If any of these metrics exceed safe operational thresholds, the rule can trigger an alert or automated mitigation action.

CloudWatch Logs

Allows to aggregate and monitor logs from applications and systems from AWS services (including CloudTrail) and from apps/systems (CloudWatch Agent can be installed on a host). Logs can be stored indefinitely (depending on the Log Group settings) and can be exported.

Elements:

Log Group

A collection of log streams that share the same retention, monitoring, and access control settings

Log Stream

A sequence of log events that share the same source

Subscription Filters

Define a filter pattern that matches events in a particular log group, send them to Kinesis Data Firehose stream, Kinesis stream, or a Lambda function

CloudWatch Monitoring & Events

CloudWatch basic aggregates data every 5min (the detailed one does that every 1 min). After the aggregation, it checks the thresholds of the alarms in case it needs to trigger one. In that case, CLoudWatch can be prepared to send an event and perform some automatic actions (AWS lambda functions, SNS topics, SQS queues, Kinesis Streams)

Agent Installation

You can install agents inside your machines/containers to automatically send the logs back to CloudWatch.

  • Create a role and attach it to the instance with permissions allowing CloudWatch to collect data from the instances in addition to interacting with AWS systems manager SSM (CloudWatchAgentAdminPolicy & AmazonEC2RoleforSSM)

  • Download and install the agent onto the EC2 instance (https://s3.amazonaws.com/amazoncloudwatch-agent/linux/amd64/latest/AmazonCloudWatchAgent.zip). You can download it from inside the EC2 or install it automatically using AWS System Manager selecting the package AWS-ConfigureAWSPackage

  • Configure and start the CloudWatch Agent

A log group has many streams. A stream has many events. And inside of each stream, the events are guaranteed to be in order.

Enumeration

# Dashboards #

## Returns a list of the dashboards of your account
aws cloudwatch list-dashboards

## Retrieves the details of the specified dashboard
aws cloudwatch get-dashboard --dashboard-name <value>

# Metrics #

## Returns a list of the specified metric
aws cloudwatch list-metrics [--namespace <value>] [--metric-name <value>] [--dimensions <value>] [--include-linked-accounts | --no-include-linked-accounts]

## Retrieves metric data (this operation can include a CloudWatch Metrics Insights query, and one or more metric math functions)
aws cloudwatch get-metric-data --metric-data-queries <value> --start-time <value> --end-time <value>

## Retrieves statistics for the specified metric and namespace over a range of time
aws cloudwatch get-metric-statistics --namespace <value> --metric-name <value> [--dimensions <value>] --start-time <value> --end-time <value> --period <value>

## Returns a list of the metric streams of your account
aws cloudwatch list-metric-streams 

## Retrieves information about the specified metric stream
aws cloudwatch get-metric-stream --name <value>

## Retrieve snapshots of the specified metric widgets
aws cloudwatch get-metric-widget-image --metric-widget <value>

# Alarms #

## Retrieves the specified alarm
aws cloudwatch describe-alarms [--alarm-names <value>] [--alarm-name-prefix <value>] [--alarm-types <value>] [--state-value <value>]

## Retrieves the alarms history, even for deleted alarms
aws cloudwatch describe-alarm-history [--alarm-name <value>] [--alarm-types <value>] [--history-item-type <ConfigurationUpdate | StateUpdate | Action>] [--start-date <value>] [--end-date <value>]

## Retrieves standard alarms based on the specified metric
aws cloudwatch escribe-alarms-for-metric --metric-name <value> --namespace <value> [--dimensions <value>]

# Anomaly Detections #

## Lists the anomaly detection models that you have created in your account
aws cloudwatch describe-anomaly-detectors [--namespace <value>] [--metric-name <value>] [--dimensions <value>]

## Lists all the Contributor Insight rules in your account
aws cloudwatch describe-insight-rules

## Retrieves the data collected over a time range for a given Contributor Insight rule	
aws cloudwatch get-insight-rule-report --rule-name <value> --start-time <value> --end-time <value> --period <value>

## Lists managed Contributor Insights rules in your account for a specified resource
aws cloudwatch list-managed-insight-rules --resource-arn <value>

# Tags #

## Lists the tags associated with the specified CloudWatch resources
aws cloudwatch list-tags-for-resource --resource-arn <value>

# CloudWatch Logs #
aws logs tail "<log_group_name>" --followaws logs get-log-events --log-group-name "<log_group_name>" --log-stream-name "<log_stream_name>" --output text > <output_file>

# CloudWatch Events #
aws events list-rules
aws events describe-rule --name <name>aws events list-targets-by-rule --rule <name>aws events list-archives
aws events describe-archive --archive-name <name>aws events list-connections
aws events describe-connection --name <name>aws events list-endpoints
aws events describe-endpoint --name <name>aws events list-event-sources
aws events describe-event-source --name <name>aws events list-replays
aws events list-api-destinations
aws events list-event-buses

Post-Exploitation / Bypass

cloudwatch:DeleteAlarms,cloudwatch:PutMetricAlarm , cloudwatch:PutCompositeAlarm

An attacker with this permissions could significantly undermine an organization's monitoring and alerting infrastructure. By deleting existing alarms, an attacker could disable crucial alerts that notify administrators of critical performance issues, security breaches, or operational failures. Furthermore, by creating or modifying metric alarms, the attacker could also mislead administrators with false alerts or silence legitimate alarms, effectively masking malicious activities and preventing timely responses to actual incidents.

In addition, with the cloudwatch:PutCompositeAlarm permission, an attacker would be able to create a loop or cycle of composite alarms, where composite alarm A depends on composite alarm B, and composite alarm B also depends on composite alarm A. In this scenario, it is not possible to delete any composite alarm that is part of the cycle because there is always still a composite alarm that depends on that alarm that you want to delete.

aws cloudwatch put-metric-alarm --cli-input-json <value> | --alarm-name <value> --comparison-operator <value> --evaluation-periods <value> [--datapoints-to-alarm <value>] [--threshold <value>] [--alarm-description <value>] [--alarm-actions <value>] [--metric-name <value>] [--namespace <value>] [--statistic <value>] [--dimensions <value>] [--period <value>]
aws cloudwatch delete-alarms --alarm-names <value>
aws cloudwatch put-composite-alarm --alarm-name <value> --alarm-rule <value> [--no-actions-enabled | --actions-enabled [--alarm-actions <value>] [--insufficient-data-actions <value>] [--ok-actions <value>] ]

The following example shows how to make a metric alarm ineffective:

  • This metric alarm monitors the average CPU utilization of a specific EC2 instance, evaluates the metric every 300 seconds and requires 6 evaluation periods (30 minutes total). If the average CPU utilization exceeds 60% for at least 4 of these periods, the alarm will trigger and send a notification to the specified SNS topic.

  • By modifying the Threshold to be more than 99%, setting the Period to 10 seconds, the Evaluation Periods to 8640 (since 8640 periods of 10 seconds equal 1 day), and the Datapoints to Alarm to 8640 as well, it would be necessary for the CPU utilization to be over 99% every 10 seconds throughout the entire 24-hour period to trigger an alarm.

{
  "Namespace": "AWS/EC2",
  "MetricName": "CPUUtilization",
  "Dimensions": [
    {
      "Name": "InstanceId",
      "Value": "i-01234567890123456"
    }
  ],
  "AlarmActions": [
    "arn:aws:sns:us-east-1:123456789012:example_sns"
  ],
  "ComparisonOperator": "GreaterThanThreshold",
  "DatapointsToAlarm": 4,
  "EvaluationPeriods": 6,
  "Period": 300,
  "Statistic": "Average",
  "Threshold": 60,
  "AlarmDescription": "CPU Utilization of i-01234567890123456 over 60%",
  "AlarmName": "EC2 instance i-01234567890123456 CPU Utilization"
}

Potential Impact: Lack of notifications for critical events, potential undetected issues, false alerts, suppress genuine alerts and potentially missed detections of real incidents.

cloudwatch:DeleteAlarmActions, cloudwatch:EnableAlarmActions , cloudwatch:SetAlarmState

By deleting alarm actions, the attacker could prevent critical alerts and automated responses from being triggered when an alarm state is reached, such as notifying administrators or triggering auto-scaling activities. Enabling or re-enabling alarm actions inappropriately could also lead to unexpected behaviors, either by reactivating previously disabled actions or by modifying which actions are triggered, potentially causing confusion and misdirection in incident response.

In addition, an attacker with the permission could manipulate alarm states, being able to create false alarms to distract and confuse administrators, or silence genuine alarms to hide ongoing malicious activities or critical system failures.

  • If you use SetAlarmState on a composite alarm, the composite alarm is not guaranteed to return to its actual state. It returns to its actual state only once any of its children alarms change state. It is also reevaluated if you update its configuration.

aws cloudwatch disable-alarm-actions --alarm-names <value>
aws cloudwatch enable-alarm-actions --alarm-names <value>
aws cloudwatch set-alarm-state --alarm-name <value> --state-value <OK | ALARM | INSUFFICIENT_DATA> --state-reason <value> [--state-reason-data <value>]

Potential Impact: Lack of notifications for critical events, potential undetected issues, false alerts, suppress genuine alerts and potentially missed detections of real incidents.

cloudwatch:DeleteAnomalyDetector, cloudwatch:PutAnomalyDetector

An attacker would be able to compromise the ability of detection and respond to unusual patterns or anomalies in metric data. By deleting existing anomaly detectors, an attacker could disable critical alerting mechanisms; and by creating or modifying them, it would be able either to misconfigure or create false positives in order to distract or overwhelm the monitoring.

aws cloudwatch delete-anomaly-detector [--cli-input-json <value> | --namespace <value> --metric-name <value> --dimensions <value> --stat <value>]
aws cloudwatch put-anomaly-detector [--cli-input-json <value> | --namespace <value> --metric-name <value> --dimensions <value> --stat <value> --configuration <value> --metric-characteristics <value>]

The following example shows how to make a metric anomaly detector ineffective. This metric anomaly detector monitors the average CPU utilization of a specific EC2 instance, and just by adding the “ExcludedTimeRanges” parameter with the desired time range, it would be enough to ensure that the anomaly detector does not analyze or alert on any relevant data during that period.

{
    "SingleMetricAnomalyDetector": {
        "Namespace": "AWS/EC2",
        "MetricName": "CPUUtilization",
        "Stat": "Average",
        "Dimensions": [
            {
                "Name": "InstanceId",
                "Value": "i-0123456789abcdefg"
            }
        ]
    }
}

Potential Impact: Direct effect in the detection of unusual patterns or security threats.

cloudwatch:DeleteDashboards, cloudwatch:PutDashboard

An attacker would be able to compromise the monitoring and visualization capabilities of an organization by creating, modifying or deleting its dashboards. This permissions could be leveraged to remove critical visibility into the performance and health of systems, alter dashboards to display incorrect data or hide malicious activities.

aws cloudwatch delete-dashboards --dashboard-names <value>
aws cloudwatch put-dashboard --dashboard-name <value> --dashboard-body <value>

Potential Impact: Loss of monitoring visibility and misleading information.

cloudwatch:DeleteInsightRules, cloudwatch:PutInsightRule ,cloudwatch:PutManagedInsightRule

Insight rules are used to detect anomalies, optimize performance, and manage resources effectively. By deleting existing insight rules, an attacker could remove critical monitoring capabilities, leaving the system blind to performance issues and security threats. Additionally, an attacker could create or modify insight rules to generate misleading data or hide malicious activities, leading to incorrect diagnostics and inappropriate responses from the operations team.

aws cloudwatch delete-insight-rules --rule-names <value>
aws cloudwatch put-insight-rule --rule-name <value> --rule-definition <value> [--rule-state <value>]
aws cloudwatch put-managed-insight-rules --managed-rules <value>

Potential Impact: Difficulty to detect and respond to performance issues and anomalies, misinformed decision-making and potentially hiding malicious activities or system failures.

cloudwatch:DisableInsightRules, cloudwatch:EnableInsightRules

By disabling critical insight rules, an attacker could effectively blind the organization to key performance and security metrics. Conversely, by enabling or configuring misleading rules, it could be possible to generate false data, create noise, or hide malicious activity.

aws cloudwatch disable-insight-rules --rule-names <value>
aws cloudwatch enable-insight-rules --rule-names <value>

Potential Impact: Confusion among the operations team, leading to delayed responses to actual issues and unnecessary actions based on false alerts.

cloudwatch:DeleteMetricStream , cloudwatch:PutMetricStream , cloudwatch:PutMetricData

An attacker with the cloudwatch:DeleteMetricStream , cloudwatch:PutMetricStream permissions would be able to create and delete metric data streams, compromising the security, monitoring and data integrity:

  • Create malicious streams: Create metric streams to send sensitive data to unauthorized destinations.

  • Resource manipulation: The creation of new metric streams with excessive data could produce a lot of noise, causing incorrect alerts, masking true issues.

  • Monitoring disruption: Deleting metric streams, attackers would disrupt the continuos flow of monitoring data. This way, their malicious activities would be effectively hidden.

Similarly, with the cloudwatch:PutMetricData permission, it would be possible to add data to a metric stream. This could lead to a DoS because of the amount of improper data added, making it completely useless.

aws cloudwatch delete-metric-stream --name <value>
aws cloudwatch put-metric-stream --name <value> [--include-filters <value>] [--exclude-filters <value>] --firehose-arn <value> --role-arn <value> --output-format <value>
aws cloudwatch put-metric-data --namespace <value> [--metric-data <value>] [--metric-name <value>] [--timestamp <value>] [--unit <value>] [--value <value>] [--dimensions <value>]

Example of adding data corresponding to a 70% of a CPU utilization over a given EC2 instance:

aws cloudwatch put-metric-data --namespace "AWS/EC2" --metric-name "CPUUtilization" --value 70 --unit "Percent" --dimensions "InstanceId=i-0123456789abcdefg"

Potential Impact: Disruption in the flow of monitoring data, impacting the detection of anomalies and incidents, resource manipulation and costs increasing due to the creation of excessive metric streams.

cloudwatch:StopMetricStreams, cloudwatch:StartMetricStreams

An attacker would control the flow of the affected metric data streams (every data stream if there is no resource restriction). With the permission cloudwatch:StopMetricStreams, attackers could hide their malicious activities by stopping critical metric streams.

aws cloudwatch stop-metric-streams --names <value>
aws cloudwatch start-metric-streams --names <value>

Potential Impact: Disruption in the flow of monitoring data, impacting the detection of anomalies and incidents.

cloudwatch:TagResource, cloudwatch:UntagResource

An attacker would be able to add, modify, or remove tags from CloudWatch resources (currently only alarms and Contributor Insights rules). This could disrupting your organization's access control policies based on tags.

aws cloudwatch tag-resource --resource-arn <value> --tags <value>
aws cloudwatch untag-resource --resource-arn <value> --tag-keys <value>

Potential Impact: Disruption of tag-based access control policies.

References

Support HackTricks
PreviousAWS - CloudTrail EnumNextAWS - Config Enum

Last updated