Writing Custom Detection Rules: A Practical Guide

Sentinel Nerd ships with over 50 built-in detection rules that cover common UniFi security scenarios. But every network is different. Custom detection rules let you encode your organization’s specific security logic — catching the threats that matter to you while ignoring the noise that doesn’t.

This guide walks you through writing custom rules from scratch, with real examples you can adapt for your own network.

When to Write Custom Rules

You should write custom rules when:

• Built-in rules don’t cover your use case — You have a specific threat scenario unique to your environment
• You need different thresholds — The default “5 failed logins in 10 minutes” doesn’t fit your team’s login patterns
• You want to monitor business logic — Track access to specific VLANs, devices, or resources
• You’re correlating across sources — Detect patterns that span Network, Protect, and Access events
• You need compliance rules — Monitor for policy violations specific to your compliance framework

Rule Anatomy

Every detection rule is defined in YAML. Here’s the structure:

id: custom-brute-force-ssh
name: SSH Brute Force Attempt
description: Detects multiple failed SSH login attempts from a single source
severity: high
category: credential-attack
enabled: true

conditions:
  - field: event.type
    operator: equals
    value: ssh_login_failed
  - field: event.source
    operator: equals
    value: unifi_network

aggregation:
  group_by: source_ip
  count: 10
  window: 5m

actions:
  - alert
  - tag: brute-force

tags:
  - ssh
  - brute-force
  - credential-attack

Let’s break down each section:

Metadata

• id — Unique identifier. Use kebab-case. Must be unique across all rules.
• name — Human-readable name shown in alerts and the dashboard.
• description — Explains what the rule detects. Appears in alert details.
• severity — One of: critical, high, medium, low. Drives alert routing.
• category — Groups related rules. Used for filtering and reporting.
• enabled — Set to false to disable without deleting.

Conditions

Conditions define what events the rule matches. All conditions must be true (AND logic) unless wrapped in an any block.

Aggregation

Optional. Triggers the rule only when conditions are met a certain number of times within a time window. Without aggregation, the rule fires on every matching event.

Actions

What happens when the rule triggers. Options include alert, tag, block, quarantine, and notify.

Your First Rule: Detecting After-Hours Access

Let’s write a rule that alerts when someone uses UniFi Access to open a door outside business hours:

id: after-hours-door-access
name: After-Hours Door Access
description: Alerts when a door is accessed outside of business hours (6 PM - 7 AM)
severity: medium
category: physical-security
enabled: true

conditions:
  - field: event.type
    operator: equals
    value: door_unlock
  - field: event.source
    operator: equals
    value: unifi_access
  - field: event.hour
    operator: not_between
    value: [7, 18]

actions:
  - alert
  - tag: after-hours

tags:
  - access-control
  - after-hours
  - physical-security

This rule matches any door unlock event from UniFi Access where the hour is outside 7 AM to 6 PM. Each match generates an alert tagged with after-hours.

Condition Operators Deep Dive

Sentinel Nerd supports these operators:

Operator	Description	Example
equals	Exact match	field: severity, value: critical
not_equals	Not equal	field: action, value: allowed
contains	Substring match	field: message, value: "brute force"
regex	Regular expression	field: user_agent, value: "^python-requests"
greater_than	Numeric comparison	field: bytes_sent, value: 1000000
less_than	Numeric comparison	field: response_time, value: 100
between	Range (inclusive)	field: port, value: [1, 1024]
not_between	Outside range	field: hour, value: [7, 18]
in	Value in list	field: country, value: ["CN", "RU", "KP"]
not_in	Value not in list	field: vlan, value: [10, 20]
exists	Field is present	field: threat_score
not_exists	Field is absent	field: user_id

OR Logic with any

To match any of several conditions (OR logic), wrap them in an any block:

conditions:
  - field: event.source
    operator: equals
    value: unifi_network
  - any:
      - field: event.type
        operator: equals
        value: ids_alert
      - field: event.type
        operator: equals
        value: ips_block

This matches Network events that are either IDS alerts OR IPS blocks.

Windowing and Thresholds

Windowed rules detect patterns over time rather than single events. The aggregation section controls this:

aggregation:
  group_by: source_ip
  count: 5
  window: 10m

This means: “Group events by source IP, and trigger when 5 or more matching events occur within a 10-minute window.”

Window Durations

Windows use shorthand notation: 30s (seconds), 5m (minutes), 1h (hours), 1d (days).

Group By Multiple Fields

Group by multiple fields to create more specific aggregations:

aggregation:
  group_by: [source_ip, destination_port]
  count: 20
  window: 1m

This detects port scanning — 20+ connections from one IP to different ports in under a minute.

Distinct Count

Use distinct_count instead of count to count unique values:

aggregation:
  group_by: source_ip
  distinct_count: destination_port
  threshold: 50
  window: 5m

This fires when a single IP connects to 50+ different ports in 5 minutes — a cleaner port scan detector.

Testing with the Simulator

Before enabling a rule in production, test it with the built-in simulator:

1. Navigate to Settings > Detection Rules > Simulator
2. Paste your YAML rule
3. Select a time range of historical events to test against
4. Click Simulate

The simulator shows:

• How many times the rule would have triggered
• Which events matched
• Timeline of triggers
• Estimated alert volume per day

This is invaluable for tuning thresholds. If a rule triggers 500 times a day, raise the threshold or narrow the conditions.

5 Useful Rules You Can Copy

1. New Device on Secure VLAN

id: new-device-secure-vlan
name: New Device on Secure VLAN
description: Alerts when an unrecognized device connects to the secure VLAN
severity: high
category: network-security
enabled: true

conditions:
  - field: event.type
    operator: equals
    value: client_connect
  - field: vlan_id
    operator: equals
    value: 10
  - field: client.is_known
    operator: equals
    value: false

actions:
  - alert
  - tag: unknown-device

2. Large Data Exfiltration

id: large-data-transfer
name: Unusual Large Data Transfer
description: Detects unusually large outbound data transfers
severity: high
category: data-exfiltration
enabled: true

conditions:
  - field: event.type
    operator: equals
    value: traffic_summary
  - field: bytes_sent
    operator: greater_than
    value: 5368709120

actions:
  - alert
  - tag: data-exfil

3. Camera Offline

id: camera-offline
name: Protect Camera Went Offline
description: Alerts when a UniFi Protect camera disconnects
severity: medium
category: device-health
enabled: true

conditions:
  - field: event.type
    operator: equals
    value: device_disconnect
  - field: event.source
    operator: equals
    value: unifi_protect
  - field: device.type
    operator: equals
    value: camera

actions:
  - alert

4. DNS Tunneling Attempt

id: dns-tunneling
name: Possible DNS Tunneling
description: Detects high-volume DNS queries that may indicate DNS tunneling
severity: high
category: exfiltration
enabled: true

conditions:
  - field: event.type
    operator: equals
    value: dns_query
  - field: query_length
    operator: greater_than
    value: 50

aggregation:
  group_by: source_ip
  count: 100
  window: 5m

actions:
  - alert
  - tag: dns-tunneling

5. Repeated Access Denied

id: repeated-access-denied
name: Repeated Physical Access Denied
description: Multiple failed door access attempts may indicate tailgating or stolen credentials
severity: high
category: physical-security
enabled: true

conditions:
  - field: event.type
    operator: equals
    value: door_access_denied
  - field: event.source
    operator: equals
    value: unifi_access

aggregation:
  group_by: credential_id
  count: 3
  window: 15m

actions:
  - alert
  - tag: access-violation

Debugging Tips

Rule not triggering? Check these common issues:

1. Field names — Use the Event Explorer to see exact field names in your events. A typo in a field name means the condition never matches.
2. Event source filter — Make sure you’re matching the right source (unifi_network, unifi_protect, unifi_access, unifi_talk).
3. Time window too narrow — If your aggregation window is 1 minute but events are sparse, you may never hit the threshold.
4. Rule disabled — Check that enabled: true is set.
5. Draft mode — New rules start in draft mode. Publish them in the dashboard.

Rule triggering too much? Tune it:

1. Raise the count threshold in aggregation
2. Narrow conditions with additional field matches
3. Exclude known-good sources using not_equals or not_in
4. Widen the time window so transient spikes don’t trigger

Check the rule audit log in Settings > Detection Rules > Audit to see rule evaluation history, match counts, and any errors.

---

Custom detection rules are what make Sentinel Nerd truly yours. Start simple, test thoroughly with the simulator, and iterate based on real-world results. The best detection rules are the ones refined over weeks of running in your environment.

Need inspiration? Browse our community rules library or share your rules on our community Discord.