Testing Spec

Defines the tests.yaml format, assertion targets, comparison operators, sample and channel selection rules, and the execution model for norq test.

Tests describe business requirements on compiled output: a subject line contains the right text, an SMS fits within a segment budget, a Slack message includes an action block. This is complementary to norq lint, which checks structural correctness; tests check content correctness.

Overview

Each notification directory may contain a tests.yaml file alongside its channel templates. When norq test runs, it discovers every notification that has a tests.yaml, parses the test cases, and evaluates each assertion against compiled output. Notifications without tests.yaml are silently skipped.

File Format

tests.yaml contains a single top-level key, tests, whose value is a sequence of test case objects.

tests:
  - name: "Email renders without errors"
    channel: email
    sample: "New user"
    assert:
      diagnostics: { errors: 0 }
 
  - name: "SMS fits in one segment"
    channel: sms
    sample: "New user"
    assert:
      sms_segments: { lte: 1 }

An empty tests array (tests: []) is valid. No test cases are executed.

A missing tests.yaml file is also valid. The notification simply has no tests.

Test Case Fields

name is a human-readable description shown in test output. It must be a non-empty string. Names need not be unique within a file, though unique names make failure messages easier to read.

channel selects a single channel by its identifier (email, sms, slack, push, whatsapp, msteams). When omitted and all_channels is false, all channels present in the notification are tested independently. When channel names a channel that does not exist in the notification, the test fails immediately with the message channel '<name>' not found in notification.

sample selects a named entry from data.samples.yaml by its name field. When omitted and all_samples is false, the first sample defined in data.samples.yaml is used. When there are no samples at all, an empty object ({}) is used as data. When sample names an entry that does not exist, the test fails immediately with the message sample '<name>' not found.

all_channels expands the test to run once per channel present in the notification. When true, the channel field is ignored even if present.

all_samples expands the test to run once per sample defined in data.samples.yaml. When true, the sample field is ignored even if present. When no samples are defined, the test runs once with an empty data object.

assert is a map from assertion target to a nested map of operator to expected value. Multiple targets may be specified in a single test case; all must pass. An empty assert map causes the test case to pass unconditionally (useful as a placeholder).

Test Expansion

When all_channels or all_samples is true, a single test case definition expands into multiple independent test runs — one per channel, per sample, or per combination. Each expanded run is reported separately. The test label for an expanded run is suffixed with [channel:sample]:

All channels render without errors [email:Full data]
All channels render without errors [sms:Full data]
All channels render without errors [email:Minimal]
All channels render without errors [sms:Minimal]

The suffix is omitted when neither all_channels nor all_samples is true.

Assertion Targets

Each entry in the assert map names a target. A target extracts a specific value from the compiled channel payload and passes it to an operator for comparison. Unknown targets cause the assertion to fail with Unknown assertion target: '<name>'.

subject

For email, extracts the compiled subject line (after template expression evaluation against the sample data). For push, extracts the push notification title. For all other channels, the value is an empty string.

assert:
  subject: { contains: "Welcome" }

The subject target supports string operators: eq, contains, not_contains, matches.

body

Extracts the primary text payload for the channel:

assert:
  body: { contains: "Acme Platform" }

The body target supports string operators: eq, contains, not_contains, matches.

html

Extracts the compiled HTML string of the email channel. For all other channels, the value is an empty string, so assertions on html for non-email channels will fail unless the expected value is also empty.

assert:
  html: { contains: "unsubscribe" }

The html target supports string operators: eq, contains, not_contains, matches.

sms_segments

Extracts the SMS segment count computed by the SMS compiler. The segment count reflects the encoding of the compiled body (GSM-7 allows 160 characters per segment; Unicode/UCS-2 allows 70 characters per segment, or 153/67 per segment for multi-part messages). For non-SMS channels, the value is 0.

assert:
  sms_segments: { lte: 1 }

The sms_segments target supports numeric operators: eq, lt, lte, gt, gte.

payload_bytes

Extracts the byte length of the compiled payload serialized as JSON. This is the full JSON serialization of the channel payload struct, not just the body text. Useful for enforcing push notification size budgets.

assert:
  payload_bytes: { lte: 4096 }

The payload_bytes target supports numeric operators: eq, lt, lte, gt, gte.

blocks

Extracts the Slack Block Kit blocks array from the compiled Slack payload. For non-Slack channels, the value is an empty JSON array. Assertions on blocks for non-Slack channels will not match any blocks.

assert:
  blocks: { any: { type: "header" } }

The blocks target supports the any operator only.

diagnostics

Runs the linter (norq lint) against the entire notification bundle and counts the diagnostics by severity. The diagnostic count is computed once per test run and is the same regardless of which channel or sample is being tested — it reflects the lint results for all channels and all templates in the notification.

assert:
  diagnostics: { errors: 0 }

The diagnostics target supports the errors operator only.

When a compile error occurs (the template cannot be parsed or compiled), and the test case contains only diagnostics assertions, the test still evaluates the diagnostic assertions using the linter's error count. This allows a single diagnostics: { errors: 0 } assertion to catch both compile-time failures and lint errors. If any non-diagnostics assertion is present and compilation fails, the test fails with the compile error message.

Assertion Operators

String Operators

String operators apply to subject, body, and html targets.

eq — exact string equality. The assertion passes if and only if the extracted string matches the expected string exactly, including case and whitespace.

subject: { eq: "Welcome, Alice!" }

contains — substring match. The assertion passes if the expected string appears anywhere in the extracted string. Case-sensitive.

subject: { contains: "Welcome" }
html: { contains: "unsubscribe" }

not_contains — substring absence. The assertion passes if the expected string does not appear anywhere in the extracted string. Case-sensitive.

body: { not_contains: "Pro access" }

matches — regular expression match. The expected value is a regex pattern (Rust regex syntax). The assertion passes if the pattern matches anywhere in the extracted string. If the pattern is not a valid regular expression, the assertion fails with an error describing the regex compilation failure.

body: { matches: "order #[A-Z]+-\\d+" }

Numeric Operators

Numeric operators apply to sms_segments and payload_bytes targets.

eq — numeric equality. The assertion passes if the actual numeric value equals the expected value. Comparison uses floating-point epsilon equality.

sms_segments: { eq: 1 }

lt — strictly less than.

payload_bytes: { lt: 4096 }

lte — less than or equal.

sms_segments: { lte: 2 }

gt — strictly greater than.

gte — greater than or equal.

sms_segments: { gte: 1 }

Block Operators

Block operators apply to the blocks target (Slack only).

any — partial JSON match against any element in the blocks array. The expected value is a JSON object; the operator checks whether at least one block in the array contains all the specified keys with matching values. Matching is partial: extra keys in the block are ignored, but every key in the expected pattern must be present with an equal value.

blocks: { any: { type: "header" } }
blocks: { any: { type: "actions" } }

The partial match is recursive: nested objects in the pattern must also be partial-matched against the corresponding values in the block.

Diagnostic Operators

Diagnostic operators apply to the diagnostics target.

errors — exact count of lint diagnostics with severity Error. The assertion passes if the number of error-severity diagnostics equals the expected integer. Warning-severity diagnostics are not counted.

diagnostics: { errors: 0 }

Test Execution

Discovery

norq test resolves the project configuration using the same config discovery strategy as all other commands: upward walk from the current directory, then downward scan. The --config <path> flag targets a specific config file or its parent directory.

norq test without a notification argument runs tests for every notification in the project that has a tests.yaml file. Notifications without tests.yaml are silently skipped.

# Test all notifications in the project
norq test
 
# Test one specific notification
norq test transactional/welcome
 
# Test with a specific config file
norq test --config ./norq.config.yaml

Execution Order

For each notification with a tests.yaml, the test runner:

1. Parses tests.yaml into a list of test cases.
2. For each test case, expands all_channels and all_samples into individual runs.
3. For each run, parses the channel template using the partial resolver, then compiles the parsed document against the sample data and project theme.
4. Evaluates every assertion in the assert map against the compiled payload.
5. Collects results: each run produces exactly one TestResult with a passed flag and a list of failure messages.

The lint error count used by diagnostics assertions is computed once per test case invocation by running the linter against the full notification bundle. It is the same value for all channel/sample combinations within a single test case.

Sample Resolution

When no sample field is specified and all_samples is false, the test uses the first entry in data.samples.yaml. If the file does not exist or defines no samples, an empty JSON object ({}) is used as data. This means templates that use {{variable}} expressions without sample data will render with empty or missing values — which may cause assertions on content to fail.

Channel Resolution

When no channel field is specified and all_channels is false, the test runs against all channels present in the notification. Each channel is compiled independently. If the notification has three channels (email, sms, slack) and a test case specifies no channel, the runner produces three TestResult entries for that test case.

Exit Codes

norq test exits with code 0 if every test case passes. It exits with code 1 if any test case fails.

Human Output

By default, norq test prints a human-readable summary:

  transactional/welcome
    ✓ Email renders without errors
    ✓ SMS fits in one segment
    ✗ Slack has header block
      blocks: no block matched {"type": "header"} in 2 blocks

1 passed, 1 failed

Each failure includes the assertion target and a message describing what was expected versus what was found. Long strings in failure messages are truncated to 100 characters.

JSON Output

The --json flag emits a machine-readable JSON envelope suitable for CI parsing.

norq test --json

Output structure:

{
  "_protocol": { "version": 1 },
  "results": [
    {
      "notification": "transactional/welcome",
      "tests": [
        {
          "name": "Email renders without errors",
          "passed": true,
          "failures": []
        },
        {
          "name": "SMS fits in one segment",
          "passed": false,
          "failures": ["sms_segments: expected lte 1, got 2"]
        }
      ]
    }
  ]
}

The _protocol.version field is always 1. The results array contains one entry per notification. Each test entry has name (string), passed (boolean), and failures (array of strings). When passed is true, failures is an empty array. When passed is false, failures contains one message per failed assertion.

Patterns

All channels render without errors

The most common starting test. Covers every channel and every sample in one declaration. Fails if any channel produces a compile error or a lint error.

tests:
  - name: "All channels render without errors"
    all_channels: true
    all_samples: true
    assert:
      diagnostics: { errors: 0 }

Add this as the first test case in every tests.yaml. It gives immediate feedback when a template change breaks compilation across any channel or sample.

Minimal sample also renders clean

Use a minimal sample (required fields only, no optional extras) to verify that templates handle sparse data gracefully.

  - name: "Minimal sample renders without errors"
    all_channels: true
    sample: "Minimal"
    assert:
      diagnostics: { errors: 0 }

Testing subject line content

Verify that dynamic expressions in the email subject render as expected for a specific sample.

  - name: "Subject contains user name"
    channel: email
    sample: "Full data"
    assert:
      subject: { contains: "Alice" }
 
  - name: "Subject contains app name"
    channel: email
    sample: "Full data"
    assert:
      subject: { contains: "Acme Platform" }

Multiple assertions per test case are also valid:

  - name: "Subject has order ID and item count"
    channel: email
    sample: "Full order"
    assert:
      subject: { contains: "ORD-2024-1847" }

Each assertion in the assert map is independent; all must pass for the test to pass.

Testing SMS segment count

Enforce that an SMS stays within a segment budget across typical and edge-case samples.

  - name: "SMS fits in one segment for minimal data"
    channel: sms
    sample: "Minimal"
    assert:
      sms_segments: { lte: 1 }
 
  - name: "SMS within two segments for long data"
    channel: sms
    sample: "Full data"
    assert:
      sms_segments: { lte: 2 }

When the SMS body contains any Unicode character outside the GSM-7 character set, the compiler switches to UCS-2 encoding, which halves the per-segment capacity. Segment count assertions catch this silently.

Testing for absence of content

Verify that conditional content is correctly suppressed when the condition is not met.

  - name: "Free user does not see Pro content"
    channel: email
    sample: "Free user"
    assert:
      body: { not_contains: "Pro access" }
 
  - name: "SMS omits promo block for minimal data"
    channel: sms
    sample: "Minimal"
    assert:
      body: { not_contains: "Upgrade now" }

not_contains is particularly useful alongside contains tests on the same channel but different samples, confirming conditional branches render correctly for both sides.

Slack structure assertions

Verify that Slack messages include specific block types.

  - name: "Slack has header block"
    channel: slack
    sample: "Full data"
    assert:
      blocks: { any: { type: "header" } }
 
  - name: "Slack has action button"
    channel: slack
    sample: "Full data"
    assert:
      blocks: { any: { type: "actions" } }

Push payload size

Enforce push notification payload limits. Most push services cap payloads at 4,096 bytes; APNs collapses oversized notifications silently.

  - name: "Push is under 4KB"
    channel: push
    sample: "Full data"
    assert:
      payload_bytes: { lte: 4096 }

HTML content assertions

Test that specific HTML content appears in the rendered email, including link text, button labels, or structural markers like unsubscribe links.

  - name: "Email has unsubscribe link"
    channel: email
    sample: "New user"
    assert:
      html: { contains: "unsubscribe" }
 
  - name: "Email has primary action button"
    channel: email
    sample: "Full data"
    assert:
      html: { contains: "Get started" }

The html target contains the fully rendered HTML string (after MJML compilation). body on the email channel contains the plain-text alternative. Use html when asserting on rendered markup; use body when asserting on plain-text content.

Edge Cases

Reference to a non-existent sample

If sample names an entry not found in data.samples.yaml, the test fails immediately without running any assertions. The failure message is sample '<name>' not found. Other test cases in the same file continue to run.

Reference to a non-existent channel

If channel names a channel that does not exist in the notification (no corresponding template file), the test fails immediately. The failure message is channel '<name>' not found in notification. Other test cases continue to run.

all_channels: true with channel also set

The channel field is silently ignored. The test expands to all channels in the notification, as if channel were not specified.

all_samples: true with sample also set

The sample field is silently ignored. The test expands to all samples, as if sample were not specified.

No samples defined

When data.samples.yaml is absent or contains no entries, and sample is not specified, the test runs with an empty data object ({}). Template expressions that reference data paths will evaluate to empty strings. all_samples: true also runs once with the empty data object.

No tests.yaml

A notification without tests.yaml produces no test results and is silently omitted from norq test output. This is not an error.

Empty tests array

tests: []

Valid. Parsed successfully. No tests are run. The notification appears in no test output.

Parse failure in tests.yaml

If tests.yaml cannot be parsed as valid YAML matching the expected structure, norq test reports a fatal error for that notification and continues with the remaining notifications.

Relationship to norq lint

norq lint and norq test are complementary. Use both in CI.

The diagnostics: { errors: 0 } assertion in tests.yaml bridges the two systems: it runs the linter and asserts on its output within a test case. This allows a single smoke-test pass to verify both compile success and zero lint errors in one declaration.

Complete Example

A tests.yaml for a transactional welcome notification with email, SMS, Slack, and push channels:

tests:
  # Smoke test — all channels, all samples
  - name: "All channels render without errors"
    all_channels: true
    all_samples: true
    assert:
      diagnostics: { errors: 0 }
 
  # Subject line
  - name: "Email subject contains user name"
    channel: email
    sample: "Full data"
    assert:
      subject: { contains: "Alice" }
 
  - name: "Email subject contains app name"
    channel: email
    sample: "Full data"
    assert:
      subject: { contains: "Acme Platform" }
 
  # Conditional branches
  - name: "Free user does not see Pro content in email body"
    channel: email
    sample: "Minimal"
    assert:
      body: { not_contains: "Pro access" }
 
  # SMS segment budget
  - name: "SMS fits in two segments for minimal data"
    channel: sms
    sample: "Minimal"
    assert:
      sms_segments: { lte: 2 }
 
  # Slack structure
  - name: "Slack has header block"
    channel: slack
    sample: "Full data"
    assert:
      blocks: { any: { type: "header" } }
 
  - name: "Slack has action button"
    channel: slack
    sample: "Full data"
    assert:
      blocks: { any: { type: "actions" } }
 
  # Push size
  - name: "Push is under 4KB"
    channel: push
    sample: "Full data"
    assert:
      payload_bytes: { lte: 4096 }
 
  # HTML content
  - name: "Email HTML has unsubscribe link"
    channel: email
    sample: "Full data"
    assert:
      html: { contains: "unsubscribe" }