props.conf vs transforms.conf Cheat Sheet (Line Breaking, TIME_FORMAT, SEDCMD, nullQueue)
Short answer: props.conf is the entry point attached to a sourcetype, source, or host that declares what happens during parsing and search (line breaking, timestamps, SEDCMD, field aliases, extracts). transforms.conf stores reusable regex, INGEST_EVAL expressions, routing, filtering (nullQueue), lookup, and metadata transforms invoked by props.conf via TRANSFORMS-, REPORT-, or LOOKUP-.
Misplaced or misconfigured props.conf and transforms.conf stanzas are the number-one cause of broken timestamps, merged stack traces, leaked PII, and failing SPLK-1003 or SPLK-2002 exam scenarios. This guide maps every directive to the exact pipeline queue, Splunk tier, and precedence rule.
1. Splunk data pipeline & tier placement
Splunk processes events across four stages: Input → Parsing → Indexing → Search. Inside the parsing and indexing pipeline (visible in metrics.log), events flow sequentially through four queues: parsingQueue → aggQueue → typingQueue → indexQueue.
parsingQueue: UTF-8 charset conversion, line breaking (LINE_BREAKER,TRUNCATE), and header/metadata tagging.aggQueue: Event merging (SHOULD_LINEMERGE,BREAK_ONLY_BEFORE,MAX_EVENTS) and timestamp extraction (TIME_PREFIX,TIME_FORMAT,MAX_TIMESTAMP_LOOKAHEAD,TZ).typingQueue: Regex and eval transformations (SEDCMD,TRANSFORMS-callingtransforms.conf/INGEST_EVAL,nullQueue,_MetaData:Index).indexQueue: Compresses_rawinto bucket journals and writesTSIDXindex files.
| Splunk Tier | Pipeline Stage | Configs That Execute Here | Operational Rule |
|---|---|---|---|
| Universal Forwarder (UF) | Input stage only | inputs.conf, outputs.conf, limited INDEXED_EXTRACTIONS (structured CSV/JSON/TSV) | Does NOT run LINE_BREAKER, SHOULD_LINEMERGE, TIME_FORMAT, SEDCMD, or TRANSFORMS routing. |
| Heavy Forwarder (HF) | Input + Parsing (parsingQueue, aggQueue, typingQueue) | Full props.conf & transforms.conf parsing, SEDCMD, INGEST_EVAL, nullQueue, _MetaData:Index | Preferred governance boundary before data reaches Splunk Cloud or indexers. |
| Indexer (IDX) | Parsing (if from UF) + Indexing (indexQueue) | All index-time props.conf / transforms.conf / fields.conf when receiving raw UF streams | Writes compressed rawdata + TSIDX files; deploy via Cluster Manager (splunk apply cluster-bundle). |
| Search Head (SH) | Search stage only | EXTRACT-, REPORT-, FIELDALIAS-, EVAL-, LOOKUP-, KV_MODE | Index-time rules placed only on a Search Head have zero effect on incoming events. |
2. props.conf vs transforms.conf responsibility matrix
| Task | props.conf | transforms.conf | Runs At |
|---|---|---|---|
| Line breaking & event boundaries | LINE_BREAKER, TRUNCATE (parsingQueue); SHOULD_LINEMERGE, BREAK_ONLY_BEFORE, MAX_EVENTS (aggQueue) | Not used | Parsing (parsingQueue / aggQueue) |
| Timestamp extraction | TIME_PREFIX, TIME_FORMAT, MAX_TIMESTAMP_LOOKAHEAD, TZ, DATETIME_CONFIG | Not used | Parsing (aggQueue) |
| PII masking / redaction | SEDCMD-<name> = s/.../.../g (inline) or TRANSFORMS-<name> = <stanza> | DEST_KEY = _raw, REGEX, FORMAT = $1XXXX$2 (or INGEST_EVAL = _raw:=replace(...)) | Parsing (typingQueue) |
| Dropping noise (nullQueue) | TRANSFORMS-<name> = drop_rule | REGEX = ..., DEST_KEY = queue, FORMAT = nullQueue (or INGEST_EVAL = queue:=if(..., "nullQueue", queue)) | Parsing (typingQueue) |
| Dynamic routing (index / host / sourcetype) | TRANSFORMS-<name> = route_rule | DEST_KEY = _MetaData:Index + FORMAT = <index_name> (or MetaData:Host + FORMAT = host::<value>, MetaData:Sourcetype + FORMAT = sourcetype::<value>; or INGEST_EVAL = index:=...) | Parsing (typingQueue) |
| Index-time field extraction | TRANSFORMS-<name> = extract_meta (+ fields.conf INDEXED = true) | REGEX = ..., WRITE_META = true, FORMAT = field::$1 | Parsing / Indexing |
| Search-time regex extraction | EXTRACT-<name> = <regex> or REPORT-<name> = <transform_stanza> | [transform_stanza] REGEX = ..., FORMAT = ..., MV_ADD = true | Search Head |
| Automatic lookups | LOOKUP-<name> = <lookup_stanza> <match_field> OUTPUT <out_fields> | [lookup_stanza] filename = lookup.csv, match_type = CIDR(...) | Search Head |
3. Line breaking & event boundaries (props.conf)
By default, Splunk sets SHOULD_LINEMERGE = true, TRUNCATE = 10000 bytes, and MAX_EVENTS = 256 lines per merged event. Leaving SHOULD_LINEMERGE = true on single-line JSON or syslog feeds wastes aggregator CPU and risks merging unrelated events.
# Single-line logs (fastest performance: disables line-merger overhead)
[custom:api:json]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TRUNCATE = 65536
# Multiline Java stack traces using LINE_BREAKER lookahead (faster than BREAK_ONLY_BEFORE)
[custom:java:app]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2})
MAX_EVENTS = 512
TRUNCATE = 0Remember that LINE_BREAKER deletes only the substring matched by its first capture group (...). Using a positive lookahead (?=...) preserves the next event's timestamp at the start of the new event.
4. Timestamp extraction & TZ precedence
Never leave Splunk guessing timestamps via fallback heuristics. Pinpoint the exact offset and format using TIME_PREFIX, TIME_FORMAT (using strptime specifiers), and MAX_TIMESTAMP_LOOKAHEAD (defaults to 128 bytes after TIME_PREFIX):
[custom:api:json]
TIME_PREFIX = "timestamp"\s*:\s*"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N%Z
MAX_TIMESTAMP_LOOKAHEAD = 40
TZ = UTC
# Use DATETIME_CONFIG = CURRENT only when logs truly lack any embedded timestamp
[legacy:heartbeat]
DATETIME_CONFIG = CURRENTTimezone resolution follows a strict order: explicit timezone inside the event (e.g. Z or -05:00) → TZ in props.conf → OS timezone of the forwarder/indexer parsing the event.
5. Masking, nullQueue, routing & INGEST_EVAL
| Mechanism | Files Required | Best For | Limitations |
|---|---|---|---|
SEDCMD-<name> | props.conf only | Simple inline regex masking or character replacement on _raw (credit cards, SSNs, tokens). | Cannot route events to nullQueue or change _MetaData:Index. |
TRANSFORMS-<name> | props.conf + transforms.conf | Dropping events (nullQueue), dynamic index/host/sourcetype routing, reusable masking, WRITE_META. | Requires matching stanza names across both files; regex capture group errors can silently drop data. |
INGEST_EVAL | props.conf + transforms.conf (Splunk 8+/9+) | Conditional routing, index-time field creation, dropping events, or replace()/sha256() without regex gymnastics. | Defined in transforms.conf and invoked via TRANSFORMS-<name> in props.conf; uses := overwrite assignment. |
Below is a complete, production-style parsing example showing SEDCMD redaction, nullQueue noise dropping, dynamic index routing, index-time field extraction (with fields.conf), and modern INGEST_EVAL alternatives:
# --- props.conf (Parsing Tier: Heavy Forwarder or Indexer) ---
[app:payment:events]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S,%3N
TZ = UTC
# 1. Inline PII masking via SEDCMD
SEDCMD-mask_cc = s/(\d{4})\d{8}(\d{4})/\1XXXXXXXX\2/g
# 2. Drop noisy debug healthchecks, route PCI events, extract indexed env field
TRANSFORMS-01_drop_health = drop_healthcheck
TRANSFORMS-02_route_pci = route_to_pci_index
TRANSFORMS-03_meta_env = extract_indexed_env
# --- transforms.conf ---
[drop_healthcheck]
REGEX = GET\s+/healthz\s+200
DEST_KEY = queue
FORMAT = nullQueue
[route_to_pci_index]
REGEX = \bcard_present=(true|1)\b
DEST_KEY = _MetaData:Index
FORMAT = pci_logs
[extract_indexed_env]
REGEX = \benv=(prod|staging|dev)\b
WRITE_META = true
FORMAT = env::$1
# --- fields.conf (required on Indexers & Search Heads for indexed fields) ---
[env]
INDEXED = trueOn Splunk 8.x/9.x, INGEST_EVAL is configured inside a transforms.conf stanza and invoked from props.conf via TRANSFORMS-<name>. Use := overwrite assignment when modifying existing fields such as queue, _raw, or index, and fall back to queue rather than hardcoding "indexQueue":
# --- props.conf ---
[app:payment:ingest_eval]
TRANSFORMS-ingest_eval = payment_ingest_eval
# --- transforms.conf ---
[payment_ingest_eval]
INGEST_EVAL = queue:=if(match(_raw, "GET\s+/healthz\s+200"), "nullQueue", queue), _raw:=replace(_raw, "(\d{4})\d{8}(\d{4})", "\1XXXXXXXX\2"), index:=if(match(_raw, "card_present=true"), "pci_logs", index)6. Config precedence & btool troubleshooting
Splunk evaluates precedence differently depending on whether the setting runs at global/index-time or search-time (app/user):
- Global / Index-time precedence (
inputs.conf,indexes.conf, index-timeprops.conf&transforms.conf):etc/system/local>etc/apps/<app>/local(ASCII A–Z) >etc/apps/<app>/default(ASCII A–Z) >etc/system/default. - Search-time precedence (search-time
EXTRACT-,FIELDALIAS-,EVAL-,LOOKUP-):etc/users/<user>/<app>/local> currentetc/apps/<app>/local> currentetc/apps/<app>/default> other exported apps (reverse ASCII Z–A) >etc/system/local>etc/system/default. - Stanza precedence &
priorityinsideprops.conf: Stanzas are evaluated by numericpriorityfirst (higher wins) and then by stanza type. Literal stanzas (exact[source::/path], exact[host::name], and literal[sourcetype]) default topriority = 100, whereas wildcard pattern stanzas ([source::*],[host::web*]) default topriority = 0. At equal priority,source::>host::>sourcetype>[default].
# Inspect merged props.conf settings and exact winning file paths
$SPLUNK_HOME/bin/splunk btool props list app:payment:events --debug
# Inspect transforms.conf and validate syntax across all conf files
$SPLUNK_HOME/bin/splunk btool transforms list route_to_pci_index --debug
$SPLUNK_HOME/bin/splunk btool check
# Deploy parsing updates to an Indexer Cluster from the Cluster Manager
$SPLUNK_HOME/bin/splunk apply cluster-bundle7. Common pitfalls
- Omitting the capture group (...) in LINE_BREAKER: Splunk deletes whatever text matches the first capturing group in LINE_BREAKER. Writing LINE_BREAKER = [\r\n]+ without parentheses fails to break lines, while capturing the timestamp itself deletes the timestamp from _raw.
- Putting index-time props.conf only on a Universal Forwarder or Search Head: Universal Forwarders ship raw bytes without running LINE_BREAKER, TIME_FORMAT, SEDCMD, or nullQueue. Deploy index-time parsing stanzas on the first full Splunk tier (Heavy Forwarders or Indexers).
- Missing MAX_TIMESTAMP_LOOKAHEAD on verbose JSON/XML: By default, Splunk searches only the first 128 bytes after TIME_PREFIX. If a timestamp appears at byte 300 of a JSON payload, Splunk falls back to heuristic scanning or ingestion time unless you raise MAX_TIMESTAMP_LOOKAHEAD.
- Editing default/ or dumping app rules into etc/system/local/: Changes inside default/ are overwritten on TA upgrades, while settings in etc/system/local/ win global precedence across every app and break modular overrides. Always place custom overrides in etc/apps/<app_name>/local/.
- Mismatched TRANSFORMS-<class> stanza names: If props.conf declares TRANSFORMS-filter = drop_healthcheck, transforms.conf must define [drop_healthcheck] verbatim. A typo fails silently with no UI alert.
Frequently asked questions
Where do props.conf and transforms.conf need to be deployed: Universal Forwarder, Heavy Forwarder, Indexer, or Search Head?
Deploy index-time rules (LINE_BREAKER, SHOULD_LINEMERGE, TIME_FORMAT, SEDCMD, INGEST_EVAL, nullQueue, and _MetaData:Index routing) on your Parsing Tier (Heavy Forwarders or Indexers). Deploy search-time rules (EXTRACT-, REPORT-, FIELDALIAS-, EVAL-, LOOKUP-) on your Search Heads. Universal Forwarders only run inputs.conf, outputs.conf, and limited INDEXED_EXTRACTIONS.
When should I use SEDCMD vs transforms.conf vs INGEST_EVAL?
Use SEDCMD in props.conf for straightforward inline regex masking on _raw. Use transforms.conf when you need nullQueue filtering, dynamic index/sourcetype/host routing, WRITE_META index-time fields, or reusable transforms across sourcetypes. Use INGEST_EVAL (inside a transforms.conf stanza invoked by TRANSFORMS-<name> in props.conf) when conditional logic or eval functions like replace() or sha256() are cleaner than multi-stanza regexes.
Why did my LINE_BREAKER or nullQueue rule not take effect?
The three most common causes are deploying the config only to a Universal Forwarder or Search Head instead of the parsing tier, forgetting the capture group (...) inside LINE_BREAKER, or a stanza mismatch between TRANSFORMS-<name> in props.conf and transforms.conf. Run ./splunk btool props list <sourcetype> --debug on the parsing node to verify.
What is the difference between global (index-time) and app/user (search-time) configuration precedence?
Global/index-time settings ignore user context and evaluate system/local first, followed by app local directories in ASCII A-Z order, app default directories in ASCII A-Z order, and finally system/default. Search-time knowledge objects prioritize user local overrides, then the current app (local before default), then other exported apps in reverse ASCII order, then system.
How do I test which props.conf file is winning on a Splunk server?
Run $SPLUNK_HOME/bin/splunk btool props list <stanza_name> --debug (and btool transforms list <stanza_name> --debug) on the tier executing that phase. Each line prints the exact file path on the left so you can spot shadowing in system/local or another app.
Practice parsing & config precedence for free
- SPLK-1003: Configuration File Structure · The btool Command
- SPLK-1003: Universal vs. Heavy Forwarders
- SPLK-1003: Event Breaking & Transformations · SPL Lab: Investigating Parse Issues
- SPLK-2002: Data Routing & Filtering · Configuration & Search Issues (btool)
- CDE (SPLK-5002): Data Onboarding & Parsing
- Related tools & guides: Interactive Splunk Architecture Diagram · SPLK-1003 Study Guide · SPLK-2002 Study Guide · SPLK-1003 Practice Exam