Skip to content

3 Effective Methods to Send Rsyslog Data to Splunk HEC (HTTP Event Collector)

If you’re looking for methods to send Rsyslog data to Splunk, then you’ve landed on the right article. In today’s data-driven IT landscape, effectively collecting and analyzing log data is crucial for maintaining system health, security, and compliance. Two powerful tools in this space are Rsyslog (the reliable system for log processing) and Splunk (the industry-leading data analytics platform).

At Ableversity, we’ve helped thousands of students master these technologies. In this post, we’ll explore the three most effective methods for forwarding Rsyslog data to Splunk, complete with configuration examples and pros/cons of each approach.

Why Send Rsyslog Data to Splunk?

Before diving into implementation methods, let’s quickly understand why this integration is valuable:

  1. Centralized Log Management: Consolidate logs from multiple sources into a single searchable platform
  2. Advanced Analytics: Leverage Splunk’s powerful search and visualization capabilities on your system logs
  3. Real-time Monitoring: Track system behavior and detect issues as they occur
  4. Security and Compliance: Maintain audit trails and detect security incidents quickly

All three methods we’ll cover use rsyslog’s omhttp module to send data to Splunk’s HTTP Event Collector (HEC). For detailed information about this module, refer to the official rsyslog documentation for omhttp.

Understanding the Configuration Format

In the configurations below, you’ll notice two types of placeholders:

  • %variablename%: These are rsyslog variables (like %hostname%, %msg%, %application%) that will be automatically filled in by rsyslogd at runtime with the actual values from your log events.
  • UPPERCASE: These are placeholders (like SPLUNKHEC.YOURDOMAIN.COM, HECTOKEN, INDEXNAME) that you need to replace with your specific environment values before implementing.

Let’s explore each approach.

Method 1: Send Rsyslog data to Splunk HEC Raw Endpoint (Without JSON)

This method sends raw log messages directly to Splunk’s HEC raw endpoint.

module(load="omhttp")   
template(name="splunk-hec" type="string" string="%msg%")
Ruleset( name="logs-to-splunk-hec"
queue.type="linkedlist"){
    action(
        type="omhttp"
        server="SPLUNKHEC.YOURDOMAIN.COM" 
        serverport="8088" 
        useHttps="on"
        skipverifyhost="on"         # CAUTION: Security risk in production
        allowunsignedcerts="on"     # CAUTION: Security risk in production
        restpath="services/collector/raw?index=INDEXNAME&host=%hostname%&sourcetype=syslog&source=%application%-rsyslogd"
        httpheaders=[
            "Authorization: Splunk HECTOKEN"
        ]
        template="splunk-hec"
        batch="off"
    )
}

Pros:

  • Simple configuration with minimal processing overhead
  • Preserves the original message format exactly as it appears in rsyslog
  • Metadata (index, host, source, sourcetype) handled via URL parameters

Cons:

  • Limited flexibility for custom formatting
  • May require additional parsing in Splunk
  • No batch processing capabilities (one event per HTTP request)

Method 2: Send Rsyslog Data to Splunk Raw Endpoint with JSON Formatting

This method sends log messages to the raw endpoint, but formats them as JSON first.

module(load="omhttp")   
template(name="splunk-hec" type="list" option.jsonf="on") {property(outname="_raw" name="msg" format="jsonf") }
Ruleset( name="logs-to-splunk-hec"
queue.type="linkedlist"){
    action(
        type="omhttp"
        server="SPLUNKHEC.YOURDOMAIN.COM" 
        serverport="8088" 
        useHttps="on"
        skipverifyhost="on"         # CAUTION: Security risk in production
        allowunsignedcerts="on"     # CAUTION: Security risk in production
        restpath="services/collector/raw?index=INDEXNAME&host=%hostname%&sourcetype=syslog&source=%application%-rsyslogd"
        httpheaders=[
            "Authorization: Splunk HECTOKEN"
        ]
        template="splunk-hec"
        batch="off"
    )
}

Pros:

  • JSON formatting helps with structured data
  • Still uses the efficient raw endpoint
  • Maintains metadata via URL parameters for consistent indexing

Cons:

  • Slightly more complex configuration
  • Still no batch processing
  • Limited to only formatting the message content as JSON

Method 3: Send Rsyslog Data to Splunk JSON Endpoint as Full JSON Event

This method formats the entire event as JSON and sends it to the standard HEC endpoint.

module(load="omhttp")   
# this template works with the json hec endpoint" restpath="services/collector"
template(name="splunk-hec" type="list" option.jsonf="on") {
  property(outname="event" name="%msg%" format="jsonf") 
  property(outname="host" value="%hostname%" format="jsonf")
  property(outname="source" value="%application%-rsyslogd" format="jsonf")
  property(outname="sourcetype" value="_json" format="jsonf")  
  property(outname="index" value="INDEXNAME" format="jsonf")
}
Ruleset( name="logs-to-splunk-hec"
queue.type="linkedlist"){
    action(
        type="omhttp"
        server="SPLUNKHEC.YOURDOMAIN.COM" 
        serverport="8088" 
        useHttps="on"
        skipverifyhost="on"         # CAUTION: Security risk in production
        allowunsignedcerts="on"     # CAUTION: Security risk in production
        restpath="services/collector/"
        httpheaders=[
            "Authorization: Splunk HECTOKEN"
        ]
        template="splunk-hec"
        batch="off"
    )
}

Pros:

  • Most flexible approach with complete control over event formatting
  • Metadata included directly in the JSON payload rather than URL parameters
  • Better for structured data and complex event formats
  • Can be extended to include additional fields and properties

Cons:

  • Most complex configuration
  • Slightly higher processing overhead
  • Still not utilizing batch processing capabilities

Security Considerations

You’ll notice in all three methods above, there are two potential security concerns:

skipverifyhost="on"         # CAUTION: Security risk in production
allowunsignedcerts="on"     # CAUTION: Security risk in production

These settings:

  • skipverifyhost="on": Disables verification of the Splunk server’s hostname against its SSL certificate
  • allowunsignedcerts="on": Allows connections to servers with self-signed or untrusted certificates

While these settings make testing and development easier, they create security vulnerabilities in production environments. For production deployments:

  1. Use properly signed SSL certificates for your Splunk HEC endpoint
  2. Set both parameters to “off” to enforce certificate validation
  3. Ensure proper certificate management on both the rsyslog and Splunk sides

Performance Optimization

While all three methods have batch="off" in our examples, enabling batch mode can significantly improve performance when dealing with high log volumes. HOWEVER, we couldn’t make batch work. Can you? Let us know!

batch="on"
batch.maxsize="1048576"  # 1MB max batch size

This will buffer multiple events and send them in a single HTTP request, reducing network overhead.

Which Method of sending Rsyslog Data to Splunk Should You Choose?

Each method has its own strengths:

  • Method 1: Best for simplicity and minimal configuration
  • Method 2: Good balance between simplicity and JSON structure
  • Method 3: Ideal for complete control and custom event formatting

At Ableversity, we recommend starting with Method 1 for basic setups, then progressing to Method 3 as your logging needs become more sophisticated.

Conclusion

Integrating rsyslogd with Splunk using the HTTP Event Collector provides a robust, scalable solution for centralized log management. The three methods outlined above offer different approaches depending on your specific requirements.

By properly configuring rsyslog to forward logs to Splunk, you’ll gain valuable insights into your systems’ behavior, security posture, and performance characteristics.

For hands-on training in Splunk and log management, check out our affordable, accessible courses at Ableversity.com. We’re committed to making tech education available to everyone.

Tags: