Blog - Cloud ComputingCommentary from our cloud experts around the globeCommentary from our cloud experts around the globePUTINGCOMCLOUDBLOG

April 27, 2018
In control: Analyze VPC flow logs from multiple AWS accounts with Kinesis Firehose and Splunk
By: Kiran Gekkula

Flow logs capture information about IP traffic going to and from network interfaces in virtual private cloud (VPC). They’re used to troubleshoot connectivity and security issues, and make sure network access and security group rules are working as expected. InfoSec and security teams also use VPC flow logs for anomaly and traffic analysis.

Thanks to recent native Kinesis integration with Splunk, it’s become easy to stream data to Splunk and extract valuable insights. For customers with multiple accounts, it is more efficient to do log analysis with centralized data and dashboards.

In AWS, custom analysis of streaming data from multiple accounts can be done by collecting federated logs for central processing. Custom apps are built using streaming data which is assembled across the accounts and delivered using CloudWatch Logs Destination, Subscriptions and Kinesis.

Kinesis Data Firehose is a fully managed, reliable and scalable solution for delivering real-time streaming data to destinations S3, Redshift, Elasticsearch Service and Splunk. It can also be configured to transform data before that data is delivered. As a platform-as-a-service solution, it provides significant cost savings.

Splunk captures and indexes data in real time and uses it to generate visualizations. There are two apps “Splunk Add-on for AWS” and “Splunk App for AWS” with built-in searches, macros dashboards and panels for VPC Traffic Analysis and VPC Security Analysis, in addition to other AWS-related visualizations.

Cross-account data sharing

Using CloudWatch Logs Destination, data can be sent from multiple sender accounts to a single receiving account. In AWS organizations, it can also be used to push down policies and control through the organizational structure. For data to be shared, both sender and recipient details are needed.

So how to get started? First, set up a log destination (Kinesis Firehose) in the data recipient account. The receiver account shares log destination information with the sender account. Access to the sender accounts needs to be granted using IAM policies.

Data-sender accounts create subscription filters, with receiving accounts designated as “log destination."

Our previous blog explained how to ingest flow logs into Splunk from an AWS account using Firehose. This time we look at how to ingest data from multiple accounts.

Solution overview

Set out below is the architecture and dataflow for VPC flow logs from multiple accounts into Kinesis Firehose, the central logging account, and from there into Splunk. This blog post will discuss the following steps:

  1. Set up Splunk HTTP Event Collector.
  2. Create Log Destinations and Kinesis Firehose in the receiving logging account and set up permissions for sender accounts to stream data.
  3. Set up Splunk HEC as the destination for Kinesis Firehose in the central logging account.
  4. Set up CloudWatch subscription filters on sender accounts with the receiving account as destination.

VPC dataflow architecture

Step-by-step process

Set up one account as receiving account (222222222222) and one as sending account (111111111111).

Create HTTP Event Collector in Splunk and then set up Kinesis Firehose and Logs Destination in receiving account 222222222222. Below is the data flow for flow logs.

Data flow for VPC flow logs

Create a Splunk HTTP Event Collector

First create HEC in Splunk. From Splunk Web, go to Data Inputs, HTTP Event Collector and add a new token for receiving data over HTTP. Check Enable indexer acknowledgement while creating token.

Creating HEC in Splunk

Select source type as aws:cloudwatchlogs:vpcflow and index as main.

Creating HEC in Splunk

Set up Kinesis Firehose with Splunk as destination

  1. Create a bucket to store failed events
    aws s3api create-bucket --bucket splunk-firehose-failed-events

  2. Create service role for Lambda.

    aws iam create-role --role-name lambda-firehose-basic-role --assume-role-policy-document file://trust-policy-for-lambda.json

    aws iam put-role-policy --role-name lambda-firehose-basic-role --policy-name lambda-firehose-policy --policy-document file://lambda-firehose-policy.json Here is the trust policy for lambda: trust-policy-for-lambda.json

      "Version": "2012-10-17",
      "Statement": [
          "Effect": "Allow",
          "Principal": {
            "Service": ""
          "Action": "sts:AssumeRole"

    Lambda permissions: lambda-firehose-policy.json

        "Version": "2012-10-17",
        "Statement": [
                "Effect": "Allow",
                "Action": "logs:CreateLogGroup",
                "Resource": "arn:aws:logs:us-east-1:222222222222:*"
                "Effect": "Allow",
                "Action": [
                "Resource": [
                    "arn:aws:logs:us-east-1: 222222222222:log-group:/aws/lambda/lambda-splunk:*"
  3. Create Lambda function for log processing (receiving account)
    Go to Lambda, create function and select blueprint. We are going to search for kinesis-firehose-cloudwatch and select the kinesis-firehose-cloudwatch-logs-processor.
    Lambda function for log processing Click on configure and name the function lambda-cw-transform, choose the Lambda service we created above (lambda-firehose-basic-role).
    Lambda function for log processing
  4. Create Firehose Stream with Splunk as destination (receiving account)
    Go to the AWS Console, select Kinesis and Data Firehose and create new data firehose stream. Enter the name and Direct PUT for source. 
    Firehose Stream with Splunk

Enable record transformation and select the Lambda function we created before.

Transform records with AWS Lambda

Select Splunk as destination.

Splunk destination

Input the Splunk HEC URL and token which we created initially.

Splunk HEC configuration

Select S3 backup bucket. We will only select to log failed events.

S3 backup

Select defaults for S3 buffer conditions, compression and encryption, error logging and for IAM choose to create a new Firehose role. Review and create delivery stream.

The final Firehose stream looks like this in its active state:

Firehose stream

Create CloudWatch Logs Destination with Kinesis Data Firehose as destination (receiving account)

We need to create a role and policy for CloudWatch Logs Destination. Create a role with the below policy.

aws iam create-role \
      --role-name cw-to-kinesis-role \
      --assume-role-policy-document file://trust-policy-for-cw.json
aws iam put-role-policy --role-name cw-to-kinesis-role --policy-name \
  cw-firehose-policy --policy-document file://cw-firehose-policy.json
copy this to cw-firehose-policy.json

Create CloudWatch Logs Destination

aws logs put-destination \
      --destination-name "awslogs-destination-splunk" \
      --target-arn "arn:aws:firehose:us-east-1:283918037053:deliverystream/firehose-splunk-delivery-stream" \
      --role-arn "arn:aws:iam::222222222222:role/cw-to-kinesis-role"

Attach a policy to Logs Destination for sender accounts to access receiving account destination.

aws logs put-destination-policy \
          --destination-name "awslogs-destination-splunk" \
          --access-policy file://sender-access-policy.json

sender-access-policy.json (Note the account id 111111111111 of the sending account)
  "Version" : "2012-10-17",
  "Statement" : [
      "Sid" : "",
      "Effect" : "Allow",
      "Principal" : {
        "AWS" : "111111111111"
      "Action" : "logs:PutSubscriptionFilter",
      "Resource" : "arn:aws:logs:us-east-1:222222222222:destination:awslogs-destination-splunk"

Setup sender accounts with CloudWatch Subscription Filters (sender account)

First setup flow logs in the sender account’s VPC.

Creating flow logs in VPC

Create a subscription filter in the data sender account

Create a subscription filter with receiving account Logs Destination awslogs-destination-splunk (note the account ID of the receiving account)

aws logs put-subscription-filter \
    --log-group-name "/aws/demovpc/flowlogs" \
    --filter-name "sender-flow-log-filter" \
    --filter-pattern "" \
    --destination-arn "arn:aws:logs:us-east-1:222222222222:destination:awslogs-destination-splunk"

You should be able to see the subscription filter in CloudWatch Logs:

CloudWatch Logs

Splunk app for AWS dashboards

It’s time to verify that events are being sent to Splunk. In just a few seconds, data is sent to Kinesis firehose and finally on to Splunk. If you search the index, you should be able to see the flow log data.

Splunk app for AWS dashboard

Go to the Splunk App, then to Insights, and check VPC Traffic Analysis and Security Analysis. These are predefined dashboards, searches and panels, all of them part of Splunk App for AWS.

VPC flow logs - security analysis

VPC flow logs - traffic analysis


AWS Services Kinesis Firehose, Log Destinations and Subscriptions filters make it easy to aggregate and stream data to destinations like Elastic Search, Redshift, Splunk and analyze it in real time. They can also be used to troubleshoot connectivity and security issues, and make sure network access and security group rules are working as expected.

Splunk has other inputs to poll data from AWS for visualizations. This push solution can be applied to other CloudWatch logs and Splunk graphs, reports, alerts and dashboards can all be easily generated.


Popular Tags

    More blogs on this topic