You must have the Firehose module to use this feature.
There are several methods by which the data can reach your systems:
Secure FTP
AWS S3
Learn more in AWS S3 - Import and export.Webhook
Fresh Relevance doesn’t store the data. This is a real-time feed and when it's gone, it's gone. Best practice is to configure the Firehose for your account as early as possible. This allows you to control how and where the data is sent.
Since October 1st 2020, the system automatically exports Firehose data to a fallback configuration - this is an AWS S3 bucket managed by Fresh Relevance.
Set up your data export
1. Create an export channel
Expand the User menu and go to Settings > Exports.
Select Data Export Channels.
Under Set up a new channel, expand the drop-down menu and select from:
Export Firehose to Webhook
Export to FTP
Export to AWS S3
Select CREATE EXPORT CHANNEL.
Enter the required details for the channel type you have chosen, then select SAVE.
2. Configure the Firehose
Fallback configuration
The system automatically detects client accounts which have the Firehose enabled on the account but no Firehose channel setup. It configures the Firehose to send data in JSON format to an S3 bucket managed by Fresh Relevance, in a folder for each client account: {website ID}/Firehose.
The configuration process creates the following:
A System Firehose Channel configured with the following options: JSON output, catchup on identify, and send non-identified person records.
A System Export Channel sending to S3.
Signal routing to send all normal signals to the Firehose.
If you decide that you want old data which was sent by the Firehose, TechOps can provide access to your folder within the S3 bucket. Contact Support to arrange this.
This access allows you to download the JSON files. The old data can’t be replayed through the Firehose and is as-is, in JSON format, without any field mapping.
Data in the Fresh Relevance managed S3 bucket is retained for 36 months or until your contract with Fresh Relevance ends.
To configure the Firehose:
Go to Data > Firehose.
Select CONFIGURE FIREHOSES.
Expand the Set up a new Firehose drop-down menu and select Send Firehose Data to an Export Channel, then select CREATE NEW FIREHOSE.
Select the Export Channel that you set up in Step 1.
Expand the drop-down menu and select whether or not to use field mapping.
Use Field Mapping (CSV: required, JSON: optional).
Required when CSV is selected in the export configuration. If JSON is selected in the export configuration, mapped fields are exported under a keymapped_fields.Do not use field mapping (JSON only).
If selected, does not run any field mapping for each signal exported. Ensure JSON is selected in the export configuration.
To configure a new mapping, select Field Mapping.
To receive notification of errors with the export, enter an email address in the Error Alert Email field.
For Firehose Minutes Between Files, enter the number in minutes between uploads from the staging area to the FTP server.
For Filename Pattern, enter the pattern for the filename to export. Leave blank to use the default format
YYYY-MM-DD-hh-mm-ss-ffff, for example,firehose-%Y-%m-%d-%H-%M-%S-%f (firehose-YYYY-MM-DD-hh-mm-ss-ffff).
Learn more about date formatting in this Python documentation.To include unidentified records in the export, select Include Anonymous Records.
To pause the export job, select Is Service Paused?.
Select Enable Firehose Catch Up to ensure that signals which have been captured for an anonymous session are sent again when the session is identified.
By default, only the most recent 30 signals are stored and are available to be sent.Select Send Person Identified Signals to include signals for sessions or person records which have not yet been identified in the Firehose data.
Select SAVE.
If you want to export Query Collection Data that came from the URL query collection, first check the configuration of Query Collection Fields. To do this, expand the User menu and go to Settings > Website Settings > Query Collection Fields.
If you want to reduce the data volume by exporting less rows, for example if you don't want product browsed data, contact Fresh Relevance Support for assistance.
Firehose destinations
Secure FTP
Fresh Relevance exports real-time data onto your FTP server, saving it in new files named with the current date-time, and you configure your third-party system to load it from there.
For further details, contact the technical support team responsible for your FTP server and the system that you are integrating with.
Fresh Relevance supports three versions - FTP, SFTP and FTPES - your technical support team should know which you need.
Once you’ve set up Fresh Relevance to export to your FTP server, wait a while and then look at the data. Each data line should have a tms_type (or type depending how you mapped it) with the following values:
Product Browsed (
pd)Product Carted (
c)Product Purchased (
t)Product Abandoned (
bfor cart abandon, orbafor browse abandon)Custom Signal (
cu)
with the other fields you chose to map. Complete the Fresh Relevance setup so it's exporting the data you want with the field names you want, before starting to load data into your third-party system.
Learn more in Firehose data format.
Send data using JSON files
Data can be sent to the Firehose using JSON files. Here’s a sample file containing a single record for a cart abandon (b) signal:
Sample file
Sample file
[
{
"signal_id": {
"$oid": "5ca4b38d32b4954a1119bb59"
},
"did": "auto90079",
"pid": "5b5b3924390cf45d2d03a808",
"cart": {
"sbr": "test",
"dont_check_marketing_pressure": true,
"curr": "GBP",
"qty": 2,
"p": [
{
"img": "https://s3-eu-west-1.amazonaws.com/tms-demo-store-images/product/test/canon-powershot-a630-8mp-digital-camera-with-4x-optical-zoom-2.jpg",
"up": "$139.99",
"uv": 139.99,
"prid": "canon-powershot-a630-8mp-digital-camera-with-4x-optical-zoom",
"qty": 1,
"u": "http://demostore.triggeredmessaging.com/canon-powershot-a630-8mp-digital-camera-with-4x-optical-zoom.html",
"n": "Shiny Camera (Test Product)",
"desc": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent molestie pulvinar tortor a suscipit. Donec iaculis placerat lectus, sed aliquam metus tempor a. Integer non tincidunt nulla. Nam nec nibh."
},
{
"img": "https://s3-eu-west-1.amazonaws.com/tms-demo-store-images/product/test/universal-camera-charger.jpg",
"up": "$19.00",
"prid": "universal-camera-charger",
"qty": 1,
"u": "http://demostore.triggeredmessaging.com/universal-camera-charger.html",
"n": "Universal Camera Charger (Test Product)",
"desc": "Lorem ipsum dolor sit amet, consectetur adipiscing elit. Praesent molestie pulvinar tortor a suscipit. Donec iaculis placerat lectus, sed aliquam metus tempor a. Integer non tincidunt nulla. Nam nec nibh."
}
],
"dt": {
"$date": "2019-04-03T13:22:21.458Z"
},
"cp": "\u00a3158.99",
"cv": 158.99
},
"dt": {
"$date": "2019-04-03T13:22:21.527Z"
},
"_id": {
"$oid": "5ca4b38e32b49549cce9bbc3"
},
"type": "b",
"evdt": {
"$date": "2019-04-03T13:22:21.458Z"
},
"unique_id": {
"$oid": "5ca4b38d32b4954a1119bb59"
}
}
]
Send data using CSV files
To configure the CSV files sent by the Firehose:
Go to Data > Firehose.
Select CONFIGURE FIREHOSES.
To create a new Firehose, choose the firehose type then select CREATE NEW FIREHOSE.
To edit an existing field mapping, select the highlighted entry in the Field Mapping column.
Set what fields in the system get loaded to the Firehose by FTP.
The following example is for a FastStats installation, but yours may vary. Learn more in Firehose data format.
Records exported by the Firehose may not be in date order, and records from around the same time may be found in more than one file. This is because of the way data is processed by multiple servers.
Also, the order of fields is not fixed and should not be relied on. The order of fields may change at any time without warning, so you should always import data using the field names in the first row.
Webhooks
The data can also be sent to a Webhook. The JSON structure documented above is sent as a the body of the request.