How to connect Fivetran with your Aptible databases

​

Overview

Fivetran is a cloud-based platform that automates data movement, allowing easy extraction, loading, and transformation of data between various sources and destinations. Fivetran is compatible with Aptible Postgres and MySQL databases.

​

Connecting with PostgreSQL Databases

⚠️ Prerequisites: A Fivetran account with the role to Create Destinations

To connect your existing Aptible PostgreSQL Database to Fivetran:

Step 1: Configure Fivetran

Follow Fivetran’s General PostgreSQL Guide, noting the following:

• The only supported “Connection method” is to Connect Directly

• pgoutput is the preferred method. All PostgreSQL databases version 10+ have this as the default logical replication plugin.

• The wal_level and max_replication_slots settings will already be present on your Aptible PostgreSQL database

  • Note: The default max_replication_slots is 10. You may need to increase this if you have many Aptible replicas or 3rd party replication using the allotted replication slots.

• The step to add a record to pg_hba.conf file can be skipped, as the settings Aptible sets for you are sufficient to allow a connection/authentication.

• Aptible PostgreSQL databases use the default value for wal_sender_timeout , so you’ll likely have to run ALTER SYSTEM SET wal_sender_timeout 0; or something similar, see related guide: How to configure Aptible PostgreSQL Databases

Step 2: Expose your database to Fivetram

You’ll need to expose the PostgreSQL Database to your Fivetran instance:

• If you’re running it as an Aptible App in the same Stack then it can access it by default.

• Otherwise, create a Database Endpoint. Be sure to only allow Fivetran’s IP addresses to connect!

​

Connecting with MySQL Databases

⚠️ Prerequisites: A Fivetran account with the role to Create Destinations

To connect your existing Aptible MySQL Database to Fivetran:

Step 1: Configure Fivetran Follow Fivetran’s General MySQL Guide, noting the following:

• The only supported “Connection method” is to Connect Directly

Step 2: Expose your database to Fivetram

You’ll need to expose the PostgreSQL Database to your Fivetran instance:

• If you’re running it as an Aptible App in the same Stack then it can access it by default.

• Otherwise, create a Database Endpoint. Be sure to only allow Fivetran’s IP addresses to connect!

​

Troubleshooting

• Fivetran replication queries can return a large amount of data per query. Fivetran support can tune down page size per query to smaller sizes, and this has resulted in positive results as a troubleshooting step.

• Very large Text / BLOB columns can have a potential impact on the Fivetran replication process. Customers have had success unblocking Fivetran replication by removing large Text / BLOB columns from the target Fivetran schema.