Hey everybody, this is Zachary Lyons, integrations developer at Hypha HubSpot Development, and in this post, we’ll be discussing the most recent puzzle we solved at the shop: building a custom integration pipeline for adding thousands of records to Constant Contact. Although a HubSpot-Constant Contact integration has been out for a few years, it didn’t quite meet our client’s needs in terms of being able to assign data to specific lists within Constant Contact; using it would have created a lot of manual assignment tasks for our client’s employees. Given this complexity, we determined that the best solution was to build our own custom integration between HubSpot and Constant Contact.
The Problem
Our client is a travel company that pairs travel agents with customers looking to book vacations based on their personal styles and budgets. They came to Hypha in need of a way to automatically upload thousands of enrolled travel agents to Constant Contact—an email marketing platform that helps users create and deploy campaigns.
The challenge was building a pipeline that would allow a constant stream of these agents to flow, via our integration, into Constant Contact, with the desired end result being complete automation of the process from signup to list assignment and email outreach. The first step: Figure out storage.
The Solution
Storing Agent Records: the Database
To build an integration pipeline enabling our client to upload thousands of contacts to their onboarding email campaign, we need a place where those records can be stored. For expediency, we decided to use Amazon Web Service’s Relational Database Service (AWS RDS). Utilizing a cloud-based database means we can quickly add additional storage if needed, as well as delegate the task of maintaining and provisioning the database to a third party—in this case, AWS.
We then quickly connected the database using a MYSQL admin tool—the free and open-source HeidiSQL to be specific—and set up a table to store our data. This included several basic pieces of information about our travel agents: first name, last name, country (for localization purposes), and enrollment date. Adding those fields to our table set the stage for storing the thousands of records we’d be processing daily! It wouldn’t be sufficient to merely create and connect a database to Constant Contact via our integration code, however. We needed to tackle the next component of our case study: Authorization.
Authorization: OAuth & Refresh Tokens
Before we could upload our Travel Agent records to the onboarding email campaign on Constant Contact, we had to authenticate our identity. Constant Contact takes protecting their marketing recipients’ information seriously, and so we had to set up a bit of infrastructure to prove we are who we say we are. This required us to build an OAuth Client application, where a human signs into their Constant Contact account, as well as a Token Manager app, which keeps that initial OAuth authorization fresh for the duration of our pipeline’s lifespan.
OAuth Client App
The first step in this process is building the OAuth Client App. For those unfamiliar, OAuth is an authorization framework that grants third parties access to your data, without giving them credentials or passwords directly.
Our OAuth Client App works by doing the following:
- Setting up a simple server with Node.js Express, to run locally
- Creating a route in your server that redirects to the Constant Contact /OAuth2 endpoint
- Redirecting browsers to Constant Contact’s sign-in page
- Constant Contact will return an access token, which exchanges for an API + Refresh token (more on this in the next section)
If any of this is confusing I recommend reading Constant Contact’s API documents, posts from their community forum, or diving in and learning a little more about OAuth itself.
Token Manager & Parameter Store
With our Access token in hand, we use the one retrieved in the previous section to fetch API + Refresh tokens from Constant Contact’s /token endpoint.
The two tokens we receive in exchange for our Access token do the following:
- API Token: This string of numbers and characters will be used every time we call the API to insert Travel Agent records from the database onto a list for onboarding emails. This token is only good for about six hours before it expires.
- Refresh Token: This string of numbers and characters can be exchanged for a new pair of API + Refresh tokens.
The challenge now is building a tool that exchanges our old tokens for new ones every six hours. We can have code that lives in an AWS Lambda, and we can schedule the code by setting up an EventBridge cron job to serve as a trigger for the Lambda. Lambda code is read-only, so we need to find a place to read and write our tokens to/from. Enter AWS Parameter Store!
Parameter Store enables us to dynamically read and write sensitive environmental variables or parameters for AWS resources. By connecting our Lambda to the Parameter Store and scheduling it to run every six hours, we can build a system where our Token Manager periodically verifies that we have fresh API tokens whenever we need them.
Now that we’ve constructed that infrastructure, we never have to think about Authorization again!
Integration Code: Tying It All Together
Now that we’ve built our Database as the entry point for our integration pipeline, as well as an OAuth Client App and Token Manager to handle our Authentication and Authorization, all that is left is building our integration code to tie all the pieces together! We’ll be building the integration in another Lambda, similar to our Token Manager.
The integration code follows a simple set of instructions to connect everything we’ve built thus far:
- Connect and fetch all new Travel Agent records for the past 24 hours from our Database—Using the Node.js mysql2/promise package, which adds async/promise support. It’s lovely!
- Fetch the Constant Contact API key from Parameter Store—Since we’re retrieving the token from Parameter Store and not from Token Manager directly, these two pieces of infrastructure never need to communicate. Yay for modularity!
- Map Database information to field names recognized by Constant Contact.
- Upload the list of Agents to Constant Contact as a batch via the /contacts_json_import endpoint.
The Impact
- With the integration acting as the connective tissue between all of the infrastructure we’ve built, we can now kick back and watch Agents get added to the onboarding campaign lists!
- The integration pipeline was constructed in modules, so if any individual piece breaks down or needs a facelift, we can make changes without impacting the rest of the system.
- Most importantly: We’ve automated away a data entry task so our client’s team can concentrate on performing other critical duties and problem solving that demand humans!
Hypha HubSpot Development: Integration Specialists
At Hypha, we specialize in developing customized integration solutions for our clients that streamline workflows, save time and resources, and improve efficiencies. From complex integrations to consultative expertise on how to best align your ecosystem with strategic goals, our highly trained team of integration, design, and demand generation experts possess the knowledge and experience to achieve and exceed all your business objectives.
