Serverless Integration Engine — ALX / ALU

Overview

African Leadership X (ALX) and the African Leadership University (ALU)—organisations within the African Leadership Group—deliver technical training and leadership programmes across Africa. As part of major enrollment drives, including the flagship “Do Hard Things Challenge” (DHTC), tens of thousands of applicants progress concurrently through multi-stage application forms, technical coding challenges, and onboarding workflows.

Managing this volume across disparate third-party platforms created operational friction. Marketing squads used Infusionsoft (Keap) as their core CRM, student selection squads evaluated candidates via Holberton School and Harver assessment APIs, application teams collected submissions via Formstack, and support teams managed incoming inquiries through ArtiBot.ai and Zendesk.

Working as an independent integration engineer, I designed and deployed ALX Integrator, an event-driven serverless middleware pipeline built on AWS Chalice. The system decoupled inbound webhooks from downstream processing, managed authentication state, and synchronised student progression records in real time without exceeding external API rate limits.

Architecture & Event Flow

The platform was engineered entirely on AWS serverless infrastructure, running in the us-east-1 region. Inbound traffic from forms and webhooks was ingested via Amazon API Gateway, queued onto Amazon SQS for asynchronous processing, and executed by dedicated AWS Lambda worker functions backed by Amazon DynamoDB for state caching:

Engineering Contributions

1. Event-Driven Decoupling & Queueing Pipeline

Direct synchronisation between marketing forms and third-party APIs risks request loss during high-volume cohort launch windows. To provide reliability:

• Built the integration services using AWS Chalice, structuring API Gateway routes and asynchronous Lambda handlers within a modular Python codebase.
• Decoupled incoming webhooks from external API calls by pushing normalised payloads directly into Amazon SQS (dhtc-events).
• Configured Lambda consumers to process SQS messages in controlled batches (batch size: 5), preventing concurrent downstream connection saturation.
• Implemented an Amazon SQS Dead-Letter Queue (dhtc-events-dead-letter) with automated daily retry schedules, ensuring transient third-party HTTP errors or schema mismatches could be isolated and inspected without dropping student records.

2. Multi-Platform Admissions & Assessment Pipeline

Engineered bi-directional data synchronisation across the applicant lifecycle:

• Holberton School Assessment Tracking: Connected ALX’s applicant funnel to Holberton’s technical evaluation endpoints. Synchronised student progression through Level 1 (Eniac and RTFM exercises), Level 2 (Website project), and Level 3 (timed test). Aggregated cumulative test errors (nb_errors_level_1, nb_errors_level_2) and scores (level_2_website_score_percentage, level_3_score) into CRM custom fields to automate admissions decisions, while generating unique authentication magic links for candidate portals.
• Formstack Webhook Ingestion & Partial Submission Recovery: Handled full application form submissions via HMAC-signed webhooks. Additionally, scheduled a recurring 5-minute CloudWatch rule to poll Formstack’s partial submission API, storing abandoned application states in DynamoDB to trigger automated applicant recovery sequences.
• ArtiBot.ai & Support Escalation: Ingested lead capture chatbot transcripts, created or updated CRM profiles with inquiry metadata, and routed escalations directly into Zendesk as support tickets.
• Multi-Touch Marketing Attribution: Captured Google Analytics query parameters (GASource, GAMedium, GACampaign) and synchronised them into first-touch attribution fields within Infusionsoft for marketing cohort analysis.

3. Rate-Limit Shielding & State Persistence

Downstream SaaS platforms impose strict throughput constraints—specifically Infusionsoft’s daily cap of 150,000 API requests.

• Engineered a caching layer in Amazon DynamoDB (dhtc-app) to track contact identifiers, applied tags, and assessment states. Before making outbound CRM calls, worker functions checked DynamoDB, eliminating redundant queries and keeping daily volume well within rate limits.
• Handled OAuth2 session lifecycles using Authlib, persisting access and refresh tokens in DynamoDB (dhtc-access-tokens). Scheduled an automated CloudWatch rule running every 23 hours to refresh credentials proactively, preventing token expiration during batch operations.

4. Operational Observability & Logging

Operating asynchronous queues without a dedicated DevOps team required straightforward observability:

• Used structured JSON logging across all Lambda functions with execution context tracing.
• Authored query patterns in Amazon CloudWatch Logs Insights (scanning over 135,000 log records in seconds) to track error rates, monitor throttled requests, and identify payload anomalies across worker queues.
• Maintained a test suite in pytest with extensive mock fixtures covering all connected service payloads, validating edge cases such as token expiration, signature verification failures, and malformed request bodies prior to deployment.