How We Built Reliable Event Aggregation at Scale

We operate an event discovery platform that aggregates events from venues across multiple cities. What started as manual data collection evolved into a fully automated workflow orchestration system that processes 6x more events with 99% less manual effort.

The Manual Era

Early on, our process was straightforward but brutal: three team members manually visiting venue websites, copying event details into spreadsheets, and updating our database. Each week consumed 9 person-hours to collect roughly 40 events from 30 businesses.

The problems were obvious: this didn't scale, data quality was inconsistent, and we couldn't experiment with improvements quickly. Every change required retraining people and updating documentation.

The Challenge

Web scraping is inherently fragile. Venue websites use different platforms (Eventbrite, custom CMSs, static HTML), layouts change without notice, and third-party APIs (Mapbox geocoding, OpenAI vision) have rate limits and intermittent failures.

Traditional approaches fall short:

• Cron jobs: No visibility into failures, retries are manual, debugging requires log diving
• AWS Step Functions: Steep learning curve, versioning is painful, JSON definitions are massive
• Airflow: Requires infrastructure maintenance, complex deployment process

We needed orchestration that was:

• Visible — know instantly when workflows fail and why
• Resilient — handle transient API failures without manual intervention
• Flexible — deploy new scraper logic without breaking in-flight jobs
• Debuggable — see inputs and outputs of every step without parsing logs

The Solution: Workflow Orchestration with JustWorkflowIt

We built two orchestrated workflows that handle the entire event aggregation pipeline:

Key Advantages We Gained

1. Automatic Retries

The Mapbox geocoding API rate-limits us at 600 requests/minute. Instead of writing custom retry logic in every Lambda, JustWorkflowIt handles exponential backoff automatically. When a step fails with a transient error (HTTP 429, timeouts), the workflow pauses, waits, and retries without our intervention.

// Before: Manual retry logic in every Lambda
try {
  const geocode = await mapboxClient.geocode(address);
} catch (err) {
  if (err.statusCode === 429) {
    await sleep(exponentialBackoff());
    // retry logic...
  }
}

// After: JustWorkflowIt handles it
// Just write the business logic, retries are automatic

2. Workflow Versioning

We've updated our venue scraper logic repeatedly in production. With versioning, we deploy a new workflow version while old in-flight jobs continue running the previous version. No crashes, no reprocessing, no rollback drama.

Example: We improved our HTML parser to detect event cancellations. When we deployed the new version, jobs already running the previous version completed successfully without touching the new code.

3. Workflow Resumption for Expensive Operations

GPT-4 Vision API calls cost money. Early on, we'd run the entire EventDetailDataAggregator workflow, realize the downstream validation logic had a bug, fix it, and rerun everything—including re-paying for the Vision API call.

Workflow resumption changed this: We run the workflow through the expensive ScrapeEventDetailsVision step once, then pause. We experiment with downstream functions (ValidateEvent, ScoreEvent) by resuming from the checkpoint. The Vision API output is already captured—we iterate on logic without re-executing expensive steps.

4. JSON Live Editing

Workflow definitions are JSON. In the JustWorkflowIt UI, we can edit the JSON directly and deploy a new version instantly—no CDK redeploy, no CI/CD pipeline wait. This is critical for rapid iteration.

Example tweak:

// Change conditional routing in 30 seconds
{
  "transitionToStep": {
    "if": [
      { "!==": [ { "var": "eventImageUrl" }, null ] },
      "EnrichEventFromImage",
      "ValidateEvent"  // Skip enrichment if no image
    ]
  }
}

5. Input/Output Visibility

Every step's inputs and outputs are visible in the UI. When ValidateEvent rejects an event, we see exactly what data it received, what the validation rule checked, and why it failed. No log parsing, no CloudWatch Insights queries—just click the step and read.

6. Complete Visibility

We can see at a glance:

• Which venues have stale data (no successful job in 7 days)
• Which scrapers are failing consistently (same step failing across jobs)
• How long each step takes (identify performance bottlenecks)
• What percentage of events pass validation

This shifted us from reactive ("Why aren't new events showing up?") to proactive ("Venue X's website changed—let's update the scraper before users notice").

Results

What used to take 9 person-hours per week now happens with a button click. Our team focuses on improving scraper logic, not copying event details.

What We Learned

Orchestration is a Force Multiplier

The individual Lambdas aren't complex. The value is in reliable coordination: retries, versioning, visibility, and resumption. We spend time improving the scrapers, not debugging why workflows half-completed.

JSON Definitions Are Underrated

Step Functions ASL is verbose and hard to read. JustWorkflowIt's JSON is clean. We version control it, review changes in PRs, and non-engineers can read it.

Visibility Changes Behavior

When failures are visible, you fix them. When inputs/outputs are visible, you debug faster. When metrics are visible, you optimize. The UI transformed how we operate.

Workflow Resumption is a Game-Changer

Being able to experiment with downstream logic without re-running expensive steps accelerated our iteration speed by weeks. It's not just about cost—it's about velocity.

Conclusion

Workflow orchestration isn't glamorous, but it's essential. JustWorkflowIt gave us retries, versioning, visibility, and resumption without forcing us to become Step Functions experts or maintain Airflow infrastructure.

We went from 40 events manually collected to 250+ events automatically aggregated. The automation works, and when it doesn't, we know why and fix it fast.

If you're scraping data, calling APIs, or running multi-step processes that occasionally fail for reasons outside your control, orchestration might be your unlock.