Skip to main content

Map from multiple sources

Everything in the MapCraft walkthrough assumed one source schema feeding one target. Real ingests often need more — a master table joined to a transaction feed, or a reference/lookup file enriching the primary records. Multi-source input lets a single mapping attach two or more source schemas, give each an @alias, and define the join that stitches their records together before transformation rules run.

By the end of this page you will have:

  • Enabled the multi-source feature flag.
  • Attached a second source schema to an existing mapping under an alias.
  • Configured a join between the two sources — via AI suggestion or manually.
  • Attached a reference schema as a lookup table (left-outer join shortcut).
  • Run a multi-source preview to see joined rows and rule outputs.
You'll need
  • A MapCraft project open at /mapcraft/{projectId} with a primary source schema already attached (see Connect a source via the Source Picker).
  • At least one more source-role schema materialized in your workspace — the attach picker lists org-wide schemas, so it can come from any project.
  • NEXT_PUBLIC_MAPCRAFT_MULTI_SOURCE=true set on the frontend deployment (see Step 1).

Step 1 — Enable the feature flag​

Multi-source v1 ships behind a progressive-rollout flag. When it's off, the UI is byte-identical to single-source MapCraft.

Set on the frontend build environment:

NEXT_PUBLIC_MAPCRAFT_MULTI_SOURCE=true   # also accepts 1 / yes / on
Flag changes require a rebuild

NEXT_PUBLIC_* values are inlined into the client bundle at build time. Flipping the flag means redeploying the frontend — it is not a runtime toggle.

Step 2 — Open the Sources panel​

With the flag on, the project page shows a Sources panel next to the source-schema panel. Your existing source appears as a chip: @<alias> with a PRIMARY role badge (green). Schemas attached later get SECONDARY (grey) or LOOKUP badges.

The alias matters: every secondary-source field is addressed as @<alias>.<field.path> in join conditions and rule expressions.

Step 3 — Attach a second source​

  1. Click Attach in the Sources panel header.
  2. In the Schema dropdown, pick the schema to attach. The list shows org-wide source schemas as name (N fields); schemas already attached to this mapping are hidden.
  3. Enter an Alias — e.g. crm. Aliases must start with a letter and contain only letters, digits, or underscores; each alias must be unique within the mapping.
  4. Click Attach.

The new chip appears with a SECONDARY badge. The first source you ever attach becomes primary automatically; you don't pick roles by hand in v1.

Removing sources

The trash icon on a chip detaches that source — with two guards: the primary can't be removed while secondaries exist, and a source referenced by the join condition can't be removed until you clear the join first.

Step 4 — Configure the join​

Once two sources are attached, the panel's footer button changes from "Attach a second source to configure a join." to Configure join. Click it to open the Join condition editor.

4a. Ask the AI first​

Click Suggest join keys. MapCraft compares field names, types, and sample evidence across the two schemas and returns ranked candidates, each shown as @left.path = @right.path with a one-line rationale and a confidence pill (green ≥ 85 %, amber ≥ 60 %).

Click a suggestion to populate the form. If the response says "No obvious join keys found", fall through to manual entry — nothing is blocked.

4b. Or enter it manually​

  • Join type — pick one of:

    OptionBehavior
    Inner (drop unmatched)Only records present on both sides survive.
    Left-outer (keep unmatched left)Every primary record survives; missing right side becomes nulls.
    Right-outer (keep unmatched right)Mirror of left-outer.
    Full-outer (keep both)Union of both sides.
    Cross (Cartesian product)Every left × right pair — no condition needed; the rows collapse automatically.

  • Left source / Right source — the @alias pair the conditions apply to.

  • Left field path = Right field path — one condition row, e.g. customer_id = id.

  • Add condition (multi-key) — adds more rows for composite keys. All rows share the same source pair (multi-key on two sources, not an N-way join).

Click Save join. The editor closes and the panel button now reads Edit join: inner on @orders.customer_id = @crm.id. On the canvas, a join node chip appears showing the join kind and the same description. Clear join (inside the editor) removes the spec entirely.

Step 5 — Attach a lookup table (optional shortcut)​

For reference data — state codes, currency tables, product catalogs — you don't need the full join editor. The attach form has a shortcut:

  1. Click Attach, pick the reference schema, give it an alias (e.g. states).
  2. Tick Attach as lookup table (left-outer join). Two extra fields appear:
    • Primary field (match left) — the field on your primary source, e.g. state_code.
    • Lookup field (match right) — the matching field on the reference schema, e.g. code.
  3. Click Attach.

This attaches the source and writes the left-outer join in one atomic step. The chip gets a LOOKUP role badge. (The option only appears once a primary source exists — a lookup needs something to anchor to.)

Step 6 — See multi-source state across the views​

  • Canvas view — the secondary source renders as a compact summary card at the top of its own lane: @alias, schema name, and field count. In v1 this card is informational — you cannot drag-connect individual secondary fields from the canvas yet (that's the planned secondary-source field-mapping slice). The join node between the lanes shows the live join spec.
  • Table view — when a mapping has ≥ 2 sources, each row gains a small @alias pill showing which source the row's path resolves from; rows resolved against the primary show —.

Step 7 — Run a multi-source preview​

Navigate to Test Preview (/mapcraft/{projectId}/preview). Below the regular single-source panel, the flag-gated multi-source test panel appears.

  1. Paste a small JSON array of sample records into the textarea for each attached source (one textarea per alias, pre-seeded with a placeholder).
  2. Paste the transformation rule under test as OperationTree JSON in the tree input.
  3. Click Run preview.

The backend executes the join over your samples, evaluates the rule, and returns the verifier verdict, the joined rows, and per-sample rule outputs rendered side-by-side.

Verify​

Multi-source input is working end-to-end if all of these hold:

CheckWhere to look
Both chips visible with correct role badgesSources panel: @orders PRIMARY, @crm SECONDARY (or LOOKUP).
The join is persistedPanel button reads Edit join: <kind> on @a.x = @b.y after a reload.
Canvas reflects itSecondary-source summary card + join node chip with the join kind.
Table view shows provenance@alias pill in the source column for rows from a secondary source.
Preview joins your samplesRun preview returns joined rows (not just primary records) and a verifier verdict.

Troubleshooting​

The Sources panel doesn't appear at all. The feature flag is off in this build. Confirm NEXT_PUBLIC_MAPCRAFT_MULTI_SOURCE=true was present at build time and redeploy the frontend.

"Alias already in use" / "Alias must start with a letter…" Aliases live inside @alias.field expression paths, so they have to be identifier-shaped (letter first, then letters/digits/underscore, max 31 chars) and unique per mapping. Pick a different one.

The trash icon on a chip is disabled. Hover it for the reason: either "Clear the join condition first" (this source is referenced by the join) or "The primary source can't be removed while secondaries exist" — detach the secondaries first.

"Attach a primary source first — lookups left-outer-join against it." The lookup checkbox path requires an existing primary. Attach your main source normally, then re-open the form for the lookup.

"No obvious join keys found." The AI heuristic + LLM pass didn't find a confident candidate. Enter the left/right field paths manually — suggestions are an accelerator, not a gate.

Saved a cross join but my conditions disappeared. That's by design: cross joins are unconditional, so switching the join type to Cross clears the condition rows before saving.

What's next​