Skip to main content

Connect a source via the Source Picker

The MapCraft walkthrough assumed your project already had a source schema attached (from StructScan). This tutorial covers the other way to put a source onto a MapCraft project: the Source Picker — a tabbed dialog that ingests from a file, a sample blob, an OpenAPI URL, a live database, a SaaS app, a webhook capture, or a PDF / HTML document.

Every tab ends the same way: a Schema is materialized under the SOURCE role on your project, stamped with a source_origin so the rest of MapCraft (provenance chip on the Schema panel, the compiler, and FlowBridge) knows where it came from.

By the end of this page you will have:

  • Opened the Source Picker for an existing project.
  • Materialized a source schema via at least one connector (File or Paste Sample is the fastest first run).
  • Seen the source_origin provenance chip on the Schema panel.
  • Learned where to go next for each of the other six connector types.
You'll need
  • An open MapCraft project at /mapcraft/{projectId}. (If you don't have one yet, follow the MapCraft walkthrough first.)
  • One of the following on hand: a JSON/CSV/XML/XSD file, a paste-able JSON or CSV sample, an OpenAPI URL, a read-only DB DSN, OAuth access to a SaaS tenant, or a PDF / HTML document.

Step 1 — Open the Source Picker​

From the project page, click Add Source Schema in the toolbar (or Replace Source Schema if a source is already attached). The Source Picker dialog opens with seven tabs across the top:

TabUse when…
FileYou already have a JSON / XML / XSD / CSV file (≤ 100 MB).
URL / OpenAPIYou have a Swagger 2 / OpenAPI 3.0 / 3.1 spec (JSON or YAML).
Paste SampleYou only have raw payloads (JSON or CSV) — no schema doc.
DatabaseYou want to introspect a live Postgres / MySQL / SQL Server / Snowflake / BigQuery / Databricks table.
SaaSYou want to pull objects from Salesforce or HubSpot via OAuth.
WebhookYou want to capture live events from a producer (Stripe, GitHub, Shopify…) and infer a schema from them.
PDF / HTMLYou have a document (spec PDF, vendor portal page) and want MapCraft to extract a field list — including via vision fallback for scanned PDFs.

The dialog defaults to the File tab every time it opens.

Step 2 — Run the golden path (File or Paste Sample)​

Pick whichever you have on hand. Both finish in under a minute.

  1. Drag-and-drop a file (or click to choose). Supported: JSON, XML, XSD, CSV, max 100 MB.
  2. The picker validates the file client-side and shows a parsed-field preview.
  3. Click Upload. The file is sent to the backend, parsed by the matching StructScan parser, and persisted as a Schema with role SOURCE.

After upload, the dialog closes and the project's Source panel populates. The source_origin chip on the Schema panel header shows the file type (e.g. json, xml).

How "AI-enriched" is gated

The purple AI-enriched banner only appears when the LLM proposed and the verifier accepted at least one enum, format, or description. Rejected proposals don't change the schema and don't trigger the banner. Hover any field's AI chip to see the rationale.

Step 3 — Pick a connector for "real" sources​

The File and Paste Sample paths are great for prototypes. For production work you usually want a connector that talks to the real system. The next five sections each walk through one connector. Skim straight to the one you need — they're independent.

3a. URL / OpenAPI​

  1. Switch to the URL / OpenAPI tab.
  2. Paste a spec URL — Swagger 2.0, OpenAPI 3.0, or 3.1; JSON or YAML.
  3. (Optional) Add auth headers as chips, e.g. Authorization: Bearer ….
  4. Click Fetch. The picker downloads, parses, and resolves $ref chains across components.schemas and inline path schemas.
  5. Pick the component or operation you want from the searchable list and click Use this schema.
Private-network URLs

Private/loopback URLs are blocked by default by an SSRF guard. To allow them in self-hosted dev environments only, set MAPCRAFT_ALLOW_PRIVATE_URLS=true on the backend. Never enable this in shared environments.

3b. Database​

The Database tab is a three-stage flow: Connect → Browse → Materialize.

Connect.

  1. Pick a Dialect tile: Postgres, MySQL, SQL Server, Snowflake, BigQuery, or Databricks.
  2. Paste a DSN. The placeholder shows the expected format per dialect, e.g.:
  3. Give the connection a Save as (label) — e.g. prod-postgres.
  4. Click Test connection. On success you'll see Connected · <dialect_version>. If the dialect supports it, the picker also confirms READ ONLY enforcement; if not, you'll see an amber "READ ONLY pragma not confirmed — defense-in-depth disabled" warning.
  5. Click Save & use. The credentials are encrypted (column-level Fernet via the credential vault) and the connection chip appears under Saved connections for next time.

Browse. The picker lists schemas — it auto-picks the first non information_schema one. Use the Filter tables… box, then click a row. Tables show column count and (where the dialect can answer cheaply) a row estimate.

Materialize. Edit the Schema name (defaults to schema.table) and click Use this table. The picker calls the backend, which introspects columns, primary keys, and foreign keys, and writes a Schema with source_origin.kind = "db".

Read-only is enforced server-side

Every query the introspection runs is wrapped in a read-only session and capped at a 10-second statement timeout. The DSN you paste is not echoed back after Save — subsequent calls use the connection ID.

3c. SaaS (Salesforce + HubSpot)​

  1. Switch to the SaaS tab.
  2. Pick Salesforce or HubSpot.
  3. Choose OAuth (preferred — pops the provider's consent flow) or Manual token for offline / CI use.
  4. After auth, the picker shows an object grid with chips marking custom objects and (for Salesforce) queryable ones.
  5. Click an object → click Use this object.

Field-prefix clusters (Billing*, Shipping*, Mailing* on Salesforce addresses) are auto-grouped in the materialized schema so the canvas stays readable.

3d. Webhook​

  1. Switch to the Webhook tab.
  2. Click Mint URL. A single-use receiver URL (/hooks/{token}) is minted and copied to your clipboard.
  3. Point your producer at the URL and trigger a few events. The picker polls every ~2 seconds and shows captured events as they arrive, tagged with known-source header hints (x-github-event, x-stripe-event, x-shopify-topic).
  4. When you have enough samples, click Finalize. The captured events funnel through the same inference engine as the Paste Sample tab, so you get LLM enrichment "for free".

The minted URL is single-use — after Finalize it stops accepting events. Re-open the tab to mint a new one.

3e. PDF / HTML​

  1. Switch to the PDF / HTML tab.
  2. Drag-drop a file (PDF or HTML) or paste a public URL.
  3. (Optional) Override the document type if auto-detection guesses wrong.
  4. Click Extract. Text-mode (pypdf for PDFs, lxml.html for HTML) runs first. If text extraction returns empty — typically a scanned PDF — the picker automatically falls back to vision mode via Anthropic's multimodal endpoint.
  5. Review the extracted field list with per-field rationale. An amber Vision chip appears on the result header when vision mode fired.
  6. Set the schema name and click Use this schema.

Step 4 — Confirm provenance landed​

Whichever connector you used, MapCraft stamped the new schema with a source_origin discriminator and re-rendered the Schema panel.

In the project, look at the source side. The panel header now shows a source-type chip — file (with file-type suffix), openapi, paste-sample, db (with dialect), salesforce / hubspot, webhook, or pdf (with a Vision modifier if vision mode fired). Hover the chip for the full provenance tooltip.

Verify​

Your source connection worked end-to-end if all of the following are true.

CheckWhere to look
The Source Picker closed itself after Use this schema/table/objectDialog dismisses; project page refreshes.
The Schema panel populated with fieldsLeft side of the canvas / table view shows source fields, not "No source schema attached".
A source_origin chip is rendered on the Schema panelTooltip shows the connector key (db, paste-sample, …) plus its specifics (dialect, table, sample count).
For Database: the saved connection chip is listed at the top of the Database tab next time you open the pickerRe-open Source Picker → Database tab → chip with your label is selectable.
For Paste Sample / Webhook: when LLM enrichment fired, the AI-enriched banner is present in the previewVisible before you click Use this schema; recorded as llm_enriched: true on the resulting schema.

If any of these are missing, see Troubleshooting below.

Troubleshooting​

"Failed: SSRF — private address blocked" (URL / OpenAPI). The URL resolves to a private, loopback, or link-local address. This is intentional. Either expose the spec on a public URL or — for self-hosted dev only — set MAPCRAFT_ALLOW_PRIVATE_URLS=true on the backend.

"READ ONLY pragma not confirmed" (Database). Your dialect doesn't surface a session-level READ ONLY flag the picker can read back. The 10-second statement timeout is still in force, but defense-in-depth is one layer thinner. Prefer a database role that's read-only at the grant level.

Inference returned 0 fields (Paste Sample). The samples couldn't be parsed under the selected format hint. Switch Format from Auto-detect to JSON or CSV to force the parser, or paste a smaller, well-formed sample.

Webhook capture page is stuck on "Waiting for events". The minted URL accepts only the first N events and is single-use. Confirm your producer is hitting /hooks/{token} exactly (no trailing slash issues) and that the token hasn't been finalized yet. If in doubt, Mint URL again.

PDF extraction returns a Vision chip but missing fields (PDF / HTML). Vision mode is best-effort for scanned PDFs. Try splitting the PDF to one section per page, or run Extract on the highest-resolution version of the file.

What's next​

  • MapCraft walkthrough — auto-map, lookups, rules, versioning, and DAG compile, starting from the source you just connected.
  • FlowBridge walkthrough — run the compiled DAG against the live source connector.
  • StructScan overview — when you want StructScan to profile a file before it becomes a MapCraft source.