Skip to main content

Deliver to Snowflake, BigQuery, or an SFTP drop

A mapping isn't done until the transformed records land somewhere your team already works. FlowBridge's Deliver lane takes the output of your mapping, serializes it to a format (CSV or JSON), and hands it to a delivery adapter — a connection that knows how to write into a destination. This page sets up three of them: a Snowflake table, a BigQuery table, and an SFTP / flat-file drop.

By the end of this page you will have:

  • Created a Snowflake, BigQuery, or SFTP delivery adapter under FlowBridge → Deliver.
  • Stored the destination's credentials safely in an encrypted Secret reference.
  • Sent a test payload as CSV and watched the rows land in the live destination.
  • Wired the adapter into a DAG so every run delivers automatically.
You'll need
  • A MapCraft project open at /flowbridge/{projectId}.
  • Credentials for one destination:
    • Snowflake — account, user, warehouse, database, schema, and a target table, plus a password or key-pair.
    • BigQuery — a GCP project, dataset, target table, and a service-account JSON key.
    • SFTP — host, username, a writable remote path, and a password or private key.
  • The destination table (Snowflake / BigQuery) should already exist with columns matching your mapped fields. Warehouse loads append into an existing table.

Step 1 — Open the Deliver lane and add an adapter

Open FlowBridge → Deliver for your project, then click New adapter. Give it a Name, and pick a Type. The form fields change to match the type you choose:

FieldExample
Accountacme-prod (your Snowflake account identifier)
UserDATACHORD_LOADER
WarehouseLOAD_WH
DatabaseANALYTICS
SchemaPUBLIC
Target tableINVOICES
Role (optional)LOADER_ROLE
Write modeappend (default) or overwrite

overwrite truncates the target table before loading; append adds rows.

Step 2 — Store credentials in a Secret reference

Never put passwords or service-account keys in the plain config fields. Instead, the form's Secret reference field points to a secret in your org's encrypted vault — DataChord decrypts it only at delivery time and merges it into the adapter's connection settings. The secret's value is a small JSON object:

{ "password": "••••••••" }

or, for key-pair auth:

{ "private_key": "-----BEGIN PRIVATE KEY-----\n…\n-----END PRIVATE KEY-----" }

Enter the name of that stored secret in Secret reference, then Save the adapter.

tip

Values from the secret override the matching config fields, so you can keep the non-sensitive parts (account, table, project) visible in the form and the sensitive parts encrypted.

Step 3 — Send a test payload

With the adapter selected, click Send test payload. In the dialog:

  1. Set Format emitter to csv (warehouses load CSV natively; json works too).

  2. Edit the sample Records — a small JSON array whose keys match your destination columns:

    [
    { "id": "1", "amount": "100.00", "vendor": "Acme" },
    { "id": "2", "amount": "250.00", "vendor": "Globex" }
    ]
  3. Click Send.

DataChord serializes the records to CSV and runs the real load — for Snowflake it stages the file and runs COPY INTO; for BigQuery it runs a load job; for SFTP it writes the file to your remote path.

Step 4 — Wire it into a DAG run

A test payload proves the connection; to deliver on every run, add a Deliver node to your DAG and point it at this adapter. On the canvas, the deliver node takes your mapped record stream, applies the format emitter, and routes the bytes through the adapter you configured. When the DAG runs (manually or on a schedule), the rows land in your destination and the run's metadata records how many bytes were delivered.


Verify

You're set up correctly when:

  • The Send test payload result banner shows Delivered with a non-zero byte count and an endpoint such as snowflake://acme-prod/ANALYTICS/PUBLIC/INVOICES.
  • The sample rows are visible in the destination — query the Snowflake/BigQuery table, or list the file on the SFTP host.
  • A DAG run that includes the deliver node finishes successfully and its run detail shows the deliver step's bytes sent.

Troubleshooting

SymptomLikely cause
snowflake-connector-python not installed / google-cloud-bigquery not installedThe warehouse SDKs are optional. Deploy the backend image built with the warehouses extra (the default Docker images include it).
asyncssh not installed (SFTP)Same — the asyncssh dependency ships in the warehouses / sftp extras.
Auth or "table not found" error in the test resultCheck the Secret reference points to a real stored secret and that the target table exists with matching columns. Warehouse loads append into an existing table.
Rows load but columns are misalignedMake sure your record keys match the destination column names; for CSV the first row is treated as the header.

What's next