Skip to main content

Streaming Data in and out of Convex

Fivetran and Airbyte are data integration platforms that allow you to sync your Convex data with other databases.

Fivetran enables streaming export from Convex to any of their supported destinations. The Convex team maintains a Convex source connector, for streaming export. Streaming import into Convex via Fivetran is not supported at the moment.

Fivetran Integration is in beta

Fivetran Integration is currently in an early access beta. For access, contact us on Discord or by email at support@convex.dev!

Using Airbyte enables streaming import from any of their supported sources into Convex and streaming export from Convex into any of their supported destinations. The Convex team maintains a Convex source connector for streaming export and a Convex destination connector for streaming import.

Airbyte integration is in beta

Airbyte integration is currently a beta feature. If you have feedback or feature requests, let us know on Discord!

Streaming Export​

Exporting data can be useful for handling workloads that aren't supported by Convex directly. Some use cases include:

  1. Analytics
    • Convex isn't optimized for queries that load huge amounts of data. A data platform like Databricks or Snowflake is more appropriate.
  2. Flexible querying
    • While Convex has powerful database queries and built-in full text search support, there are still some queries that are difficult to write within Convex. If you need very dynamic sorting and filtering for something like an "advanced search" view, databases like ElasticSearch can be helpful.
  3. Machine learning training
    • Convex isn't optimized for queries running computationally intensive machine learning algorithms.

Streaming export using the Convex source connector requires a Convex Professional Plan. Contact us about Fivetran or see the Airbyte docs to learn how to set up a streaming export.

Streaming Import​

Adopting new technologies can be a slow, daunting process, especially when the technologies involve databases. Streaming import enables adopting Convex alongside your existing stack without having to write your own migration or data sync tooling. Some use cases include:

  1. Prototyping how Convex could replace your project's existing backend using its own data.
  2. Building new products faster by using Convex alongside existing databases.
  3. Developing a reactive UI-layer on top of an existing dataset.
  4. Migrating your data to Convex (if the CLI tool doesn't meet your needs).
Make imported tables read-only

A common use case is to "mirror" a table in the source database to Convex to build something new using Convex. We recommend leaving imported tables as read-only in Convex because syncing the results back to the source database could result in dangerous write conflicts. While Convex doesn't yet have access controls that would ensure a table is read-only, you can make sure that there are no mutations or actions writing to imported tables in your code and avoid editing documents in imported tables in the dashboard.

Streaming import is included with all Convex plans. See the Airbyte docs on how to set up the Convex destination connector here.