Using Convex with Airbyte
Airbyte is a data-integration platform that allows you to sync your Convex data with other databases.
Using Airbyte enables streaming import from any of their supported sources into Convex and streaming export from Convex into any of their supported destinations. The Convex team maintains a Convex source connector for streaming export and a Convex destination connector for streaming import.
Exporting data can be useful for handling workloads that aren't supported by Convex directly. Some use cases include:
- Convex isn't optimized for queries that load huge amounts of data. A data platform like Databricks or Snowflake is more appropriate.
- Flexible querying
- While Convex has powerful database queries and built-in full text search support, there are still some queries that are difficult to write within Convex. If you need very dynamic sorting and filtering for something like an "advanced search" view, databases like ElasticSearch can be helpful.
- Machine learning training
- Convex isn't optimized for queries running computationally intensive machine learning algorithms.
Streaming export using the Convex source connector requires a Convex Professional Plan. See the Airbyte docs on how to set up the Convex source connector here.
Adopting new technologies can be a slow, daunting process, especially when the technologies involve databases. Streaming import enables adopting Convex alongside your existing stack without having to write your own migration or data sync tooling. Some use cases include:
- Prototyping how Convex could replace your project's existing backend using its own data.
- Building new products faster by using Convex alongside existing databases.
- Developing a reactive UI-layer on top of an existing dataset.
- Migrating your data to Convex (if the CLI tool doesn't meet your needs).
A common use case is to "mirror" a table in the source database to Convex to build something new using Convex. We recommend leaving imported tables as read-only in Convex because syncing the results back to the source database could result in dangerous write conflicts. While Convex doesn't yet have access controls that would ensure a table is read-only, you can make sure that there are no mutations or actions writing to imported tables in your code and avoid editing documents in imported tables in the dashboard.
Streaming import is included with all Convex plans. See the Airbyte docs on how to set up the Convex destination connector here.