Artie continuously streams data changes as they occur to downstream data stores. Streaming data means significantly lower data latency compared to batching, which always results in some lag. As companies’ data volume grows, each batch takes longer to process, which means batch processing is an unscalable solution long term. 
‍
Artie leverages change data capture (CDC) which means we only sync over changed data vs traditional methods of periodically taking an entire snapshot of source tables. With CDC and a streaming based approach, we are able to achieve real-time data syncs without impacting performance on the source database. A benefit of CDC is lower network traffic and compute costs as less data is being processed. 

Artie performs database replication using the log-based replication method. This method reads the transaction logs of the database, which record all changes to the data. When set up correctly, it's a non-intrusive way to replicate data in real-time and is highly efficient as it does not impact the performance of the source database. 

Log-based replication rely on transaction logs which are a fundamental part of a database’s architecture and are how databases internally replicate, which means there is no additional load or queries performed on the database. 

Read more about different database replication methods here. 

Artie leverages Apache Kafka and commits the offset upon a successful response from the destination. In the event of a crash or an outage, Artie will simply recover by seeking the previous offset and continue streaming. This means a full re-sync is not required and you should expect no data loss!

Artie scales with you as you grow and will be able to maintain the same speed of transfer whether your data volume is in the gigabytes or terabytes. We are able to accomplish this because our architecture scales horizontally and Artie automatically scales up or down based on your data’s time-of-day traffic.

Artie does not store any data. We simply transfer data from your database to your data warehouse/data lake - and all data is encrypted at rest and in-transit. Learn more about our security/compliance policies at trust.artie.so.

Read our documentation to see the list of sources and destinations that Artie supports: https://docs.artie.so/.

If you don’t see your sources or destinations listed, get in touch with us at hi@artie.so. Our roadmap is informed by customer requests - we’d love to hear from you!

Yes! When Artie initially connects to a new table, we will trigger a historical backfill to capture the current state of your table(s) and then continue to stream ongoing changes after. 

Initial table snapshots are free!

One bonus about our streaming architecture is that instead of waiting days/weeks for large table(s) to complete the initial backfill, you’ll be able to see data flow into downstream data stores in a matter of seconds.

Yes, our deployment model is customizable such that we can accommodate the following (among others):
‍
- Hybrid deployment: we can deploy the end-to-end data plane within your VPC, so no data leaves your environment, with control plane access for pipeline management.
- If you want to host your own Kafka: we can deploy our readers to help grab and publish CDC messages to Kafka. Artie Transfer can read directly from your internal Kafka topic and consume the CDC messages.

Yes! This is supported in our Enterprise plan with custom pricing. 

Artie’s control plane and data plane are split. This means we can easily deploy the data plane within your VPC, so no data leaves your environment. Our control plane will require access to the data plane within your VPC for pipeline management and maintenance.

This is a great option for companies that have strict data privacy requirements or are in a heavily regulated industry.

Yes. Artie has built-in composite keys support.

Yes. Artie is able to handle TOAST columns. We do not require higher level replica identities for the table which may increase load on your source database. Artie is configured to disregard unchanged TOAST values by integrating with Debezium's unavailable value placeholder.