datafusion-remote-table

Features

1. Execute SQL queries on remote databases and stream results as datafusion table provider
2. Insert data into remote databases
3. Support inferring schema or user specified schema
4. Support pushing down filters and limit to remote databases
5. Execution plan can be serialized for distributed execution
6. Record batches can be transformed before outputting to next plan node

Usage

1. Execute SQL queries on remote database

#[tokio::main]
pub async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let options = PostgresConnectionOptions::new("localhost", 5432, "user", "password");
    let remote_table = RemoteTable::try_new(options, "select * from supported_data_types").await?;

    let ctx = SessionContext::new();
    ctx.register_table("remote_table", Arc::new(remote_table))?;

    ctx.sql("select * from remote_table").await?.show().await?;

    Ok(())
}

2. Insert data into remote database

#[tokio::main]
pub async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let options = PostgresConnectionOptions::new("localhost", 5432, "user", "password");
    let remote_table = RemoteTable::try_new(options, vec!["public", "test_table"]).await?;

    let ctx = SessionContext::new();
    ctx.register_table("remote_table", Arc::new(remote_table))?;

    ctx.sql("insert into remote_table values (1, 'Tom')").await?.show().await?;

    Ok(())
}

Supported databases

• Postgres
  • Int2 / Int4 / Int8
  • Float4 / Float8 / Numeric
  • Char / Varchar / Text / Bpchar / Bytea
  • Date / Time / Timestamp / Timestamptz / Interval
  • Bool / Oid / Name / Json / Jsonb / Geometry(PostGIS) / Xml / Uuid
  • Int2[] / Int4[] / Int8[]
  • Float4[] / Float8[]
  • Char[] / Varchar[] / Bpchar[] / Text[] / Bytea[]
• MySQL
  • TinyInt (Unsigned) / Smallint (Unsigned) / MediumInt (Unsigned) / Int (Unsigned) / Bigint (Unsigned)
  • Float / Double / Decimal
  • Date / DateTime / Time / Timestamp / Year
  • Char / Varchar / Binary / Varbinary
  • TinyText / Text / MediumText / LongText
  • TinyBlob / Blob / MediumBlob / LongBlob
  • Json / Geometry
• Oracle
  • Number / BinaryFloat / BinaryDouble / Float
  • Varchar2 / NVarchar2 / Char / NChar / Long / Clob / NClob
  • Raw / Long Raw / Blob
  • Date / Timestamp
  • Boolean / SDE.ST_GEOMETRY
• SQLite
  • Null / Integer / Real / Text / Blob
• DM (达梦数据库)
  • TinyInt / Smallint / Int / Bigint
  • Real / Float / Double / Numeric / Decimal
  • Char / Varchar / Text
  • Binary / Varbinary / Image
  • Bit / Timestamp / Time / Date

Thanks

• datafusion-table-providers