Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .prettierignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
.github/
.husky/
.ranger/
.svelte-kit/
.vscode/
.idea/
Expand Down
15 changes: 13 additions & 2 deletions docs/docs/developers/build/connectors/connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,7 +164,6 @@ Rill is continually evaluating additional OLAP engines that can be added. For a
content="Connect to SQLite databases for lightweight, file-based data storage and querying."
link="/developers/build/connectors/data-source/sqlite"
linkLabel="Learn more"
referenceLink="sqlite"
/>
<ConnectorIcon
icon={<img src="/img/build/connectors/icons/Logo-Supabase.svg" alt="Supabase" className="duckdb-icon"/>}
Expand Down Expand Up @@ -216,6 +215,19 @@ Rill is continually evaluating additional OLAP engines that can be added. For a

</div>

## Table Formats
### Apache Iceberg

<div className="connector-icon-grid">
<ConnectorIcon
icon={<img src="/img/build/connectors/icons/Logo-Iceberg.svg" alt="Apache Iceberg" />}
header="Apache Iceberg"
content="Read Iceberg tables directly from object storage through compatible query engines."
link="/developers/build/connectors/data-source/iceberg"
linkLabel="Learn more"
/>
</div>

## Other Data Connectors
### External DuckDB
### Google Sheets
Expand Down Expand Up @@ -262,7 +274,6 @@ Rill is continually evaluating additional OLAP engines that can be added. For a
content="Connect to Salesforce to extract data from objects and queries using the Salesforce API."
link="/developers/build/connectors/data-source/salesforce"
linkLabel="Learn more"
referenceLink="salesforce"
/>

</div>
Expand Down
23 changes: 20 additions & 3 deletions docs/docs/developers/build/connectors/data-source/data-source.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ Rill supports connecting your data to both [DuckDB](/developers/build/connectors
### MySQL
### PostgreSQL
### SQLite
### Supabase

<div className="connector-icon-grid">
<ConnectorIcon
Expand All @@ -108,7 +109,14 @@ Rill supports connecting your data to both [DuckDB](/developers/build/connectors
content="Connect to SQLite databases for lightweight, file-based data storage and querying."
link="/developers/build/connectors/data-source/sqlite"
linkLabel="Learn more"
referenceLink="sqlite"
/>
<ConnectorIcon
icon={<img src="/img/build/connectors/icons/Logo-Supabase.svg" alt="Supabase" className="duckdb-icon"/>}
header="Supabase"
content="Connect to Supabase's managed PostgreSQL databases with SSL support and standard connection methods."
link="/developers/build/connectors/data-source/supabase"
linkLabel="Learn more"
referenceLink="supabase"
/>
</div>

Expand Down Expand Up @@ -146,9 +154,19 @@ Rill supports connecting your data to both [DuckDB](/developers/build/connectors
linkLabel="Learn more"
referenceLink="azure"
/>
</div>

## Table Formats
### Apache Iceberg


<div className="connector-icon-grid">
<ConnectorIcon
icon={<img src="/img/build/connectors/icons/Logo-Iceberg.svg" alt="Apache Iceberg" />}
header="Apache Iceberg"
content="Read Iceberg tables directly from object storage through compatible query engines."
link="/developers/build/connectors/data-source/iceberg"
linkLabel="Learn more"
/>
</div>

## Other Data Connectors
Expand Down Expand Up @@ -197,7 +215,6 @@ Rill supports connecting your data to both [DuckDB](/developers/build/connectors
content="Connect to Salesforce to extract data from objects and queries using the Salesforce API."
link="/developers/build/connectors/data-source/salesforce"
linkLabel="Learn more"
referenceLink="salesforce"
/>

</div>
Expand Down
134 changes: 134 additions & 0 deletions docs/docs/developers/build/connectors/data-source/iceberg.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
---
title: Apache Iceberg
description: Read Iceberg tables from object storage
sidebar_label: Apache Iceberg
sidebar_position: 27
---

## Overview

[Apache Iceberg](https://iceberg.apache.org/) is an open table format for large analytic datasets. Rill supports reading Iceberg tables directly from object storage through compatible query engine integrations. Today, this is powered by DuckDB's native [Iceberg extension](https://duckdb.org/docs/extensions/iceberg/overview.html).

:::note Direct file access only
Rill reads Iceberg tables by scanning the table's metadata and data files directly from object storage. Catalog-based access (e.g., through a Hive Metastore, AWS Glue, or REST catalog) is not currently supported.
:::

## Storage Backends

Iceberg tables can be read from any of the following storage backends:

| Backend | URI format | Authentication |
|---|---|---|
| Amazon S3 | `s3://bucket/path/to/table` | Requires an [S3 connector](/developers/build/connectors/data-source/s3) |
| Google Cloud Storage | `gs://bucket/path/to/table` | Requires a [GCS connector](/developers/build/connectors/data-source/gcs) with HMAC keys |
| Azure Blob Storage | `azure://container/path/to/table` | Requires an [Azure connector](/developers/build/connectors/data-source/azure) |
| Local filesystem | `/path/to/table` | No authentication needed |

For cloud storage backends, you must first configure the corresponding storage connector with valid credentials. Rill uses these credentials to authenticate when reading the Iceberg table files.

## Using the UI

1. Click **Add Data** in your Rill project
2. Select **Apache Iceberg** as the data source type
3. Choose your storage backend (S3, GCS, Azure, or Local)
4. Enter the path to your Iceberg table directory
5. Optionally configure advanced parameters (allow moved paths, snapshot version)
6. Enter a model name and click **Create**

For cloud storage backends, the UI will prompt you to set up the corresponding storage connector if one doesn't already exist.

## Manual Configuration

Create a model that uses DuckDB's `iceberg_scan()` function to read the table.

### Reading from S3

Create `models/iceberg_data.yaml`:

```yaml
type: model
connector: duckdb
create_secrets_from_connectors: s3
materialize: true

sql: |
SELECT *
FROM iceberg_scan('s3://my-bucket/path/to/iceberg_table')
```

### Reading from GCS

:::info HMAC keys required
DuckDB's `iceberg_scan()` authenticates to GCS using HMAC keys, not JSON service account credentials. When configuring your [GCS connector](/developers/build/connectors/data-source/gcs), use the `key_id` and `secret` (HMAC) properties instead of `google_application_credentials`.
:::

```yaml
type: model
connector: duckdb
create_secrets_from_connectors: gcs
materialize: true

sql: |
SELECT *
FROM iceberg_scan('gs://my-bucket/path/to/iceberg_table')
```

### Reading from Azure

```yaml
type: model
connector: duckdb
create_secrets_from_connectors: azure
materialize: true

sql: |
SELECT *
FROM iceberg_scan('azure://my-container/path/to/iceberg_table')
```

### Reading from local filesystem

```yaml
type: model
connector: duckdb
materialize: true

sql: |
SELECT *
FROM iceberg_scan('/path/to/iceberg_table')
```

## Optional Parameters

The `iceberg_scan()` function accepts additional parameters:

| Parameter | Type | Description |
|---|---|---|
| `allow_moved_paths` | boolean | Allow reading tables where data files have been moved from their original location. Defaults to `true` in the UI. |
| `version` | string | Read a specific Iceberg snapshot version instead of the latest. |

Example with optional parameters:

```sql
SELECT *
FROM iceberg_scan('s3://my-bucket/path/to/iceberg_table',
allow_moved_paths = true,
version = '2')
```

## Deploy to Rill Cloud

Since Iceberg tables are read through DuckDB using your existing storage connector credentials, deploying to Rill Cloud follows the same process as the underlying storage connector:

- **S3**: Follow the [S3 deployment guide](/developers/build/connectors/data-source/s3#deploy-to-rill-cloud)
- **GCS**: Follow the [GCS deployment guide](/developers/build/connectors/data-source/gcs#deploy-to-rill-cloud)
- **Azure**: Follow the [Azure deployment guide](/developers/build/connectors/data-source/azure#deploy-to-rill-cloud)

Ensure your storage connector credentials are configured in your Rill Cloud project before deploying.

## Limitations

- **Direct file access only**: Rill reads Iceberg metadata and data files directly from storage. Catalog integrations (Hive Metastore, AWS Glue, REST catalog) are not supported.
- **DuckDB engine**: Iceberg support is currently provided through DuckDB's Iceberg extension. Additional engine support (e.g., ClickHouse) is planned.
- **GCS requires HMAC keys**: DuckDB's `iceberg_scan()` only supports HMAC authentication for GCS, not JSON service account credentials.
- **Read-only**: Rill reads from Iceberg tables but does not write to them.
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ sidebar_position: 65

## Local credentials

When using Rill Developer on your local machine, you will need to provide your credentials via a connector file. We would recommend not using plain text to create your file and instead use the `.env` file. For more details on your connector, see [connector YAML](/reference/project-files/connectors#salesforce) for more details.
When using Rill Developer on your local machine, you will need to provide your credentials via a connector file. We would recommend not using plain text to create your file and instead use the `.env` file. For more details on your connector, see [connector YAML](/reference/project-files/connectors) for more details.

:::tip Updating the project environmental variable

Expand Down Expand Up @@ -44,7 +44,7 @@ If this project has already been deployed to Rill Cloud and credentials have bee

## Deploy to Rill Cloud

When deploying a project to Rill Cloud, Rill requires you to explicitly provide Salesforce credentials used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors#salesforce) for more information.
When deploying a project to Rill Cloud, Rill requires you to explicitly provide Salesforce credentials used in your project. Please refer to our [connector YAML reference docs](/reference/project-files/connectors) for more information.

If you subsequently add sources that require new credentials (or if you simply entered the wrong credentials during the initial deploy), you can update the credentials by pushing the `Deploy` button to update your project or by running the following command in the CLI:
```
Expand Down
17 changes: 9 additions & 8 deletions docs/docs/developers/build/connectors/data-source/sqlite.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,22 +13,23 @@ sidebar_position: 80

## Connect to SQLite

SQLite databases are read through DuckDB's [SQLite extension](https://duckdb.org/docs/extensions/sqlite.html) using the `sqlite_scan()` function. No separate connector is needed.

In many cases, since SQLite is used as an in-process database, credentials are not required. Instead, Rill will need to know the path to the SQLite database file so that it can be read accordingly.
Create a model file (e.g., `models/my_sqlite_data.yaml`):

```yaml
type: connector
driver: sqlite
type: model
connector: duckdb
materialize: true

dsn: "file:mydatabase.db"
sql: |
SELECT *
FROM sqlite_scan('data/mydatabase.db', 'my_table')
```

Alternatively, you can create the connector directly using the [connector YAML reference documentation](/reference/project-files/connectors#sqlite).


:::tip

If you plan to deploy the project to Rill Cloud, it is recommended that you move the SQLite database file to a `data` folder in your Rill project home directory. You can then use the relative path of the db file in your source definition (e.g., `data/test_sqlite.db`).
If you plan to deploy the project to Rill Cloud, place the SQLite database file in a `data` folder in your Rill project directory and use the relative path (e.g., `data/mydatabase.db`).

:::

Expand Down
56 changes: 0 additions & 56 deletions docs/docs/reference/project-files/connectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,6 @@ Connector YAML files define how Rill connects to external data sources and OLAP
### _Databases_
- [**MySQL**](#mysql) - MySQL databases
- [**PostgreSQL**](#postgres) - PostgreSQL databases
- [**SQLite**](#sqlite) - SQLite databases
- [**Supabase**](#supabase) - Supabase (managed PostgreSQL)

### _Object Storage_
Expand All @@ -42,7 +41,6 @@ Connector YAML files define how Rill connects to external data sources and OLAP

### _Other_
- [**HTTPS**](#https) - Public files via HTTP/HTTPS
- [**Salesforce**](#salesforce) - Salesforce data

:::warning Security Recommendation
For all credential parameters (passwords, tokens, keys), use environment variables with the syntax `{{ .env.KEY_NAME }}`. This keeps sensitive data out of your YAML files and version control. See our [credentials documentation](/developers/build/connectors/credentials/) for complete setup instructions.
Expand Down Expand Up @@ -1127,43 +1125,6 @@ endpoint: "https://my-s3-endpoint.com" # Optional custom endpoint URL for S3-com
region: "us-east-1" # AWS region of the S3 bucket
```

## Salesforce

### `driver`

_[string]_ - Refers to the driver type and must be driver `salesforce` _(required)_

### `username`

_[string]_ - Salesforce account username _(required)_

### `password`

_[string]_ - Salesforce account password (secret)

### `key`

_[string]_ - Authentication key for Salesforce (secret)

### `endpoint`

_[string]_ - Salesforce API endpoint URL _(required)_

### `client_id`

_[string]_ - Client ID used for Salesforce OAuth authentication _(required)_

```yaml
# Example: Salesforce connector configuration
type: connector # Must be `connector` (required)
driver: salesforce # Must be `salesforce` _(required)_
username: "myusername" # Salesforce account username
password: "{{ .env.SALESFORCE_PASSWORD }}" # Salesforce account password (secret)
key: "{{ .env.SALESFORCE_KEY }}" # Authentication key for Salesforce (secret)
endpoint: "https://login.salesforce.com" # Salesforce API endpoint URL
client_id: "my-client-id" # Client ID used for Salesforce OAuth authentication
```

## Slack

### `driver`
Expand Down Expand Up @@ -1275,21 +1236,4 @@ type: connector
driver: snowflake
dsn: "{{ .env.SNOWFLAKE_DSN }}" # define SNOWFLAKE_DSN in .env file
parallel_fetch_limit: 2
```

## SQLite

### `driver`

_[string]_ - Refers to the driver type and must be driver `sqlite` _(required)_

### `dsn`

_[string]_ - DSN(Data Source Name) for the sqlite connection _(required)_

```yaml
# Example: SQLite connector configuration
type: connector # Must be `connector` (required)
driver: sqlite # Must be `sqlite` _(required)_
dsn: "file:mydatabase.db" # DSN for the sqlite connection
```
14 changes: 0 additions & 14 deletions docs/docs/reference/project-files/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -673,20 +673,6 @@ _[object]_ - Settings related to glob file matching.

_[string]_ - Size of a batch (e.g., '100MB')

## Additional properties when `connector` is `salesforce` or [named connector](./connectors#salesforce) of salesforce

### `soql`

_[string]_ - SOQL query to execute against the Salesforce instance.

### `sobject`

_[string]_ - Salesforce object (e.g., Account, Contact) targeted by the query.

### `queryAll`

_[boolean]_ - Whether to include deleted and archived records in the query (uses queryAll API).

## Examples

```yaml
Expand Down
Loading
Loading