Run duckdb query on remote server
Compatibility Matrix
This table covers all DuckDB releases that have not reached end-of-life, including LTS releases, and shows the 2 most recent patch releases of each. Older patch releases are not listed, but the extension may be available for them as well.
| Platform | 1.4.4 | 1.4.5 | 1.5.5 | 1.5.6 | 2.0.0 |
|---|---|---|---|---|---|
| Linux AMD64 | |||||
| Linux ARM64 | |||||
| macOS AMD64 | |||||
| macOS ARM64 | |||||
| Windows AMD64 | |||||
| Windows ARM64 | |||||
| WebAssembly |
Installing and Loading
INSTALL duckherder FROM community;
LOAD duckherder;
Example
SELECT duckherder_start_local_server(8815); ATTACH DATABASE 'localhost:8815' AS dh (TYPE duckherder);
About duckherder
This extension is built on storage extension, to execute query on remote server and transfer data back with arrow flight. For remote and distributed execution, there're two key components: driver and worker(s). As their names suggest, driver node is the control plane, and worker nodes do the real execution tasks. Users are allowed to implement their own driver and worker(s), and register to the duckdb client-side, as long they speaks duckherder dialect (i.e., grpc stubs and arrow flight). From users' perspective, all DML and DDL SQL statements should be used exactly the same as local duckdb.
Added Functions
| function_name | function_type | description | comment | examples |
|---|---|---|---|---|
| distributed_scan | table | Scans a table stored on a Duckherder server. | NULL | |
| duckherder_clear_query_recorder_stats | scalar | Clears all query history records collected by the Duckherder query recorder. | NULL | [SELECT duckherder_clear_query_recorder_stats();] |
| duckherder_connected_query | table | Runs the SQL of a connection CONNECT-ed to a Duckherder database on its server as is. | NULL | [CONNECT dh;] |
| duckherder_get_query_execution_stats | table | Returns execution mode, merge strategy, duration, worker, task, and start-time statistics for distributed queries. | NULL | [SELECT * FROM duckherder_get_query_execution_stats();] |
| duckherder_get_query_history | table | Returns recorded SQL queries and their observed execution latencies. | NULL | [SELECT * FROM duckherder_get_query_history();] |
| duckherder_get_worker_count | scalar | Returns the number of workers registered with the local Duckherder driver server. | NULL | [SELECT duckherder_get_worker_count();] |
| duckherder_load_extension | scalar | Loads an extension on the attached Duckherder server and attempts to load it on the client. | NULL | [SELECT duckherder_load_extension('parquet');] |
| duckherder_query | table | Runs a query, a DML statement or a CREATE TABLE AS on the server of an attached Duckherder database. | NULL | [SELECT * FROM duckherder_query('dh', 'SELECT count(*) FROM main.orders');] |
| duckherder_register_worker | scalar | Registers a worker node with the local Duckherder driver server. | NULL | [SELECT duckherder_register_worker('worker-1', 'grpc://localhost:8816');] |
| duckherder_start_local_server | scalar | Starts a local Duckherder driver server on the given port, optionally with local workers. | NULL | [SELECT duckherder_start_local_server(8815, 4);] |
| duckherder_start_standalone_worker | scalar | Starts a standalone Duckherder worker on the given port. | NULL | [SELECT duckherder_start_standalone_worker(8816);] |
| duckherder_stop_local_server | scalar | Stops the local Duckherder driver server and its managed standalone workers. | NULL | [SELECT duckherder_stop_local_server();] |
Overloaded Functions
This extension does not add any function overloads.
Added Types
This extension does not add any types.
Added Settings
| name | description | input_type | scope | aliases | typed_value |
|---|---|---|---|---|---|
| duckherder_async_threads | Worker RPC thread limit (0: min(4 * DuckDB threads, 256)); applied on next dispatch | BIGINT | GLOBAL | [] | 0 |
| duckherder_query_pushdown | Push whole queries reading only remote tables to the server | BOOLEAN | GLOBAL | [] | true |
| duckherder_retry_initial_backoff_ms | Initial Duckherder RPC retry backoff in milliseconds | BIGINT | GLOBAL | [] | 25 |
| duckherder_retry_jitter_ratio | Jitter ratio applied to Duckherder RPC retry backoff | DOUBLE | GLOBAL | [] | 0.2 |
| duckherder_retry_max_attempts | Maximum number of attempts for retryable Duckherder RPCs | UBIGINT | GLOBAL | [] | 3 |
| duckherder_retry_max_backoff_ms | Maximum Duckherder RPC retry backoff in milliseconds | BIGINT | GLOBAL | [] | 250 |
| temp_directory | Set the directory to which temporary files are written. | VARCHAR | GLOBAL | [] | post.db.tmp |