Secure remote secrets storage with OPAQUE PAKE authentication, email/password registration, MFA login, and automatic ducklake mounting
Maintainer(s):
dforsber
Installing and Loading
INSTALL boilstream FROM community;
LOAD boilstream;
Example
-- Setup: Install and load boilstream extension
INSTALL httpfs; LOAD httpfs;
INSTALL boilstream FROM community;
LOAD boilstream;
-- Option 1: Login with email/password/MFA (establishes OPAQUE session)
-- NOTE: Remember to clean up your ~/.duckdb/history
PRAGMA boilstream_login('https://api.example.com/user@example.com', 'password', '123456');
┌────────────────────────┬─────────────────────┬────────────────────┐
│ status │ expires_at │ ducklakes_attached │
├────────────────────────┼─────────────────────┼────────────────────┤
│ Session token obtained │ 2025-11-21 10:30:00 │ 2 │
└────────────────────────┴─────────────────────┴────────────────────┘
-- Option 2: Authenticate with bootstrap token (from SSO/web UI)
PRAGMA boilstream_bootstrap_session('https://api.example.com/secrets:TOKEN');
┌────────────────────────┬─────────────────────┬────────────────────┐
│ status │ expires_at │ ducklakes_attached │
├────────────────────────┼─────────────────────┼────────────────────┤
│ Session token obtained │ 2025-11-21 10:30:00 │ 2 │
└────────────────────────┴─────────────────────┴────────────────────┘
-- List all available ducklakes
FROM boilstream_ducklakes();
┌──────────────┬──────────────┬─────────────────┐
│ catalog_id │ catalog_name │ description │
├──────────────┼──────────────┼─────────────────┤
│ 5a805647-... │ my_analytics │ Sales analytics │
└──────────────┴──────────────┴─────────────────┘
-- List all cached secrets with expiration
FROM boilstream_secrets();
┌─────────────────────┬──────────┬─────────────────────┐
│ name │ type │ expires_at │
├─────────────────────┼──────────┼─────────────────────┤
│ my_analytics │ ducklake │ 2026-11-19 20:00:00 │
│ my_analytics_s3 │ s3 │ 2026-11-19 20:00:00 │
│ my_analytics_pg │ postgres │ 2026-11-19 20:00:00 │
└─────────────────────┴──────────┴─────────────────────┘
-- Create new ducklake
PRAGMA boilstream_create_ducklake('new_catalog', 'My new ducklake');
-- Query ducklake (automatically attached)
USE my_analytics;
SHOW TABLES;
About boilstream
Boilstream extension provides enterprise-grade remote secrets storage for multi-tenant DuckDB deployments.
Features:
- 🔐 OPAQUE PAKE Authentication - Password-less authentication (IETF RFC 9807)
- 🔒 End-to-end Encryption - AES-256-GCM/ChaCha20-Poly1305 for secrets and responses
- 🔑 Cryptographic Key Derivation - HKDF-SHA256 with domain separation
- 🛡️ Defense in Depth - HMAC verification before AEAD decryption
- 🎯 Session Protection - Lock-step sequence protocol prevents hijacking
- 🗄️ Ducklake Support - Auto-attach multi-tenant data catalogs
- 🌐 SSO Integration - Bootstrap token exchange for web environments
Security:
- Facebook's audited OPAQUE implementation (Rust + C++ FFI)
- Protects against MITM attacks even if TLS is compromised
- One-time bootstrap tokens with anonymous authentication
- Session resumption with encrypted refresh tokens
- Comprehensive test vectors and security specification
API Functions:
PRAGMA boilstream_register_user(url_with_email, password)- Register new user with MFA (displays QR code)PRAGMA boilstream_verify_mfa(totp_code)- Complete registration and get backup codesPRAGMA boilstream_login(url_with_email, password, mfa_code)- Email/password/MFA loginPRAGMA boilstream_bootstrap_session(url_with_token)- Authenticate with bootstrap tokenboilstream_ducklakes()- List available ducklakesboilstream_secrets()- List cached secrets with expirationPRAGMA boilstream_create_ducklake(name, description)- Create new ducklakePRAGMA boilstream_help- Show all available commands
Requirements:
- httpfs extension (auto-loaded)
- postgres_scanner extension (auto-loaded)
- ducklake extension (auto-loaded)
- Compatible REST API server (see API specification)
For detailed documentation, security specification, and setup instructions, visit the extension repository.
Added Functions
| function_name | function_type | description | comment | examples |
|---|---|---|---|---|
| boilstream_bootstrap_session | pragma | NULL | NULL | |
| boilstream_create_ducklake | pragma | NULL | NULL | |
| boilstream_ducklakes | table | NULL | NULL | |
| boilstream_help | pragma | NULL | NULL | |
| boilstream_login | pragma | NULL | NULL | |
| boilstream_register_user | pragma | NULL | NULL | |
| boilstream_secrets | table | NULL | NULL | |
| boilstream_verify_mfa | pragma | NULL | NULL | |
| ducklake_add_data_files | table | NULL | NULL | |
| ducklake_cleanup_old_files | table | NULL | NULL | |
| ducklake_current_snapshot | table | NULL | NULL | |
| ducklake_delete_orphaned_files | table | NULL | NULL | |
| ducklake_expire_snapshots | table | NULL | NULL | |
| ducklake_flush_inlined_data | table | NULL | NULL | |
| ducklake_last_committed_snapshot | table | NULL | NULL | |
| ducklake_list_files | table | NULL | NULL | |
| ducklake_merge_adjacent_files | table | NULL | NULL | |
| ducklake_options | table | NULL | NULL | |
| ducklake_rewrite_data_files | table | NULL | NULL | |
| ducklake_set_commit_message | table | NULL | NULL | |
| ducklake_set_option | table | NULL | NULL | |
| ducklake_snapshots | table | NULL | NULL | |
| ducklake_table_changes | table_macro | NULL | NULL | |
| ducklake_table_deletions | table | NULL | NULL | |
| ducklake_table_info | table | NULL | NULL | |
| ducklake_table_insertions | table | NULL | NULL | |
| pg_clear_cache | table | NULL | NULL | |
| postgres_attach | table | NULL | NULL | |
| postgres_execute | table | NULL | NULL | |
| postgres_query | table | NULL | NULL | |
| postgres_scan | table | NULL | NULL | |
| postgres_scan_pushdown | table | NULL | NULL |
Added Settings
| name | description | input_type | scope | aliases |
|---|---|---|---|---|
| auto_fallback_to_full_download | Allows automatically falling back to full file downloads when possible. | BOOLEAN | GLOBAL | [] |
| ca_cert_file | Path to a custom certificate file for self-signed certificates. | VARCHAR | GLOBAL | [] |
| ducklake_max_retry_count | The maximum amount of retry attempts for a ducklake transaction | UBIGINT | GLOBAL | [] |
| ducklake_retry_backoff | Backoff factor for exponentially increasing retry wait time | DOUBLE | GLOBAL | [] |
| ducklake_retry_wait_ms | Time between retries | UBIGINT | GLOBAL | [] |
| enable_curl_server_cert_verification | Enable server side certificate verification for CURL backend. | BOOLEAN | GLOBAL | [] |
| enable_server_cert_verification | Enable server side certificate verification. | BOOLEAN | GLOBAL | [] |
| force_download | Forces upfront download of file | BOOLEAN | GLOBAL | [] |
| hf_max_per_page | Debug option to limit number of items returned in list requests | UBIGINT | GLOBAL | [] |
| http_keep_alive | Keep alive connections. Setting this to false can help when running into connection failures | BOOLEAN | GLOBAL | [] |
| http_retries | HTTP retries on I/O error | UBIGINT | GLOBAL | [] |
| http_retry_backoff | Backoff factor for exponentially increasing retry wait time | FLOAT | GLOBAL | [] |
| http_retry_wait_ms | Time between retries | UBIGINT | GLOBAL | [] |
| http_timeout | HTTP timeout read/write/connection/retry (in seconds) | UBIGINT | GLOBAL | [] |
| httpfs_client_implementation | Select which is the HTTPUtil implementation to be used | VARCHAR | GLOBAL | [] |
| pg_array_as_varchar | Read Postgres arrays as varchar - enables reading mixed dimensional arrays | BOOLEAN | GLOBAL | [] |
| pg_connection_cache | Whether or not to use the connection cache | BOOLEAN | GLOBAL | [] |
| pg_connection_limit | The maximum amount of concurrent Postgres connections | UBIGINT | GLOBAL | [] |
| pg_debug_show_queries | DEBUG SETTING: print all queries sent to Postgres to stdout | BOOLEAN | GLOBAL | [] |
| pg_experimental_filter_pushdown | Whether or not to use filter pushdown | BOOLEAN | GLOBAL | [] |
| pg_null_byte_replacement | When writing NULL bytes to Postgres, replace them with the given character | VARCHAR | GLOBAL | [] |
| pg_pages_per_task | The amount of pages per task | UBIGINT | GLOBAL | [] |
| pg_use_binary_copy | Whether or not to use BINARY copy to read data | BOOLEAN | GLOBAL | [] |
| pg_use_ctid_scan | Whether or not to parallelize scanning using table ctids | BOOLEAN | GLOBAL | [] |
| pg_use_text_protocol | Whether or not to use TEXT protocol to read data. This is slower, but provides better compatibility with non-Postgres systems | BOOLEAN | GLOBAL | [] |
| s3_access_key_id | S3 Access Key ID | VARCHAR | GLOBAL | [] |
| s3_endpoint | S3 Endpoint | VARCHAR | GLOBAL | [] |
| s3_kms_key_id | S3 KMS Key ID | VARCHAR | GLOBAL | [] |
| s3_region | S3 Region | VARCHAR | GLOBAL | [] |
| s3_requester_pays | S3 use requester pays mode | BOOLEAN | GLOBAL | [] |
| s3_secret_access_key | S3 Access Key | VARCHAR | GLOBAL | [] |
| s3_session_token | S3 Session Token | VARCHAR | GLOBAL | [] |
| s3_uploader_max_filesize | S3 Uploader max filesize (between 50GB and 5TB) | VARCHAR | GLOBAL | [] |
| s3_uploader_max_parts_per_file | S3 Uploader max parts per file (between 1 and 10000) | UBIGINT | GLOBAL | [] |
| s3_uploader_thread_limit | S3 Uploader global thread limit | UBIGINT | GLOBAL | [] |
| s3_url_compatibility_mode | Disable Globs and Query Parameters on S3 URLs | BOOLEAN | GLOBAL | [] |
| s3_url_style | S3 URL style | VARCHAR | GLOBAL | [] |
| s3_use_ssl | S3 use SSL | BOOLEAN | GLOBAL | [] |
| unsafe_disable_etag_checks | Disable checks on ETag consistency | BOOLEAN | GLOBAL | [] |