Search Shortcut cmd + k | ctrl + k
boilstream

Secure remote secrets storage with OPAQUE PAKE authentication, email/password registration, MFA login, and automatic ducklake mounting

Maintainer(s): dforsber

Installing and Loading

INSTALL boilstream FROM community;
LOAD boilstream;

Example

-- Setup: Install and load boilstream extension
INSTALL httpfs; LOAD httpfs;
INSTALL boilstream FROM community;
LOAD boilstream;

-- Option 1: Login with email/password/MFA (establishes OPAQUE session)
--     NOTE: Remember to clean up your ~/.duckdb/history
PRAGMA boilstream_login('https://api.example.com/user@example.com', 'password', '123456');
┌────────────────────────┬─────────────────────┬────────────────────┐
         status              expires_at       ducklakes_attached 
├────────────────────────┼─────────────────────┼────────────────────┤
 Session token obtained  2025-11-21 10:30:00                   2 
└────────────────────────┴─────────────────────┴────────────────────┘

-- Option 2: Authenticate with bootstrap token (from SSO/web UI)
PRAGMA boilstream_bootstrap_session('https://api.example.com/secrets:TOKEN');
┌────────────────────────┬─────────────────────┬────────────────────┐
         status              expires_at       ducklakes_attached 
├────────────────────────┼─────────────────────┼────────────────────┤
 Session token obtained  2025-11-21 10:30:00                   2 
└────────────────────────┴─────────────────────┴────────────────────┘

-- List all available ducklakes
FROM boilstream_ducklakes();
┌──────────────┬──────────────┬─────────────────┐
 catalog_id    catalog_name    description   
├──────────────┼──────────────┼─────────────────┤
 5a805647-...  my_analytics  Sales analytics 
└──────────────┴──────────────┴─────────────────┘

-- List all cached secrets with expiration
FROM boilstream_secrets();
┌─────────────────────┬──────────┬─────────────────────┐
        name            type        expires_at      
├─────────────────────┼──────────┼─────────────────────┤
 my_analytics         ducklake  2026-11-19 20:00:00 
 my_analytics_s3      s3        2026-11-19 20:00:00 
 my_analytics_pg      postgres  2026-11-19 20:00:00 
└─────────────────────┴──────────┴─────────────────────┘

-- Create new ducklake
PRAGMA boilstream_create_ducklake('new_catalog', 'My new ducklake');

-- Query ducklake (automatically attached)
USE my_analytics;
SHOW TABLES;

About boilstream

Boilstream extension provides enterprise-grade remote secrets storage for multi-tenant DuckDB deployments.

Features:

  • 🔐 OPAQUE PAKE Authentication - Password-less authentication (IETF RFC 9807)
  • 🔒 End-to-end Encryption - AES-256-GCM/ChaCha20-Poly1305 for secrets and responses
  • 🔑 Cryptographic Key Derivation - HKDF-SHA256 with domain separation
  • 🛡️ Defense in Depth - HMAC verification before AEAD decryption
  • 🎯 Session Protection - Lock-step sequence protocol prevents hijacking
  • 🗄️ Ducklake Support - Auto-attach multi-tenant data catalogs
  • 🌐 SSO Integration - Bootstrap token exchange for web environments

Security:

  • Facebook's audited OPAQUE implementation (Rust + C++ FFI)
  • Protects against MITM attacks even if TLS is compromised
  • One-time bootstrap tokens with anonymous authentication
  • Session resumption with encrypted refresh tokens
  • Comprehensive test vectors and security specification

API Functions:

  • PRAGMA boilstream_register_user(url_with_email, password) - Register new user with MFA (displays QR code)
  • PRAGMA boilstream_verify_mfa(totp_code) - Complete registration and get backup codes
  • PRAGMA boilstream_login(url_with_email, password, mfa_code) - Email/password/MFA login
  • PRAGMA boilstream_bootstrap_session(url_with_token) - Authenticate with bootstrap token
  • boilstream_ducklakes() - List available ducklakes
  • boilstream_secrets() - List cached secrets with expiration
  • PRAGMA boilstream_create_ducklake(name, description) - Create new ducklake
  • PRAGMA boilstream_help - Show all available commands

Requirements:

  • httpfs extension (auto-loaded)
  • postgres_scanner extension (auto-loaded)
  • ducklake extension (auto-loaded)
  • Compatible REST API server (see API specification)

For detailed documentation, security specification, and setup instructions, visit the extension repository.

Added Functions

function_name function_type description comment examples
boilstream_bootstrap_session pragma NULL NULL  
boilstream_create_ducklake pragma NULL NULL  
boilstream_ducklakes table NULL NULL  
boilstream_help pragma NULL NULL  
boilstream_login pragma NULL NULL  
boilstream_register_user pragma NULL NULL  
boilstream_secrets table NULL NULL  
boilstream_verify_mfa pragma NULL NULL  
ducklake_add_data_files table NULL NULL  
ducklake_cleanup_old_files table NULL NULL  
ducklake_current_snapshot table NULL NULL  
ducklake_delete_orphaned_files table NULL NULL  
ducklake_expire_snapshots table NULL NULL  
ducklake_flush_inlined_data table NULL NULL  
ducklake_last_committed_snapshot table NULL NULL  
ducklake_list_files table NULL NULL  
ducklake_merge_adjacent_files table NULL NULL  
ducklake_options table NULL NULL  
ducklake_rewrite_data_files table NULL NULL  
ducklake_set_commit_message table NULL NULL  
ducklake_set_option table NULL NULL  
ducklake_snapshots table NULL NULL  
ducklake_table_changes table_macro NULL NULL  
ducklake_table_deletions table NULL NULL  
ducklake_table_info table NULL NULL  
ducklake_table_insertions table NULL NULL  
pg_clear_cache table NULL NULL  
postgres_attach table NULL NULL  
postgres_execute table NULL NULL  
postgres_query table NULL NULL  
postgres_scan table NULL NULL  
postgres_scan_pushdown table NULL NULL  

Added Settings

name description input_type scope aliases
auto_fallback_to_full_download Allows automatically falling back to full file downloads when possible. BOOLEAN GLOBAL []
ca_cert_file Path to a custom certificate file for self-signed certificates. VARCHAR GLOBAL []
ducklake_max_retry_count The maximum amount of retry attempts for a ducklake transaction UBIGINT GLOBAL []
ducklake_retry_backoff Backoff factor for exponentially increasing retry wait time DOUBLE GLOBAL []
ducklake_retry_wait_ms Time between retries UBIGINT GLOBAL []
enable_curl_server_cert_verification Enable server side certificate verification for CURL backend. BOOLEAN GLOBAL []
enable_server_cert_verification Enable server side certificate verification. BOOLEAN GLOBAL []
force_download Forces upfront download of file BOOLEAN GLOBAL []
hf_max_per_page Debug option to limit number of items returned in list requests UBIGINT GLOBAL []
http_keep_alive Keep alive connections. Setting this to false can help when running into connection failures BOOLEAN GLOBAL []
http_retries HTTP retries on I/O error UBIGINT GLOBAL []
http_retry_backoff Backoff factor for exponentially increasing retry wait time FLOAT GLOBAL []
http_retry_wait_ms Time between retries UBIGINT GLOBAL []
http_timeout HTTP timeout read/write/connection/retry (in seconds) UBIGINT GLOBAL []
httpfs_client_implementation Select which is the HTTPUtil implementation to be used VARCHAR GLOBAL []
pg_array_as_varchar Read Postgres arrays as varchar - enables reading mixed dimensional arrays BOOLEAN GLOBAL []
pg_connection_cache Whether or not to use the connection cache BOOLEAN GLOBAL []
pg_connection_limit The maximum amount of concurrent Postgres connections UBIGINT GLOBAL []
pg_debug_show_queries DEBUG SETTING: print all queries sent to Postgres to stdout BOOLEAN GLOBAL []
pg_experimental_filter_pushdown Whether or not to use filter pushdown BOOLEAN GLOBAL []
pg_null_byte_replacement When writing NULL bytes to Postgres, replace them with the given character VARCHAR GLOBAL []
pg_pages_per_task The amount of pages per task UBIGINT GLOBAL []
pg_use_binary_copy Whether or not to use BINARY copy to read data BOOLEAN GLOBAL []
pg_use_ctid_scan Whether or not to parallelize scanning using table ctids BOOLEAN GLOBAL []
pg_use_text_protocol Whether or not to use TEXT protocol to read data. This is slower, but provides better compatibility with non-Postgres systems BOOLEAN GLOBAL []
s3_access_key_id S3 Access Key ID VARCHAR GLOBAL []
s3_endpoint S3 Endpoint VARCHAR GLOBAL []
s3_kms_key_id S3 KMS Key ID VARCHAR GLOBAL []
s3_region S3 Region VARCHAR GLOBAL []
s3_requester_pays S3 use requester pays mode BOOLEAN GLOBAL []
s3_secret_access_key S3 Access Key VARCHAR GLOBAL []
s3_session_token S3 Session Token VARCHAR GLOBAL []
s3_uploader_max_filesize S3 Uploader max filesize (between 50GB and 5TB) VARCHAR GLOBAL []
s3_uploader_max_parts_per_file S3 Uploader max parts per file (between 1 and 10000) UBIGINT GLOBAL []
s3_uploader_thread_limit S3 Uploader global thread limit UBIGINT GLOBAL []
s3_url_compatibility_mode Disable Globs and Query Parameters on S3 URLs BOOLEAN GLOBAL []
s3_url_style S3 URL style VARCHAR GLOBAL []
s3_use_ssl S3 use SSL BOOLEAN GLOBAL []
unsafe_disable_etag_checks Disable checks on ETag consistency BOOLEAN GLOBAL []