Search Shortcut cmd + k | ctrl + k
scalarfs

A collection of virtual filesystems for working with scalars

Maintainer(s): teaguesterling

Installing and Loading

INSTALL scalarfs FROM community;
LOAD scalarfs;

Example

LOAD scalarfs;

-- Read JSON from a variable
SET VARIABLE config = '{"debug": true, "port": 8080}';
SELECT * FROM read_json('variable:config');

-- Read CSV from inline content
SELECT * FROM read_csv('data+varchar:name,score
Alice,95
Bob,87');

-- Use a file path stored in a variable
SET VARIABLE data_path = '/data/reports/monthly.csv';
SELECT * FROM read_csv('pathvariable:data_path');

SET VARIABLE data_paths = ['/data/reports/monthly.csv', '/archive/reports/*.csv'];
SELECT * FROM read_csv('pathvariable:data_paths');

-- Write query results to a variable
COPY (SELECT * FROM my_table WHERE active) TO 'variable:exported' (FORMAT json);
SELECT getvariable('exported');

-- Store query results as native values (not serialized text)
COPY (SELECT path FROM files WHERE active) TO 'variable:paths' (FORMAT variable);
SELECT * FROM read_csv('pathvariable:paths');  -- Read all files from the list

-- Select which files to read via a catalog macro (pathmacro:)
CREATE TABLE reports AS
  SELECT * FROM (VALUES ('west', '/data/west.csv'), ('east', '/data/east.csv')) t(region, path);
CREATE MACRO region_files(p) AS (SELECT list(path) FROM reports WHERE region = p['region']);
SET allowed_pathmacro_macros = 'region_files';
SELECT * FROM read_csv('pathmacro:region_files?region=east');  -- reads only the east file

-- Build pathmacro: URLs safely (keys/values URL-encoded)
SELECT to_pathmacro_url('region_files', {region: 'west'});  -- pathmacro:region_files?region=west

About scalarfs

DuckDB's file functions (read_csv, read_json, COPY TO, etc.) expect file paths. scalarfs bridges the gap when your content is already in memory, allowing the same functions to work with:

Variables — Store data in DuckDB variables and read/write them as files Path Variables — Use file paths stored in variables for dynamic file resolution Inline literals — Embed content directly in your queries without temporary files Catalog macros — Select which files to read from an index/catalog via a macro (pathmacro:)

Protocol Purpose Mode
variable: DuckDB variable as file Read/Write
pathvariable: File path(s) stored in variable Read/Write*
data: RFC 2397 data URI (base64/url-encoded) Read
data+varchar: Raw VARCHAR content as file Read
data+blob: Escaped BLOB content as file Read
decompress+gz: Gzip decompression wrapper Read
decompress+zstd: Zstd decompression wrapper Read
pathmacro: Paths resolved by an allow-listed macro Read/Write
  • Can only write to a pathvariable: that's a scalar path (not lists).

Helper functions to_pathmacro_url() / from_pathmacro_url() build and parse pathmacro: URLs.

For full documentation, see: https://scalarfs.readthedocs.io/

Note: This extension was written primarily using Claude and Claude Code as an exercise in AI-driven development.

Added Functions

function_name function_type description comment examples
from_blob_uri scalar NULL NULL  
from_data_uri scalar NULL NULL  
from_pathmacro_url scalar NULL NULL  
from_scalarfs_uri scalar NULL NULL  
from_varchar_uri scalar NULL NULL  
to_blob_uri scalar NULL NULL  
to_data_uri scalar NULL NULL  
to_pathmacro_url scalar NULL NULL  
to_scalarfs_uri scalar NULL NULL  
to_varchar_uri scalar NULL NULL  

Overloaded Functions

This extension does not add any function overloads.

Added Types

This extension does not add any types.

Added Settings

name description input_type scope aliases
allowed_pathmacros Comma-separated list of scalar macros invokable via the pathmacro: filesystem VARCHAR GLOBAL []
scalarfs_max_decompressed_bytes Maximum number of bytes a scalarfs decompress+ read may materialize (decompression-bomb guard; 0 disables the cap) UBIGINT GLOBAL []