A collection of virtual filesystems for working with scalars
Installing and Loading
INSTALL scalarfs FROM community;
LOAD scalarfs;
Example
LOAD scalarfs;
-- Read JSON from a variable
SET VARIABLE config = '{"debug": true, "port": 8080}';
SELECT * FROM read_json('variable:config');
-- Read CSV from inline content
SELECT * FROM read_csv('data+varchar:name,score
Alice,95
Bob,87');
-- Use a file path stored in a variable
SET VARIABLE data_path = '/data/reports/monthly.csv';
SELECT * FROM read_csv('pathvariable:data_path');
SET VARIABLE data_paths = ['/data/reports/monthly.csv', '/archive/reports/*.csv'];
SELECT * FROM read_csv('pathvariable:data_paths');
-- Write query results to a variable
COPY (SELECT * FROM my_table WHERE active) TO 'variable:exported' (FORMAT json);
SELECT getvariable('exported');
-- Store query results as native values (not serialized text)
COPY (SELECT path FROM files WHERE active) TO 'variable:paths' (FORMAT variable);
SELECT * FROM read_csv('pathvariable:paths'); -- Read all files from the list
-- Select which files to read via a catalog macro (pathmacro:)
CREATE TABLE reports AS
SELECT * FROM (VALUES ('west', '/data/west.csv'), ('east', '/data/east.csv')) t(region, path);
CREATE MACRO region_files(p) AS (SELECT list(path) FROM reports WHERE region = p['region']);
SET allowed_pathmacro_macros = 'region_files';
SELECT * FROM read_csv('pathmacro:region_files?region=east'); -- reads only the east file
-- Build pathmacro: URLs safely (keys/values URL-encoded)
SELECT to_pathmacro_url('region_files', {region: 'west'}); -- pathmacro:region_files?region=west
About scalarfs
DuckDB's file functions (read_csv, read_json, COPY TO, etc.) expect file paths. scalarfs bridges the gap when your content is already in memory, allowing the same functions to work with:
Variables — Store data in DuckDB variables and read/write them as files Path Variables — Use file paths stored in variables for dynamic file resolution Inline literals — Embed content directly in your queries without temporary files Catalog macros — Select which files to read from an index/catalog via a macro (pathmacro:)
| Protocol | Purpose | Mode |
|---|---|---|
| variable: | DuckDB variable as file | Read/Write |
| pathvariable: | File path(s) stored in variable | Read/Write* |
| data: | RFC 2397 data URI (base64/url-encoded) | Read |
| data+varchar: | Raw VARCHAR content as file | Read |
| data+blob: | Escaped BLOB content as file | Read |
| decompress+gz: | Gzip decompression wrapper | Read |
| decompress+zstd: | Zstd decompression wrapper | Read |
| pathmacro: | Paths resolved by an allow-listed macro | Read/Write |
- Can only write to a
pathvariable:that's a scalar path (not lists).
Helper functions to_pathmacro_url() / from_pathmacro_url() build and parse pathmacro: URLs.
For full documentation, see: https://scalarfs.readthedocs.io/
Note: This extension was written primarily using Claude and Claude Code as an exercise in AI-driven development.
Added Functions
| function_name | function_type | description | comment | examples |
|---|---|---|---|---|
| from_blob_uri | scalar | NULL | NULL | |
| from_data_uri | scalar | NULL | NULL | |
| from_pathmacro_url | scalar | NULL | NULL | |
| from_scalarfs_uri | scalar | NULL | NULL | |
| from_varchar_uri | scalar | NULL | NULL | |
| to_blob_uri | scalar | NULL | NULL | |
| to_data_uri | scalar | NULL | NULL | |
| to_pathmacro_url | scalar | NULL | NULL | |
| to_scalarfs_uri | scalar | NULL | NULL | |
| to_varchar_uri | scalar | NULL | NULL |
Overloaded Functions
This extension does not add any function overloads.
Added Types
This extension does not add any types.
Added Settings
| name | description | input_type | scope | aliases |
|---|---|---|---|---|
| allowed_pathmacros | Comma-separated list of scalar macros invokable via the pathmacro: filesystem | VARCHAR | GLOBAL | [] |
| scalarfs_max_decompressed_bytes | Maximum number of bytes a scalarfs decompress+ read may materialize (decompression-bomb guard; 0 disables the cap) | UBIGINT | GLOBAL | [] |