Search Shortcut cmd + k | ctrl + k
hdfs
Downloads 243this week
GitHub stars 0
Extension repository on GitHub
Extension descriptor (YAML)

Read and write hdfs:// URLs via a native HDFS filesystem — no JVM or libhdfs required

Maintainer(s): casperhart

Installing and Loading 

INSTALL hdfs FROM community;
LOAD hdfs;

Example 

-- Read any format DuckDB supports directly from HDFS
SELECT * FROM 'hdfs://namenode:8020/data/events/*.parquet';
SELECT * FROM read_csv('hdfs://namenode:8020/data/users.csv');

-- Write with COPY ... TO, including PARTITION_BY
COPY (SELECT * FROM big_table)
TO 'hdfs://namenode:8020/out/table.parquet' (FORMAT parquet);

-- List directories with full FileStatus metadata
SELECT name, type, size, owner, permissions
FROM hdfs_ls('hdfs://namenode:8020/data');

-- Metadata for a single path, and existence checks
SELECT * FROM hdfs_stat('hdfs://namenode:8020/data');
SELECT hdfs_exists('hdfs://namenode:8020/data/events');

About hdfs

The hdfs extension adds a native HDFS filesystem to DuckDB: read and write hdfs:// URLs directly — no JVM, no libhdfs. It is backed by the pure-Rust hdfs-native client through a thin C FFI bridge.

Features:

  • Reading any file format DuckDB supports (Parquet, CSV, JSON, …) over hdfs://, with concurrent positional reads for DuckDB's parallel Parquet reader.
  • Writing via COPY ... TO, including PARTITION_BY (HDFS is append-only).
  • Globbing (*, ?, [abc], **, plus Hadoop-style {a,b} alternation), directory listing, file metadata, and directory/file management.
  • Metadata SQL functions: hdfs_ls, hdfs_stat, hdfs_exists.
  • HDFS encryption zones (Transparent Data Encryption): data is decrypted/encrypted client-side, unwrapping keys via the KMS over TLS.
  • Kerberos-secured clusters are supported via the standard ticket/keytab mechanism.

Configuration: connection config is resolved from the standard Hadoop environment, exactly as the Hadoop CLI tools do — HADOOP_CONF_DIR (or HADOOP_HOME/etc/hadoop) for core-site.xml/hdfs-site.xml, and HADOOP_USER_NAME for the effective user on non-secure clusters. The scheme-only form hdfs:///path uses fs.defaultFS from your Hadoop config.

For details, see the extension repository.

Added Functions

function_name function_type description comment examples
hdfs_exists scalar NULL NULL  
hdfs_ls table NULL NULL  
hdfs_stat table NULL NULL  

Overloaded Functions

This extension does not add any function overloads.

Added Types

This extension does not add any types.

Added Settings

name description input_type scope aliases
hdfs_include_hidden Return hidden HDFS entries (names starting with '' or '.', e.g. _temporary, _SUCCESS) from listings, globs and scans; by default they are excluded unless a glob component names them explicitly (e.g. '*') BOOLEAN GLOBAL []
hdfs_list_parallelism Maximum number of concurrent HDFS directory-listing RPCs used by listings and globs that walk more than one directory (e.g. hdfs_ls('/path/**')); 1 disables parallelism UBIGINT GLOBAL []
hdfs_skip_permission_errors Prune subtrees whose listing fails with a permission error during HDFS globs and recursive listings instead of failing the query; an error on the listed path or glob root itself still fails BOOLEAN GLOBAL []