Glossary

This glossary defines important terms used in Xorq documentation. If you encounter a technical term you don’t recognize, check here first.

Fundamentals

These terms describe fundamental Xorq concepts that appear throughout the documentation.

Backend

An execution engine that runs your queries. Xorq supports multiple backends, including DuckDB, DataFusion, Snowflake, PostgreSQL, Pandas, and more. You can switch backends without changing your code.

Expression

A query or computation that you build but don’t execute immediately. Expressions are like blueprints. They describe what you want to do, but computation happens only when you call .execute().

Deferred execution

Xorq’s approach of building expression graphs before running computation. Operations are queued up and optimized together, then executed only when you explicitly request results.

Lazy evaluation

The practice of deferring computation until results are needed. Xorq operations are lazy by default. They build expression graphs but don’t execute until you call .execute().

Manifest

A YAML file that captures an expression as a portable artifact. Manifests let you share computations across teams and environments without code dependencies.

Build

A compiled version of an expression saved as a reusable artifact. Builds include the expression logic, dependencies, and metadata needed to reproduce the computation.

Build hash

A unique identifier (like 12287e173c17) computed from a build’s contents. The same expression always produces the same hash, enabling reproducible builds.

Node hash

A unique identifier that’s computed for each operation in an expression graph. Node hashes enable caching and help track computation dependencies.

Profile

A configuration that stores backend connection settings. Profiles let you switch between different databases and engines without hardcoding credentials.

Schema

The structure of a table that defines column names and data types. Schemas ensure type safety and help Xorq optimize queries.

Memtable

A table created from in-memory Python data (like lists or dictionaries). Useful for testing and combining Python data with database queries.

Accessor

A way to reference columns without knowing the complete table schema. In Xorq, xo._ is an accessor that lets you reference columns like xo._.sepal_length.

Register

To load data into a backend and make it available as a table expression. Registered tables can be queried and transformed.

Functions and extensions

These terms describe how you can extend Xorq with custom functions and remote services.

User-defined function (UDF)

Custom Python functions you write that run within Xorq expressions. UDFs let you add domain-specific logic that isn’t available in standard SQL or pandas operations.

User-defined exchange function (UDXF)

A specialized UDF that runs in a separate process or remote service using Apache Arrow Flight. UDXFs are ideal for external API calls, ML model inference, or resource-intensive computations.

Exchange

A UDXF endpoint that can be called remotely. Exchanges receive data, apply transformations, and return results over the Flight protocol.

Data operations

These terms describe common data transformation operations you can perform in Xorq.

Filter

Select rows that meet certain conditions. Similar to SQL WHERE clauses or pandas boolean indexing.

Group by

Split data into groups based on column values, then apply operations to each group separately.

Aggregate (agg)

Compute summary statistics (like sum, mean, count) across groups or entire tables.

Mutate

Add new columns or modify existing ones based on calculations from other columns.

Join

Combine data from multiple tables by matching values in specified columns.

AsOf join

A type of join that matches rows based on the most recent value before or at a specific point in time. Useful for time-series data and temporal queries.

Pipe

Chain operations together, passing the result of one expression as input to the next.

Into backend

Move data between execution engines. Xorq transfers data efficiently using Apache Arrow’s zero-copy protocol.

Window function

An operation that performs calculations across a set of rows related to the current row. Examples include running totals, moving averages, and ranking functions.

Temporal

Relating to time or time-based data. Temporal operations handle time-series data, point-in-time queries, and time-based joins.

Time-series

Data collected over time with timestamps. Time-series operations include temporal joins, window functions, and time-based aggregations.

Cardinality

The number of unique values in a column. High cardinality means many unique values. Low cardinality means few unique values.

Bucket

A group or partition created by hashing values. Used in train/test splits to assign rows to partitions deterministically.

Machine learning

These terms describe ML concepts and operations in Xorq.

Step

A single operation in an ML pipeline, typically wrapping a scikit-learn estimator (like a scaler or classifier).

Pipeline

A sequence of Steps that process data and train models. Pipelines chain preprocessing, feature engineering, and model training together.

FittedPipeline

A Pipeline that has been trained on data. Fitted pipelines can make predictions or transform new data.

FittedStep

A Step that has been trained on data. Fitted steps can transform or predict on new data.

Estimator

A scikit-learn component that can learn from data. Examples include classifiers, regressors, and transformers like scalers.

Classifier

A type of estimator that predicts categorical labels (like “spam” or “not spam”). Examples include k-nearest neighbors, random forests, and support vector machines.

Scaler

A transformer that normalizes or standardizes feature values. Common scalers include StandardScaler, which centers and scales data to have a mean of 0 and a variance of 1.

Preprocessing

Data transformations applied before model training. Includes scaling, encoding, imputation, and feature engineering.

Feature engineering

The process of creating new features from raw data to improve model performance. Includes transformations, combinations, and domain-specific calculations.

Train/test split

Dividing data into separate sets for training models and evaluating their performance. Xorq provides deterministic, reproducible splits.

Validation set

A subset of data used to tune hyperparameters and evaluate model performance during training. Separate from both training and test sets.

Holdout set

A reserved portion of data that’s used only for final model evaluation and never used during training or validation to ensure unbiased performance estimates.

Deferred fit/predict

Training and prediction operations that follow deferred execution. Models train only when you explicitly execute, enabling caching of trained models.

Score

A metric that measures model performance. Standard scores include accuracy, precision, recall, F1-score, and ROC-AUC.

Score expression

A deferred expression that computes a model’s performance score. Created with .score_expr() and executed to get the actual score value.

Data and storage

These terms describe how Xorq stores and manages data.

Cache

Stored results from previous expression executions. Xorq automatically reuses cached results when expressions haven’t changed, avoiding redundant computation.

Cache invalidation

The process of marking cached data as outdated when source data changes. Xorq automatically invalidates caches to ensure results stay fresh.

Cache key

A unique identifier computed from an expression’s structure and data sources and used to determine if cached results match a new expression.

Content-addressed hashing

A method of identifying data by computing a hash from its contents. Identical computations produce identical hashes, enabling reliable caching and versioning.

SourceCache

A cache that automatically invalidates when upstream data files change. Ensures cached results stay fresh without manual intervention.

SourceSnapshotCache

A cache that stores results without automatic invalidation. Useful when you want manual control over when the cache is cleared.

ParquetCache

A cache that stores results as Parquet files on disk. Useful for persisting intermediate results across sessions.

ParquetSnapshotCache

A cache that stores results as Parquet files without automatic invalidation. Ideal for reproducible research where you want fixed snapshots.

Snapshot

A fixed point-in-time copy of data. Snapshots don’t change even if the source data changes, making them useful for reproducible analysis.

Materialize

To execute an expression and store the results. Materialized data is actual data, not an expression description.

Parquet

A columnar storage format optimized for analytical workloads. Parquet files compress well and support efficient column-level reads.

Columnar

A storage format where columns rather than rows organize data. Columnar formats like Parquet and Arrow are efficient for analytical queries.

Compression

Reducing file size by encoding data more efficiently. Parquet files use compression to save storage space and improve read performance.

Durable persistence

Storing data in a way that survives system restarts. Disk-based caches provide durable persistence, while in-memory caches are temporary.

Infrastructure

These terms describe Xorq’s infrastructure components and systems.

Compute catalog

A registry that stores builds, aliases, and metadata. The catalog helps teams discover, share, and version computations across projects.

Alias

A human-readable name pointing to a specific build hash. Aliases make it easier to reference builds without having to remember long hash strings.

Entry

A catalog record that stores a build with its metadata. Entries have unique IDs and can have multiple revisions.

Entry ID

A unique identifier (UUID) for a catalog entry. Entry IDs track builds across revisions and aliases.

Revision

A version of a catalog entry. Revisions let you track changes to builds over time while maintaining the same entry ID.

Metadata

Information about builds, expressions, or data. Metadata includes schemas, timestamps, dependencies, and other descriptive information.

Apache Arrow

An in-memory columnar format that enables zero-copy data transfer between processes and engines. Xorq uses Arrow to move data efficiently between backends.

Apache Arrow Flight

A protocol for high-performance data transfer over networks. Xorq uses Flight for UDXFs and distributed execution.

Flight Server

A service that hosts UDXFs and serves them as network endpoints. Clients connect to Flight servers to execute remote computations.

Flight Client

A connection to a Flight Server that sends data and receives results. Xorq automatically manages Flight clients when you use UDXFs.

Multi-engine

The ability to run the same expression across different execution engines (like DuckDB, Snowflake, or DataFusion) without code changes.

Embedded backend

Xorq’s default execution engine based on DataFusion. Fast, supports all Xorq features, and requires no external setup.

Stateless

A system that doesn’t store information between requests. Stateless execution makes systems easier to scale and reason about.

Stateful

A system that maintains information between requests. Stateful systems can be more complex but enable features like caching and session management.

Tracking and governance

These terms describe how Xorq tracks data flow and enforces quality standards.

Lineage

The record of where data comes from and how it’s transformed. Xorq automatically tracks column-level lineage, showing the full path from source to result.

Column-level lineage

Tracking that shows the origin and transformation history of individual columns, not just entire tables.

Validation

Checking that data or models meet specific quality standards. Validation can catch errors early and ensure correctness.

Production and deployment

These terms describe concepts related to deploying Xorq in production environments.

Point-in-time correctness

Ensuring that features and predictions use only data that was available at a specific point in time. Prevents data leakage in ML pipelines.

Temporal leakage

A bug where models train on information they shouldn’t have access to at prediction time. Xorq’s point-in-time correctness prevents this.

Feature store

A system for storing and serving features (computed attributes) used in ML models. Xorq can build feature stores with proper versioning and lineage.

Feature pipeline

A sequence of transformations that compute features from raw data. Feature pipelines ensure consistent feature computation across training and serving.

Serve

Deploy an expression or model as a network endpoint that can be called remotely. Xorq can serve expressions via Flight servers.

Unbound expression

An expression that doesn’t require a specific backend connection. Unbound expressions can serve as endpoints and be executed with different backends.

Hermetic

An environment that is entirely isolated and reproducible. Hermetic builds include all dependencies and produce identical results regardless of the host system.

Reproducible

Producing the same results whenever given the same inputs. Reproducible environments and builds ensure consistency across development and production.

Dependency locking

Pinning exact versions of all dependencies to ensure reproducible builds. Tools like uv lock dependencies to specific versions.

Environment promotion

Moving builds or configurations through stages like development, staging, and production. Ensures tested code reaches production safely.

Technical terms

These terms describe technical concepts and tools used in Xorq.

Artifact

A packaged version of a computation that includes code, dependencies, and metadata. Builds produce artifacts that can be shared and executed independently.

Expression graph

The internal representation of your query as a tree of operations. Xorq builds this graph before execution to enable optimization.

Zero-copy transfer

Moving data between processes without serialization overhead. Apache Arrow enables zero-copy transfers, making backend switching fast and memory-efficient.

Serialize

To convert data or code into a format that can be stored or transmitted. Xorq serializes expressions to YAML manifests.

Deserialize

To convert serialized data back into its original form. Xorq deserializes YAML manifests back into executable expressions.

Ibis

The underlying library Xorq uses for cross-engine expression trees. Ibis expressions are serialized to YAML for portability.

DataFusion

An embedded query engine optimized for Arrow operations. Xorq’s default embedded backend is based on DataFusion.

uv

A Python package manager that Xorq uses for reproducible environment management. Ensures consistent execution across development and production.

Deterministic

Producing the same results whenever given the same inputs. Xorq’s splits and hashes are deterministic, making experiments reproducible.

Declarative

Describing what you want rather than how to compute it. Declarative manifests let Xorq automatically optimize execution.

Predicate

A condition used to filter rows. Predicates are boolean expressions, such as sepal_length > 6.

Field

A single column in a table. Fields have names and data types defined in the schema.

Relation

A table or the result of a query operation. Relations are the building blocks of expression graphs.

Operation

A single transformation or computation step. Operations include filter, group_by, aggregate, join, and many others.

Node

A single element in an expression graph representing an operation or data source. Nodes connect to form the complete expression tree.

Definitions

The section of a manifest that declares schemas, nodes, and other reusable components before they’re used in expressions.

PyArrow

The Python library for working with Apache Arrow data. Xorq returns results as PyArrow Tables for efficient data manipulation.

Imputation

Filling in missing values in data. Imputation strategies include mean, median, mode, or model-based approaches.

Roundtrip

The ability to convert data or code from one format to another and back again without loss. Xorq expressions can roundtrip between Python code and YAML manifests.

Deferred reads

Data loading operations that are deferred until execution. Deferred reads are captured in deferred_reads.yaml files within builds.

Model Context Protocol (MCP)

A protocol for integrating AI assistants like Claude with external tools and services. Xorq can be exposed as an MCP tool.

Semantic layer

An abstraction layer that provides business-friendly views of data. Xorq can build semantic layers for analytics tools.

Data CI

Continuous integration practices applied to data pipelines. Ensures data quality and catches issues before production.

Fail-fast

Stopping execution immediately when an error is detected, rather than continuing and potentially producing incorrect results.

Compile-time validation

Checking for errors and policy violations before code executes. Catches issues early in the development cycle.

Schema evolution

The process of changing table schemas over time while maintaining compatibility. Xorq handles schema changes gracefully.

Arrow schema

A schema definition using Apache Arrow’s type system. Arrow schemas enable efficient data transfer and type safety.

sdist

A source distribution containing Python package source code. Generated as part of Xorq builds when using uv-build.

Time To Live (TTL)

The duration that cached data remains valid before expiration. TTL helps manage cache storage and freshness.