Performance problems

Identify and resolve performance issues

This page covers slow or hanging queries, out-of-memory, and evaluation that is too slow. By the end, you will know when to switch backends and how to use streaming or sampling.

Troubleshooting

Issue: Queries slow or hanging

Cause: Large datasets or a backend that does not match the workload.

Solution: Use a backend that fits the use case:

  • DuckDB — Analytical queries on moderate-sized data (for example, Parquet, AsOf joins).
  • Pandas — Small data and fast iteration.
  • Snowflake or BigQuery — Large-scale production workloads.

See Switch between backends and Supported backends.

Issue: Out of memory

Error: MemoryError: Unable to allocate array

Cause: The dataset or result is too large for available memory.

Solution:

  1. Stream data: for batch in data.to_pyarrow_batches(chunk_size=100000): ...
  2. Lower chunk size (for example, 10,000) if needed.
  3. Process one file or partition at a time instead of loading everything.

For connection timeouts while reading large data, see Backend connection issues.

Issue: Evaluation too slow

Cause: The test set or metric computation is too large.

Solution: Reduce work per run by sampling the test set and running the metric on the sample:

test_sample = test.limit(1000)
accuracy = deferred_sklearn_metric(
    expr=fitted.predict(test_sample),
    target="target",
    pred_col="predict",
    metric_fn=accuracy_score
).execute()

Issue: Pipeline fitting is slow

Cause: The training data is large or the model is expensive to fit.

Solution: Use caching to avoid re-fitting on every run:

from xorq.caching import ParquetCache

cache = ParquetCache(path="./cache")
pipeline = Pipeline.from_instance(
    SKPipeline([("scaler", StandardScaler()), ("model", LogisticRegression())]),
    cache=cache,
)

See Cache expression results for details.

Next steps