Performance problems
This page covers slow or hanging queries, out-of-memory, and evaluation that is too slow. By the end, you will know when to switch backends and how to use streaming or sampling.
Troubleshooting
Issue: Queries slow or hanging
Cause: Large datasets or a backend that does not match the workload.
Solution: Use a backend that fits the use case:
- DuckDB — Analytical queries on moderate-sized data (for example, Parquet, AsOf joins).
- Pandas — Small data and fast iteration.
- Snowflake or BigQuery — Large-scale production workloads.
Issue: Out of memory
Error: MemoryError: Unable to allocate array
Cause: The dataset or result is too large for available memory.
Solution:
- Stream data:
for batch in data.to_pyarrow_batches(chunk_size=100000): ... - Lower chunk size (for example, 10,000) if needed.
- Process one file or partition at a time instead of loading everything.
For connection timeouts while reading large data, see Backend connection issues.
Issue: Evaluation too slow
Cause: The test set or metric computation is too large.
Solution: Reduce work per run by sampling the test set and running the metric on the sample:
test_sample = test.limit(1000)
accuracy = deferred_sklearn_metric(
expr=fitted.predict(test_sample),
target="target",
pred_col="predict",
metric_fn=accuracy_score
).execute()Issue: Pipeline fitting is slow
Cause: The training data is large or the model is expensive to fit.
Solution: Use caching to avoid re-fitting on every run:
from xorq.caching import ParquetCache
cache = ParquetCache(path="./cache")
pipeline = Pipeline.from_instance(
SKPipeline([("scaler", StandardScaler()), ("model", LogisticRegression())]),
cache=cache,
)See Cache expression results for details.
Next steps
- Backend connection issues for connection timeouts
- Switch between backends for backend setup
- Overview for all topics