What’s the openaleph Library All About?

At its core, openaleph is a simple Python library that comes with a command‑line tool, all built to interact with OpenAleph. I primarily use it to ingest data from my disk, streamlining the process and making it more efficient and reliable.

I used the original alephclient (which this new library is a fork of) extensively at my old job, but I kept running into limitations that made simple processes take longer than necessary. Back then, I never had the time to fix these issues. Now that I control my own schedule, I decided it was time to tackle them.

You can install it via the Python package manager: pip install openaleph.

New Features

The core changes affect the crawldir module: One key upgrade is the introduction of a local ingestion state. This lets you cancel and resume an ongoing ingestion process without losing track of what’s already been processed - a real game‑changer for working with large datasets. It also now keeps track of files that ultimately fail, allowing you to debug these separately.

Another neat addition is the ignore file logic. Mimicking some of the behaviors you might be familiar with (think .gitignore), you can now specify patterns to filter out files or folders that aren’t meant to be ingested. This ensures that only the relevant files are processed, keeping your workflow lean and your datasets clean.

Under the Hood

The openaleph crawldir module employs a small, self‑contained SQLite database that stores the path of every file uploaded successfully. The same database also logs records of failed ingests for easier troubleshooting. There's a brief summary printed to stdout at the end of the ingestion process.

The .openalephignore file is applied during the globbing process, which ensures that any rules you’ve set up are enforced before processing begins, keeping things efficient. I decided to remove the original --no-junk option in favor of the file-based mode.

The remaining functionality is identical to that of the alephclient library. Details on how to use the tool are available in the README file.

Feedback wanted!

I'd love to hear from anyone who's used the new library and wants to share their thoughts There's a thread in our forum for feedback, and we'll also be monitoring the GitHub repo for any issues or pull requests.