Open source · Zig 0.16.0
Chi
Streaming, checkpointable building blocks for text-corpus curation when the data cannot—or should not—fit in memory all at once.
Implemented scope
Four composable primitives
Each component documents borrowed lifetimes, keeps steady-state memory bounded, and exposes a versioned checkpoint for resuming or moving in-flight work.
Bounded ingestion
Read files, deterministic repository trees, or pull-based streams without indexing an entire corpus in memory.
Streaming cleanup
Decode UTF-8, UTF-16, Latin-1, or Windows-1252 and preserve decoder and formatting state across arbitrary chunk boundaries.
Document gates
Apply deterministic length, UTF-8, visibility, replacement, cleaning-change, and repeated-run policies with early rejection.
Deduplication signals
Combine chunk-stable exact fingerprints, SimHash candidates, sampled duplicate spans, and a transactional index boundary.
Verification
The implementation is public
The complete v0.1.0 source is licensed under Apache-2.0. Its test suite runs on Windows and Linux against Zig 0.16.0 and covers arbitrary chunk boundaries, checkpoint transfer, deterministic traversal, duplicate classification, and rollback behavior.
This is an engineering foundation and a public record of the work, not a benchmark claim or a promise of a hosted service.
Boundaries
What v0.1.0 does not claim
- Library components, not a turnkey distributed pipeline
- No NFC or NFKC normalization in v0.1.0
- No built-in language or safety classifiers
- No concrete database or distributed-index backend
- No published performance claims
Questions and reports
Keep the discussion with the code
Use the GitHub repository for bugs and design discussion. Send sensitive reports to security@yycore.ai or read the YY security page.