bunny-lab/Borealis-Github-Replica
1
0
Fork 0
mirror of https://github.com/bunny-lab-io/Borealis.git synced 2025-12-16 06:05:48 -07:00
Code Issues Packages Projects Releases Wiki Activity

Assembly Management Rework - Stage 1 Complete

Browse Source
This commit is contained in:
Nicole Rappe
2025-11-02 15:17:02 -07:00
parent d2ec04284f
commit a86f3117f1
7 changed files with 955 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
Data/Engine/Assemblies/DB_MIGRATION_TRACKER.md
View File

@@ -9,15 +9,15 @@
## 1. Implement multi-database assembly persistence with payload indirection
[ ] Define three SQLite databases (`official.db`, `community.db`, `user_created.db`) stored under `Data/Engine/Assemblies/` and mirrored to `/Engine/Assemblies/` at runtime.
[ ] Standardize a shared schema (per your column list) and enable WAL + shared-cache on all connections.
[ ] Add payload GUID indirection so large scripts/workflows/binaries live under `Data/Engine/Assemblies/Payloads/<GUID>` and `/Engine/Assemblies/Payloads/<GUID>`; the DB stores GUID references, not raw base64.
[ ] Build a startup loader that opens each database, validates schema, and hydrates an in-memory cache keyed by `assembly_id` with metadata, payload handles, and source domain.
[ ] Implement a write queue service that stages mutations in memory, marks cache entries as “dirty,” persists them on a 60-second cadence (configurable), and handles graceful shutdown flushing.
[ ] Expose cache state so callers can detect queued writes vs. persisted rows.
[x] Define three SQLite databases (`official.db`, `community.db`, `user_created.db`) stored under `Data/Engine/Assemblies/` and mirrored to `/Engine/Assemblies/` at runtime.
[x] Standardize a shared schema (per your column list) and enable WAL + shared-cache on all connections.
[x] Add payload GUID indirection so large scripts/workflows/binaries live under `Data/Engine/Assemblies/Payloads/<GUID>` and `/Engine/Assemblies/Payloads/<GUID>`; the DB stores GUID references, not raw base64.
[x] Build a startup loader that opens each database, validates schema, and hydrates an in-memory cache keyed by `assembly_id` with metadata, payload handles, and source domain.
[x] Implement a write queue service that stages mutations in memory, marks cache entries as “dirty,” persists them on a 60-second cadence (configurable), and handles graceful shutdown flushing.
[x] Expose cache state so callers can detect queued writes vs. persisted rows.
### Details
```
1. Under `Data/Engine/engine/assemblies/`, add a new package (e.g., `persistence/`) containing:
1. Under `Data/Engine/assembly_management/`, add the management package containing:
* `databases.py` for connection management (WAL, pragmas, attach logic).
* `models.py` defining dataclasses for assemblies/payload metadata.
@@ -40,6 +40,13 @@
5. Include integrity checks/logging: detect missing payload files, log warnings, and surface errors to API callers.
```
**Stage Notes**
- Added `Data/Engine/assembly_management/` with `databases.py`, `models.py`, `payloads.py`, and `bootstrap.py` to manage multi-domain SQLite storage, payload GUID indirection, and the timed write queue.
- `AssemblyDatabaseManager.initialise()` now creates `official.db`, `community.db`, and `user_created.db` in the staging tree with WAL/shared-cache pragmas and mirrors them to `/Engine/Assemblies/`.
- `PayloadManager` persists payload content beneath `Payloads/<GUID>` in both staging and runtime directories, computing SHA-256 checksums for metadata.
- `AssemblyCache` hydrates all domains at startup, exposes `describe()` for dirty state inspection, and flushes staged writes on a configurable cadence (default 60 s) with an atexit shutdown hook.
- `initialise_assembly_runtime()` is invoked from both `create_app` and `register_engine_api`, wiring the cache onto `EngineContext` and ensuring graceful shutdown flushing.
## 2. Update Engine services and APIs for multi-domain assemblies
[ ] Refactor existing assembly REST endpoints to read from the cache instead of filesystem JSON.
[ ] Add source metadata (`official`, `community`, `user`) to API responses.
@@ -132,4 +139,4 @@
* Database layout.
* Dev Mode usage instructions.
* Backup guidance (even if future work, note current expectations).
```
```