bunny-lab/Borealis-Github-Replica
1
0
Fork 0
mirror of https://github.com/bunny-lab-io/Borealis.git synced 2025-12-14 22:35:47 -07:00
Code Issues Packages Projects Releases Wiki Activity

Added Assembly DB Migration Tracker

Browse Source
This commit is contained in:
Nicole Rappe
2025-11-02 13:56:20 -07:00
parent 0b61b29e64
commit 5e2dcda397
2 changed files with 125 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files

125
Data/Engine/Assemblies/DB_MIGRATION_TRACKER.md Normal file
View File

@@ -0,0 +1,125 @@
## 1. Implement multi-database assembly persistence with payload indirection
[ ] Define three SQLite databases (`official.db`, `community.db`, `user_created.db`) stored under `Data/Engine/Assemblies/` and mirrored to `/Engine/Assemblies/` at runtime.
[ ] Standardize a shared schema (per your column list) and enable WAL + shared-cache on all connections.
[ ] Add payload GUID indirection so large scripts/workflows/binaries live under `Data/Engine/Assemblies/Payloads/<GUID>` and `/Engine/Assemblies/Payloads/<GUID>`; the DB stores GUID references, not raw base64.
[ ] Build a startup loader that opens each database, validates schema, and hydrates an in-memory cache keyed by `assembly_id` with metadata, payload handles, and source domain.
[ ] Implement a write queue service that stages mutations in memory, marks cache entries as “dirty,” persists them on a 60-second cadence (configurable), and handles graceful shutdown flushing.
[ ] Expose cache state so callers can detect queued writes vs. persisted rows.
### Details
```
1. Under `Data/Engine/engine/assemblies/`, add a new package (e.g., `persistence/`) containing:
* `databases.py` for connection management (WAL, pragmas, attach logic).
* `models.py` defining dataclasses for assemblies/payload metadata.
* `payloads.py` handling GUID generation, filesystem read/write, and mirroring between staging/runtime paths.
2. Write a startup hook in `Data/Engine/server.py` (or appropriate service bootstrap) that:
* Ensures the three `.db` files exist (creating `user_created.db`, `community.db` if missing).
* Runs migrations (idempotent CREATE TABLE IF NOT EXISTS).
* Loads all assemblies into an in-memory cache structure (`AssemblyCache` singleton) with domain tagging.
3. Implement `AssemblyCache` with:
* Read APIs returning cached copies.
* Mutation methods that enqueue write operations into an internal async queue.
* Background worker that flushes dirty entries every 60s or on-demand; allow config overrides.
* Graceful shutdown flush in existing shutdown sequence.
4. Add payload GUID support:
* Modify save/update flows to write payload files (`.json` for workflows, raw text for scripts) under `Payloads/`.
* Update cache entries to track payload GUID + type; adapt runner interfaces to resolve GUID back to content.
5. Include integrity checks/logging: detect missing payload files, log warnings, and surface errors to API callers.
```
## 2. Update Engine services and APIs for multi-domain assemblies
[ ] Refactor existing assembly REST endpoints to read from the cache instead of filesystem JSON.
[ ] Add source metadata (`official`, `community`, `user`) to API responses.
[ ] Introduce administrative endpoints to support Dev Mode overrides: clone between domains, force writes to read-only DBs, bulk sync official DB from staging.
[ ] Provide queue status (dirty vs. persisted) in list/detail responses for UI pill rendering.
### Details
```
1. Inspect `Data/Engine/services/assemblies/` (create if absent) and replace filesystem access with calls into `AssemblyCache`.
2. Extend response serializers to include:
* `source` field.
* `is_dirty` (true when queued for persistence).
* `payload_guid` (for internal tooling; hide from non-admin if required).
3. Implement new endpoints:
* `POST /assemblies/{id}/clone` with body specifying target domain.
* `POST /assemblies/dev-mode/switch` to toggle Dev Mode (admin role only).
* `POST /assemblies/dev-mode/write` to persist staged changes immediately to a chosen domain (bypasses read-only).
* `POST /assemblies/official/sync` to purge and repopulate official DB from staging copy on engine boot.
4. Add role guards: normal operators can only mutate `user_created`; admin+Dev Mode can modify any domain.
5. Update background boot sequence to call `official_db_sync()` before cache hydration.
6. Adjust error handling to surface concurrency/locking issues with retries and user-friendly messages.
```
## 3. Implement Dev Mode authorization and UX toggles
[ ] Gate privileged writes behind Admin role + Dev Mode toggle.
[ ] Store Dev Mode state server-side (per-user session or short-lived token) to prevent unauthorized access.
[ ] Ensure Dev Mode audit logging for all privileged operations.
### Details
```
1. Extend authentication middleware in `Data/Engine/services/auth/` to include role checks and Dev Mode status.
2. Persist Dev Mode state per operator (e.g., in Redis/in-memory with TTL) and expose REST endpoints to enable/disable it.
3. Record Dev Mode actions in audit logs (`Engine/Logs/engine.log` or dedicated file) with user, time, target domain, and operation.
4. Update error responses for insufficient privileges to guide admins to enable Dev Mode.
```
## 4. Enhance Assembly Editor WebUI
[ ] Add “Source” column to AG Grid with domain filter badges.
[ ] Display yellow “Queued to Write to DB” pill for assemblies whose cache entry is dirty.
[ ] Implement Import/Export dropdown in `Assembly_Editor.jsx`:
[ ] Export: bundle metadata + payload contents into legacy JSON format for download.
[ ] Import: parse JSON, populate editor form, and default to saving into `user_created`.
[ ] Add Dev Mode banner/toggles and domain picker when Dev Mode is active; otherwise show read-only warnings.
[ ] Ensure admin-only controls are hidden for non-authorized users.
### Details
```
1. Modify `Data/Engine/web-interface/src/Assemblies/Assembly_List.jsx` (or equivalent) to:
* Fetch new API fields (`source`, `is_dirty`).
* Render Source column with badges (Official/Community/User-Created).
* Show yellow pill when `is_dirty` true.
2. In `Assembly_Editor.jsx`:
* Add Import/Export dropdown; hook export to new REST endpoint or client-side JSON assembly packaging.
* On import, validate schema, generate new payload GUID, and stage in editor state.
* Display Dev Mode status, domain selection dropdown, and warnings when editing read-only domains without Dev Mode.
3. Create reusable components for domain badges/pills to maintain consistent styling.
4. Wire UI actions to new API endpoints (clone, Dev Mode enable/disable, immediate persist).
5. Update i18n/strings files if the UI uses localization.
```
## 5. Support JSON import/export endpoints
[ ] Implement backend utilities to translate between DB model and legacy JSON structure.
[ ] Ensure exports include payload content (decoded) and metadata for compatibility.
[ ] On import, map JSON back into DB schema, creating payload GUIDs and queuing writes.
### Details
```
1. Add `serialization.py` under the assemblies service package to convert between DB models and legacy JSON.
2. Implement `POST /assemblies/import` and `GET /assemblies/{id}/export` endpoints leveraging the serializer.
3. Validate imported JSON (schema, payload size limits) before staging to cache.
4. Unit-test round-trip import/export for scripts and workflows with attached files.
```
## 6. Testing and tooling
[ ] Unit tests for cache, payload storage, Dev Mode permissions, and import/export.
[ ] Integration tests simulating concurrent edits across domains.
[ ] CLI or admin script to run official DB sync and verify counts.
[ ] Update developer documentation with new workflows.
### Details
```
1. Under `Data/Engine/tests/assemblies/`, add:
* Cache tests covering load, dirty flag, flush scheduling.
* Permission tests ensuring Dev Mode restrictions.
* Import/export round-trip tests with payload files.
2. Create an integration test (pytest) mocking concurrent writer/reader scenarios using WAL and ensuring retries succeed.
3. Add a CLI helper (`python Data/Engine/tools/assemblies.py sync-official`) to trigger staging sync manually.
4. Update `Docs/` (e.g., `Docs/assemblies.md`) with:
* Database layout.
* Dev Mode usage instructions.
* Backup guidance (even if future work, note current expectations).
```

52
Data/Engine/CODE_MIGRATION_TRACKER.md
View File

@@ -1,52 +0,0 @@
# Migration Prompt
You are working in the Borealis Automation Platform repo (root: <ProjectRoot>). The legacy runtime lives under Data/Server/server.py. Your objective is to introduce a new Engine runtime under Data/Engine that will progressively take over responsibilities (API first, then WebUI, then WebSocket). Execute the migration in the stages seen below (be sure to not overstep stages, we only want to work on one stage at a time, until I give approval to move onto the next stage):
Everytime you do work, you indicate the current stage you are on by writing to the file in <ProjectRoot>/Data/Engine/CODE_MIGRATION_TRACKER.md, inside of this file, you will keep an up-to-date ledger of the overall task list seen below, as well as the current stage you are on, and what task within that stage you are working on. You will keep this file up-to-date at all times whenever you make progress, and you will reference this file whenever making changes in case you forget where you were last at in the codebase migration work. You will never make modifications to the "# Migration Prompt" section, only the "# Borealis Engine Migration Tracker" section.
Lastly, everytime that you complete a stage, you will create a pull request named "Stage <number> - <Stage Description> Implemented" I will merge your pull request associated with that stage into the "main" branch of the codebase, then I will create a new gpt-5-codex conversation to keep teh conversation fresh and relevant, instructing the agent to work from the next stage in-line, and I expect the Codex agent to read the aforementioned <ProjectRoot>/Data/Engine/CODE_MIGRATION_TRACKER.md to understand what it has already done thus far, and what it needs to work on next. Every time that I start the new conversation, I will instruct gpt-5-codex to read <ProjectRoot>/Data/Engine/CODE_MIGRATION_TRACKER.md to understand it's tasks to determine what to do.
# Borealis Engine Migration Tracker
## Task Ledger
- [x] **Stage 1 — Establish the Engine skeleton and bootstrapper**
- [x] Add Data/Engine/__init__.py plus service subpackages with placeholder modules and docstrings.
- [x] Scaffold Data/Engine/server.py with the create_app(config) factory and stub service registration hooks.
- [x] Return a shared context object containing handles such as the database path, logger, and scheduler.
- [x] Update project tooling so the Engine runtime can be launched alongside the legacy path.
- [x] **Stage 2 — Port configuration and dependency loading into the Engine factory**
- [x] Extract configuration loading logic from Data/Server/server.py into Data/Engine/config.py helpers.
- [x] Verify context parity between Engine and legacy startup.
- [x] Initialize logging to Logs/Server/server.log when Engine mode is active.
- [x] Document Engine launch paths and configuration requirements in module docstrings.
- [x] **Stage 3 — Introduce API blueprints and service adapters**
- [x] Create domain-focused API blueprints and register_api entry point.
- [x] Mirror route behaviour from the legacy server via service adapters.
- [x] Add configuration toggles for enabling API groups incrementally.
- [x] **Stage 4 — Build unit and smoke tests for Engine APIs**
- [x] Add pytest modules under Data/Engine/Unit_Tests exercising API blueprints.
- [x] Provide fixtures that mirror the legacy SQLite schema and seed data.
- [x] Assert HTTP status codes, payloads, and side effects for parity.
- [x] Integrate Engine API tests into CI/local workflows.
- [x] **Stage 5 — Bridge the legacy server to Engine APIs**
- [x] Delegate API blueprint registration to the Engine factory from the legacy server.
- [x] Replace legacy API routes with Engine-provided blueprints gated by a flag.
- [x] Emit transitional logging when Engine handles requests.
- [ ] **Stage 6 — Plan WebUI migration**
- [x] Move static/template handling into Data/Engine/services/WebUI.
- [x] Ensure that data from /Data/Server/WebUI is copied into /Engine/web-interface during engine Deployment via Borealis.ps1
- [x] Preserve TLS-aware URL generation and caching.
- [ ] Add migration switch in the legacy server for WebUI delegation.
- [x] Extend tests to cover critical WebUI routes.
- [ ] Port device API endpoints into Engine services (device + admin coverage in progress).
- [x] Move authentication/token stack onto Engine services without legacy fallbacks.
- [x] Port enrollment request/poll flows to Engine services and drop legacy imports.
- [ ] **Stage 7 — Plan WebSocket migration**
- [ ] Extract Socket.IO handlers into Data/Engine/services/WebSocket.
- [x] Ported quick_job_result handler to keep device activity statuses in sync.
- [ ] Provide register_realtime hook for the Engine factory.
- [ ] Add integration tests or smoke checks for key events.
- [ ] Update legacy server to consume Engine WebSocket registration.
## Current Status
- **Stage:** Stage 6 — Plan WebUI migration
- **Active Task:** Continue Stage 6 device/admin API migration (focus on remaining device and admin endpoints now that auth, token, and enrollment paths are Engine-native).