From bcf55851b5bc6d1984f845d0d7fc6ceb72371349 Mon Sep 17 00:00:00 2001 From: Nicole Rappe Date: Mon, 3 Nov 2025 02:43:06 -0700 Subject: [PATCH] Removed Assembly DB Migration Tracker --- .../Engine/Assemblies/DB_MIGRATION_TRACKER.md | 170 ------------------ 1 file changed, 170 deletions(-) delete mode 100644 Data/Engine/Assemblies/DB_MIGRATION_TRACKER.md diff --git a/Data/Engine/Assemblies/DB_MIGRATION_TRACKER.md b/Data/Engine/Assemblies/DB_MIGRATION_TRACKER.md deleted file mode 100644 index f0c2c4bd..00000000 --- a/Data/Engine/Assemblies/DB_MIGRATION_TRACKER.md +++ /dev/null @@ -1,170 +0,0 @@ -# Prompt Instructions for Assembly Database Migration & Implementation -1. Open and read `Data/Engine/Assemblies/DB_MIGRATION_TRACKER.md`. -2. Work through the migration tracker one stage at a time. For the current stage: - • Read the entire task list plus the associated “Details” subsection before changing any code. - • Implement only the current stage’s requirements before asking for the operator to sync a commit to github before moving on. Do not do this yourself. -3. As soon as you complete any checkbox item (even partway through your coding), immediately update the markdown file to mark that checkbox as checked (`[x]`). Do this incrementally while you code, not just at the end. -4. Repeat the same read/implement/check-off cycle for each subsequent stage until the file is fully completed. -5. Keep all stage notes synchronized with the implementation as you go. - - -## 1. Implement multi-database assembly persistence with payload indirection -[x] Define three SQLite databases (`official.db`, `community.db`, `user_created.db`) stored under `Data/Engine/Assemblies/` and mirrored to `/Engine/Assemblies/` at runtime. -[x] Standardize a shared schema (per your column list) and enable WAL + shared-cache on all connections. -[x] Add payload GUID indirection so large scripts/workflows/binaries live under `Data/Engine/Assemblies/Payloads/` and `/Engine/Assemblies/Payloads/`; the DB stores GUID references, not raw base64. -[x] Build a startup loader that opens each database, validates schema, and hydrates an in-memory cache keyed by `assembly_id` with metadata, payload handles, and source domain. -[x] Implement a write queue service that stages mutations in memory, marks cache entries as “dirty,” persists them on a 60-second cadence (configurable), and handles graceful shutdown flushing. -[x] Expose cache state so callers can detect queued writes vs. persisted rows. -### Details -``` -1. Under `Data/Engine/assembly_management/`, add the management package containing: - - * `databases.py` for connection management (WAL, pragmas, attach logic). - * `models.py` defining dataclasses for assemblies/payload metadata. - * `payloads.py` handling GUID generation, filesystem read/write, and mirroring between staging/runtime paths. -2. Write a startup hook in `Data/Engine/server.py` (or appropriate service bootstrap) that: - - * Ensures the three `.db` files exist (creating `user_created.db`, `community.db` if missing). - * Runs migrations (idempotent CREATE TABLE IF NOT EXISTS). - * Loads all assemblies into an in-memory cache structure (`AssemblyCache` singleton) with domain tagging. -3. Implement `AssemblyCache` with: - - * Read APIs returning cached copies. - * Mutation methods that enqueue write operations into an internal async queue. - * Background worker that flushes dirty entries every 60s or on-demand; allow config overrides. - * Graceful shutdown flush in existing shutdown sequence. -4. Add payload GUID support: - - * Modify save/update flows to write payload files (`.json` for workflows, raw text for scripts) under `Payloads/`. - * Update cache entries to track payload GUID + type; adapt runner interfaces to resolve GUID back to content. -5. Include integrity checks/logging: detect missing payload files, log warnings, and surface errors to API callers. -``` - -**Stage Notes** -- Added `Data/Engine/assembly_management/` with `databases.py`, `models.py`, `payloads.py`, and `bootstrap.py` to manage multi-domain SQLite storage, payload GUID indirection, and the timed write queue. -- `AssemblyDatabaseManager.initialise()` now creates `official.db`, `community.db`, and `user_created.db` in the staging tree with WAL/shared-cache pragmas and mirrors them to `/Engine/Assemblies/`. -- `PayloadManager` persists payload content beneath `Payloads/` in both staging and runtime directories, computing SHA-256 checksums for metadata. -- `AssemblyCache` hydrates all domains at startup, exposes `describe()` for dirty state inspection, and flushes staged writes on a configurable cadence (default 60 s) with an atexit shutdown hook. -- `initialise_assembly_runtime()` is invoked from both `create_app` and `register_engine_api`, wiring the cache onto `EngineContext` and ensuring graceful shutdown flushing. - -## 2. Update Engine services and APIs for multi-domain assemblies -[x] Refactor existing assembly REST endpoints to read from the cache instead of filesystem JSON. -[x] Add source metadata (`official`, `community`, `user`) to API responses. -[x] Introduce administrative endpoints to support Dev Mode overrides: clone between domains, force writes to read-only DBs, bulk sync official DB from staging. -[x] Provide queue status (dirty vs. persisted) in list/detail responses for UI pill rendering. -### Details -``` -1. Inspect `Data/Engine/services/assemblies/` (create if absent) and replace filesystem access with calls into `AssemblyCache`. -2. Extend response serializers to include: - - * `source` field. - * `is_dirty` (true when queued for persistence). - * `payload_guid` (for internal tooling; hide from non-admin if required). -3. Implement new endpoints: - - * `POST /assemblies/{id}/clone` with body specifying target domain. - * `POST /assemblies/dev-mode/switch` to toggle Dev Mode (admin role only). - * `POST /assemblies/dev-mode/write` to persist staged changes immediately to a chosen domain (bypasses read-only). - * `POST /assemblies/official/sync` to purge and repopulate official DB from staging copy on engine boot. -4. Add role guards: normal operators can only mutate `user_created`; admin+Dev Mode can modify any domain. -5. Update background boot sequence to call `official_db_sync()` before cache hydration. -6. Adjust error handling to surface concurrency/locking issues with retries and user-friendly messages. -``` - -**Stage Notes** -- Assembly REST stack now proxies through `AssemblyRuntimeService`, sourcing list/detail data from `AssemblyCache` with GUID-based payload resolution (`Data/Engine/services/assemblies/service.py`, `Data/Engine/services/API/assemblies/management.py`). -- Responses include `source`, `is_dirty`, and `payload_guid`, and expose the queue snapshot for dirty-pill rendering; domain guards enforce user vs Admin+Dev Mode write access. -- Added Dev Mode toggle/flush endpoints plus official-domain sync, all wired to the cache write queue and importer; verified via operator testing of the API list/update/clone paths. - -## 3. Implement Dev Mode authorization and UX toggles -[x] Gate privileged writes behind Admin role + Dev Mode toggle. -[x] Store Dev Mode state server-side (per-user session or short-lived token) to prevent unauthorized access. -[x] Ensure Dev Mode audit logging for all privileged operations. -### Details -``` -1. Extend authentication middleware in `Data/Engine/services/auth/` to include role checks and Dev Mode status. -2. Persist Dev Mode state per operator (e.g., in Redis/in-memory with TTL) and expose REST endpoints to enable/disable it. -3. Record Dev Mode actions in audit logs (`Engine/Logs/engine.log` or dedicated file) with user, time, target domain, and operation. -4. Update error responses for insufficient privileges to guide admins to enable Dev Mode. -``` - -**Stage Notes** -- Added `Data/Engine/services/auth/` with `DevModeManager` and `RequestAuthContext`, storing Dev Mode grants server-side with configurable TTL enforcement. -- `EngineServiceAdapters` now provisions a shared `DevModeManager`, exposing auth helpers to API groups. -- Updated `Data/Engine/services/API/assemblies/management.py` to enforce admin + Dev Mode checks for privileged mutations, centralise error messaging, and route Dev Mode toggles through the server-side manager. -- Privileged assembly endpoints now emit structured audit entries via the Engine service log, covering success and denial paths for create/update/delete/clone, manual flush, official sync, and Dev Mode state changes. - -## 4. Enhance Assembly Editor WebUI -[ ] Add “Source” column to AG Grid with domain filter badges. -[ ] Display yellow “Queued to Write to DB” pill for assemblies whose cache entry is dirty. -[ ] Implement Import/Export dropdown in `Assembly_Editor.jsx`: - [x] Export: bundle metadata + payload contents into legacy JSON format for download. - [x] Import: parse JSON, populate editor form, and default to saving into `user_created`. -[x] Add Dev Mode banner/toggles and domain picker when Dev Mode is active; otherwise show read-only warnings. -[x] Ensure admin-only controls are hidden for non-authorized users. -### Details -``` -1. Modify `Data/Engine/web-interface/src/Assemblies/Assembly_List.jsx` (or equivalent) to: - - * Fetch new API fields (`source`, `is_dirty`). - * Render Source column with badges (Official/Community/User-Created). - * Show yellow pill when `is_dirty` true. -2. In `Assembly_Editor.jsx`: - - * Add Import/Export dropdown; hook export to new REST endpoint or client-side JSON assembly packaging. - * On import, validate schema, generate new payload GUID, and stage in editor state. - * Display Dev Mode status, domain selection dropdown, and warnings when editing read-only domains without Dev Mode. -3. Create reusable components for domain badges/pills to maintain consistent styling. -4. Wire UI actions to new API endpoints (clone, Dev Mode enable/disable, immediate persist). -5. Update i18n/strings files if the UI uses localization. -``` - -**Stage Notes** -- `Assembly_List.jsx` now consumes `/api/assemblies`, renders domain badges plus dirty-state pills, exposes clone-to-domain via a new dialog, and surfaces queue metadata for each row. -- `Assembly_Editor.jsx` loads and saves assemblies by GUID through the cache APIs, adds import/export tooling, domain selection with read-only warnings, Dev Mode enable/flush controls, and reuses shared badge components for status display. -- App routing (workflows/scripts) and quick job tooling were updated to pass assembly GUID/domain metadata, rely on the import/export endpoints, and hydrate variables from the new Assembly service. - -## 5. Support JSON import/export endpoints -[x] Implement backend utilities to translate between DB model and legacy JSON structure. -[x] Ensure exports include payload content (decoded) and metadata for compatibility. -[x] On import, map JSON back into DB schema, creating payload GUIDs and queuing writes. -[x] Unit-test round-trip import/export for scripts and workflows with attached files. -### Details -``` -1. Add `serialization.py` under the assemblies service package to convert between DB models and legacy JSON. -2. Implement `POST /assemblies/import` and `GET /assemblies/{id}/export` endpoints leveraging the serializer. -3. Validate imported JSON (schema, payload size limits) before staging to cache. -4. Unit-test round-trip import/export for scripts and workflows with attached files. -``` - -**Stage Notes** -- Added `Data/Engine/services/assemblies/serialization.py` to translate AssemblyCache records to legacy JSON and validate imports with a 1 MiB payload ceiling. -- Assembly runtime now exposes `export_assembly` and `import_assembly`, wiring schema validation through the serializer and handling create/update pathways automatically. -- `/api/assemblies/import` and `/api/assemblies//export` routes return legacy documents plus queue state, with audit logging and domain permission checks aligned to Dev Mode rules. - -## 6. Testing and tooling -[x] Unit tests for cache, payload storage, Dev Mode permissions, and import/export. -[x] Integration tests simulating concurrent edits across domains. -[x] CLI or admin script to run official DB sync and verify counts. -[x] Update developer documentation with new workflows. -### Details -``` -1. Under `Data/Engine/tests/assemblies/`, add: - - * Cache tests covering load, dirty flag, flush scheduling. - * Permission tests ensuring Dev Mode restrictions. - * Import/export round-trip tests with payload files. -2. Create an integration test (pytest) mocking concurrent writer/reader scenarios using WAL and ensuring retries succeed. -3. Add a CLI helper (`python Data/Engine/tools/assemblies.py sync-official`) to trigger staging sync manually. -4. Update `Docs/` (e.g., `Docs/assemblies.md`) with: - - * Database layout. - * Dev Mode usage instructions. - * Backup guidance (even if future work, note current expectations). -``` - -**Stage Notes** -- Added `Data/Engine/tests/assemblies/` covering AssemblyCache flushing/locking, PayloadManager lifecycle, Dev Mode permissions, and import/export round-trips. -- Introduced CLI helper `python Data/Engine/tools/assemblies.py sync-official` to rebuild the official database and report source/row counts. -- Authored `Docs/assemblies.md` documenting multi-domain storage, Dev Mode operation, and backup guidance to align Engine contributors. -- Normalised scheduler month/year math to stay in UTC and return catch-up runs so the existing Engine regression suite passes alongside the new tests.