Catalog: residual objects-view members + mod-time staleness follow-ons (index completeness & large-project build done)

[ako]

Summary

The catalog (.mxcli/catalog.db) is a useful name→type/index substrate (it now backs describe type auto-detection, marketplace-style scans, code search, etc.). The original three limitations below have largely been resolved; what remains is a small residual follow-on plus one deliberately-deferred item.

1. The objects view is not a complete index — ✅ DONE

The objects view now indexes every describable top-level document type the catalog builds. Closed out across three increments:

• v3 — associations folded in (ObjectType = ASSOCIATION); describe auto-detect drops its separate associations query.
• v4 — image collections, JavaScript actions, data transformers cataloged (new tables + generic buildSimpleNamedDocs, no CatalogReader/backend change).
• v5 — agent-editor docs cataloged (agent / model / knowledge base / consumed MCP service; Name is a top-level BSON field, so same raw-unit pattern, no backend change). Verified live against test3-app.

Also fixed SHOW CATALOG TABLES, whose hand-maintained list had drifted and never showed the new views — now complete and guarded by TestTables_CoversAllViews (every catalog VIEW must be listed).

1a. Residual: sub-document members (small follow-on, low priority)

A few describable things the issue originally listed are sub-document members, not top-level named documents, so they're intentionally not in objects yet: attributes (Module.Entity.Attribute), module/user roles, navigation, fragments. Folding these in (or exposing a superset view) is worth doing only if a consumer actually needs to resolve them by qualified name — describe handles them today via explicit type / live scan.

2. Staleness signal is mod-time only (deferred — acceptable for now)

isCacheValid invalidates purely on .mpr file mod-time. That catches edits saved to disk, but cannot see a project's unsaved in-memory changes — e.g. when working through the MCP/PED backend against a project open in Studio Pro. The cache can then disagree with the live model.

Deliberately deprioritized: users rarely have hundreds of unsaved documents, and MCP/PED save functionality is expected to land soon (which makes disk mod-time a sufficient signal again). Worth revisiting if that changes — options would be a content hash or backend-liveness awareness. (describe already gates its catalog read on mod-time freshness and falls back to a live scan otherwise.)

3. First build is slow on large projects — ✅ DONE

refresh catalog source was O(N²): it resolved each document by re-reading and re-bson.Unmarshaling every unit on every describe call (#651-scale: ~3.3k microflows / ~33k activities → ~6 hours). Now O(1) via a one-time $Type + qualified-name → unit name index (decoding only Name), with incremental progress reported every 2s instead of a silent multi-minute parse. Hours → minutes on cloud-portal-scale projects. (The generic 5-minute statement-guard timeout was the separately-fixed #651.)

Remaining scope

• 1a — fold sub-document members (attributes / roles / navigation / fragments) into objects if/when a consumer needs qualified-name resolution for them.
• 2 — revisit the mod-time-only staleness signal if MCP/PED save doesn't land or unsaved-change drift becomes a real problem.

Both are low-priority / conditional; the core ask (a complete unified index + a build that finishes in reasonable time) is delivered.

References

• mdl/catalog/tables.go (objects view), mdl/catalog/catalog.go (Tables()), mdl/executor/cmd_catalog.go (isCacheValid)
• cmd/mxcli/cmd_describe.go (resolveViaCatalog / catalogMatchesProject)
• REFRESH CATALOG SOURCE leading to timeout (5mins) #651 (catalog build timeout on large projects — fixed)

[github-actions]

Issue Triage

Summary:
The issue requests enhancements to mxcli’s catalog so the objects view becomes a complete index, improves cache‑staleness detection, and adds incremental progress for large‑project builds.

Checklist (feature request):

• ✅ Use case – clearly explains why the feature is needed (unified index, reliable cache, better UX on large projects).
• ⚠️ Proposed syntax or behavior – describes the desired behavior (fold missing types into objects, use content‑hash or backend‑liveness for staleness, show incremental progress) but does not give concrete CLI flags, config changes, or API details.
• ❌ Mendix version relevance – not mentioned; indicate whether the changes apply to all Mendix versions or are version‑specific.

Next steps:
Could you let us know if these catalog improvements are intended to work across all Mendix versions (e.g., 9.x – 10.x) or if they depend on a particular version? Also, if you have any specific ideas for how the changes should be exposed (e.g., new flags, config options, or adjustments to existing commands), please share them so we can refine the proposal. Thank you!

---

Automated triage via OpenRouter — workflow source

[github-actions]

Enhancement Implementation Guide

We are given an enhancement request about the catalog (.mxcli/catalog.db) and its limitations.

The request has three parts, but the primary focus (priority 1) is to make the objects view a complete index by including missing document types.

The missing document types listed are:

• Associations
• Image collections, JavaScript actions, data transformers
• Agent-editor docs (agent / model / knowledge base / consumed MCP service)
• Navigation, module/user roles, fragments, attributes

We are to focus on the first part (making the objects view complete) as the highest priority.

We are to produce an implementation guide that includes:

1. Completeness Assessment
2. Overlap Check
3. Proposed MDL Syntax
4. Implementation Roadmap
5. BSON Investigation
6. Testing Strategy
7. Gotchas & Risks

Let's break it down.

1. Completeness Assessment

The issue is clear: the objects view in the catalog is missing several document types. We need to know:

• Which Mendix version? (But since we are adding to the catalog view, it's likely version-agnostic as long as the document types exist in the version we are targeting. However, note that some types like agent-editor docs require Mendix 11.9+ and the AgentEditorCommons module.)
• We don't need an example of desired syntax because we are modifying an existing view (the objects view) and not adding a new MDL statement. However, we must ensure that the view includes the missing types.

But note: the issue says that the objects view is defined in mdl/catalog/tables.go. We are to extend that view to include the missing types.

So, the issue is clear enough. We don't need to ask for more details.

2. Overlap Check

Can this be done with existing syntax?

• The objects view is a SQLite view that unions several tables. Currently, it does not include the missing types.
• We can already query the individual tables (like associations, image_collections, etc.) but the request is to have a unified view.

Therefore, there is no existing way to get a unified index without querying multiple tables or views. So, we must extend the objects view.

3. Proposed MDL Syntax

We are not adding a new MDL statement, but modifying an existing view. However, note that the objects view is used by the describe command for auto-detection and by other catalog consumers.

We don't need to change the MDL syntax for users. Instead, we are changing the internal catalog view.

But note: the issue says that the objects view is used by describe for auto-detection. By extending the view, we make describe auto-detection work for the missing types without needing to query multiple tables.

Therefore, we don't propose a new MDL syntax. We propose to change the definition of the objects view in mdl/catalog/tables.go.

However, for completeness, we note that the objects view is not directly exposed in MDL syntax. It is an internal view. But we can query it via SQL:

select * from CATALOG.objects;

So, we are not changing the MDL language, but we are changing the catalog schema.

4. Implementation Roadmap

We need to modify the objects view in mdl/catalog/tables.go to include the missing types.

Steps:

1. Identify the missing types and their corresponding tables in the catalog.

   • From the issue:
     • Associations: already have an associations table.
     • Image collections: we have a table for image collections? (We need to check the catalog schema.)
     • JavaScript actions: likely a table for JavaScript actions.
     • Data transformers: table for data transformers.
     • Agent-editor docs: agent, model, knowledge base, consumed MCP service -> we have tables for these? (We added support for agent documents in a previous feature, so we should have tables for them.)
     • Navigation: we have a table for navigation? (We have navigation documents, so likely a table.)
     • Module/user roles: we have tables for module roles and user roles.
     • Fragments: we have a table for fragments? (Fragments are a type of page, so they might be in the pages table? But note: the issue says fragments are missing. We must check.)
     • Attributes: we have a table for attributes? (Attributes are part of entities, so they might be in the entity_attributes table? But note: the issue says attributes are missing. We must check.)

   However, note that the objects view currently unions:
   modules, entities, microflows, nanoflows, pages, snippets, layouts, constants, java actions, enumerations, workflows, odata client/service, rest client, published rest service, business event service, database connection, external entity, json structure, import/export mapping, contract entity/action/message, jar dependency, business event, external action.

   We need to add the missing ones.

   Let's map the missing types to their catalog tables (if they exist) or we may need to create new tables? But note: the catalog is built by scanning the project and extracting metadata. We already have reflection data for these types? We must check.

   Actually, the catalog is built by the refresh catalog command, which uses the MDL executor to read the project and insert into the catalog tables. We already have support for these types in the MDL executor? (We added agent documents, image collections, etc. in previous features.)

   Therefore, we assume that the catalog already has tables for these missing types. We just need to union them into the objects view.

   We must check the current catalog schema to see what tables exist.

   But note: the issue says that associations are in their own associations table. So we can assume that the other missing types also have tables.

   We can look in mdl/catalog/tables.go for the existing table definitions and the objects view.

   We are going to:
   - For each missing type, if there is a corresponding table in the catalog, we add a SELECT statement to the objects view union.

   If a table does not exist, we must first create the table and populate it during catalog refresh. But note: the issue says that the catalog already backs describe type auto-detection, etc., and that the missing types are describable. So we assume that the tables exist because we can describe them.

   However, to be safe, we should verify that the catalog has tables for:
   - image_collections
   - javascript_actions
   - data_transformers
   - agents, models, knowledge_bases, consumed_mcp_services (for agent-editor docs)
   - navigation
   - module_roles, user_roles
   - fragments
   - attributes

   But note: attributes are not top-level documents? They are part of entities. However, the issue says that attributes are missing from the objects view. We must check if we have an attributes table in the catalog.

   Looking at the current objects view in mdl/catalog/tables.go (we don't have the code, but we can reason): it includes entities, but not attributes. Similarly, it includes modules but not module roles.

   We must check the existing catalog tables. Since we don't have the code, we will assume that the tables for these missing types exist because we have MDL support for them (we can describe them). Therefore, we can add them to the view.

   However, note: the issue says that the objects view is missing:
   - Associations (we have an associations table, so we just need to add it)
   - Image collections, JavaScript actions, data transformers
   - Agent-editor docs (agent, model, knowledge base, consumed MCP service)
   - Navigation, module/user roles, fragments, attributes

   We will add SELECT statements for each of these.

   But note: the objects view currently selects:
   - id, qualified_name, type_name, documentation, module_name, etc.

   We must ensure that the SELECT for each new table has the same columns.

   We can look at the existing SELECTs in the view to see the pattern.

   Example from the existing view (for modules):
   SELECT
   m.ID AS id,
   m.QualifiedName AS qualified_name,
   'module' AS type_name,
   m.Documentation AS documentation,
   m.ModuleName AS module_name,
   ...
   FROM modules m

   We must do the same for each missing table.

   Steps in mdl/catalog/tables.go:
   - Find the objects view definition (a string constant).
   - For each missing type, add a UNION ALL SELECT ... FROM

   ...
   - Make sure the column list and aliases match.

   We must also check that the tables we are unioning have the required columns (id, qualified_name, etc.). If not, we may need to adjust the table schema or the view.

   However, note that the catalog tables are defined in the same file. We can check the table definitions for the missing types.

   If a table does not have a qualified_name column, we must add it? But note: the catalog is built from the project, and we know that every document has a qualified name. So we assume that the tables have the necessary columns.

   We will proceed to add the missing types to the view.

2. BSON Investigation
   We are not changing the BSON structure of the documents. We are only changing the catalog view, which is built from the existing catalog tables. The catalog tables are populated by the refresh catalog command, which reads the project and inserts into the catalog tables.

   Therefore, we do not need to investigate BSON for this change. However, we must ensure that the catalog tables for the missing types are being populated correctly. But that is out of scope for this issue (it assumes that the catalog already has the data for these types, because we can describe them).

   However, to be safe, we should verify that the catalog tables for the missing types exist and are populated. We can do that by:
   - Creating a test project that has one of each missing type.
   - Running refresh catalog and then querying the catalog tables.

   But note: the issue says that the catalog already backs describe type auto-detection, and that the missing types are describable. So we assume the tables are there and populated.

   Therefore, for the purpose of this issue, we do not need to investigate BSON.

3. Testing Strategy
   We must test that the objects view now includes the missing types.

   We can do:
   - Unit test: in mdl/catalog/tables_test.go, we can test the view definition? Or we can test that the view returns the expected rows for a known project.

   However, note that the catalog is built from a project. We can create a test project that has one of each missing type and then:
   - Refresh the catalog.
   - Query the objects view and check that it returns rows for each missing type.

   We can also check that the row counts match the sum of the individual tables.

   We should also test that the view does not break existing functionality (i.e., the existing types are still there).

   We can add a test in mdl/catalog/tables_test.go that:
   - Creates an in-memory SQLite database.
   - Creates the catalog schema (by calling the function that creates the tables and views).
   - Inserts known data into each table (including the missing ones).
   - Queries the objects view and verifies that it returns the union of all tables.

   We must also update any existing tests that might be affected by the change in the view.

   Additionally, we should test that the describe command's auto-detection works for the missing types (since it uses the objects view for auto-detection). But note: the describe command also has fallbacks. We can test that when we run describe <qualified_name> for a missing type, it works without needing to specify the type.

   However, note: the issue says that describe auto-detect works around the limitation by also querying the associations table and falling back to a live scan. So we want to remove the need for that fallback for the missing types.

   We can test:
   - Create a project with a missing type (e.g., an image collection).
   - Run describe <image_collection_name> without specifying the type and see if it auto-detects.

   But note: the describe command first tries to auto-detect the type by querying the objects view. If it finds exactly one match, it uses that. If it finds multiple, it asks for clarification.

   We can test that for each missing type, the objects view returns exactly one row when qualified by the qualified name.

   We can do this in an integration test.

4. Gotchas & Risks

   • The objects view must have the same columns for every unioned SELECT. We must ensure that the SELECT for each missing table returns the same number and types of columns, and that the aliases match.
   • If a table does not have a column that the view expects (e.g., documentation), we must use a default (like NULL) or adjust the view to use a different column or compute it.
   • We must be cautious about performance: the view might become slower if we add many tables. But the number of missing types is fixed and not huge.
   • We must ensure that the catalog tables for the missing types are indeed being populated. If they are not, then the view will be missing data. But note: the issue states that these types are describable, so we assume the catalog tables are populated.

   However, note: the catalog is built by the refresh catalog command. We must make sure that the refresh catalog command does populate the tables for the missing types. This is outside the scope of this change, but we should verify that the existing refresh catalog command does insert into these tables.

   Since we have MDL support for these types (we can create and describe them), the refresh catalog command (which uses the MDL executor to read the project) should already be inserting into the catalog tables for these types.

   Therefore, the risk is low.

   Another risk: the objects view might exceed some SQLite limit (like the maximum number of UNION ALLs). But we are only adding about 10-15 more, and the current view already has many. We should check the current count and make sure we don't hit a limit. However, SQLite can handle hundreds of UNION ALLs.

   We will proceed.

Implementation Steps in Detail

1. Open mdl/catalog/tables.go.

2. Find the constant that defines the objects view (look for objectsView or similar).

3. For each missing type, add a UNION ALL SELECT ... clause.

   • We must match the column list of the existing SELECTs.

   Example of the existing SELECT for modules (we assume):

    SELECT
        m.ID AS id,
        m.QualifiedName AS qualified_name,
        'module' AS type_name,
        m.Documentation AS documentation,
        m.ModuleName AS module_name,
        ... 
    FROM modules m
   

   We will do the same for, say, the associations table:

    UNION ALL
    SELECT
        a.ID AS id,
        a.QualifiedName AS qualified_name,
        'association' AS type_name,
        a.Documentation AS documentation,
        a.ModuleName AS module_name,
        ... 
    FROM associations a
   

   We must check the actual column names in the table. We can look at the table definition for associations in the same file.

4. We must do this for:
   - associations
   - image_collections
   - javascript_actions
   - data_transformers
   - agents
   - models
   - knowledge_bases
   - consumed_mcp_services
   - navigation
   - module_roles
   - user_roles
   - fragments
   - attributes [if we have an attributes table]

   But note: do we have an attributes table? Attributes are not top-level documents. They are children of entities. However, the issue says they are missing. We must check.

   Alternatively, note that the issue says: "attributes" are missing. But in the catalog, we might not have a top-level table for attributes. Instead, attributes are stored in the entity_attributes table? And the objects view currently does not include attributes because they are not top-level.

   However, the issue says that the objects view is meant to be "everything with a QualifiedName". Do attributes have a qualified name? Yes, an attribute has a qualified name like Module.Entity.Attribute.

   Therefore, we must have a way to get attributes in the catalog. We might have an entity_attributes table that includes the qualified name of the attribute.

   We must check the catalog schema for an attributes table.

   If we don't have one, we may need to create it. But note: the issue says that the catalog already backs describe type auto-detection, and that describe can describe attributes. So we must have a way to get attributes.

   Let's assume we have an entity_attributes table (or similar) that stores attributes and has a qualified name.

   We will add it if it exists.

   If we don't, then we must create the table and populate it during catalog refresh. But note: the issue is about making the objects view complete, and if the data isn't in the catalog, we cannot put it in the view. So we must first ensure that the catalog has the data.

   However, the issue states that the missing types are describable, so we assume that the catalog already has tables for them (because we can describe them). Therefore, we will assume that there is a table for attributes (or that attributes are stored in a way that we can query them with a qualified name).

   We will check the existing catalog tables in mdl/catalog/tables.go to see what is available.

   Since we don't have the file, we will list the missing types as per the issue and hope that the tables exist.

   If we find during implementation that a table does not exist, we will need to create it and update the catalog refresh to populate it.

---

Automated guide via OpenRouter — workflow source

[ako]

Progress — item 1 (objects-view completeness), partial.

Associations are now in the objects view (ObjectType = ASSOCIATION); catalog schema bumped to v3 so cached files rebuild on next refresh catalog. describe auto-detect drops its separate associations query and resolves them through the single objects lookup (cross-module association ~3.7s via catalog vs ~8.5s via the live-scan fallback). Test added (TestObjectsView_IncludesAssociations).

So objects is now complete for every type that is cataloged. The remaining omissions — image collections, JavaScript actions, data transformers, agent-editor docs (agent/model/knowledge base/consumed MCP service) — have no catalog table at all, so finishing the index for them is bigger: the builder has to catalog those types first (new <name>_data tables + builder population), then they can be unioned into objects. That builder work is the rest of item 1.

[ako]

Progress — item 1, three more types cataloged (schema v4).

Image collections, JavaScript actions, and data transformers now have catalog tables and are unioned into objects (IMAGE_COLLECTION, JAVASCRIPT_ACTION, DATA_TRANSFORMER). A single generic builder (buildSimpleNamedDocs) populates them from the raw-unit surface (ListRawUnitsByType + Name-only decode), so no CatalogReader/backend change was needed — staying clear of the in-flight backend rewrite on the modelsdk branch. describe auto-detect now resolves image collections and data transformers by bare name.

So objects is complete for all the regular named-unit document types. Remaining: the agent-editor docs (agent / model / knowledge base / consumed MCP service). Those are CustomBlobDocuments with the name inside a JSON blob (discriminated by CustomDocumentType), so they need blob parsing in the builder rather than a top-level Name decode — a separate, slightly bigger follow-up (and there's no agent-editor fixture in the test projects to verify against yet).

[ako]

Progress — item 1 complete (agent-editor docs cataloged, schema v5).

Agents, AI models, knowledge bases, and consumed MCP services are now cataloged (agents / ai_models / knowledge_bases / consumed_mcp_services tables) and unioned into objects (AGENT, AI_MODEL, KNOWLEDGE_BASE, CONSUMED_MCP_SERVICE). describe auto-detect resolves all four by bare name. Catalog schema bumped to v5.

Correction to my previous note: these CustomBlobDocuments turned out to carry the document Name as a top-level field of the BSON wrapper (only the type-specific config lives in the inner JSON blob, which the catalog doesn't need). So no blob parsing was required — they slot into the same raw-unit pattern as the v4 types (ListRawUnitsByType + a small struct decode, routed by the CustomDocumentType discriminator), again with no CatalogReader/backend change.

Also found a fixture after all — test3-app has a real agent stack — so this is verified live (8 agent-editor docs across all four types; describe AgentEditorCommons.TranslationAgent round-trips), not just unit-tested.

Item 1 is now done. objects is a complete index for every describable document type that the catalog builds. Status of the three sub-items:

• 1. objects-view completeness — ✅ complete (associations v3 → image collections/JS actions/data transformers v4 → agent-editor v5). The remaining gaps the issue listed — attributes, module/user roles, navigation, fragments — are sub-document members rather than top-level named documents; folding those in would be a separate, lower-value follow-up if a consumer ever needs them.
• 2. staleness signal (mod-time only) — unchanged, still deliberately deprioritized pending MCP/PED save.
• 3. slow first build on large projects — addressed separately: refresh catalog source was O(N²) (re-scanning every unit per describe); now O(1) via a one-time name index, with incremental progress. (See REFRESH CATALOG SOURCE leading to timeout (5mins) #651-scale projects: hours → minutes.)

Side fix while here: SHOW CATALOG TABLES had drifted — its hand-maintained table list never included the new v4/v5 views (nor the pre-existing navigation_profiles view), so the views were built and queryable but invisible to the command. Now listed, plus a drift-guard test (TestTables_CoversAllViews) asserting every catalog VIEW appears in the list, so a future document type can't be silently omitted again.

Given item 1 is complete and 2/3 are tracked/handled, this issue can probably be narrowed to just the residual sub-document members (attributes/roles/navigation/fragments) or closed — your call.