Investigate paginated prefetch for repository pipelines and common API path support

[MariusStorhaug]

Summary

Investigate prefetching paginated GitHub API/GraphQL results so pipelines like:

Get-GitHubRepository | Remove-GitHubRepository

can continue collecting future pages while downstream commands process already-emitted objects.

The immediate motivating case is repository deletion, where current behavior is effectively:

• fetch 100 repos
• process/delete those 100
• fetch the next 100
• repeat

The goal is to improve throughput while keeping the authoring/caller experience clean.

Current findings

PowerShell pipeline behavior

Plain PowerShell pipeline behavior is backpressured and synchronous at the emission boundary:

• producer emits one object
• downstream consumer processes that object
• producer resumes only after downstream processing returns

So plain functions alone do not overlap page fetching with downstream processing.

Working prototype direction

A working prototype exists using:

• a foreground pipeline emitter
• a background runspace producer
• System.Collections.Concurrent.BlockingCollection[object]

That pattern works cross-platform on PowerShell 7 (macOS/Windows/Linux) and does not require Add-Type.

Repo/worktree state

Created worktree/branch for source changes:

• Branch: prefetch-repository-pipeline
• Worktree: /Users/AD08640/Repos/github.com/PSModule/GitHub/prefetch-repository-pipeline
• Base commit: 1c76a38

Current source changes in that worktree:

 M src/functions/private/Repositories/Get-GitHubMyRepositories.ps1
 M src/functions/private/Repositories/Get-GitHubRepositoryListByOwner.ps1
 M src/functions/public/API/Invoke-GitHubGraphQLQuery.ps1
?? src/functions/private/Core/Invoke-GitHubGraphQLConnectionPrefetch.ps1

Current diff summary:

 src/functions/private/Repositories/Get-GitHubMyRepositories.ps1        | 37 +++++++++++++++----------------------
 src/functions/private/Repositories/Get-GitHubRepositoryListByOwner.ps1 | 41 +++++++++++++++++------------------------
 src/functions/public/API/Invoke-GitHubGraphQLQuery.ps1                 | 13 +++++++++++++
 3 files changed, 45 insertions(+), 46 deletions(-)

Note: the new private helper file is currently untracked and not reflected in the diff stat above.

Design conclusion so far

For the biggest impact across all call types, the best long-term shape is:

1. Invoke-GitHubAPI

   • Own the generic transport/pagination/prefetch primitive
   • This is the highest-leverage location because it sits under both REST and GraphQL

2. Invoke-GitHubGraphQLQuery

   • Own GraphQL-specific connection semantics (nodes, pageInfo, endCursor, hasNextPage)
   • Avoid leaking GraphQL-specific concepts into the generic transport layer

Practical interpretation:

• Invoke-GitHubAPI should eventually grow a generic page-prefetch capability
• Invoke-GitHubGraphQLQuery should layer a GraphQL connection-streaming mode on top of that

Why not only GraphQL?

Current call-site counts from the source tree:

• GraphQL call sites: 19
• Private REST / Invoke-GitHubAPI call sites: 114

So changing only GraphQL helps some list functions, but the broadest architectural payoff is still at Invoke-GitHubAPI.

What currently works

An installed-module experiment was done directly in the locally installed module copy at:

/Users/AD08640/.local/share/powershell/Modules/GitHub/0.43.1/GitHub.psm1

That experiment successfully made this safe validation path work:

Get-GitHubRepository -Context 'psmodule-user' -PerPage 2 -Verbose |
    Select-Object -First 6 |
    Remove-GitHubRepository -Context 'psmodule-user' -WhatIf -Verbose

That proves the general prefetch idea is viable for the motivating repo-list pipeline.

However, that installed-module edit is only a local experiment and should not be treated as the maintainable implementation.

Source-side prototype in worktree

A cleaner source-side prototype was started in the prefetch-repository-pipeline worktree:

• Add opt-in connection-prefetch behavior to Invoke-GitHubGraphQLQuery
• Add private helper Invoke-GitHubGraphQLConnectionPrefetch
• Convert the two repository list functions to use the common GraphQL path instead of bespoke per-function pagination loops

This keeps caller code clean, for example:

Invoke-GitHubGraphQLQuery -Query $query -Variables $variables -ConnectionPath @('viewer', 'repositories')

That prototype parses cleanly at the source-file level, but it has not yet been taken through a full module build/import/test path from source.

Scope answer from analysis

Does the current prototype cover all GraphQL functions?

No.

Current source-side prototype only targets the two repository list paths used by Get-GitHubRepository list scenarios.

Does it cover REST API paths?

No.

REST still uses the existing Invoke-GitHubAPI behavior.

Is it a bigger change to make this work for all functions?

Yes.

• Broader GraphQL rollout is moderate: mostly convert paginated GraphQL connection queries to use the common connection-streaming mode.
• Broader REST rollout is a larger architectural change: Invoke-GitHubAPI would need a generic, backwards-compatible prefetch/page-stream model.

Recommended next step

Resume from the source worktree and do this in order:

1. Finish the source-side GraphQL common-path prototype in prefetch-repository-pipeline
2. Validate it through the repo's actual build/import/test path
3. If successful, design a generic page-prefetch primitive in Invoke-GitHubAPI
4. Keep GraphQL connection interpretation in Invoke-GitHubGraphQLQuery
5. Roll out gradually to other paginated GraphQL list functions before attempting REST

Open questions

• What is the preferred backwards-compatible API surface for generic prefetching in Invoke-GitHubAPI?
• Should prefetching be opt-in everywhere, or enabled only for known paginated list paths?
• What is the repo's preferred build/test path for validating a source-side module refactor like this?
• Are there pagination-sensitive commands where mutating the same collection being paged should remain explicitly non-prefetched?