Support new versioning scheme#137
Conversation
…y file. Update GLOWS L3b to handle new major.minor version format.
…s minor version only for compatibility
# Conflicts: # tests/periodically_run_tests.json
pleasant-menlo
changed the title
|
@bryan-harter @tech3371 @vineetbansal , can one of you take a look at this? |
…me to use VersionMap with a version per output descriptor.
…me to use VersionMap for GLOWS processing L3b, L3c, and L3d
|
Note we probably shouldn't merge this until we have all the metadata updates for release 1 done, so we can still build new containers against those. |
…s per product descriptor for GLOWS L3bcde
# Conflicts: # pyproject.toml # tests/glows/l3e/test_glows_l3e_utils.py # tests/periodically_run_tests.json # uv.lock
| "astropy_healpix==1.1.2", | ||
| "bitstring==4.3.0", | ||
| "imap-data-access==0.42.0.dev0", | ||
| "imap-data-access>=0.42.0.dev0", |
There was a problem hiding this comment.
0.42.0.dev1 is breaking backwards compatibility for versions. We could leave this at ==0.42.0.dev0 for the time being until 0.42.0.dev2 is released.
There was a problem hiding this comment.
I'm pushing to get 0.42.0 final released. The Great Renaming happened over the weekend (thanks Bryan), so is backward compatibility still a concern?
| l3d_initializer_result = GlowsL3DInitializer.should_process_l3d(l3bc_initializer_data.external_dependencies, | ||
| l3bs, l3cs) | ||
| l3bs, l3cs, l3d_major_version) | ||
| if l3d_initializer_result is None: |
There was a problem hiding this comment.
Can this early return just go away? If L3e needs to be regenerated due to a new major version, that should not be gated behind updates or a new major version of L3d
| if not latest_l3d_cr: | ||
| first_updated_cr = find_first_updated_cr(l3d_output.l3d_cdf_file_path, previous_l3d) | ||
| if first_updated_cr is None: | ||
| return None |
There was a problem hiding this comment.
This early return prevents us from generating any L3e if L3d has not changed. We may still want to process if the L3e major version has changed
…are no changes between the output L3d and the existing L3d
…query result to accommodate major_version and minor_version keys.
| if previous_l3d is not None: | ||
| first_updated_cr = find_first_updated_cr(l3d_output.l3d_cdf_file_path, previous_l3d) | ||
| if first_updated_cr is None: | ||
| first_updated_cr = first_cr |
There was a problem hiding this comment.
This will cause all L3e to be reprocessed when there is no difference in L3d. Could first_updated_cr be None or something here so no L3e processing is triggered based on L3d updates, only L3e major version updates.
2 participants
Change Summary
Overview
Implemented handling of major version in IMAP L3 processing including the jobs that manage their own version numbers (cron jobs for GLOWS and maps).
The dependency JSON can now contain additional information about the requested version:
The parallel change to imap_processing was made in IMAP-Science-Operations-Center/imap_processing#3313.
See IMAP-Science-Operations-Center/sds-data-manager#1397 for the top-level issue.
Closes #118.
File changes
Version objects are now created from the dependency JSON when they contain version numbers and otherwise fall back to creating version objects from the command line arguments.
Testing
Invoking any job from the command line using a dependency JSON that includes the version numbers should result in an output with the corresponding major and minor version numbers.