BUG: "numpy.int64' object has no attribute 'lower'" Error

[lacoak21]

During the version renaming migration, some files were not copied over to the new version schema vMMM.mmmm. @tech3371 and @vineetbansal and I looked into it and confirmed that these files had problems with the metadata that caused problems when using write_cdf to write the file back to s3 with the updated metadata 'Data_version' attribute.

This was confirmed by running a quick script using a CDF file that didnt get renamed.

`
from imap_processing.cdf.utils import load_cdf, write_cdf

bad_file = "data/imap/hit/l1a/2026/02/imap_hit_l1a_hk_20260226_v005.cdf"
ds = load_cdf(bad_file)
write_cdf(ds)
`
This errors out with: numpy.int64' object has no attribute 'lower'

The solution was to copy them to the new names without editing the CDFs themselves. This means that these files dont have updated data_version attributes.

These may be old CDFs that have since been updated and fixed but we need to confirm this. The L3 files I dont think have been fixed and @pleasant-menlo should take a look at these.

Here are the logs for the dry run and real run. These logs give a record of what files we needed to manually copy.

copy_missing_new_version_files2.py
other_rename_dry_run_after_renamed.txt
other_rename_dry_run.txt
other_rename_real_run.txt

[vineetbansal]

A small script to see where the error is happening is below. I've also posted the errors I see with the latest version of all the libraries - some are with load_cdf, others with write_cdf.

import os
import traceback
from imap_processing.cdf.utils import load_cdf, write_cdf
from imap_data_access import download

imap_data_dir = os.getenv("IMAP_DATA_DIR")

files = (
    # load_cdf => KeyError: 'energy_h'
    "imap_codice_l1b_hi-ialirt_20251008_v003.cdf",

    # write_cdf => AttributeError: 'numpy.float32' object has no attribute 'lower'
    "imap_codice_l2_hi-direct-events_20250814_v003.cdf",

    # load_cdf => Exception: ERROR: Variable "spin_angle_delta" contains the dimensions
    # ('spin_angle', 'spin_angle_dim'). Dimension "spin_angle" is a data variable, but
    # its dimensions must be equal to itself.  Instead, its dimensions are
    # ('spin_angle', 'spin_angle_dim').  This is likely due to issues with "DEPEND_X"
    # attributes in either spin_angle_delta or spin_angle
    "imap_glows_l3a_hist_20260701-repoint00296_v001.cdf"
)

failures = {}

for file in files:
    print(file)
    try:
        file_path = download(file)
        cdf = load_cdf(file_path)
        write_cdf(cdf, istp=True, terminate_on_warning=False)
    except Exception as e:
        failures[file] = e
        print(f"  FAILED: {type(e).__name__}: {e}")
        traceback.print_exc()

if failures:
    print(f"\n{len(failures)} of {len(files)} file(s) failed:")
    for file, e in failures.items():
        print(f"  {file}: {type(e).__name__}: {e}")

[vineetbansal]

The one write_cdf error (AttributeError: 'numpy.float32' object has no attribute 'lower') I'm seeing above (and @lacoak21 saw as well) was fixed in commit #50420.

It seems to me like these kinds of errors are expected if the fixes was put in after detecting the error in the CDF (as they would have to be). Perhaps only running these checks against the release=True column is the way to go..

[pleasant-menlo]

We noticed that not every day of data listed in the dry run log actually got copied. For example, imap_hit_l3_direct-events_20260402 only appears in the dry run log and not in the real run.

Should we simply reprocess the affected l3 data products? Our thought is that the new files will then exist on S3 and have Data_version and Parents using the new version format.