Version 3.7.0 and duplicated keys in a YAML file

This issue has been tracked since 2021-04-08.

In contrast to Version 3.6.1,
in Version 3.7.0 on both Linux and Windows, when I encrypt a YAML file that accidentally has a duplicated key,
I cannot decrypt it afterwards.

Reproduce with sops test.yaml , duplicate the first line, and save.
Then try to decrypt, and get:

Error unmarshalling input yaml: yaml: unmarshal errors:
  line 2: mapping key "hello" already defined at line 1

The only way I found to recover the file would be to delete the offending key from the encrypted file
hello: ENC[AES256_GCM,data:nvp56OqVzo33WWLSyJubbsx4peUHNd/VQ3N4KXq2DRs4o5pf/3V3Hj8vg7DQUg==,iv:23E6rL8i155Ne1/SeGvsuc5OHku09sK9mhJv44Ky0gI=,tag:T0Dv5t+h3FMDYP7d2x01uw==,type:str]
and decrypt with sops --ignore-mac test.yaml

felixfontein wrote this answer on 2021-04-09

3.7.0 uses a new YAML parser (see the changelog), that's why the behavior is different from 3.6.x or earlier.

I'm a bit surprised that it is possible to encrypt a file with duplicate keys. I would expect it to fail with the same error.

felixfontein wrote this answer on 2021-05-08

I've looked at this a bit. There are two parts:

  1. The YAML to sops.TreeBranches and back conversion code does not mind duplicate keys.
  2. The code which loads the sops metadata (LoadEncryptedFile() in stores/yaml/store.go) uses yaml.Unmarshal(in, &metadataHolder) to parse the metadata, and that produces the error.

In other words: when using yaml.v3 to deserialize (or serialize from) yaml.Node, it does not care about duplicate keys. But when using yaml.v3 to deserialize into Go structs, it does mind.

I'm not sure what's the best way to proceed here. We can add code to prevent to parse YAML files with duplicate keys in all cases (but we'd have to do that manually), or we can try to work around the issue with yaml.v3's Go struct deserialization to be able to handle YAML files with duplicate keys in all cases.

@autrilla @ajvb what do you think should the behavior be?

jaythamke wrote this answer on 2022-02-28

I also found this error message in version 3.7.1. I expect sops editor should fail the file encryption at the first step.

Bdw, is there a new release/patch with fix? Thanks

More Details About Repo
Owner Name mozilla
Repo Name sops
Full Name mozilla/sops
Language Go
Created Date 2015-08-13
Updated Date 2022-12-01
Star Count 11439
Watcher Count 109
Fork Count 667
Issue Count 334

YOU MAY BE INTERESTED

Issue Title Created Date Updated Date