A vulnerability report can flag a library that has already been patched—or miss one that remains vulnerable. Even when a supplier provides an accurate assessment, the tools consuming it may leave the wrong warning in place or remove a warning about software the supplier never assessed*.
Our PyYAML experiment investigates how these errors arise. An SBOM generator identifies the software installed in a container, a scanner matches those components against vulnerability advisories, and a supplier’s VEX statement describes whether a particular product is affected or fixed. For this process to work, all three must refer to the same software. We examine what happens when package identities, ownership relationships, or installation details are lost or interpreted differently as information passes between tools.
1. A small experiment with an independent check
We built seven container images using the same Ubuntu 20.04 base, Python version, and application. Each image contains a different installation of PyYAML: an unpatched Ubuntu package, Ubuntu’s patched package, version 5.3.1 installed from PyPI, Git, or a local source tree, a locally patched 5.3.1 build labelled 5.3.1+acme1, or the fixed upstream release 6.0.2. This setup lets us compare installation methods while including patched and unpatched cases whose vulnerability status we can verify independently.
The target vulnerability, CVE-2020-14343, allows an attacker to execute Python code when an affected application processes attacker-controlled YAML using yaml.full_load. To establish ground truth independently of the scanners, a test script runs a harmless exploit payload inside each container. The payload instructs PyYAML to evaluate 6*7; vulnerable installations return (42,), demonstrating that parsing the YAML caused Python code to execute. Patched installations reject the payload. We support this distinction by tracing the backported fix to Ubuntu’s security patch and comparing SHA-256 hashes of the constructor.py file Python actually loads against known fixed versions. Together, these checks provide stronger evidence than an unsuccessful exploit alone, which could fail for reasons unrelated to a security fix.
We save each container image as an archive and use Syft 1.54.1 and Trivy 0.75.0 to inventory its installed packages, producing SBOMs in CycloneDX format. Each SBOM is then scanned by both Grype 0.120.1 and Trivy, giving four generator–scanner combinations per image. Using vexctl 0.4.4, we create supplier OpenVEX statements that describe whether each shipped package is affected or fixed. We test two ways of applying these statements: the scanner’s --vex option filters findings during the scan, while vexctl filter processes the results afterwards, using a SARIF report; the standard format for exchanging analysis results. This lets us compare whether both routes remove the intended findings while retaining unrelated ones.
That gives 28 combinations per VEX mode, or 84 verdicts. Tools and the base image are pinned; vulnerability databases are frozen.
Syft identifies the installed library twice: once from the Debian package records and once from the Python packaging metadata inside that package. The Python entry receives the Package URL (PURL) pkg:pypi/pyyaml@5.3.1. Because that metadata retains the upstream version, the scanner matches it against the PyPI advisory’s affected range, <5.4, and reports a vulnerability even though Ubuntu has patched the installed code.
Syft detects that the Python metadata files also appear in the Debian package’s installed-file list. It therefore links the Python entry to the Debian entry through an ownership-by-file-overlap relationship, indicating that the entries describe overlapping parts of the same installation.
Although the main experiment uses CycloneDX, a follow-up probe compared Syft’s output formats for the same image (see §4). Syft’s native JSON preserves this ownership link explicitly, while its SPDX output preserves it as an OTHER relationship with an explanatory comment. Grype recognises both representations and excludes the duplicate Python entry from vulnerability matching. Syft’s CycloneDX output omits the link, so Grype treats the Python entry independently and reports the false positive. This result reflects what Syft exports and what Grype understands; it does not establish that CycloneDX itself cannot represent ownership. A similar loss of ownership information was reported upstream in May 2026.
2. Compatibility failures can hide each other
Ubuntu’s security advisories identify the affected source package as pyyaml, but the package installed in the container is named python3-yaml. A scanner must connect these two names to find the relevant advisory. Both SBOM generators record that connection, but in different fields: Grype understands Syft’s representation, while Trivy understands its own. In the tested configurations, neither scanner uses the other generator’s representation.
As a result, Grype fails to report the vulnerable Debian package when reading Trivy’s SBOM. Trivy also fails to match the Debian package when reading Syft’s SBOM, yet still reports the vulnerability because Syft includes the separate Python entry described above. The final warning is correct, but it comes from the duplicate entry and conceals the failed Debian-package match. Checking only whether a report contains the CVE would miss this underlying compatibility problem.
3. VEX needs the right identity and the right scope
When VEX is applied inside the scanner using --vex, all four generator–scanner combinations correctly suppress the local backport’s false positive. The version label 5.3.1+acme1 distinguishes the patched rebuild from the vulnerable upstream release, allowing the supplier’s statement to name the build it assessed.
Successful matching also depends on how the vulnerability and package are identified. Grype reports this Python vulnerability under its GitHub Security Advisory (GHSA) identifier, while Trivy uses its CVE identifier, so the statement needs both to work with both scanners. Additional PURL attributes, called qualifiers, can prevent matching if they appear in the statement but are absent from the SBOM entry. Installation origins can also become indistinguishable: PyPI and local installations share a PURL, and Trivy gives the Git installation that same identity. Syft preserves more origin information, but retaining that evidence does not ensure scanners use it when applying VEX.
Applying VEX after scanning produces different results. In vexctl 0.4.4, the SARIF filter checks the vulnerability identifier but ignores the product named in the statement. Consequently, a statement marking the same CVE as fixed in an unrelated npm package removes a genuine PyYAML finding from Trivy’s report. This behavior was reported in 2023. Across the experiment, post-filtering leaves four incorrect verdicts, compared with three when VEX is applied inside the scanners.
Scope matters when interpreting these results. Our statements identify a package as the assessed product, so they can apply wherever that package identity appears—not just within one container image. Grype already documents image-scoped statements that identify the container by its digest and the package as a subcomponent. We have not tested that approach here, so the cross-image suppression probes do not demonstrate a fundamental limitation of VEX.
4. The research question beyond this case
A complementary question is whether an SBOM describes what a project declares or what its built image actually contains. In our follow-up probe, source-directory scans turn stale dependency pins into false positives, and both generators omit the tested Git requirement. These results show why the two views need to be distinguished: a manifest records intended dependencies, while installed-package evidence describes what reached the image.
This investigation builds on existing work. Benedetti et al. examine how Python SBOM generation affects vulnerability detection. The SBOMproof preprint studies cross-tool interoperability, while the component-variants preprint examines component identity and VEX handling.
A research direction is to establish which information must survive installation, SBOM export, and downstream consumption for vulnerability decisions to remain correct. That includes identifying the exact build, retaining package ownership relationships, recording evidence of applied patches, and preserving the scope of the supplier’s assessment. A broader benchmark could test these requirements across ecosystems and real backports, evaluate image-scoped VEX, and measure whether preserving this information reduces both missed vulnerabilities and suppression of findings outside the assessed product.
Takeaway. For current pipelines, we recommend validating the exact combination of generator, SBOM format, scanner, and VEX application method against known patched and vulnerable artifacts. Include negative controls: a statement about an unrelated product should leave the target’s findings unchanged. Retain suppressed findings and the reasons for their suppression so that reviewers can distinguish an assessed vulnerability from a missed match. A report with no warnings should come with evidence explaining why.
>> Part of the Agentic AI Security research project.
*These findings come from a controlled study of one vulnerability in PyYAML, using Ubuntu 20.04 and the tool versions listed above. They demonstrate failures in the tested configurations; they do not establish how common those failures are across other packages, distributions, or tool versions. The code, recorded evidence, and reproduction instructions are available in the repository.