Compromised PyTorch Dependency Chain

Compromised PyTorch-nightly Dependency Chain: A Supply Chain Attack via Dependency Confusion

Between December 25 and December 30, 2022, Linux users installing PyTorch’s pre-release version, PyTorch-nightly, through pip were unknowingly exposed to a supply chain attack. The attack exploited a technique known as dependency confusion, where a malicious package with the same name as a legitimate PyTorch dependency was uploaded to the Python Package Index (PyPI). Due to pip’s default behavior of prioritizing PyPI over PyTorch’s official repository, the malicious package—named torchtriton—was installed instead of the intended dependency. This incident affected only users of PyTorch-nightly on Linux systems, while stable PyTorch packages remained unaffected.

The malicious torchtriton package contained a compromised binary that executed when the triton module was imported. This required explicit user action, meaning the attack did not trigger automatically during standard PyTorch operations. However, once imported, the binary exfiltrated sensitive system information, including the first 1,000 files in the user’s home directory (limited to files under 99,999 bytes) and all files under 99,999 bytes in the .ssh directory. The collected data was uploaded via encrypted DNS queries to the domain h4ck.cfd using the DNS server wheezy.io. The binary’s SHA256 hash was identified as 2385b29489cd9e35f92c072780f903ae2e517ed422eae67246ae50a5cc738a0e.

The attack was not an isolated incident but part of a broader trend in dependency confusion attacks. The individual responsible for uploading the malicious package later claimed it was intended as ethical research, though the collected data included sensitive files such as SSH keys, gitconfig, hosts, passwd, and terminal command histories. The domain h4ck.cfd was registered on December 21, 2022, just days before the attack.

PyTorch responded swiftly upon discovering the compromise on December 30, 2022. The organization announced the incident and advised users to uninstall PyTorch-nightly and the malicious torchtriton dependency immediately. PyTorch also removed all nightly packages dependent on torchtriton from its official repository and renamed the dependency to pytorch-triton to mitigate future risks. Additionally, a dummy package was registered on PyPI to prevent further confusion attacks targeting the same package name.

The incident highlighted critical vulnerabilities in Python’s package management ecosystem. Pip’s default behavior of prioritizing PyPI over private or official repositories creates an inherent risk for dependency confusion attacks. While such attacks require precise timing and knowledge of target dependencies, they can have severe consequences if exploited. The case underscores the need for improved package verification mechanisms, such as digital signatures or cryptographic hashes, to ensure the integrity of installed dependencies. PyTorch’s response serves as a model for transparency and rapid mitigation in the face of supply chain compromises.

Users can verify whether their system was affected by running the following Python command, which checks for the presence of the malicious binary in the triton module:

python3 -c "import pathlib;import importlib.util;s=importlib.util.find_spec('triton');affected=any(x.name == 'triton' for x in (pathlib.Path(s.submodule_search_locations[0] if s is not None else '/' ) / 'runtime').glob('*'));print('You are {}affected'.format('' if affected else 'not '))"

If the output indicates the system is affected, users should uninstall PyTorch-nightly and torchtriton immediately and update to the latest nightly binaries released after December 30, 2022.

Technical Mechanics of the Attack

The attack leveraged a well-documented supply chain vulnerability known as dependency confusion. This technique exploits the way package managers resolve dependencies by registering a malicious package on a public repository (PyPI) with the same name as a private or official dependency. When pip installs packages, it defaults to the highest-priority repository, which in this case was PyPI rather than PyTorch’s official repository. This design flaw allows attackers to intercept legitimate dependencies if they can register a conflicting package name first.

The malicious torchtriton package included a compromised triton binary located at PYTHON_SITE_PACKAGES/triton/runtime/triton. The binary was designed to remain dormant until explicitly triggered by importing the triton module in user code. Once activated, the binary performed the following actions:

• Collected system metadata, including IP address, username, and current working directory.
• Scanned the user’s home directory for files under 99,999 bytes, uploading the first 1,000 such files.
• Specifically targeted the .ssh directory, uploading all files under 99,999 bytes, which could include private SSH keys.
• Uploaded the gathered data to the domain h4ck.cfd via encrypted DNS queries using the wheezy.io DNS server.

The binary also employed anti-VM techniques to evade detection, further complicating analysis. Unlike many proof-of-concept exploits, which are often detectable due to their overt behavior, this attack used obfuscation and embedded the malicious payload directly within the Linux ELF binary.

The attacker’s claim of ethical research was contradicted by the scope of data exfiltration. While the notice on h4ck.cfd suggested the data was used solely for identifying vulnerable systems, the binary’s behavior indicated broader data collection. The incident aligns with previous cases where researchers or attackers exploited dependency confusion for unauthorized data access, such as the 2022 hijacking of the ctx and PHPass libraries to steal AWS credentials.

Mitigation and Lessons Learned

PyTorch’s response to the incident demonstrated proactive measures to contain and prevent future attacks.

• Renaming the torchtriton dependency to pytorch-triton to avoid future confusion with malicious packages on PyPI.
• Registering a dummy package on PyPI under the torchtriton name to prevent further hijacking attempts.

The reliance on repository priority in pip’s resolution process creates inherent risks for dependency confusion attacks.

• Mandatory cryptographic verification of package origins, such as digital signatures or hashes, to ensure package integrity.

The PyTorch-nightly dependency confusion attack serves as a case study in the evolving threat landscape of open-source software. While the immediate impact was limited to a specific timeframe and user base, the incident underscores the importance of vigilance in dependency management. Organizations and developers must remain proactive in adopting security measures to protect against supply chain attacks, which are increasingly sophisticated and targeted.

For developers using PyTorch-nightly, the incident reinforces the necessity of regularly updating dependencies and verifying package sources.

Sources

• atlas.mitre.orghttps://atlas.mitre.org/studies/AML.CS0015
• pytorch.orghttps://pytorch.org/blog/compromised-nightly-dependency/
• bleepingcomputer.comhttps://www.bleepingcomputer.com/news/security/pytorch-discloses-malicious-dependency-ch…

Case study data from MITRE ATLAS, used under the Apache License 2.0.