A team of computational reproducibility researchers set out to do something rare: systematically rerun the code behind more than 50 published computational papers. They had the original studies, the methodology descriptions, and a clear plan. What they did not have was access to the code. Roughly half of the targeted studies had deposited their analysis scripts and data in cloud-hosted repositories—GitHub, Zenodo, institutional cloud drives. The funding agency that commissioned the audit, a European biomedical funder, had a standing policy: no cloud storage could be used for any project it funded. The rule, written years earlier to safeguard sensitive data, also applied to the audit itself. The reviewers could not log in to those repositories. The audit collapsed.

This episode is not primarily about technical failure or sloppy science. It illustrates how a well-intentioned infrastructure rule, designed for one era of research, silently blocked a verification effort in another. Computational reproducibility—the ability to rerun analyses and obtain the same results—is widely seen as a cornerstone of trustworthy science. But that trust depends on access. When a funding policy makes that access impossible, the entire exercise unravels before it begins.

The episode, which has not been formally published, was described to me by two researchers involved in the audit, who spoke on condition of anonymity. They requested that neither the agency nor their names be disclosed, to avoid potential professional fallout from discussing a funder's internal policy. Their account offers a rare glimpse into the friction between institutional rules and the practical demands of reproducibility. It also raises uncomfortable questions about who bears the cost of verification, and whether the current incentive structure in computational science makes such audits doomed from the start.

A Simple Rule That Silenced a Reproducibility Check

The policy in question was straightforward: all data and code generated under the agency's grants must be stored on local drives or institutional servers within the funder's home country. Cloud services—whether commercial providers like Amazon Web Services or academic platforms like Google Cloud—were banned. The rationale was data security. The agency, which funds a mix of sensitive biomedical and physical science research, wanted to ensure that no project data ever left controlled, auditable infrastructure.

The rule was drafted roughly a decade ago, when cloud storage was still emerging in academic research. Many funders were wary of sending data to servers they could not physically inspect. The policy was seen as prudent, even necessary. But it was written before the reproducibility movement gained momentum, before journals began requiring code deposits, and before cloud-hosted repositories became the default way for computational scientists to share their work. The audit team, which included a statistician and two computational scientists, was tasked with evaluating the reproducibility of a batch of papers that the agency had funded. They planned to download the code, set up the computing environment, and re-run the analyses. The first step was to locate the materials. For about half the papers, the authors had posted code on GitHub or similar platforms. The team requested access through normal channels. The agency's legal office replied that the policy prohibited using cloud services for any work related to the grant—including the audit itself.

The team appealed, arguing that accessing a public repository to download code was not the same as storing project data on a cloud server. The agency held firm. The rule made no distinction between reading and writing, between temporary access and persistent storage. The reviewers could not log in to GitHub. They could not download the code. The audit of those papers was abandoned.

The result is an incomplete audit that remains unpublished. The team has no timeline for revisiting the work. The agency, for its part, has not revised the policy. The episode is a quiet casualty of a rule that no one thought to examine from the perspective of reproducibility.

The Audit That Could Not Run

The audit targeted a diverse set of computational papers: climate model simulations, genomic association studies, and machine learning analyses in materials science. The team had selected them because they represented a range of computational intensity and methodological complexity. Each paper had passed peer review and was published in a reputable journal. The audit was meant to be a routine check, not an expose.

Of the 52 papers initially selected, 24 had code hosted on cloud-based repositories. For those 24, the audit could not proceed. The remaining 28 had code on institutional servers or local drives that the team could access. But the loss of nearly half the sample meant the audit's conclusions would be biased: the cloud-hosted papers might have been systematically different in their computational practices. For instance, researchers who used cloud resources might have been more likely to employ large-scale data analysis pipelines that are harder to replicate without the original cloud environment. The team decided that publishing a partial audit could mislead. They shelved the results.

The audit methodology involved several steps. First, the team attempted to locate the code and data for each paper using information provided in the manuscript or supplementary materials. They then assessed whether the materials were complete and whether the computational environment could be recreated. For the 24 cloud-hosted papers, the team could not even begin this assessment because the policy barred them from accessing the repositories. The team also planned to document any discrepancies between the original results and their re-runs, but for the blocked papers, no such comparison was possible.

One researcher involved described the frustration: “We had the methods, we had the papers, we had the time. We just couldn't touch the code. It was locked behind a policy that was never meant to block reproducibility, but that's exactly what it did.” The agency's legal office, he said, was sympathetic but unwilling to create an exception. They feared that any waiver would set a precedent that could undermine the entire security framework.

The episode highlights a growing tension in computational science: the tools that enable open sharing—cloud repositories—are the same tools that some funders forbid. The result is a patchwork of access that makes large-scale reproducibility audits nearly impossible. If every funder has its own storage policy, the effort to verify published results becomes a logistical nightmare. The team has since discussed the episode informally at conferences. Several colleagues have told them they encountered similar barriers. But no one has published an account of it. The fear of alienating funders, who control future grant money, is strong. The audit remains a cautionary tale told in whispers.

Why Funders Ban the Cloud

Data security concerns are the primary driver behind cloud bans. For agencies that fund research involving human subjects, classified information, or proprietary data, the idea of storing files on servers owned by a foreign corporation is unacceptable. Even for non-sensitive data, some funders worry about jurisdiction: if a cloud server is located in another country, the data may be subject to that country's laws, including government access requests.

IT departments at many universities and research institutes also prefer local storage for control. They can manage backups, monitor access, and ensure compliance with institutional policies. Cloud services introduce a third party whose security practices may not align with the institution's standards. For funders that already have robust local infrastructure, the simplest path is to require its use.

But the landscape has shifted. Many cloud providers now offer encryption, audit logs, and compliance certifications that meet or exceed institutional capabilities. Services like Amazon Web Services, Google Cloud, and Microsoft Azure have dedicated academic programs with security features tailored to research data. Some funders, including the National Institutes of Health in the United States, have embraced cloud storage for large-scale data sharing. Others remain cautious.

The agency in question revisited its policy in 2022 but decided not to change it. A review committee concluded that the security risks of cloud storage, however small, outweighed the convenience. The reproducibility implications were not part of the discussion. The policy remains in effect, and the audit that failed because of it is unlikely to be the last.

“The irony is that the agency funds a lot of computational work that depends on cloud resources,” one of the researchers said. “The scientists they fund use cloud services all the time. The policy just means they can't admit it, and they can't share code in the way that reproducibility demands.”

The Hidden Cost of Infrastructure Rules

Infrastructure policies have hidden costs that are rarely accounted for. Local storage, while secure, is expensive. Maintaining large server farms, paying for electricity and cooling, and staffing IT support all consume grant money that could otherwise fund research. For large datasets—genomic sequences, climate model outputs, high-resolution imaging—local storage can be prohibitively costly. For example, a single genomic dataset can exceed several terabytes, and storing such data on local servers requires significant capital investment. Cloud storage, by contrast, offers flexibility. Researchers can pay for only what they use, scale up when needed, and share data with collaborators across institutions. Many computational scientists have gravitated toward the cloud precisely because it lowers the barrier to collaboration. The agency's policy forces them into a more expensive, less collaborative model—or drives them to ignore the rule quietly.

The cost of the policy extends to reproducibility. When code is stored on local drives, it is often inaccessible to anyone outside the institution. Even within the same institution, code can be lost when a graduate student graduates or a professor moves to another university. Cloud repositories, with their persistent URLs and version control, are designed to prevent exactly this kind of loss. The policy that bans them undermines the very infrastructure that reproducibility requires.

A related issue is the division of grant budgets. Cloud storage costs are often covered by separate infrastructure grants or institutional allocations, not by the individual research grants that fund the actual science. When a reproducibility audit comes along, the reviewers have no budget for cloud access. The policy creates a gap between how research is practiced and how it can be verified.

The result is a system where the rules for doing science and the rules for checking science are misaligned. Researchers are encouraged to share code, but the funder's storage policy makes sharing difficult. Auditors are asked to verify results, but the policy blocks the verification. The misalignment is not intentional; it is a byproduct of policies that were never designed with reproducibility in mind.

What the Failed Audit Reveals About Incentives

The failed audit is a symptom of a deeper problem: the incentive structure in computational science prioritizes new results over verification. Researchers are rewarded for publishing novel findings, not for making their code reusable. Journals increasingly require code deposits, but enforcement is weak. Funders track publications and grant money, not whether the code behind those publications can be rerun.

Reproducibility audits are rare and underfunded. They require time, expertise, and access to computing resources. They do not produce glamorous results. The audit that failed would have taken months and produced a report that might have been read by a few hundred people. The agency that commissioned it saw it as a low-priority project. When the cloud policy blocked it, there was little pressure to find a workaround.

“The agency was not hostile to reproducibility,” one researcher said. “They just didn't see it as their problem. Their job is to fund science, not to check it.” That attitude is widespread. Funding agencies are evaluated by the number of papers and patents their grants produce, not by the reproducibility of those papers. The audit was a voluntary add-on, not a core mission. When it hit a wall, it was easier to abandon it than to change the policy.

Some researchers argue that the burden of verification should fall on journals, not funders. But journals have limited resources and no authority over researchers' storage choices. Others argue that the scientific community should self-police through post-publication peer review. But that model relies on volunteers and has no mechanism to enforce access. The failed audit shows that without institutional alignment, reproducibility efforts will remain ad hoc and fragile.

“We need to think about reproducibility as an infrastructure problem, not just a methodological one,” said a computational scientist who has studied the issue. “If the pipes are blocked, the water doesn't flow. In this case, the pipe was a policy that said 'no cloud.'”

Practical Paths Forward for Funders

There are straightforward steps funders can take to prevent similar failures. The most obvious is to create an exception process for reproducibility audits. A reviewer who needs to download code from a cloud repository could be granted temporary, read-only access without compromising the funder's security posture. The agency in question considered this but worried about precedent. A carefully scoped exception, however, would not open the floodgates. For instance, the exception could require that the downloaded code be stored on a local machine and deleted after the audit, with a signed agreement from the reviewer.

Another option is to approve a set of vetted cloud services that meet the funder's security standards. Several cloud providers already offer compliance certifications for academic research. Funders could negotiate institutional agreements that guarantee data protection. This would allow researchers to use cloud storage without violating policy, and it would give auditors a clear path to access code. The agency could start by piloting such an arrangement with a single provider and evaluating its security.

Funders could also require that code be deposited in recognized, non-commercial repositories such as Zenodo or institutional repositories that the funder already trusts. These platforms are not cloud services in the traditional sense; they are designed for long-term archiving and are often hosted by academic organizations. Requiring deposit in such a repository would align the funder's storage policy with open science goals. The agency could update its grant conditions to mandate that all code be deposited in an approved repository within a year of publication.

Finally, funders could pilot joint audits that share infrastructure. If multiple agencies agreed on a common set of storage rules and a shared audit platform, the cost and complexity of verification would drop. The failed audit was a solo effort by one agency. A coordinated approach could distribute the burden and create a standard that all parties could follow. For example, a consortium of European funders could establish a shared reproducibility unit with access to vetted cloud services, reducing the risk for any single agency.

None of these solutions are radical. They require only that funders recognize that their storage policies have consequences beyond data security. The rule that blocked the audit was not malicious. It was simply outdated. Updating it would not compromise security. It would, however, open the door to the kind of verification that computational science needs. The story of the audit that could not run is a reminder that reproducibility is not just a technical challenge. It is a policy challenge, an infrastructure challenge, and an incentive challenge. The code exists. The desire to check exists. What is missing is the alignment between how we fund science and how we verify it. Until that alignment improves, more audits will fail—and the science will be the poorer for it.