By Tim Willense & Vincent Traag, CWTS

Introduction

The EASY case study investigates the impact of data availability on its usage by researchers. It explores whether data shared through national repositories experiences higher uptake compared to other platforms like international or disciplinary repositories. Focused on the EASY repository managed by DANS in the Netherlands, the study analyzes datasets from the SSH, with a specific emphasis on Dutch research. By utilizing DataCite, it correlates data with related publications to assess citation rates, providing insights into the effectiveness of national repositories in promoting data dissemination and scholarly collaboration.

Why was this particular study selected to support testing and operationalization of Open Science indicators?

This study on the cross-cutting effects of open research data from a national repository, specifically focusing on the EASY database, was chosen to support testing and operationalization of Open Science indicators for several reasons. The case study's unique focus on data availability and its impact on data use aligns closely with some of the core objectives of Open Science. By investigating whether scientists exhibit different usage patterns based on the repository from which data is made available, the study provides a nuanced understanding on some of the dynamics involved. The EASY database, managed by DANS in the Netherlands, stands out as a case due to its commitment to the FAIR principles, making it an ideal candidate for assessing the effectiveness of open data repositories. This enables this case study to leverage insights for refining Open Science indicators related to data sharing practices.

Why do you think this is study is important for the broader Open Science context? 

In the broader Open Science context, this study holds importance for multiple reasons. Firstly, it addresses a critical gap in understanding the relationship between open data repositories, data reuse, and data referencing. The investigation into the effectiveness of open data repositories in making research data openly available contributes valuable insights. Additionally, by examining how data is referenced in research and its compliance with open data reuse and repositories, the study provides a foundation for developing more effective Open Science practices. Understanding these dynamics is crucial for fostering a transparent, collaborative, and accessible research environment, aligning with the overarching principles of Open Science.

How will this study contribute to the main aims of the project? 

The EASY case study significantly contributes to the main aims of the PathOS project by offering insight into the impact pathways of open science. Specifically, the study sheds light on the impact pathway of open data repositories on open data reuse. By examining factors such as accessibility, findability, data quality, and the role of various contextual elements, the case study highlights mechanisms within the broader open science system. This understanding is pivotal for refining and operationalizing Open Science indicators, providing a comprehensive view of how data repositories influence data uptake and reuse.

What kind of impact is expected to be generated by the results/outcomes of the study for different stakeholder groups? 

• Government: While not directly affected, governments that provide funding for building national data repositories, will have a keen interest in the results. Insights from the study can inform future decisions regarding funding allocation and the development of national-level data repositories.
• National data repository: The results would be of interest to the national data repository by offering valuable insights into its usage.
• Scientists: Scientists who contribute data to repositories will benefit from understanding the differences in data uptake across various repositories. The study's outcomes can guide scientists in making informed decisions about where to share their data for optimal visibility and impact within the research community.

The CWTS team

Tim Willemse

Researcher

Vincent Traag

Senior researcher and bibliometric consultant

By Simon Apartis & Tommaso Venturini, CNRS

Introduction

The presented interview of case study leaders examines how French Open Science platforms — HAL, OpenEdition, and RechercheDataGouv— shape the impact of Open Science policies by analyzing connection logs. It focuses on user access patterns to explore the societal impact of Open Science beyond citation metrics.

Why was this particular study selected to support testing and operationalization of Open Science indicators?

Over the past decade, France has implemented ambitious Open Science (OS) policies underpinned by two National Plans for Open Science (2018–2021 and 2021–2024). These initiatives aim to foster "an ecosystem where scientific research becomes more cumulative, better supported by data, and more transparent, with faster and more universal access to results." The ultimate vision is to make "knowledge accessible to all, serving research, education, the economy, and society." 

As incentives for researchers to publish in Open Access generalized and public funding for OS expanded, two French platforms established in 2000—HAL (Hyper Articles en Ligne) and OpenEdition—have emerged as central pillars in France's open publication and Open Data ecosystem. HAL, in particular, has seen remarkable growth, with total deposits rising from 500,000 in 2010 to 3.5 million by 2024.  

More recently, the RechercheDataGouv Platform, funded as part of the National Plan for Open Science, was launched to promote data recognition as an independent research output. The platform aims to improve reproducibility, reduce redundancy, and encourage the open sharing of research data to increase citation rates. It intends to support researchers in managing their data effectively, enabling them to retrieve, reuse, and decide whether to preserve, share, or delete data, thereby hoping to minimize the digital footprint. 

Along with expanding OS practices and platforms and following the principle of evidence-based steering of public policies, there has been an increasing need for advanced monitoring tools to properly understand and demonstrate the effects fostered and expected by OS policies on science, society and the economy. Launched in 2018, the Open Science Barometer aims to address this need, but much remains to be done to align it with the indicators set by UNESCO´s working group on Open Science monitoring, to dive into the complexity of societal effects, especially on the long run, and not only to monitor but to counterfactually assess and evaluate the effects as consequences causally attributable to OS policies and OS outputs.  

Why do you think this is study is important for the broader Open Science context? 

Because Open Science Platforms act as a pivotal meeting point between, on the one hand, scientific offer and, on the other hand, scientific, economic and societal demand for science, they are crucial to study in order better to understand the effects of OS policies and practices. Access to Open Science through platforms is the first step for its reuse by a wide variety of heterogeneous players, whose societal, economic and scientific practices would not have been impacted and thus be made to impact the world around them in unprecedented ways if OS platforms had not been available in the first place.  

Our study primarily focuses on the preliminary step for impact: access. Access can be studied at the platform level by focusing on the connection logs. Our approach boils down to three core questions. Who are the leading private and public organizations using these platforms? When are these platforms used the most and why? Who are the websites referring to these platforms and how are they connected? 

We will tackle those questions by analysing the connection logs to HAL, OpenEdition and RechercheDataGouv and enriching them with data based on the IP address, referrer-based data and data from OpenAlex.

Figure 1 shows the first result of the first version of this automated classification script run on the list of all the identifiable organizations (~10%) which accessed OpenEdition during the first four days of October 2022 (96% of organizations (4267) domains divided into 22 categories, 5,07% (228) couldn’t be appropriately classified).

Figure 1: Preliminary results - distribution of identifiable organizations accessing OpenEdition, categorized by sector

Our goal is to analyze how patterns of OS usage (specifically, which categories of users access certain types of resources) vary based on their degree of openness. For example, we might find that articles published through diamond Open Access are accessed by a larger proportion of non-academic users compared to articles available only as preprints. This trend could be an indicator of the potential for greater societal impact of scientific work published in Open Access formats.

How will this study contribute to the main aims of the project?  

This case study seeks to understand the role of OS platforms as central nodes in the impact pathways of Open Science. Rather than focusing solely on citations—which primarily reflect the academic diffusion and impact of OS—we expand our analysis to include 1) referrers pointing to OS artifacts and, more importantly, 2) connection logs. This approach aims to develop replicable and scalable tools for assessing the societal impact of OS while laying the foundation for further qualitative analyses of how OS artifacts are used and reused by various actors in society.

Our methodology is closely aligned with PathOS' framework. It is structured around data sprints, fostering collaboration with platform technical teams and experts from the French Ministry of Higher Education. Following PathOS' iterative process, we integrate the causality model insights from meta-analyses on OS impact assessment and recommendations from the Open Science handbook At the same time and in a four-step iterative manner, we address technical challenges, explore the possibilities and constraints of log analysis, and incorporate the practical knowledge of platform technical teams alongside the policy needs reported by the ministry collaborators.

Who are the main actors involved and why are they important within the R&I ecosystem represented in this study?  

Based on a generalist conception of the R&I ecosystem, which still remains a widely discussed term[1], it is possible to distinguish between four main groups of stakeholders that are either indirectly challenged or directly involved in OS platforms. (1) Concurring science dissemination infrastructures, (2) academic science producers and consumers, (3) non-academic science producers and consumers, (4) Open Science professionals who manage the platforms.  Among those four categories, the most significant stakeholders are large commercial publishers; small commercial publishers; not-for-profit publishers; university libraries; universities, research communities and individual researchers at research performing organizations (RPOs); companies; patient groups; citizen sciences, non-governmental organizations (NGOs), activists and citizens; technical, administrative and support department of platforms at RPOs. It is important to note, however, that the extent and nature of these stakeholders' involvement and transformations in the R&I ecosystem remain under investigation. This inquiry includes both quantitative and econometric methods like those employed by PathOS and detailed ethnographic approaches to research ecosystems.

What kind of impact is expected to be generated by the results/outcomes of the study? 

By leading this case study we are already strengthening collaboration between Open Science platforms and the technical teams that run them. Connection logs are the common denominator of thousands of Open Science platforms, and working on a standardized log analysis method could also help achieve broader strategic goals.

For easy future upscaling, we are coding a Node.js middleware for the Ezpaarse toolkit, a set of COUNTER5 compatible digital tools for eletronic resources dedicated to the detection, collection, enrichment and dynamic visualization of usage data, which is already being used by hundreds of platforms. Our development script is available here and will be published on Ezpaarse’s own GitHub repository once it has been approved for production.

Because it relies on widely used standards and Open Source software, our method could be easily replicated and applied to greater sets of connection logs, both from for-pay and Open Access platforms, allowing for greater comparability and scale effects due to the volume of data and an even stronger contractual assessing method if for-pay and Open Access data can be used respectively as a test and control group.

The possible impacts will stem from our tools enabling everyone to easily visualize and intuitively grasp how strongly different degrees of openness influences the kind of actors who access science, possibly advocating for lasting and significant funding and support of OS policies at the highest political level, at times when science is expected to bridge its gap with society, address societal challenges and support social innovation.

[1] See for instance :

• Altman, Micah et Philip N. Cohen (2022). “The Scholarly Knowledge Ecosystem: Challenges and Opportunities for the Field of Information,” Frontiers in Research Metrics and Analytics, vol. 6. https://doi.org/10.3389/frma.2021.751553
• Kuehn, Evan F. (2022). “The information ecosystem concept in information literacy: A theoretical approach and definition,” JASIST, pp. 1–10. https://doi.org/10.1002/asi.24733
• Lyle, Peter, Henrik Korsgaard et Susanne Bødker (2020). “What’s in an Ecology? A Review of Artifact, Communicative, Device and Information Ecologies,” NordiCHI ’20, October 25–29, 2020, Tallinn, Estonia. https://doi.org/10.1145/3419249.3420185
• Mounier, Pierre et Simon Dumas Primbault (2023). “Sustaining Knowledge and Governing its Infrastructure in the Digital Age: An Integrated View”, Zenodo, 2023. https://doi.org/10.5281/zenodo.10036402
• Star, Susan Leigh (ed.) (1995). Ecologies of Knowledge: Work and Politics in Science and Technology. New York, State University of New York Press.

Sound causal inference is crucial for advancing the study of science. Incorrectly interpreting predictive effects as causal might be ineffective or even detrimental. Many publications in science studies lack appropriate methods to substantiate their causal claims. In this preprint we provide an introduction to structural causal models. Such models allow researchers to make their causal assumptions transparent and provide a foundation for causal inference. We illustrate how to use structural causal models based on simulated data of a hypothetical structural causal model of Open Science. We hope our introduction helps researchers in science studies to consider causality explicitly.

The PathOS context

Concerns of causality are centre stage for the PathOS project. Without a proper understanding of causality, it is impossible to provide proper policy recommendations. For example, imagine we observe that published research using open data is less reproducible. Even if open data does in fact have a positive effect on reproducibility, this negative association might appear if journals select research based on open data and rigour. That is, journals may be more likely to publish research if it has open data, but also if it is more rigorous. If published research has no open data, it therefore tends to be more rigorous, otherwise it would not be published at all. Research that is more rigorous tends to be more reproducible, and this effect might be stronger than the effect of open data. For this reason, the association between open data and reproducibility might be negative, even if the actual causal effect is positive. If we incorrectly interpret the negative association as causal, and then recommend not to incentivise open data, we would be providing ill advice.

What's next?

Having a common understanding of causality and structural causal modelling helps the PathOS project interpret the existing literature and the identification of impact pathways. This requires us to differentiate the impact of open science from the effect of openness on that impact. That is: how does the fact that something is open—be it publication, data, code, review—have a causal effect on its impact? The introduction to causality provides such a common understanding. This will be especially important as PathOS builds upon the knowledge gained through our evidence scoping and intervention logic definition to further map and validate Open Science impact pathways and their verification methods (work on which is well underway – watch this space!).