Showing posts with label Data. Show all posts
Showing posts with label Data. Show all posts

Wednesday, April 16, 2025

Open Source Electronic Lab Notebooks (ELN) in Academic Research: Balancing Openness, Sustainability, and Institutional Readiness




The increasing complexity of research environments in higher education has prompted a shift toward digital tools that support data stewardship, reproducibility, and collaborative inquiry. Among these, Electronic Lab Notebooks (ELNs) have become central. Open source ELNs, in particular, offer alignment with the academic world's values of code transparency, FAIR data principals, low entry cost, and the possibility of customized connections to other data tools. On the other hand use of open source ELNs also raises important questions related to deployment, hosting, integration with infrastructure, sustainability, security, compliance and institutional capacity. 

Recent developments in the open source ecosystem include the appearance of enterprise grade open systems with available paid support packages and managed hosting and deployment options. These types of systems address earlier limitations of "home brew" open source ELNs, positioning the former as more viable options for institutions seeking scalable research data solutions. This article examines the advantages and challenges of open source ELNs within the academic development landscape, with attention to early career researchers, student learning, data integrity, legal compliance, cost, and reproducibility.

Affordability and Strategic Cost Management 

Open source ELNs can be free to download and use, assuming that the user has the technical aptitude to configure a complex client-server system themselves, and that they have access to a suitable environment for hosting the server and all of its data. If that is not the case, then the good news is that some modern open source ELN systems offer paid support tiers that include cloud hosting, deployment assistance, integration services, and technical assistance. This hybrid model allows small labs or even entire institutions to enjoy the benefits of a professionally managed system, but reduce dependency on high-cost proprietary software while retaining full environment and data control (Kanza et al., 2017). Some examples of systems that use this hybrid open source with paid support model are RSpace, SciNote, ElabFTW and the chemistry-focused tool chemotion. The price paid by institutions for managed deployments of these systems varies between 50 and 500 USD per user per year depending on the system, the number of users, and many other factors, but this price range is pretty reasonable when compared to commercial data management products used by large pharmaceutical companies. It’s also very reasonable considering that the acquisition, integrity, reproducibility and long-term safety of data is arguably the primary raison d'être of university research labs, and therefore, this is something that certainly should be made a top priority. Theoretically, the flexibility of hybrid open source systems also enables cost-effective pilot systems, and fast deployment of a reliable production system at scale, aligning with institutional priorities for both fiscal responsibility and digital transformation, as well as better awareness of research activities and data collection institution-wide (European Commission, 2016). It also allows smaller or less-resourced institutions to participate in the digital research ecosystem without compromising access or functionality.

Capacity Building for Early Career Researchers 

For early career researchers, open source ELNs foster habits of transparency, reproducibility, and collaboration. These tools promote structured, searchable, and shareable record keeping, supporting the development of data literacy and research integrity (Freedman et al., 2015). When paired with centrally managed support systems, ELNs can be integrated into research training and mentoring programs, encouraging best practices in data stewardship from the outset of researchers’ careers (Lowndes et al., 2017). Use of Open Source ELNs is especially desirable because of the greater range of cost strategies, deployment options and data mobility, making it easier for data or copies of data to travel with the scientist as they move between institutions, and as their rank, funding and oversight and data publication needs change over the course of their career.

Expanding Visibility and Continuity of Student Research 

A growing use case for ELNs lies in undergraduate and postgraduate education. Research projects conducted as part of coursework or dissertations often generate valuable data or insights that are not destined for journal publication. ELNs provide a platform for preserving, showcasing, and potentially publishing this work in a semi-formal context. Making student research visible in this way enhances its discoverability, value, and impact (Borghi et al., 2018). ELNs also offers professional development benefits—students can share their research data with potential employers or collaborators and revisit their work in future research contexts, enhancing research continuity and identity formation as emerging scholars (Brew, 2023). Some systems even have the ability make selected work fully available for indexing by google and other search engines such that web searches by researchers working in specialized fields will find relevant studies performed by students, possibly introducing those students to new professional connections and future job opportunities. Again, the flexibility of open source systems can offer an enhanced range of strategies for students at any point in their training and early career paths.

Reproducibility and Research Integrity

Open source ELNs support reproducibility through features such as time-stamped entries, version control, well defined, distributable protocol templates and integration with computational workflows (Karkkainen et al., 2022). These features help ensure research can be verified, reused, or built upon, aligning with funder mandates and journal expectations around open and transparent research. Universities that adopt ELNs institution-wide can foster consistent standards in research documentation, ensuring robust audit trails and institutional readiness for external evaluation or legal challenges. The lower cost of open source systems helps them compete with other cheap, but less that idea alternatives such as MS SharePoint or google docs, which although attractive and familiar, absolutely do NOT offer the same advantages for research integrity. Only a fully 21CFR11 compliant ELN system guarantees the accuracy and integrity of electronically collected digital records.

Legal Compliance and Ethical Considerations 

Historically, open source tools have struggled with full regulatory compliance, in part because to be fully compliant they often need to be hosted on a robust, validated environment and not on a graduate student’s home server, and they may need to integrate with university SSO systems in order to be sure of user identity and affiliation. However, modern hybrid systems like RSpace offer optional support packages essentially as “add on” services to help with issues like GDPR compliance, encrypted storage, backup, and university-managed access control and formal training, making them fully 21CFR11 compliant and suitable for GMP research or research involving sensitive or regulated data. Some also offer delegated, hierarchical data oversight and system administration. Ensuring institutional oversight and guidance in these areas is critical. Without proper governance, individual researchers may lack the knowledge required to manage legal and ethical risks, particularly when dealing with human subjects or biomedical data. 

Sustainability and Institutional Integration

Concerns around sustainability have long hindered the broader adoption of open source software in academic environments. Community-driven development, while dynamic, can lead to uncertainty around long-term viability. The introduction of support contracts and hosted enterprise deployments mitigates these concerns by ensuring continuity, updates, and compliance that is also managed by a professional team with a long-term financial stake in the system’s continued existence (Petrisor et al., 2021). Institutional IT teams can now treat open source ELNs as enterprise-grade services, integrating them with identity management systems and aligning them with cybersecurity policies and Research Data Management (RDM) pipelines. This opens new possibilities for institution-wide adoption, from small labs to large collaborative networks. Best of all, the engineering collaborations between the vendor and user base can greatly enhance each institution's own vertical integration objectives (Plankytė et al 2025). It's true that this may involve recruiting university software engineers to work on complex problems and spend university time contributing to the system, but this is hardly "lost time" since presumably the only alternative is creation of the features the university wants outside of the ELN, and most likely this would take at least as long. By building on top of a managed system, the university gains access to the vendor as a full partner who can assist with testing, quality control, technical advice and distribution of the new features to other universities for the benefit of all. Involvement of university developers also helps alleviate sustainability fears because developers who already know the product and are familiar with its code would be in a better position to take over development and upkeep if the vendor were to end its formal support for the system.

Conclusion

Open Source, Professionally Supported, and Pedagogically Valuable Open source Electronic Lab Notebooks now offer a pragmatic, sustainable, and pedagogically rich alternative to proprietary systems. Their alignment with open science values, coupled with new support models, makes them a compelling option for institutions seeking to enhance research integrity, student engagement, and cost-effectiveness. For early career researchers and students alike, ELNs provide not only a tool for rigorous documentation but also a platform for scholarly visibility and future research continuity. Academic developers and institutional leaders who invest in these systems—not only as technologies but as pedagogical and professional development tools—stand to cultivate a more open, inclusive, and future-oriented research culture. Of the three best known ELN systems of this type (RSpace, Scinote and ElabFTW) Scinote is focussed primarily on commercial customers, and ElabFTW is focussed mainly on smaller lab deployments, with only RSpace designed specifically for large-scale, long-term deployment to entire universities, and this is reflected in the specialized SSO, filestore and RDM integration, integration with a plethora of well-known third-party research tools, hierarchic administration and free large-scale hosting services offered as part of the standard RSpace deployment package, as well as exhaustive documentation and a thriving github community.

Join the RSpace subreddit here: https://www.reddit.com/r/RSpaceELN

References 

Borghi, J., Van Gulick, A. E., & Hodges, T. L. (2018). Support for student data management at U.S. academic institutions. Journal of eScience Librarianship, 7(1), e1124. https://doi.org/10.7191/jeslib.2018.1124 

Brew, A. (2023). Researcher development and student identity: Changing understandings of academic practice. International Journal for Academic Development, 28(1), 1–14. https://doi.org/10.1080/1360144X.2022.2132134 

European Commission. (2016). Open Innovation, Open Science, Open to the World: A Vision for Europe. https://op.europa.eu/en/publication-detail/-/publication/3213b335-1cbc-11e6-ba9a-01aa75ed71a1 

Freedman, L. P., Venugopalan, G., & Wisman, R. (2015). Reproducibility2020: Progress and priorities. FASEB Journal, 29(9), 3729–3735. https://doi.org/10.1096/fj.15-100100 

Kanza, S., Sword, S., Gibbins, N., et al. (2017). Electronic Lab Notebooks: Can they replace paper?. Journal of Cheminformatics, 9(1), 31. https://doi.org/10.1186/s13321-017-0221-3 

Karkkainen, T., Aavik, G., Vahdat, M., et al. (2022). Electronic lab notebooks improve transparency and reproducibility in research. Nature Communications, 13, 5175. https://doi.org/10.1038/s41467-022-32885-2 

Lowndes, J. S. S., Best, B. D., Scarborough, C., et al. (2017). Our path to better science in less time using open data science tools. Nature Ecology & Evolution, 1, 160. https://doi.org/10.1038/s41559-017-0160 

Petrisor, A., Gavrilescu, A., & Vlad, C. (2021). Evaluating the sustainability of digital infrastructure for scientific research: The case of open source lab notebooks. Journal of Open Research Software, 9(1), 1–10. https://doi.org/10.5334/jors.315 

Plankytė, V., Edmunds, R., and Macneil, R.: Case Study of Vertical Interoperability Between Research Tools Enabling an End-to-End Sample Workflow from Collection, to Management, to Archiving, EGU General Assembly 2025, Vienna, Austria, 27 Apr–2 May 2025, EGU25-9046, https://doi.org/10.5194/egusphere-egu25-9046, 2025.

Tuesday, November 19, 2024

Coming in CERF 6: Improved support for using custom apps to perform mission-critical tasks and analyses on files stored in CERF.

Imagine you are a research organization that works with data files in some specialized format. A genetics lab working with GenBank .GBK or snapgene .DNA sequence files would be a good example. Now imagine your software engineers have written a custom app designed to perform some calculation or processing task on your data files, with the result or summary output to a new file.

Let's further imagine, that as a data manager, you need to have a good record of when any analyses where performed, who performed the analysis and precisely where the results of the analysis are stored. As it happens, this is a workflow that CERF ELN is very well pre-adapted to perform.

In most cases, users typically use CERF in conjunction with the default, industry standard applications on their local computer. An MS Word file, for example, may automatically open in MS Word, whilst your .DNA files may open in, say, snapgene. This workflow illustrates one of the unique advantages of a combined ELN and document management system that uses a desktop application to process your files. CERF carefully logs the interaction between the user and the files stored on the CERF server and displays all activity in the secure audit trail, so that managers are aware of current and past activity and access. In some cases, it may be advantageous to work with highly specialized applications that you've written yourself, designed specifically for performing specialized tasks on data that you stored in CERF. With CERF ELN, users can specify local applications on their computer that they would like to use to check out and edit specific file types. This allows users to optionally checkout files from CERF and open them in a non-default local application.

Lab-Ally has been working with bioinformatics students at the University of Maryland to create a toolbox of small accessory applications that can be used for processing various data files stored in CERF. Each academic term, as part of a capstone bioinformatics class, small groups of students (supervised by Lab-Ally) design, build and test an application of their choice. The application is designed to solve some common bioinformatics problem. An example is described below.

One team of 4 students recently built a GenBank extractor to make parsing genomic data easier and by utilizing this program you get a simplified output from the GenBank files that is readily compatible with CERF and the CERF search feature. The application can be used as a standalone tool or can be used integrated with CERF ELN to allow for superior record keeping, better efficiency and improved organization-wide collaboration.  This parser is designed to extract essential information from GenBank files and output a readable .rtf file.

What Does the GenBank Parser Do?

The parser extracts important data from GenBank files, such as:

  • Accession
  • Organism (Genus species)
  • Taxon data
  • Gene(s)
  • Genetic Sequence

It then organizes this data into an .rtf file, which is easy to read and compatible with most

platforms. Below is an example of what you will find in the output: 


RTF file showing various metadata retrieved from within a sequence file



How to Use the Parser as a Standalone Application


Install the Application:

  • Run the installation file on your computer
Launching the Application:
  • Navigate to the executable file of the program: go to C:\Program Files\GenBankParser and double-click on GenBankParser.exe.
  • A window should pop up with an "Open File" button.
Troubleshooting Display Issues:
  • If the window doesn’t show up properly, try resizing the window. Some users have experienced this issue, and resizing the window can often solve it.
Processing a GenBank File:
  • After clicking the Open File button, choose a .gb or .gbff GenBank file from your system.
  • The application will process the file and save an .rtf file to your desktop.

How to Use the Parser with CERF

  • If you’ve installed the parser, the next step is to configure CERF so it can summon files from the CERF server on demand and utilize the parser tool. Without this step, CERF would simply open GenBank files in whatever the default sequence editing application is on the user's local machine.
  • In CERF, navigate to Tools > Options > Applications.
  • Add the GenBankParser by pointing to the .exe in C:\Program Files\GenBankParser.
  • Set the MIME type to chemical/x-genbank. This helps CERF to understand what types of files you would like to open with the specified applicaiton

This is how Tools > Options > Applications should look once it's set up: 


CERF external application selector


Viewing GenBank Files with CERF:

  • Locate any .gb or .gbff file in CERF’s collections.
  • Right-click the file, select View-in, and choose GenBankParser from the list.
  • The parser will open, allowing you to process the file

CERF modified "View In..."  right-click options



File Output:

  • The application will process the file and save an .rtf file containing the results of the parser analysis to a specified local location.




Pasting as a Relation:

  • The file can then be dragged from the desktop into CERF, and specifically onto the associated file to have it pasted as a relation. This has the advantage that once added to CERF, the .rtf file is immediately indexed for searching so that users with the correct access permissions can search for target text that is located in the .rtf, and once they find THAT file, they can also locate the parent file containing the original raw sequence data.


New in CERF 6, the system will offer the option to automatically associate new files produced by custom applications (containing the results of some analysis) with the parent file containing the raw data. Since CERF offers outstanding version control, it will also be possible to perform these sorts of analysis with different versions of the original data file, associating the results with the correct version of the data in each case, and recording the entire process accurately in the CERF audit trail. We also hope to eventually offer this student-built parser and many other "add-on" tools for use with CERF on our website some time after the release of CERF 6 in 2025.

If you would like to see this tool in action or take a look at the code for the tool that these students built, or if you are a student or developer who would like to work with us to create additional tools like this genetic parser, we would love to hear from you. You can find contact information on the Lab-Ally website.



Wednesday, February 18, 2015

Research Data Management (RDM) in the UK, factors affecting development of a coherent national strategy.

Research Data Management (RDM) should be a topic of discussion for all academic and government labs. How can data be coherently protected, archived, searched and made available in ways that further national and institutional research goals? How can (and should?) organizations co-operate to achieve these goals in a consistent way? Frankly, the US lags in this area because to some extent, most of the big academic research institutes see themselves as partial competitors and also because so many of the faculty tend to see themselves as independent agents rather than members of a national team striving towards some common goal for the greater good. In the UK, there tends to be more government sponsored co-ordination of national research goals, so the discussion of a national co-operative research policy appears to be more advanced, although not without its challenges. The article below appears with kind permission from my colleagues at Research Space, makers of the RSpace ELN, a solution that is focussed on the needs of institutional ELN deployments at large academic and government research organizations.
The original appears at:
http://www.dcc.ac.uk/blog/reflections-idcc15-why-road-broader-take-rdm-opening
The Digital Curation Centre (DCC) website at http://www.dcc.ac.uk is a great place to visit regularly for anyone with an interest in scientific data management strategies at the institutional and national levels.
The three fundamental factors influencing RDM take up
The ‘Why is it taking so long panel’ discussion touched on two themes that are crucial in understanding the kind of environment that is conducive to take up of RDM. Geoff Bilder repeatedly, and correctly in my view, hammered home the point that until the right infrastructure is in place you can’t expect researchers to be enthusiastic about engaging with RDM, in fact you can’t expect them to do it at all. 
Geoff pointed to a second, and in his view underlying, issue, namely funding -- without an appropriate funding model infrastructure will develop too slowly to support, and stimulate, take up of RDM. Geoff sees the problem as originating in the current funding model, which tries to squeeze infrastructure development out from grant funding.
Up to a point I also agree with this second strand of Geoff’s argument. But I would suggest that it’s possible to dig down and identify a third, even more fundamental, factor which lies beneath the funding conundrum. This is what could be termed ‘culture’, specifically researchers’ attitudes to RDM infrastructure and tools, and their views on RDM’s priority or lack of priority in the context of their broader need for support.
If researchers don’t view RDM as a priority they are not going to pressure funders or their host institutions to provide the necessary infrastructure and tools to make it possible. No amount of cajoling or encouraging is going to change that, and until recently the RDM community has mostly been in the position of fighting that uphill battle.
So by culture I mean an understanding on the part of researchers’ of the usefulness of a particular bit of infrastructure or tool, and a desire on their part to adopt or use it because they think it will benefit their research. I would argue that when the culture and the infrastructure or tool are there, funding will follow. Depending on the circumstances – the cost of the bit of infrastructure or tool, the institutional set up, budget, funding cycles, etc. -- this may take longer or happen more quickly, but it will happen. My amended picture of the three key factors driving RDM uptake is displayed in the following diagram.


Figure 1 Factors influencing RDM take up

The three constituencies in action:  ELNs at the Universities of Manchester and Edinburgh
A second point to understanding the critical minimum circumstances for RDM to be taken up, and to take off, is that RDM happens in particular institutions. We saw this point illustrated in the many presentations at #IDCC15 where people from a wide variety of institutions talked about RDM at their institution – how it is developing, challenges, progress, issues, etc. In each case the status and prospect of RDM is inseparable from the institutional environment.
Let me here make a second assertion  -- a culture conducive to RDM take up requires buy-in or actually enthusiastic support from three key constituencies in research institutions: researchers, IT managers and administrators – data librarians and research data administrators. Absent support from all three constituencies the culture will not develop, and without that the requisite pressure to find funding for RDM will not happen.



Figure 2 Three key constituencies in research institutions influencing RDM take up
To bring this point home I’ll end by recounting a recent personal experience. In January my colleague Richard Adams and I gave a talk at the University of Manchester about our RSpace ELN and in particular how it had been integrated into the RDM infrastructure at the University of Edinburgh. Our host Mary Mcderby had advertised the session in advance and, to our amazement, more than 60 people showed up: an equal mix of researchers, IT managers and research administrators. Even more amazingly, we were greeted like rock stars (ok, not quite like rock stars), and peppered by a volley of interested questions and comments from all three sections of the audience.
What became clear as the discussion progressed was not only that all three constituencies had an active interest in adopting an ELN, but that they were aware of each other’s interests and by and large seemed supportive of each other and happy to work together. That is what I mean by a culture conducive to RDM take up. I’m confident that Manchester will find a way, sooner rather than later, to adopt an institutional ELN, because (a) the will is there across all three constituencies, and, crucially (b) this is a tool that all three constituencies can see will bring benefits.
The road to a broadly dispersed RDM culture and sustainable funding models is opening up
Pioneering institutions like Manchester and Edinburgh may have to be a bit creative, and come up with innovative and ad hoc solutions, to fund take up of RDM infrastructure. But, as they are now beginning to show the way, pressure will grow on funders to put forward sustainable and well considered funding solutions that are replicable more broadly as the culture at other institutions develops to the point where the majority of research institutions find themselves in a position to follow in the footsteps of the pioneers.
Rory Macneil
Research Space

Tuesday, June 10, 2014

Study says research data is lost at alarming rates

Study says research data is lost at alarming rates

"Eighty per cent of scientific data are lost within two decades, according to a new study that tracks the accessibility of data over time.
The culprits? Old e-mail addresses and obsolete storage devices.
“Publicly funded science generates an extraordinary amount of data each year,” says Tim Vines, a visiting scholar at the University of British Columbia. “Much of these data are unique to a time and place, and is thus irreplaceable, and many other datasets are expensive to regenerate."

Wednesday, April 16, 2014

Integrating an ELN with a university's long term Datashare archive.

This article discusses a recent initiative at the University of Edinburgh to solve this problem by integrating the RSpace ELN with their DataShare long-term archiving system. Longevity of digital data is always an issue, and as more and more scientists make the jump to ELNs, the importance of archiving data for future generations is often overlooked. Imagine what a loss it would be if there were no equivalent to "Down House" (where Darwin's notes and data are kept).

http://datablog.is.ed.ac.uk/2014/04/15/using-an-electronic-lab-notebook-to-deposit-data/