Journal:Cyberbiosecurity: An emerging new discipline to help safeguard the bioeconomy

From LIMSWiki
Jump to navigationJump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.
Full article title Cyberbiosecurity: An emerging new discipline to help safeguard the bioeconomy
Journal Frontiers in Bioengineering and Biotechnology
Author(s) Murch, Randall S.; So, William K.; Buchholz, Wallace G.; Raman, Sanjay; Peccoud, Jean
Author affiliation(s) Virginia Tech, Federal Bureau of Investigation, University of Nebraska, Colorado State University
Primary contact Email: rmurch at vt dot edu
Editors Berns, Kenneth I.
Year published 2018
Volume and issue 6
Page(s) 39
DOI 10.3389/fbioe.2018.00039
ISSN 2296-4185
Distribution license Creative Commons Attribution 4.0 International
Website https://www.frontiersin.org/articles/10.3389/fbioe.2018.00039/full
Download https://www.frontiersin.org/articles/10.3389/fbioe.2018.00039/pdf (PDF)

Abstract

Cyberbiosecurity is being proposed as a formal new enterprise which encompasses cybersecurity, cyber-physical security, and biosecurity as applied to biological and biomedical-based systems. In recent years, an array of important meetings and public discussions, commentaries, and publications have occurred that highlight numerous vulnerabilities. While necessary first steps, they do not provide a systematized structure for effectively promoting communication, education and training, elucidation, and prioritization for analysis, research, development, testing and evaluation, and implementation of scientific and technological standards of practice, policy, or regulatory or legal considerations for protecting the bioeconomy. Further, experts in biosecurity and cybersecurity are generally not aware of each other's domains, expertise, perspectives, priorities, or where mutually supported opportunities exist for which positive outcomes could result. Creating, promoting, and advancing a new discipline can assist with formal, beneficial, and continuing engagements. Recent key activities and publications that inform the creation of cyberbiosecurity are briefly reviewed, as is the expansion of cyberbiosecurity to include biomanufacturing, which is supported by a rigorous analysis of a biomanufacturing facility. Recommendations are provided to initialize cyberbiosecurity and place it on a trajectory to establish a structured and sustainable discipline, forum, and enterprise.

Keywords: cyberbiosecurity, bioeconomy, biosecurity, biomanufacturing, cybersecurity, cyber-physical security, supply chain

Introduction

We propose “cyberbiosecurity” as an emerging hybridized discipline at the interface of cybersecurity, cyber-physical security, and biosecurity. Initially, we define this term as “understanding the vulnerabilities to unwanted surveillance, intrusions, and malicious and harmful activities which can occur within or at the interfaces of commingled life and medical sciences, cyber, cyber-physical, supply chain, and infrastructure systems, and developing and instituting measures to prevent, protect against, mitigate, investigate, and attribute such threats as it pertains to security, competitiveness, and resilience.” We emphasize this is an initial definition; we fully expect that the definition and the landscape will rapidly evolve, requiring the definition to be revised. We also contend that, because of its diversity and extent, cyberbiosecurity needs its own systematics, so that it can be better communicated, organized, explored, advanced, and implemented. Here, we also posit that cyberbiosecurity contributes to a larger strategic objective of “safeguarding the bioeconomy,”[1] a concept advanced in the U.S. which seeks to increase security and resilience of the bioeconomy to protect its rapidly changing cyber-life science topology.

Thus far, what we are proposing to call cyberbiosecurity has primarily been initiated out of two principal sets of activities. The first set of activities involved a study[2] and three workshops[1][3][4], which were primarily focused on security issues with respect to “big data” and the relationship with the “bioeconomy.” The second set was a first-ever systems analysis of a biomanufacturing facility, which expands the view to include a different “target set” and approach to understanding vulnerabilities with sharp acuity. This tasked study was conducted to comprehensively understand the vulnerabilities with respect to a wide range of unwanted intrusions and nefarious activities in the life science, cyber, cyber-physical, infrastructure, and supply chain aspects, and determine what measures could be taken or developed and implemented to anticipate, detect, identify, prevent, mitigate, respond to, and attribute such potential exploitation. The first published paper on cyberbiosecurity primarily focuses on the security of the biotechnology interface with cyberspace.[5] In addition to the system analysis as part of the second set, a small workshop was held in the U.S. that sought to scope and stimulate interest in the government, academic, corporate, and non-profit sectors, create a core constituency, understand what topics and themes could constitute cyberbiosecurity, identify priorities, and begin to develop a campaign and timeline. The workshop was highly successful. These endeavors, together with additional recent activities and publications[6][7][8][9], have added to scoping the future of cyberbiosecurity yet to come.

Background

Simply stated, since its inception, biosecurity has been primarily focused on reducing the risks associated with the misuse of science which could cause harm to humans, animals, plants, and the environment through the creation, production, and deliberate or accidental release of infectious disease agents or their byproducts (e.g., toxins). Cybersecurity has been a separate field which has been primarily focused on the security of information technology systems, from personal computers and communications devices to large infrastructures and networks. Up until just the past few years, the “cyber” overlaps with biosecurity have not been realized or fleshed out. The important interrelationship between biosecurity and cybersecurity is gaining increasing attention. We posit that the two must work collaboratively and will not be effective working separately. Cyberbiosecurity actually started with thinking about a particular set of problems being confronted by the life sciences. As a result of our recent work, described below, other dimensions are being added. Establishing a unifying discipline, crafting its systematics, and identifying an evolutionary path forward are within reach.

The economic strength and growth of the United States have been due to a culture and environment that foster innovation. Those developments could not be possible without significant contributions by science and engineering. The intersection among economic growth and the biological sciences contributions—the bioeconomy—has recently been recognized as an important component of national security. For the U.S., the bioeconomy accounts for an estimated $4 trillion annually, nearly 25% of the GDP. That contribution ranges from pharmaceuticals to renewable energy, from environmental remediation to public health resilience, and from agriculture to emerging disease response. As part of the U.S. national security architecture, “safeguarding the sciences” is a priority. In doing so, the U.S. Federal Bureau of Investigation (FBI) and other federal agencies also fulfill the U.S. obligation to the Biological Toxins and Weapons Convention (BTWC) and compliance to the United Nations Security Council Resolution (UNSCR) 1540, preventing the misuse of biological material, technology, and expertise, and encouraging the enforcement of the related statutes. The FBI also sponsors and actively engages the International Genetically Engineered Machine (iGEM) competition to inculcate a culture of security among international students, who will become leaders of research, industry, and policymaking. At the same time, the FBI works with U.S. policymakers[10] to redefine the scope of the biosecurity spectrum for the twenty-first century, a century with an unprecedented pace of biological research and innovation, and the use of diverse and large datasets (big data) to assist global scientific and societal priorities and opportunities. Concomitant to both realized and future benefits and growth, the life sciences are becoming increasingly digitized—while at the same time intellectual property protection, cyber intrusion, and the protection of personal medical and genomic information becoming more important—and the impacts on science, trade, and commerce loom large. Engagements with the science media[6] and testimonies[10] have raised these issues to advance both U.S. competitiveness and national security.

In 2014, the American Association for the Advancement of Science (AAAS), FBI, and the United Nations Interregional Crime and Justice Research Institute (UNICRI) published a report entitled “National and Transnational Security Implications of Big Data in the Life Sciences.[2] Briefly, this report starts by helping to understand “big data”; massive, diverse data sets that are created, reside, are analyzed in, and move in information ecosystems. For the life sciences, big data refers to datasets including “raw data, combined data, or published data from the health-care system, pharmaceutical industry, genomics and other –omics fields, clinical research, environment, agriculture, and microbiome efforts.” Further, they state that big data also includes analytic technologies and outputs, such as from “data integration, data mining, data fusion, image and speech recognition, natural language processing, machine learning, social media analysis, and Bayesian analysis.” A number of areas that have drawn and need attention are pointed out, such as the security of the cyber infrastructure and data repositories, and the privacy and confidentiality of individuals. In our view, their focus on the security risks of big data in the life sciences falls into just two major categories, i.e., inappropriate access to data and analytic technologies through vulnerabilities in the data and cyber infrastructure. As such, the use of big data technologies to integrate current data and enable the design of a harmful biological agent should be revisited and refined. Thanks to this team's efforts, not only do we have a useful topology of big data, the beginnings of a structure for thinking about security implications at the bio-cyber interface (technical, legal, institutional, and individual) and a set of high-level recommendations for a path forward.

From 2014 to 2016, three workshops were organized by the U.S. National Academies on behalf of the FBI under the theme of “Safeguarding the Bioeconomy.” The first[1] laid the foundation for the next two. Presentations and discussions focused on the security implications of the convergence in the life and chemical sciences with physical, mathematical, computational, engineering, and social and behavioral sciences. In addition to broader contexts, two specific technologies received focus: neuromorphic computing and 3-D bioprinting. The second workshop[3] introduced a range of new threats to and vulnerabilities of the bioeconomy, which at the time had not received focused consideration with respect to U.S. “competiveness, security, economic growth, and global leadership in research and innovation.” This workshop was built on three major themes: the role of informatics in the bioeconomy, criminal threats and vulnerabilities in the existing and near-future bioeconomy, and securing and flourishing the bioeconomy for the future. Rapid growth of this sector creates increasing security risks to proprietary materials and informatics, brings about an increase in frequency in industrial espionage and data hacks, and decreases the effectiveness of traditional security measures. Still, alternative and adaptive security measures could be implemented even with the inherent openness of emerging technologies upon which the bioeconomy is dependent. Workshop participants not only provided more detail on the threats and vulnerabilities but also both comprehensive categories and specific approaches that could be taken to address the problems and concerns identified. The third workshop[4] principally focused on data generation and access with respect to the bioeconomy within several categories of both clinical and non-clinical data, from the perspectives of biosecurity, data policy and regulation, future implications, technology advances, data sovereignty and sharing, cybersecurity, and international implications. Taken together, these events significantly expanded the view of what the emerging discipline of cyberbiosecurity could encompass.

Pauwels and her co-authors also raise important concerns and recommendations for the security of biotechnology in cyberspace. In the first, she and Vidyarthi[7] raise concerns over data breaches of health care information and what it means for the biotechnology industry. Protecting digital DNA and personal medical information is highlighted, as well as the fact that a then recent U.S. Presidential cybersecurity initiative put significant resources into shoring up cyberinfrastructure. Unfortunately, the need for improvements to protecting the bioeconomy, which is heavily dependent on information systems and infrastructure, was not recognized. The report outlined the implications of not protecting the bioeconomy dimension. Their recommendations were primarily focused on protecting genomic data. In the second report, Pauwels and Dunlap[9] go into more depth framing potential cyber-vulnerabilities for specific types of biotechnologies: genome-editing; DNA assembly, synthesis and printing; portable genomic sequencers; artificial intelligence for understanding biological complexity; autonomous systems and robotics in cloud labs; and lab-on-a-chip and microfluidic technologies, all of which have cyber-physical interfaces. These authors also suggest governance systems and policy recommendations which might be harnessed to address the lab-focused concerns they raise.

Other recent publications also highlight the complexity of the enterprise we are terming “cyberbiosecurity” and concerns over security, robustness, and resiliency. These include:

  • security of personal genomic data when foreign companies purchase all or part of a U.S. company or are contracted for genomic or health care data services, which provides access to sensitive personal information[8];
  • the continuing vulnerability of electronic health records[11] and health care systems[12][13][14];
  • imposing control over DNA sequencing through DNA-encoded malware[15];
  • synthetic biology supply chain vulnerabilities[16];
  • cybersecurity compromise of large industrial biopharma companies[17][18]; and
  • high-level studies which are systematically examining U.S. biodefense programs and capabilities.[19][20]

The darkweb/darknet[21][22][23] could be included as it interfaces with dual use life science endeavors and biopharma research, development, intellectual property, and products, compromising the integrity of critical life science and health cyber-supported technologies and infrastructures. Because of the reliance on bioinformatics, the security of synthetic DNA could also be included, as well.[24] Clearly, this rapidly expanding galaxy does needs a universally accepted definition, common terms of reference, and defined boundaries and structure for best value, ordered evolution, and impact.

Adding another dimension: Cyberbiosecurity systems analysis of a biomanufacturing facility

Now we add another dimension to cyberbiosecurity and take an approach that we posit that should be incorporated with other aspects discussed earlier. The biopharma industry itself has its own substantial equities and investments in the research, development, production, and sale of vaccines, therapeutics, and prophylactics for the global market. The U.S. Government has substantial investments in the development and production of critical vaccines and biotherapeutics for both civilian and military purposes. Concomitantly, experts are increasingly recognizing that biomanufacturing itself is potentially vulnerable to unwanted or illicit activities which could result in damaging outcomes. These could include the theft of intellectual property, disruption of the supply chain, manipulation of the bioprocess development and bioproduction, cyberattacks on key information technology components and cyberphysical interfaces, the corruption of critical data, and manipulation of security systems and infrastructure upon which secure and safe facility operations are dependent. Our sponsor was not interested in generalizations or esoteric approximations about the security vulnerabilities of a biomanufacturing facility but wanted a comprehensive, detailed, actionable analysis.

Thus, we undertook an in-depth, multidimensional analysis of an existing biomanufacturing facility to identify and project security gaps and vulnerabilities, make recommendations with respect to addressing those identified and projected, and set the stage for more specific and comprehensive measures to be undertaken, whether they exist or have to be developed and validated. The systems analysis approach used was designed to assess the state of security at present, determine what an acceptable state of security would be, and provide guidance and recommendations to take the facility from its current state to the desired state.

The bioprocess development/bioproduction facility used as the “test bed” for this analysis designs, develops, and produces clinical trial quantities of protein-based biotherapeutics and the associated documentation for commercial and government clients. If the outputs from this “test bed” meet client expectations and the client receives government approval, the client scales up production and the product is marketed. This facility was studied as a system, consisting of four key, interrelated subsystems: end-to-end bioprocess development/biomanufacturing; the supply chain; the supporting information systems infrastructure and cyber-physical interfaces with bioprocess development and biomanufacturing; and facility infrastructure, including its relationship to the facility's host infrastructure. The systems analysis was a phased process with project management methods applied. The facility or any of its components or operations were not compromised, corrupted, or altered during this project in any manner or form. Rather, it was studied thoroughly yet benignly. The analysis included human factors and “downstream” considerations, as well.

The systems aspects of a biomanufacturing facility which are potentially vulnerable to security threats and the solutions required are summarized in Figure 1.


Fig1 Murch FrontBioengBiotech2019 6.jpg

Fig. 1 A systems view of protecting a biomanufacturing facility. For each defensive set identified, multiple threats and impacts were identified and potentially more than one pathway or technique could be used by an adversary to achieve their objectives. GMP, Good Manufacturing Practice; IT, Information Technology; IS, Information Systems; VPN, Virtual Private Network; IP, Intellectual Property.

Due to space limitations, we provide only a top-level view of the analysis. Key overarching findings include:

  • Vulnerabilities can exist across the entire system, from bioprocess development and GMP to supply chain, to cyber-physical and infrastructure; there are potentially more than one might anticipate a priori.
  • Successful exploitation of vulnerabilities can occur through passive and active means for passive and active purposes, depending upon adversaries' intentions, objectives, accesses, knowledge and resources, and outcomes sought.
  • Exploitation of some vulnerabilities require direct access to facilities or components; personnel and physical security aspects should not be overlooked.
  • Adversaries can use combinations and sequences of methods and targeting, both subtle and not, to attempt to and achieve their objectives.
  • The operational capabilities of adversaries, not just technical, must be considered and accounted for in planning for and implementing security measures.

We emphasize that, while there are general principles that apply and observations that will derive from such analyses, the analysis design and execution and the resulting solution set, should be tailored on a facility-by-facility basis. We note that the defensive areas noted may not be singular, but rather require combinations of defensive approaches and techniques to be identified and implemented to ensure optimal security robustness.

What is considerably important from this analysis is that a rigorous study of a facility such as this can result in the identification and characterization of discrete vulnerabilities, gaps, shortfalls, and opportunities for which readily-available solutions can be implemented, or otherwise can be developed, tested, and implemented. We did not conduct detailed studies regarding how genomics can be compromised as it relates to biomanufacturing because we were directed not to, but we are well aware of plausible scenarios and what the effects could be.

Our analysis demonstrates that biomanufacturing facilities can benefit from comprehensive, multidisciplinary analyses to identify security vulnerabilities, leading to solutions to mitigate or address them. This, in turn, raises the prospect of the development and validation of a set of methods or protocols would be in order which could be used by facility staff or external service providers to shore up individual facilities, from do-it-yourself to large biopharma. Walking this out, guidelines or standards could be developed, established, and accepted to ensure consistency and quality of the analyses conducted, the credentials of the personnel doing so, and the quality and effectiveness of measures undertaken. While sophisticated adversaries could design and execute sophisticated attacks, it is likely in many instances that relatively straightforward methods and practices could raise the bar considerably to reduce risk. Lastly, combining analyses of this sort could be used as a basis for informed investments in research, development, testing, and evaluation for solutions to the most worrisome current and future threats.

Moving cyberbiosecurity forward

Many other critical cyber-enabled life science and biomedical technologies, systems, and applications naturally lend themselves to inclusion within cyberbiosecurity. These include, but are not limited to, personalized genomics, medical and fitness technologies, 3-D printing of critical personalized medical devices, and medical laboratory and surgical robotics. A more comprehensive system is warranted. Cyberbiosecurity could be expanded to include cyber-bio systems within agriculture and farm-to-table food production, processing and distribution systems, and within natural resource and environmental management. Direct and ordered engagements of the pertinent sectors of the life sciences, biosecurity, and cyber-cybersecurity communities should occur. Academia, industry, government, and non-profits (including policy, regulatory, and legal experts) need to begin to learn to communicate with and educate each other, harmoniously identify and develop priorities and opportunities, and specify “next steps.” A major opportunity exists right now to propose a unified structure and common vernacular. Lastly, while definition and assemblage of cyberbiosecurity is occurring, national or international strategies should be pursued to harmonize the emerging enterprise and foster measurable value, success, and sustainability.

Acknowledgements

Author contributions

RM: Co-originator of cyberbiosecurity concept; Lead author, responsible for structure, content and figure; responsible for considering, incorporating co-author contributions and suggested modifications; responsible for final version. WS: Co-originator of Safeguarding the Bioeconomy concept and campaign; contributed or reviewed and edited paragraphs related to the AAAS-FBI-UNICRI Big Data report, the US National Academies workshops and the relevant FBI programs and initiatives. SR: Co-originator of cyberbiosecurity concept; overall quality assurance and readability reviews and modifications; contributions to section on cyberbiosecurity applied to biomanufacturing. WB: Contributions to section on cyberbiosecurity applied to biomanufacturing; reviews to ensure content quality and readability of paper. JP: Co-originator of cyberbiosecurity concept; review and critique of manuscript to ensure complementarity and alignment with first paper published on cyberbiosecurity (biotechnology focus) in another journal (he is the lead author; RM, SR, and WB are co-authors).

Funding

The funding for the systems analysis of the Test Bed Facility was provided through a contract award (Contract FA4600-12-D-9000, Task Order 0065) from the U. S. Department of Defense, United States Strategic Command to the National Strategic Research Institute, University of Nebraska.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

References

  1. 1.0 1.1 1.2 Board on Chemical Sciences and Technology; Board on Life Sciences (2014). Meeting Recap: Workshop - Convergence: Safeguarding Technology in the Bioeconomy. The National Academies of Sciences, Engineering, and Medicine. 
  2. 2.0 2.1 FBI WMD Directorate, American Association for the Advancement of Science, United Nations Interregional Crime and Justice Research Institute (2014). "National and Transnational Security Implications of Big Data in the Life Sciences" (PDF). AAAS. http://www.aaas.org/sites/default/files/AAAS-FBI-UNICRI_Big_Data_Report_111014.pdf. 
  3. 3.0 3.1 Board on Chemical Sciences and Technology (2015) (PDF). Meeting Recap: Safeguarding the Bioeconomy: Applications and Implications of Emerging Science. The National Academies of Sciences, Engineering, and Medicine. https://www.ehidc.org/sites/default/files/resources/files/Safeguarding%20the%20Bioeconomy_II_Recap%20Final%20090815.pdf. 
  4. 4.0 4.1 Board on Life Sciences and Board on Chemical Sciences and Technology (2016) (PDF). Meeting Recap: Safeguarding the Bioeconomy III: Securing Life Sciences Data. The National Academies of Sciences, Engineering, and Medicine. https://www.ibpforum.org/sites/default/files/Safeguarding_the_Bioeconomy_III_Recap.pdf. 
  5. Peccoud, J.; Gallegos, J.E.; Murch, R. et al. (2018). "Cyberbiosecurity: From Naive Trust to Risk Awareness". Trends in Biotechnology 36 (1): 4–7. doi:10.1016/j.tibtech.2017.10.012. PMID 29224719. 
  6. 6.0 6.1 Kozminski, K.G.; Drubin, D.G. (2015). "Biosecurity in the age of Big Data: A conversation with the FBI". Molecular Biology of the Cell 26 (22): 3894–97. doi:10.1091/mbc.E14-01-0027. PMC PMC4710219. PMID 26543195. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4710219. 
  7. 7.0 7.1 Pauwels, E.; Vidyarthi, A. (29 March 2016). "How Our Unhealthy Cybersecurity Infrastructure Is Hurting Biotechnology". Wilson Briefs. Wilson Center. https://www.wilsoncenter.org/publication/how-our-unhealthy-cybersecurity-infrastructure-hurting-biotechnology. 
  8. 8.0 8.1 Pauwels, E.; Vidyarthi, A. (19 November 2017). "Who Will Own The Secrets In Our Genes? A U.S. – China Race in Artificial Intelligence and Genomics". Wilson Briefs. Wilson Center. https://www.wilsoncenter.org/publication/who-will-own-the-secrets-our-genes-us-china-race-artificial-intelligence-and-genomics. 
  9. 9.0 9.1 Pauwels, E.; Dunlap, E. (7 September 2017). "The Intelligent and Connected Bio-Labs of the Future: Promise and Peril in the Fourth Industrial Revolution". Wilson Briefs. Wilson Center. https://www.wilsoncenter.org/publication/the-intelligent-and-connected-bio-labs-the-future-promise-and-peril-the-fourth. 
  10. 10.0 10.1 You, E.H. (16 March 2017). "Safeguarding the Bioeconomy: U.S. Opportunities and Challenges - Testimony for the U.S.-China Economic and Security Review Commission" (PDF). https://www.ehidc.org/sites/default/files/resources/files/Ed_You_Testimony_USCC.pdf. 
  11. Weise, E. (5 February 2015). "Millions of Anthem customers alerted to hack". USA Today. https://www.usatoday.com/story/tech/2015/02/05/anthem-health-care-computer-security-breach/22917635/. 
  12. Hackett, R. (17 July 2015). "UCLA Health System data breach may affect millions". Fortune. http://fortune.com/2015/07/17/ucla-health-system-data-breach/. 
  13. Winton, R. (18 February 2016). "Hollywood hospital pays $17,000 in bitcoin to hackers; FBI investigating". Los Angeles Times. https://www.latimes.com/business/technology/la-me-ln-hollywood-hospital-bitcoin-20160217-story.html. 
  14. Griffin, A. (12 May 2017). "NHS hack: Cyber attack takes 16 hospitals offline as patients are turned away". Independent. https://www.independent.co.uk/news/uk/home-news/nhs-cyber-attack-hack-hospitals-16-patients-turned-away-wanna-decryptor-a7733196.html. 
  15. Greenberg, A. (10 August 2017). "Biohackers encoded malware in a strand of DNA". Wired. https://www.wired.com/story/malware-dna-hack/. 
  16. Frazar, S.L.; Hund, G.E.; Bonheyo, G.T. et al. (2017). "Defining the synthetic biology supply chain". Health Security 15 (4): 392-400. doi:10.1089/hs.2016.0083. PMID 28767286. 
  17. Collier, K. (30 June 2017). "Merck IT systems still crippled in Petya's aftermath". CyberScoop. https://www.cyberscoop.com/merck-petya-ransomware-ukraine/. 
  18. Shaban, H.; Nakashima, E. (27 June 2017). "Pharmaceutical giant rocked by ransomware attack". The Washington Post. https://www.washingtonpost.com/news/the-switch/wp/2017/06/27/pharmaceutical-giant-rocked-by-ransomware-attack. 
  19. Blue Ribbon Study Panel on Biodefense (28 October 2015). "A National Blueprint for Biodefense: Leadership and Major Reform Needed To Optimize Efforts". Hudson Institute. https://www.hudson.org/research/11824-a-national-blueprint-for-biodefense-leadership-and-major-reform-needed-to-optimize-efforts. 
  20. Center for the Study of Weapons of Mass Destruction (2017). Emergence and Convergence Deep Dive: the Age of Genomic Data. Technical Executive Summary. National Defense University. 
  21. Beckett, A. (25 November 2009). "The dark side of the internet". The Guardian. https://www.theguardian.com/technology/2009/nov/26/dark-side-internet-freenet. 
  22. INTERPOL (24 February 2015). "Pharmaceutical Crime on the Darknet" (PDF). https://www.gwern.net/docs/sr/2015-interpol-pharmaceuticals.pdf. 
  23. Langewiesche, W. (11 September 2016). "Welcome to the dark net, a wilderness where invisible world wars are fought and hackers roam free". Vanity Fair. https://www.vanityfair.com/news/2016/09/welcome-to-the-dark-net?verso=true. 
  24. Adam, L.; Kozar, M.; Letort, G. et al. (2011). "Strengths and limitations of the federal guidance on synthetic DNA". Nature Biotechnology 29 (3): 208-10. doi:10.1038/nbt.1802. PMID 21390018. 

Notes

This presentation is faithful to the original, with only a few minor changes to presentation, grammar, and punctuation. In some cases important information was missing from the references, and that information was added.