Difference between revisions of "Journal:Handling metadata in a neurophysiology laboratory"

From LIMSWiki
Jump to navigationJump to search
(Created stub. Saving and adding more.)
 
(Saving and editing more.)
Line 29: Line 29:
==Introduction==
==Introduction==
Technological advances in neuroscience during the last decades have led to methods that nowadays enable researchers to simultaneously record the activity from tens to hundreds of neurons simultaneously, in vitro or in vivo, using a variety of techniques<ref name="NicolelisMulti02">{{cite journal |title=Multielectrode recordings: The next steps |journal=Current Opinion in Neurobiology |author=Nicolelis, M.A.; Ribeiro, S. |volume=12 |issue=5 |pages=602–6 |year=2002 |doi=10.1016/S0959-4388(02)00374-4 |pmid=12367642}}</ref><ref name="VerkhratskyFrom06">{{cite journal |title=From Galvani to patch clamp: The development of electrophysiology |journal=Pflügers Archiv |author=Verkhratsky, A.; Krishtal, O.A.; Petersen, O.H. |volume=453 |issue=3 |pages=233–47 |year=2006 |doi=10.1007/s00424-006-0169-z |pmid=17072639}}</ref><ref name="ObienRevealing15">{{cite journal |title=Revealing neuronal function through microelectrode array recordings |journal=Frontiers in Neuroscience |author=Obien, M.E.; Deligkaris, K.; Bullmann, T. et al. |volume=8  |pages=423 |year=2015 |doi=10.3389/fnins.2014.00423 |pmid=25610364 |pmc=PMC4285113}}</ref> in combination with sophisticated stimulation methods, such as optogenetics.<ref name="DeisserothEngineer13">{{cite journal |title=Engineering approaches to illuminating brain structure and dynamics |journal=Neuron |author=Deisseroth, K.; Schnitzer, M.J. |volume=80 |issue=3 |pages=568–77 |year=2013 |doi=10.1016/j.neuron.2013.10.032 |pmid=24183010 |pmc=PMC5731466}}</ref><ref name="MiyamotoTheFiber16">{{cite journal |title=The fiber-optic imaging and manipulation of neural activity during animal behavior |journal=Neuroscience Research |author=Miyamoto, D.; Murayama, M. |volume=103 |pages=1–9 |year=2016 |doi=10.1016/j.neures.2015.09.004 |pmid=26427958}}</ref> In addition, recordings can be performed in parallel from multiple brain areas, together with behavioral measures such as eye or limb movements.<ref name="MaldonadoSynchro08">{{cite journal |title=Synchronization of neuronal responses in primary visual cortex of monkeys viewing natural images |journal=Journal of Neurophysiology |author=Maldonado, P.; Babul, C.; Singer, W. et al. |volume=100 |issue=3 |pages=1523-32 |year=2008 |doi=10.1152/jn.00076.2008 |pmid=18562559}}</ref><ref name="VargasIrwinDecoding10">{{cite journal |title=Decoding complete reach and grasp actions from local primary motor cortex populations |journal=Journal of Neuroscience |author=Vargas-Irwin, C.E.; Shakhnarovich, G.; Yadollahpour, P. et al. |volume=30 |issue=29 |pages=9659-69 |year=2010 |doi=10.1523/JNEUROSCI.5443-09.2010 |pmid=20660249 |pmc=PMC2921895}}</ref> Such recordings enable researchers to study network interactions and cross-area coupling, and to relate neuronal processing to the behavioral performance of the subject.<ref name="BerenyiLarge14">{{cite journal |title=Large-scale, high-density (up to 512 channels) recording of local circuits in behaving animals |journal=Journal of Neurophysiology |author=Berényi, A.; Somogyvári, Z.; Nagy, A.J. et al. |volume=111 |issue=5 |pages=1132-49 |year=2014 |doi=10.1152/jn.00785.2013 |pmid=24353300 |pmc=PMC3949233}}</ref><ref name="LewisRecording15">{{cite journal |title=Recording of brain activity across spatial scales |journal=Current Opinion in Neurobiology |author=Lewis, C.M.; Bosman, C.A.; Fries, P. et al. |volume=32 |pages=68–77 |year=2015 |doi=10.1016/j.conb.2014.12.007 |pmid=25544724}}</ref><ref name="LismanTheChall15">{{cite journal |title=The challenge of understanding the brain: Where we stand in 2015 |journal=Neuron |author=Lisman, J. |volume=86 |issue=4 |pages=864-882 |year=2015 |doi=10.1016/j.neuron.2015.03.032 |pmid=25996132 |pmc=PMC4441769}}</ref> These approaches lead to increasingly complex experimental designs that are difficult to parameterize, e.g., due to multidimensional characterization of natural stimuli<ref name="GeislerVisual08">{{cite journal |title=Visual perception and the statistical properties of natural scenes |journal=Annual Review of Psychology |author=Geisler, W.S. |volume=59 |pages=167–92 |year=2008 |doi=10.1146/annurev.psych.58.110405.085632 |pmid=17705683}}</ref> or high-dimensional movement parameters for almost freely behaving subjects.<ref name="SchwarzChronic14">{{cite journal |title=Chronic, wireless recordings of large-scale brain activity in freely moving rhesus monkeys |journal=Nature Methods |author=Schwarz, D.A.; Lebedev, M.A.; Hanson, T.L. et al. |volume=11 |issue=6 |pages=670-6 |year=2014 |doi=10.1038/nmeth.2936 |pmid=24776634 |pmc=PMC4161037}}</ref> It is a serious challenge for researchers to keep track of the overwhelming amount of metadata generated at each experimental step and to precisely extract all the information relevant for data analysis and interpretation of results. Various aspects such as the parametrization of the experimental task, filter settings and sampling rates of the setup, the quality of the recorded data, broken electrodes, preprocessing steps (e.g., spike sorting), or the condition of the subject need to be considered. Nevertheless, the organization of these metadata is of utmost importance for conducting research in a reproducible manner, i.e., the ability to faithfully reproduce the experimental procedures and subsequent analysis steps.<ref name="LaineReproduc07">{{cite journal |title=Reproducible research: Moving toward research the public can really trust |journal=Annals of Internal Medicine |author=Laine, C.; Goodman, S.N.; Griswold, M.E.; Sox, H.C. |volume=146 |issue=6 |pages=450–3 |year=2007 |doi=10.7326/0003-4819-146-6-200703200-00154 |pmid=17339612}}</ref><ref name="PengReproduc11">{{cite journal |title=Reproducible research in computational science |journal=Science |author=Peng, R.D. |volume=334 |issue=6060 |pages=1226-7 |year=2011 |doi=10.1126/science.1213847 |pmid=22144613 |pmc=PMC3383002}}</ref><ref name="TomaselloMethod11">{{cite journal |title=Methodological challenges in the study of primate cognition |journal=Science |author=Tomasello, M.; Call, J. |volume=334 |issue=6060 |pages=1227-8 |year=2011 |doi=10.1126/science.1213443 |pmid=22144614}}</ref> Moreover, detailed knowledge of the complete recording and analysis processes is crucial for the correct interpretation of results, and is a minimal requirement to enable researchers to verify published results and build their own research on the previous findings.
Technological advances in neuroscience during the last decades have led to methods that nowadays enable researchers to simultaneously record the activity from tens to hundreds of neurons simultaneously, in vitro or in vivo, using a variety of techniques<ref name="NicolelisMulti02">{{cite journal |title=Multielectrode recordings: The next steps |journal=Current Opinion in Neurobiology |author=Nicolelis, M.A.; Ribeiro, S. |volume=12 |issue=5 |pages=602–6 |year=2002 |doi=10.1016/S0959-4388(02)00374-4 |pmid=12367642}}</ref><ref name="VerkhratskyFrom06">{{cite journal |title=From Galvani to patch clamp: The development of electrophysiology |journal=Pflügers Archiv |author=Verkhratsky, A.; Krishtal, O.A.; Petersen, O.H. |volume=453 |issue=3 |pages=233–47 |year=2006 |doi=10.1007/s00424-006-0169-z |pmid=17072639}}</ref><ref name="ObienRevealing15">{{cite journal |title=Revealing neuronal function through microelectrode array recordings |journal=Frontiers in Neuroscience |author=Obien, M.E.; Deligkaris, K.; Bullmann, T. et al. |volume=8  |pages=423 |year=2015 |doi=10.3389/fnins.2014.00423 |pmid=25610364 |pmc=PMC4285113}}</ref> in combination with sophisticated stimulation methods, such as optogenetics.<ref name="DeisserothEngineer13">{{cite journal |title=Engineering approaches to illuminating brain structure and dynamics |journal=Neuron |author=Deisseroth, K.; Schnitzer, M.J. |volume=80 |issue=3 |pages=568–77 |year=2013 |doi=10.1016/j.neuron.2013.10.032 |pmid=24183010 |pmc=PMC5731466}}</ref><ref name="MiyamotoTheFiber16">{{cite journal |title=The fiber-optic imaging and manipulation of neural activity during animal behavior |journal=Neuroscience Research |author=Miyamoto, D.; Murayama, M. |volume=103 |pages=1–9 |year=2016 |doi=10.1016/j.neures.2015.09.004 |pmid=26427958}}</ref> In addition, recordings can be performed in parallel from multiple brain areas, together with behavioral measures such as eye or limb movements.<ref name="MaldonadoSynchro08">{{cite journal |title=Synchronization of neuronal responses in primary visual cortex of monkeys viewing natural images |journal=Journal of Neurophysiology |author=Maldonado, P.; Babul, C.; Singer, W. et al. |volume=100 |issue=3 |pages=1523-32 |year=2008 |doi=10.1152/jn.00076.2008 |pmid=18562559}}</ref><ref name="VargasIrwinDecoding10">{{cite journal |title=Decoding complete reach and grasp actions from local primary motor cortex populations |journal=Journal of Neuroscience |author=Vargas-Irwin, C.E.; Shakhnarovich, G.; Yadollahpour, P. et al. |volume=30 |issue=29 |pages=9659-69 |year=2010 |doi=10.1523/JNEUROSCI.5443-09.2010 |pmid=20660249 |pmc=PMC2921895}}</ref> Such recordings enable researchers to study network interactions and cross-area coupling, and to relate neuronal processing to the behavioral performance of the subject.<ref name="BerenyiLarge14">{{cite journal |title=Large-scale, high-density (up to 512 channels) recording of local circuits in behaving animals |journal=Journal of Neurophysiology |author=Berényi, A.; Somogyvári, Z.; Nagy, A.J. et al. |volume=111 |issue=5 |pages=1132-49 |year=2014 |doi=10.1152/jn.00785.2013 |pmid=24353300 |pmc=PMC3949233}}</ref><ref name="LewisRecording15">{{cite journal |title=Recording of brain activity across spatial scales |journal=Current Opinion in Neurobiology |author=Lewis, C.M.; Bosman, C.A.; Fries, P. et al. |volume=32 |pages=68–77 |year=2015 |doi=10.1016/j.conb.2014.12.007 |pmid=25544724}}</ref><ref name="LismanTheChall15">{{cite journal |title=The challenge of understanding the brain: Where we stand in 2015 |journal=Neuron |author=Lisman, J. |volume=86 |issue=4 |pages=864-882 |year=2015 |doi=10.1016/j.neuron.2015.03.032 |pmid=25996132 |pmc=PMC4441769}}</ref> These approaches lead to increasingly complex experimental designs that are difficult to parameterize, e.g., due to multidimensional characterization of natural stimuli<ref name="GeislerVisual08">{{cite journal |title=Visual perception and the statistical properties of natural scenes |journal=Annual Review of Psychology |author=Geisler, W.S. |volume=59 |pages=167–92 |year=2008 |doi=10.1146/annurev.psych.58.110405.085632 |pmid=17705683}}</ref> or high-dimensional movement parameters for almost freely behaving subjects.<ref name="SchwarzChronic14">{{cite journal |title=Chronic, wireless recordings of large-scale brain activity in freely moving rhesus monkeys |journal=Nature Methods |author=Schwarz, D.A.; Lebedev, M.A.; Hanson, T.L. et al. |volume=11 |issue=6 |pages=670-6 |year=2014 |doi=10.1038/nmeth.2936 |pmid=24776634 |pmc=PMC4161037}}</ref> It is a serious challenge for researchers to keep track of the overwhelming amount of metadata generated at each experimental step and to precisely extract all the information relevant for data analysis and interpretation of results. Various aspects such as the parametrization of the experimental task, filter settings and sampling rates of the setup, the quality of the recorded data, broken electrodes, preprocessing steps (e.g., spike sorting), or the condition of the subject need to be considered. Nevertheless, the organization of these metadata is of utmost importance for conducting research in a reproducible manner, i.e., the ability to faithfully reproduce the experimental procedures and subsequent analysis steps.<ref name="LaineReproduc07">{{cite journal |title=Reproducible research: Moving toward research the public can really trust |journal=Annals of Internal Medicine |author=Laine, C.; Goodman, S.N.; Griswold, M.E.; Sox, H.C. |volume=146 |issue=6 |pages=450–3 |year=2007 |doi=10.7326/0003-4819-146-6-200703200-00154 |pmid=17339612}}</ref><ref name="PengReproduc11">{{cite journal |title=Reproducible research in computational science |journal=Science |author=Peng, R.D. |volume=334 |issue=6060 |pages=1226-7 |year=2011 |doi=10.1126/science.1213847 |pmid=22144613 |pmc=PMC3383002}}</ref><ref name="TomaselloMethod11">{{cite journal |title=Methodological challenges in the study of primate cognition |journal=Science |author=Tomasello, M.; Call, J. |volume=334 |issue=6060 |pages=1227-8 |year=2011 |doi=10.1126/science.1213443 |pmid=22144614}}</ref> Moreover, detailed knowledge of the complete recording and analysis processes is crucial for the correct interpretation of results, and is a minimal requirement to enable researchers to verify published results and build their own research on the previous findings.
To achieve reproducibility, experimenters have typically developed their own lab procedures and practices for performing experiments and their documentation. Within the lab, crucial [[information]] about the experiment is often transmitted by personal communication, through handwritten [[laboratory notebook]]s, or implicitly by trained experimental procedures. However, at least when it comes to data sharing across labs, essential information is often missed in the exchange.<ref name="HinesSorting14">{{cite journal |title=Sorting out the FACS: A devil in the details |journal=Cell Reports |author=Hines, W.C.; Su, Y.; Kuhn, I. et al. |volume=6 |issue=5 |pages=779-81 |year=2014 |doi=10.1016/j.celrep.2014.02.021 |pmid=24630040}}</ref><ref name="OSCPsychology15">{{cite journal |title=Estimating the reproducibility of psychological science |journal=Science |author=Open Science Collaboration |volume=349 |issue=6251 |pages=aac4716 |year=2015 |doi=10.1126/science.aac4716 |pmid=26315443}}</ref> Moreover, if collaborating groups have different scientific backgrounds, for example experimenters and theoreticians, implicit domain-specific knowledge is often not communicated or is communicated in an ambiguous fashion that leads to misunderstandings. To avoid such scenarios, the general principle should be to keep as much information about an experiment as possible from the beginning on, even if information seems to be trivial or irrelevant at the time. Furthermore, one should annotate the data with these metadata in a clear and concise fashion.
In order to provide metadata in an organized, easily accessible, but also machine-readable way, Grewe ''et al.''<ref name="GreweABottom11">{{cite journal |title=A Bottom-up Approach to Data Annotation in Neurophysiology |journal=Frontiers in Neuroinformatics |author=Grewe, J.; Wachtler, T.; Benda, J. |volume=5 |pages=16 |year=2011 |doi=10.3389/fninf.2011.00016 |pmid=21941477 |pmc=PMC3171061}}</ref> introduced odML (open metadata Markup Language) as a simple file format in analogy to SBML in systems biology<ref name="HuckaTheSys03">{{cite journal |title=The systems biology markup language (SBML): A medium for representation and exchange of biochemical network models |journal=Bioinformatics |author=Hucka, M.; Finney, A.; Sauro, H.M. et al. |volume=19 |issue=4 |pages=524–31 |year=2003 |doi=10.1093/bioinformatics/btg015 |pmid=12611808}}</ref>, or NeuroML in neuroscientific simulation studies.<ref name="GleesonNeuroML10">{{cite journal |title=NeuroML: a language for describing data driven models of neurons and networks with a high degree of biological detail |journal=PLoS Computational Biology |author=Gleeson, P.; Crook, S.; Cannon, R.C. et al. |volume=6 |issue=6 |pages=e1000815 |year=2010 |doi=10.1371/journal.pcbi.1000815 |pmid=20585541 |pmc=PMC2887454}}</ref><ref name="CrookCreating12">{{cite journal |title=Creating, documenting and sharing network models |journal=Network |author=Crook, S.M.; Bednar, J.A.; Berger, S. et al. |volume=23 |issue=4 |pages=131-49 |year=2012 |doi=10.3109/0954898X.2012.722743 |pmid=22994683}}</ref> However, lacking to date is a detailed investigation on how to incorporate metadata management in the daily lab routine in terms of (i) organizing the metadata in a comprehensive collection, (ii) practically gathering and entering the metadata, and (iii) profiting from the resulting comprehensive metadata collection in the process of analyzing the data. Here we address these points, both conceptually and practically, in the context of a complex behavioral experiment that involves neuronal recordings from a large number of electrodes that yield massively parallel spike and local field potential (LFP) data.<ref name="RiehleMapping13">{{cite journal |title=Mapping the spatio-temporal structure of motor cortical LFP and spiking activities during reach-to-grasp movements |journal=Frontiers in Neural Circuits |author=Riehle, A.; Wirtssohn, S.; Grün, S.; Brochier, T. |volume=7 |pages=48 |year=2013 |doi=10.3389/fncir.2013.00048 |pmid=23543888 |pmc=PMC3608913}}</ref> To illustrate how to organize a comprehensive metadata collection (i), we introduce in the next section the concept of metadata and demonstrate the rich diversity of metadata that arise in the context of the example experiment. To demonstrate why the effort of creating a comprehensive metadata collection is time well spent, we next describe five use cases that summarize where the access to metadata becomes relevant when working with the data (iii). Afterwards we provide detailed guidelines and assistance on how to create, structure, and hierarchically organize comprehensive metadata collections (i and ii). Complementing these guidelines, we provide in the supplementary material section a thorough practical introduction on how to embed a metadata management tool, such as the odML library, into the experimental and analysis workflow. Finally, we close by critically contrasting the importance of proper metadata handling against its difficulties and deriving future challenges.


==References==
==References==

Revision as of 22:25, 20 December 2017

Full article title Handling metadata in a neurophysiology laboratory
Journal Frontiers in Neuroinformatics
Author(s) Zehl, Lyuba; Jaillet, Florent; Stoewer, Adrian; Grewe, Jan; Sobolev, Andrey; Wachtler, Thomas; Brochier, Thomas G.; Riehle, Alexa; Denker, Michael; Grün, Sonja
Author affiliation(s) Jülich Research Centre, Aix-Marseille Université, Ludwig-Maximilians-Universität München, Eberhard-Karls-Universität Tübingen, Aachen University
Primary contact Email: l dot zehl at fz-juelich dot de
Editors Luo, Qingming
Year published 2016
Volume and issue 10
Page(s) 26
DOI 10.3389/fninf.2016.00026
ISSN 1662-5196
Distribution license Creative Commons Attribution 4.0 International
Website https://www.frontiersin.org/articles/10.3389/fninf.2016.00026/full
Download https://www.frontiersin.org/articles/10.3389/fninf.2016.00026/pdf (PDF)

Abstract

To date, non-reproducibility of neurophysiological research is a matter of intense discussion in the scientific community. A crucial component to enhance reproducibility is to comprehensively collect and store metadata, that is, all information about the experiment, the data, and the applied preprocessing steps on the data, such that they can be accessed and shared in a consistent and simple manner. However, the complexity of experiments, the highly specialized analysis workflows, and a lack of knowledge on how to make use of supporting software tools often overburden researchers to perform such a detailed documentation. For this reason, the collected metadata are often incomplete, incomprehensible for outsiders, or ambiguous. Based on our research experience in dealing with diverse datasets, we here provide conceptual and technical guidance to overcome the challenges associated with the collection, organization, and storage of metadata in a neurophysiology laboratory. Through the concrete example of managing the metadata of a complex experiment that yields multi-channel recordings from monkeys performing a behavioral motor task, we practically demonstrate the implementation of these approaches and solutions, with the intention that they may be generalized to other projects. Moreover, we detail five use cases that demonstrate the resulting benefits of constructing a well-organized metadata collection when processing or analyzing the recorded data, in particular when these are shared between laboratories in a modern scientific collaboration. Finally, we suggest an adaptable workflow to accumulate, structure and store metadata from different sources using, by way of example, the odML metadata framework.

Introduction

Technological advances in neuroscience during the last decades have led to methods that nowadays enable researchers to simultaneously record the activity from tens to hundreds of neurons simultaneously, in vitro or in vivo, using a variety of techniques[1][2][3] in combination with sophisticated stimulation methods, such as optogenetics.[4][5] In addition, recordings can be performed in parallel from multiple brain areas, together with behavioral measures such as eye or limb movements.[6][7] Such recordings enable researchers to study network interactions and cross-area coupling, and to relate neuronal processing to the behavioral performance of the subject.[8][9][10] These approaches lead to increasingly complex experimental designs that are difficult to parameterize, e.g., due to multidimensional characterization of natural stimuli[11] or high-dimensional movement parameters for almost freely behaving subjects.[12] It is a serious challenge for researchers to keep track of the overwhelming amount of metadata generated at each experimental step and to precisely extract all the information relevant for data analysis and interpretation of results. Various aspects such as the parametrization of the experimental task, filter settings and sampling rates of the setup, the quality of the recorded data, broken electrodes, preprocessing steps (e.g., spike sorting), or the condition of the subject need to be considered. Nevertheless, the organization of these metadata is of utmost importance for conducting research in a reproducible manner, i.e., the ability to faithfully reproduce the experimental procedures and subsequent analysis steps.[13][14][15] Moreover, detailed knowledge of the complete recording and analysis processes is crucial for the correct interpretation of results, and is a minimal requirement to enable researchers to verify published results and build their own research on the previous findings.

To achieve reproducibility, experimenters have typically developed their own lab procedures and practices for performing experiments and their documentation. Within the lab, crucial information about the experiment is often transmitted by personal communication, through handwritten laboratory notebooks, or implicitly by trained experimental procedures. However, at least when it comes to data sharing across labs, essential information is often missed in the exchange.[16][17] Moreover, if collaborating groups have different scientific backgrounds, for example experimenters and theoreticians, implicit domain-specific knowledge is often not communicated or is communicated in an ambiguous fashion that leads to misunderstandings. To avoid such scenarios, the general principle should be to keep as much information about an experiment as possible from the beginning on, even if information seems to be trivial or irrelevant at the time. Furthermore, one should annotate the data with these metadata in a clear and concise fashion.

In order to provide metadata in an organized, easily accessible, but also machine-readable way, Grewe et al.[18] introduced odML (open metadata Markup Language) as a simple file format in analogy to SBML in systems biology[19], or NeuroML in neuroscientific simulation studies.[20][21] However, lacking to date is a detailed investigation on how to incorporate metadata management in the daily lab routine in terms of (i) organizing the metadata in a comprehensive collection, (ii) practically gathering and entering the metadata, and (iii) profiting from the resulting comprehensive metadata collection in the process of analyzing the data. Here we address these points, both conceptually and practically, in the context of a complex behavioral experiment that involves neuronal recordings from a large number of electrodes that yield massively parallel spike and local field potential (LFP) data.[22] To illustrate how to organize a comprehensive metadata collection (i), we introduce in the next section the concept of metadata and demonstrate the rich diversity of metadata that arise in the context of the example experiment. To demonstrate why the effort of creating a comprehensive metadata collection is time well spent, we next describe five use cases that summarize where the access to metadata becomes relevant when working with the data (iii). Afterwards we provide detailed guidelines and assistance on how to create, structure, and hierarchically organize comprehensive metadata collections (i and ii). Complementing these guidelines, we provide in the supplementary material section a thorough practical introduction on how to embed a metadata management tool, such as the odML library, into the experimental and analysis workflow. Finally, we close by critically contrasting the importance of proper metadata handling against its difficulties and deriving future challenges.

References

  1. Nicolelis, M.A.; Ribeiro, S. (2002). "Multielectrode recordings: The next steps". Current Opinion in Neurobiology 12 (5): 602–6. doi:10.1016/S0959-4388(02)00374-4. PMID 12367642. 
  2. Verkhratsky, A.; Krishtal, O.A.; Petersen, O.H. (2006). "From Galvani to patch clamp: The development of electrophysiology". Pflügers Archiv 453 (3): 233–47. doi:10.1007/s00424-006-0169-z. PMID 17072639. 
  3. Obien, M.E.; Deligkaris, K.; Bullmann, T. et al. (2015). "Revealing neuronal function through microelectrode array recordings". Frontiers in Neuroscience 8: 423. doi:10.3389/fnins.2014.00423. PMC PMC4285113. PMID 25610364. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4285113. 
  4. Deisseroth, K.; Schnitzer, M.J. (2013). "Engineering approaches to illuminating brain structure and dynamics". Neuron 80 (3): 568–77. doi:10.1016/j.neuron.2013.10.032. PMC PMC5731466. PMID 24183010. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5731466. 
  5. Miyamoto, D.; Murayama, M. (2016). "The fiber-optic imaging and manipulation of neural activity during animal behavior". Neuroscience Research 103: 1–9. doi:10.1016/j.neures.2015.09.004. PMID 26427958. 
  6. Maldonado, P.; Babul, C.; Singer, W. et al. (2008). "Synchronization of neuronal responses in primary visual cortex of monkeys viewing natural images". Journal of Neurophysiology 100 (3): 1523-32. doi:10.1152/jn.00076.2008. PMID 18562559. 
  7. Vargas-Irwin, C.E.; Shakhnarovich, G.; Yadollahpour, P. et al. (2010). "Decoding complete reach and grasp actions from local primary motor cortex populations". Journal of Neuroscience 30 (29): 9659-69. doi:10.1523/JNEUROSCI.5443-09.2010. PMC PMC2921895. PMID 20660249. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2921895. 
  8. Berényi, A.; Somogyvári, Z.; Nagy, A.J. et al. (2014). "Large-scale, high-density (up to 512 channels) recording of local circuits in behaving animals". Journal of Neurophysiology 111 (5): 1132-49. doi:10.1152/jn.00785.2013. PMC PMC3949233. PMID 24353300. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3949233. 
  9. Lewis, C.M.; Bosman, C.A.; Fries, P. et al. (2015). "Recording of brain activity across spatial scales". Current Opinion in Neurobiology 32: 68–77. doi:10.1016/j.conb.2014.12.007. PMID 25544724. 
  10. Lisman, J. (2015). "The challenge of understanding the brain: Where we stand in 2015". Neuron 86 (4): 864-882. doi:10.1016/j.neuron.2015.03.032. PMC PMC4441769. PMID 25996132. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4441769. 
  11. Geisler, W.S. (2008). "Visual perception and the statistical properties of natural scenes". Annual Review of Psychology 59: 167–92. doi:10.1146/annurev.psych.58.110405.085632. PMID 17705683. 
  12. Schwarz, D.A.; Lebedev, M.A.; Hanson, T.L. et al. (2014). "Chronic, wireless recordings of large-scale brain activity in freely moving rhesus monkeys". Nature Methods 11 (6): 670-6. doi:10.1038/nmeth.2936. PMC PMC4161037. PMID 24776634. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4161037. 
  13. Laine, C.; Goodman, S.N.; Griswold, M.E.; Sox, H.C. (2007). "Reproducible research: Moving toward research the public can really trust". Annals of Internal Medicine 146 (6): 450–3. doi:10.7326/0003-4819-146-6-200703200-00154. PMID 17339612. 
  14. Peng, R.D. (2011). "Reproducible research in computational science". Science 334 (6060): 1226-7. doi:10.1126/science.1213847. PMC PMC3383002. PMID 22144613. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3383002. 
  15. Tomasello, M.; Call, J. (2011). "Methodological challenges in the study of primate cognition". Science 334 (6060): 1227-8. doi:10.1126/science.1213443. PMID 22144614. 
  16. Hines, W.C.; Su, Y.; Kuhn, I. et al. (2014). "Sorting out the FACS: A devil in the details". Cell Reports 6 (5): 779-81. doi:10.1016/j.celrep.2014.02.021. PMID 24630040. 
  17. Open Science Collaboration (2015). "Estimating the reproducibility of psychological science". Science 349 (6251): aac4716. doi:10.1126/science.aac4716. PMID 26315443. 
  18. Grewe, J.; Wachtler, T.; Benda, J. (2011). "A Bottom-up Approach to Data Annotation in Neurophysiology". Frontiers in Neuroinformatics 5: 16. doi:10.3389/fninf.2011.00016. PMC PMC3171061. PMID 21941477. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3171061. 
  19. Hucka, M.; Finney, A.; Sauro, H.M. et al. (2003). "The systems biology markup language (SBML): A medium for representation and exchange of biochemical network models". Bioinformatics 19 (4): 524–31. doi:10.1093/bioinformatics/btg015. PMID 12611808. 
  20. Gleeson, P.; Crook, S.; Cannon, R.C. et al. (2010). "NeuroML: a language for describing data driven models of neurons and networks with a high degree of biological detail". PLoS Computational Biology 6 (6): e1000815. doi:10.1371/journal.pcbi.1000815. PMC PMC2887454. PMID 20585541. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2887454. 
  21. Crook, S.M.; Bednar, J.A.; Berger, S. et al. (2012). "Creating, documenting and sharing network models". Network 23 (4): 131-49. doi:10.3109/0954898X.2012.722743. PMID 22994683. 
  22. Riehle, A.; Wirtssohn, S.; Grün, S.; Brochier, T. (2013). "Mapping the spatio-temporal structure of motor cortical LFP and spiking activities during reach-to-grasp movements". Frontiers in Neural Circuits 7: 48. doi:10.3389/fncir.2013.00048. PMC PMC3608913. PMID 23543888. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3608913. 

Notes

This presentation is faithful to the original, with only a few minor changes to presentation. In some cases important information was missing from the references, and that information was added. The original article lists references alphabetically, but this version — by design — lists them in order of appearance.