データ駆動科学と機械学習

http://art.ist.hokudai.ac.jp/~takigawa/index_ja.html

x!y<latexit sha1_base64="kaK7x7wFN3GZpVpMGGOJaQQWYoA=">AAACBXicbVC7SgNBFL0bXzG+opY2E4NgFXZF0DJoYxnBPCC7htnJJBkys7PMzIrLktpfsNXeTmz9Dlu/xEmyhSYeuHA4517O5YQxZ9q47pdTWFldW98obpa2tnd298r7By0tE0Vok0guVSfEmnIW0aZhhtNOrCgWIaftcHw99dsPVGkmozuTxjQQeBixASPYWOneD0X2OPErvpF+Je2Vq27NnQEtEy8nVcjR6JW//b4kiaCRIRxr3fXc2AQZVoYRTiclP9E0xmSMh7RraYQF1UE2+3qCTqzSRwOp7EQGzdTfFxkWWqcitJsCm5Fe9Kbiv14oFpLN4DLIWBQnhkZkHjxIODISTStBfaYoMTy1BBPF7O+IjLDCxNjiSrYUb7GCZdI6q3luzbs9r9av8nqKcATHcAoeXEAdbqABTSCg4Ble4NV5ct6cd+djvlpw8ptD+APn8wfECJj5</latexit><latexit sha1_base64="kaK7x7wFN3GZpVpMGGOJaQQWYoA=">AAACBXicbVC7SgNBFL0bXzG+opY2E4NgFXZF0DJoYxnBPCC7htnJJBkys7PMzIrLktpfsNXeTmz9Dlu/xEmyhSYeuHA4517O5YQxZ9q47pdTWFldW98obpa2tnd298r7By0tE0Vok0guVSfEmnIW0aZhhtNOrCgWIaftcHw99dsPVGkmozuTxjQQeBixASPYWOneD0X2OPErvpF+Je2Vq27NnQEtEy8nVcjR6JW//b4kiaCRIRxr3fXc2AQZVoYRTiclP9E0xmSMh7RraYQF1UE2+3qCTqzSRwOp7EQGzdTfFxkWWqcitJsCm5Fe9Kbiv14oFpLN4DLIWBQnhkZkHjxIODISTStBfaYoMTy1BBPF7O+IjLDCxNjiSrYUb7GCZdI6q3luzbs9r9av8nqKcATHcAoeXEAdbqABTSCg4Ble4NV5ct6cd+djvlpw8ptD+APn8wfECJj5</latexit><latexit sha1_base64="kaK7x7wFN3GZpVpMGGOJaQQWYoA=">AAACBXicbVC7SgNBFL0bXzG+opY2E4NgFXZF0DJoYxnBPCC7htnJJBkys7PMzIrLktpfsNXeTmz9Dlu/xEmyhSYeuHA4517O5YQxZ9q47pdTWFldW98obpa2tnd298r7By0tE0Vok0guVSfEmnIW0aZhhtNOrCgWIaftcHw99dsPVGkmozuTxjQQeBixASPYWOneD0X2OPErvpF+Je2Vq27NnQEtEy8nVcjR6JW//b4kiaCRIRxr3fXc2AQZVoYRTiclP9E0xmSMh7RraYQF1UE2+3qCTqzSRwOp7EQGzdTfFxkWWqcitJsCm5Fe9Kbiv14oFpLN4DLIWBQnhkZkHjxIODISTStBfaYoMTy1BBPF7O+IjLDCxNjiSrYUb7GCZdI6q3luzbs9r9av8nqKcATHcAoeXEAdbqABTSCg4Ble4NV5ct6cd+djvlpw8ptD+APn8wfECJj5</latexit><latexit sha1_base64="kaK7x7wFN3GZpVpMGGOJaQQWYoA=">AAACBXicbVC7SgNBFL0bXzG+opY2E4NgFXZF0DJoYxnBPCC7htnJJBkys7PMzIrLktpfsNXeTmz9Dlu/xEmyhSYeuHA4517O5YQxZ9q47pdTWFldW98obpa2tnd298r7By0tE0Vok0guVSfEmnIW0aZhhtNOrCgWIaftcHw99dsPVGkmozuTxjQQeBixASPYWOneD0X2OPErvpF+Je2Vq27NnQEtEy8nVcjR6JW//b4kiaCRIRxr3fXc2AQZVoYRTiclP9E0xmSMh7RraYQF1UE2+3qCTqzSRwOp7EQGzdTfFxkWWqcitJsCm5Fe9Kbiv14oFpLN4DLIWBQnhkZkHjxIODISTStBfaYoMTy1BBPF7O+IjLDCxNjiSrYUb7GCZdI6q3luzbs9r9av8nqKcATHcAoeXEAdbqABTSCg4Ble4NV5ct6cd+djvlpw8ptD+APn8wfECJj5</latexit>
x<latexit sha1_base64="BLB8K/n7QYAsE73zsDEUiBvCSV8=">AAAB/XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXRjcsK9gHtUDJppo1NMkOSEctQ/AW3uncnbv0Wt36JaTsLbT1w4XDOvZzLCRPBjfW8L1RYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0wbXis7uw4YYEkA8UjTol1UrMbyuxx0itXvKo3A14mfk4qkKPeK393+zFNJVOWCmJMx/cSG2REW04Fm5S6qWEJoSMyYB1HFZHMBNns2wk+cUofR7F2oyyeqb8vMiKNGcvQbUpih2bRm4r/eqFcSLbRZZBxlaSWKToPjlKBbYynVeA+14xaMXaEUM3d75gOiSbUusJKrhR/sYJl0jyr+l7Vvz2v1K7yeopwBMdwCj5cQA1uoA4NoHAPz/ACr+gJvaF39DFfLaD85hD+AH3+ADzJlfc=</latexit><latexit sha1_base64="BLB8K/n7QYAsE73zsDEUiBvCSV8=">AAAB/XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXRjcsK9gHtUDJppo1NMkOSEctQ/AW3uncnbv0Wt36JaTsLbT1w4XDOvZzLCRPBjfW8L1RYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0wbXis7uw4YYEkA8UjTol1UrMbyuxx0itXvKo3A14mfk4qkKPeK393+zFNJVOWCmJMx/cSG2REW04Fm5S6qWEJoSMyYB1HFZHMBNns2wk+cUofR7F2oyyeqb8vMiKNGcvQbUpih2bRm4r/eqFcSLbRZZBxlaSWKToPjlKBbYynVeA+14xaMXaEUM3d75gOiSbUusJKrhR/sYJl0jyr+l7Vvz2v1K7yeopwBMdwCj5cQA1uoA4NoHAPz/ACr+gJvaF39DFfLaD85hD+AH3+ADzJlfc=</latexit><latexit sha1_base64="BLB8K/n7QYAsE73zsDEUiBvCSV8=">AAAB/XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXRjcsK9gHtUDJppo1NMkOSEctQ/AW3uncnbv0Wt36JaTsLbT1w4XDOvZzLCRPBjfW8L1RYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0wbXis7uw4YYEkA8UjTol1UrMbyuxx0itXvKo3A14mfk4qkKPeK393+zFNJVOWCmJMx/cSG2REW04Fm5S6qWEJoSMyYB1HFZHMBNns2wk+cUofR7F2oyyeqb8vMiKNGcvQbUpih2bRm4r/eqFcSLbRZZBxlaSWKToPjlKBbYynVeA+14xaMXaEUM3d75gOiSbUusJKrhR/sYJl0jyr+l7Vvz2v1K7yeopwBMdwCj5cQA1uoA4NoHAPz/ACr+gJvaF39DFfLaD85hD+AH3+ADzJlfc=</latexit><latexit sha1_base64="BLB8K/n7QYAsE73zsDEUiBvCSV8=">AAAB/XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXRjcsK9gHtUDJppo1NMkOSEctQ/AW3uncnbv0Wt36JaTsLbT1w4XDOvZzLCRPBjfW8L1RYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0wbXis7uw4YYEkA8UjTol1UrMbyuxx0itXvKo3A14mfk4qkKPeK393+zFNJVOWCmJMx/cSG2REW04Fm5S6qWEJoSMyYB1HFZHMBNns2wk+cUofR7F2oyyeqb8vMiKNGcvQbUpih2bRm4r/eqFcSLbRZZBxlaSWKToPjlKBbYynVeA+14xaMXaEUM3d75gOiSbUusJKrhR/sYJl0jyr+l7Vvz2v1K7yeopwBMdwCj5cQA1uoA4NoHAPz/ACr+gJvaF39DFfLaD85hD+AH3+ADzJlfc=</latexit>

David Hand
All models are wrong, but some are useful
(George Box)
Theory-driven models can be wrong
But data-driven models cannot be wrong

David Hand
(George Box)
or right

David Hand
(George Box)
or right
Data-driven are not trying to describe an underlying reality.
so they could be poor or useless, but not wrong
But are merely intended to be useful

With enough data, the numbers
speak for themselves.
Chris Anderson (2008)
cf.

REVIEW
Inverse molecular design using
machine learning: Generative models
for matter engineering
Benjamin Sanchez-Lengeling1
and Alán Aspuru-Guzik2,3,4
*
The discovery of new materials can bring enormous societal and technological progress. In this
context, exploring completely the large space of potential materials is computationally
intractable. Here, we review methods for achieving inverse design, which aims to discover
tailored materials from the starting point of a particular desired functionality. Recent advances
from the rapidly growing field of artificial intelligence, mostly from the subfield of machine
learning, have resulted in a fertile exchange of ideas, where approaches to inverse molecular
design are being proposed and employed at a rapid pace. Among these, deep generative models
have been applied to numerous classes of materials: rational design of prospective drugs,
synthetic routes to organic compounds, and optimization of photovoltaics and redox flow
batteries, as well as a variety of other solid-state materials.
M
any of the challenges of the 21st century
(1), from personalized health care to
energy production and storage, share a
common theme: materials are part of
the solution (2). In some cases, the solu-
tions to these challenges are fundamentally
limited by the physics and chemistry of a ma-
terial, such as the relationship of a materials
bandgap to the thermodynamic limits for the
generation of solar energy (3).
Several important materials discoveries arose
by chance or through a process of trial and error.
For example, vulcanized rubber was prepared in
the 19th century from random mixtures of com-
pounds, based on the observation that heating
with additives such as sulfur improved the
rubber’s durability. At the molecular level, in-
dividual polymer chains cross-linked, forming
bridges that enhanced the macroscopic mechan-
ical properties (4). Other notable examples in
this vein include Teflon, anesthesia, Vaseline,
Perkin’s mauve, and penicillin. Furthermore,
these materials come from common chemical
compounds found in nature. Potential drugs
either were prepared by synthesis in a chem-
ical laboratory or were isolated from plants,
soil bacteria, or fungus. For example, up until
2014, 49% of small-molecule cancer drugs were
natural products or their derivatives (5).
In the future, disruptive advances in the dis-
covery of matter could instead come from unex-
plored regions of the set of all possible molecular
and solid-state compounds, known as chemical
space (6, 7). One of the largest collections of
molecules, the chemical space project (8), has
mapped 166.4 billion molecules that contain at
most 17 heavy atoms. For pharmacologically rele-
vant small molecules, the number of structures is
estimated to be on the order of 1060
(9). Adding
consideration of the hierarchy of scale from sub-
nanometer to microscopic and mesoscopic fur-
ther complicates exploration of chemical space
in its entirety (10). Therefore, any global strategy
for covering this space might seem impossible.
Simulation offers one way of probing this
space without experimentation. The physics
and chemistry of these molecules are governed
by quantum mechanics, which can be solved via
the Schrödinger equation to arrive at their ex-
act properties. In practice, approximations are
used to lower computational time at the cost of
accuracy.
Although theory enjoys enormous progress,
now routinely modeling molecules, clusters, and
perfect as well as defect-laden periodic solids, the
size of chemical space is still overwhelming, and
smart navigation is required. For this purpose,
machine learning (ML), deep learning (DL), and
artificial intelligence (AI) have a potential role
to play because their computational strategies
automatically improve through experience (11).
In the context of materials, ML techniques are
often used for property prediction, seeking to
learn a function that maps a molecular material
to the property of choice. Deep generative models
are a special class of DL methods that seek to
model the underlying probability distribution of
both structure and property and relate them in a
nonlinear way. By exploiting patterns in massive
datasets, these models can distill average and
salient features that characterize molecules (12, 13).
Inverse design is a component of a more
complex materials discovery process. The time
scale for deployment of new technologies, from
discovery in a laboratory to a commercial pro-
duct, historically, is 15 to 20 years (14). The pro-
cess (Fig. 1) conventionally involves the following
steps: (i) generate a new or improved material
concept and simulate its potential suitability; (ii)
synthesize the material; (iii) incorporate the ma-
terial into a device or system; and (iv) characterize
and measure the desired properties. This cycle
generates feedback to repeat, improve, and re-
fine future cycles of discovery. Each step can take
up to several years.
In the era of matter engineering, scientists
seek to accelerate these cycles, reducing the
FRONTIERS IN COMPUTATION
Sanchez-Lengeling et al., Science 361, 360–365 (2018) 27 July 2018 1 of 6
1
Department of Chemistry and Chemical Biology, Harvard
University 12 Oxford Street, Cambridge, MA 02138, USA.
2
Department of Chemistry and Department of Computer
Science, University of Toronto, Toronto Ontario, M5S 3H6,
Canada. 3
Vector Institute for Artificial Intelligence, Toronto,
Ontario M5S 1M1, Canada. 4
Canadian Institute for Advanced
Research (CIFAR) Senior Fellow Toronto, Ontario M5S 1M1,
Canada.
*Corresponding author. Email: aspuru@utoronto.ca
Fig. 1. Schematic comparison of material discovery paradigms. The current paradigm is
outlined at left and exemplified in the center with organic redox flow batteries (92). A closed-loop
paradigm is outlined at right. Closing the loop requires incorporating inverse design, smart software
(93), AI/ML, embedded systems, and robotics (87) into an integrated ecosystem.
IMAGE:ADAPTEDBYK.HOLOSKI
onJuly26,2018http://science.sciencemag.org/Downloadedfrom
REVIEW https://doi.org/10.1038/s41586-018-0337-2
Machine learning for molecular and
materials science
Keith T. Butler1
, Daniel W. Davies2
, Hugh Cartwright3
, Olexandr Isayev4
* & Aron Walsh5,6
*
Here we summarize recent progress in machine learning for the chemical sciences. We outline machine-learning
techniques that are suitable for addressing research questions in this domain, as well as future directions for the field.
We envisage a future in which the design, synthesis, characterization and application of molecules and materials is
accelerated by artificial intelligence.
T
he Schrödinger equation provides a powerful structure–
property relationship for molecules and materials. For a given
spatial arrangement of chemical elements, the distribution of
electrons and a wide range of physical responses can be described. The
development of quantum mechanics provided a rigorous theoretical
foundationforthechemicalbond.In1929,PaulDiracfamouslyproclaimed
that the underlying physical laws for the whole of chemistry are “completely
known”1
. John Pople, realizing the importance of rapidly developing
computer technologies, created a program—Gaussian 70—that could
perform ab initio calculations: predicting the behaviour, for molecules
of modest size, purely from the fundamental laws of physics2
. In the 1960s,
the Quantum Chemistry Program Exchange brought quantum chemistry
to the masses in the form of useful practical tools3
. Suddenly, experi-
mentalists with little or no theoretical training could perform quantum
calculations too. Using modern algorithms and supercomputers,
systems containing thousands of interacting ions and electrons can now
be described using approximations to the physical laws that govern the
world on the atomic scale4–6
.
The field of computational chemistry has become increasingly pre-
dictive in the twenty-first century, with activity in applications as wide
ranging as catalyst development for greenhouse gas conversion, materials
discovery for energy harvesting and storage, and computer-assisted drug
design7
. The modern chemical-simulation toolkit allows the properties
of a compound to be anticipated (with reasonable accuracy) before it has
been made in the laboratory. High-throughput computational screening
has become routine, giving scientists the ability to calculate the properties
of thousands of compounds as part of a single study. In particular, den-
sity functional theory (DFT)8,9
, now a mature technique for calculating
the structure and behaviour of solids10
, has enabled the development of
extensive databases that cover the calculated properties of known and
hypothetical systems, including organic and inorganic crystals, single
molecules and metal alloys11–13
.
The emergence of contemporary artificial-intelligence methods has
the potential to substantially alter and enhance the role of computers in
science and engineering. The combination of big data and artificial intel-
ligence has been referred to as both the “fourth paradigm of science”14
and the “fourth industrial revolution”15
, and the number of applications
in the chemical domain is growing at an astounding rate. A subfield of
artificial intelligence that has evolved rapidly in recent years is machine
learning. At the heart of machine-learning applications lie statistical algo-
rithms whose performance, much like that of a researcher, improves with
training. There is a growing infrastructure of machine-learning tools for
generating, testing and refining scientific models. Such techniques are
suitable for addressing complex problems that involve massive combi-
natorial spaces or nonlinear processes, which conventional procedures
either cannot solve or can tackle only at great computational cost.
As the machinery for artificial intelligence and machine learning
matures, important advances are being made not only by those in main-
stream artificial-intelligence research, but also by experts in other fields
(domain experts) who adopt these approaches for their own purposes. As
we detail in Box 1, the resources and tools that facilitate the application
of machine-learning techniques mean that the barrier to entry is lower
than ever.
In the rest of this Review, we discuss progress in the application of
machine learning to address challenges in molecular and materials
research. We review the basics of machine-learning approaches, iden-
tify areas in which existing methods have the potential to accelerate
research and consider the developments that are required to enable more
wide-ranging impacts.
Nuts and bolts of machine learning
With machine learning, given enough data and a rule-discovery algo-
rithm, a computer has the ability to determine all known physical laws
(and potentially those that are currently unknown) without human
input. In traditional computational approaches, the computer is little
more than a calculator, employing a hard-coded algorithm provided
by a human expert. By contrast, machine-learning approaches learn
the rules that underlie a dataset by assessing a portion of that data
and building a model to make predictions. We consider the basic steps
involved in the construction of a model, as illustrated in Fig. 1; this
constitutes a blueprint of the generic workflow that is required for the
successful application of machine learning in a materials-discovery
process.
Data collection
Machine learning comprises models that learn from existing (train-
ing) data. Data may require initial preprocessing, during which miss-
ing or spurious elements are identified and handled. For example, the
Inorganic Crystal Structure Database (ICSD) currently contains more
than 190,000 entries, which have been checked for technical mistakes
but are still subject to human and measurement errors. Identifying
and removing such errors is essential to avoid machine-learning
algorithms being misled. There is a growing public concern about
the lack of reproducibility and error propagation of experimental data
1
ISIS Facility, Rutherford Appleton Laboratory, Harwell Campus, Harwell, UK. 2
Department of Chemistry, University of Bath, Bath, UK. 3
Department of Chemistry, Oxford University, Oxford, UK.
4
Eshelman School of Pharmacy, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA. 5
Department of Materials Science and Engineering, Yonsei University, Seoul, South Korea.
6
Department of Materials, Imperial College London, London, UK. *e-mail: olexandr@olexandrisayev.com; a.walsh@imperial.ac.uk
2 6 J U L Y 2 0 1 8 | V O L 5 5 9 | N A T U R E | 5 4 7
© 2018 Springer Nature Limited. All rights reserved.
DNA to be sequences into distinct pieces,
parcel out the detailed work of sequencing,
and then reassemble these independent ef-
forts at the end. It is not quite so simple in the
world of genome semantics.
Despite the differences between genome se-
quencing and genetic network discovery, there
are clear parallels that are illustrated in Table 1.
In genome sequencing, a physical map is useful
to provide scaffolding for assembling the fin-
ished sequence. In the case of a genetic regula-
tory network, a graphical model can play the
same role. A graphical model can represent a
high-level view of interconnectivity and help
isolate modules that can be studied indepen-
dently. Like contigs in a genomic sequencing
project, low-level functional models can ex-
plore the detailed behavior of a module of genes
in a manner that is consistent with the higher
level graphical model of the system. With stan-
dardized nomenclature and compatible model-
ing techniques, independent functional models
can be assembled into a complete model of the
cell under study.
To enable this process, there will need to
be standardized forms for model representa-
tion. At present, there are many different
modeling technologies in use, and although
models can be easily placed into a database,
they are not useful out of the context of their
specific modeling package. The need for a
standardized way of communicating compu-
tational descriptions of biological systems ex-
tends to the literature. Entire conferences
have been established to explore ways of
mining the biology literature to extract se-
mantic information in computational form.
Going forward, as a community we need
to come to consensus on how to represent
what we know about biology in computa-
tional form as well as in words. The key to
postgenomic biology will be the computa-
tional assembly of our collective knowl-
edge into a cohesive picture of cellular and
organism function. With such a comprehen-
sive model, we will be able to explore new
types of conservation between organisms
and make great strides toward new thera-
peutics that function on well-characterized
pathways.
References
1. S. K. Kim et al., Science 293 , 2087 (2001).
2. A. Hartemink et al., paper presented at the Pacific
Symposium on Biocomputing 2000, Oahu, Hawaii, 4
to 9 January 2000.
3. D. Pe’er et al., paper presented at the 9th Conference
on Intelligent Systems in Molecular Biology (ISMB),
Copenhagen, Denmark, 21 to 25 July 2001.
4. H. McAdams, A. Arkin, Proc. Natl. Acad. Sci. U.S.A.
94 , 814 ( 1997 ).
5. A. J. Hartemink, thesis, Massachusetts Institute of
Technology, Cambridge (2001).
V I E W P O I N T
Machine Learning for Science: State of the
Art and Future Prospects
Eric Mjolsness* and Dennis DeCoste
Recent advances in machine learning methods, along with successful
applications across a wide variety of fields such as planetary science and
bioinformatics, promise powerful new tools for practicing scientists. This
viewpoint highlights some useful characteristics of modern machine learn-
ing methods and their relevance to scientific applications. We conclude
with some speculations on near-term progress and promising directions.
Machine learning (ML) (1) is the study of
computer algorithms capable of learning to im-
prove their performance of a task on the basis of
their own previous experience. The field is
closely related to pattern recognition and statis-
tical inference. As an engineering field, ML has
become steadily more mathematical and more
successful in applications over the past 20
years. Learning approaches such as data clus-
tering, neural network classifiers, and nonlinear
regression have found surprisingly wide appli-
cation in the practice of engineering, business,
and science. A generalized version of the stan-
dard Hidden Markov Models of ML practice
have been used for ab initio prediction of gene
structures in genomic DNA (2). The predictions
correlate surprisingly well with subsequent
gene expression analysis (3). Postgenomic bi-
ology prominently features large-scale gene ex-
pression data analyzed by clustering methods
(4), a standard topic in unsupervised learning.
Many other examples can be given of learning
and pattern recognition applications in science.
Where will this trend lead? We believe it will
lead to appropriate, partial automation of every
element of scientific method, from hypothesis
generation to model construction to decisive
experimentation. Thus, ML has the potential to
amplify every aspect of a working scientist’s
progress to understanding. It will also, for better
or worse, endow intelligent computer systems
with some of the general analytic power of
scientific thinking.
Machine Learning at Every Stage of
the Scientific Process
Each scientific field has its own version of the
scientific process. But the cycle of observing,
creating hypotheses, testing by decisive exper-
iment or observation, and iteratively building
up comprehensive testable models or theories is
shared across disciplines. For each stage of this
abstracted scientific process, there are relevant
developments in ML, statistical inference, and
pattern recognition that will lead to semiauto-
matic support tools of unknown but potentially
broad applicability.
Increasingly, the early elements of scientific
method—observation and hypothesis genera-
tion—face high data volumes, high data acqui-
sition rates, or requirements for objective anal-
ysis that cannot be handled by human percep-
tion alone. This has been the situation in exper-
imental particle physics for decades. There
automatic pattern recognition for significant
events is well developed, including Hough
transforms, which are foundational in pattern
recognition. A recent example is event analysis
for Cherenkov detectors (8) used in neutrino
oscillation experiments. Microscope imagery in
cell biology, pathology, petrology, and other
fields has led to image-processing specialties.
So has remote sensing from Earth-observing
satellites, such as the newly operational Terra
spacecraft with its ASTER (a multispectral
thermal radiometer), MISR (multiangle imag-
ing spectral radiometer), MODIS (imaging
Machine Learning Systems Group, Jet Propulsion Lab-
oratory/California Institute of Technology, Pasadena,
CA, 91109, USA.
*To whom correspondence should be addressed. E-
mail: mjolsness@jpl.nasa.gov
Table 1. Parallels between genome sequencing
and genetic network discovery.
Genome
sequencing
Genome semantics
Physical maps Graphical model
Contigs Low-level functional
models
Contig
reassembly
Module assembly
Finished genome
sequence
Comprehensive model
www.sciencemag.org SCIENCE VOL 293 14 SEPTEMBER 2001 2051
C O M P U T E R S A N D S C I E N C E
onAugust29,2018http://science.sciencemag.org/Downloadedfrom
Nature, 559 
pp. 547–555 (2018)
Science, 293
pp. 2051-2055 (2001)
Science, 361
pp. 360-365 (2018)
Science is changing, the tools of science are changing. And that
requires different approaches. (Erich Bloch, 1925-2016)

CH3
N
H3C
H
NS
N
O
CH3
N
OH
x ˆyˆy = f✓(x)

N
NH
OO
HH
H
H H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
O
O
O
O
O
O
Cl
H
H
H
H
H
HH
H
H
H
H
H
H
H
H
H
H
Br
Br O P
O
O Br
Br
O
Br
Br
H
H
H
H
H
H
H
H
H
H
HH
H
HH
N
S
N
N
H
H
H
H
H
H
H
H
H
H
H
H
H
H
H
O
N
O
O
H
H
H
O
O
H
H
N
O
O
Cl
ClCl
H
H
H
H
H
H H
N
O
O
H
H
H
H
H
H
H H
H
N
O
O
H
H
H
H
H
H
H
N
H
N
O
O
N
O
O
H
H
H
H
H
H
H
H
N
CH3
O
O
H
N Cl
Cl
Cl
Cl
Cl
H3C
O O
O
O
O
O
H3C
CH3
CH2
O
HN
O
O
NH
CH3
HO
OH
CH3
N
O
O
CH3
N
N
H
N
H
H3C
N
H3C
H3C
NH
O
N
O
NO
CH3
O N
NH2
O
CH3
Br
CH3
N
H3C
H
NS
N
O
CH3
N
OH
CH3
CH3N
N
N
CH3H3C
H2N NH2
H
OH
O
HO
CH3
H
H
O
CH3
H
O
OH3C HH
H
O
H3C
S
CH3
O
H
H
O
CH3
CH3
OO
HO
H3CH
HO
F
H
O
H3C
NH2
O
N
HO
HO
O
H
H
O
O
OH3C
O
O
O
CH3
O
CH3
HO
CH3
H
O
O
CH3
H
H
N
H
N O
H3C
O
O
O

“ ”
J’aime la
musique I love music
CH3
N
H3C
H
NS
N
O
CH3
N
OH

ML modelx<latexit sha1_base64="7MVBSNoyoDWzaMSxoBhJg5NShso=">AAAB+HicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48JmAckS5id9CZDZnaXmVkxLvkCr3r3Jl79G69+iZNkD5pY0FBUdVNNBYng2rjul1NYW9/Y3Cpul3Z29/YPyodHLR2nimGTxSJWnYBqFDzCpuFGYCdRSGUgsB2Mb2d++wGV5nF0byYJ+pIOIx5yRo2VGo/9csWtunOQVeLlpAI56v3yd28Qs1RiZJigWnc9NzF+RpXhTOC01Es1JpSN6RC7lkZUovaz+aNTcmaVAQljZScyZK7+vsio1HoiA7spqRnpZW8m/usFcinZhNd+xqMkNRixRXCYCmJiMmuBDLhCZsTEEsoUt78TNqKKMmO7KtlSvOUKVknrouq5Va9xWand5PUU4QRO4Rw8uIIa3EEdmsAA4Rle4NV5ct6cd+djsVpw8ptj+APn8wcecJOi</latexit><latexit sha1_base64="7MVBSNoyoDWzaMSxoBhJg5NShso=">AAAB+HicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48JmAckS5id9CZDZnaXmVkxLvkCr3r3Jl79G69+iZNkD5pY0FBUdVNNBYng2rjul1NYW9/Y3Cpul3Z29/YPyodHLR2nimGTxSJWnYBqFDzCpuFGYCdRSGUgsB2Mb2d++wGV5nF0byYJ+pIOIx5yRo2VGo/9csWtunOQVeLlpAI56v3yd28Qs1RiZJigWnc9NzF+RpXhTOC01Es1JpSN6RC7lkZUovaz+aNTcmaVAQljZScyZK7+vsio1HoiA7spqRnpZW8m/usFcinZhNd+xqMkNRixRXCYCmJiMmuBDLhCZsTEEsoUt78TNqKKMmO7KtlSvOUKVknrouq5Va9xWand5PUU4QRO4Rw8uIIa3EEdmsAA4Rle4NV5ct6cd+djsVpw8ptj+APn8wcecJOi</latexit><latexit sha1_base64="7MVBSNoyoDWzaMSxoBhJg5NShso=">AAAB+HicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48JmAckS5id9CZDZnaXmVkxLvkCr3r3Jl79G69+iZNkD5pY0FBUdVNNBYng2rjul1NYW9/Y3Cpul3Z29/YPyodHLR2nimGTxSJWnYBqFDzCpuFGYCdRSGUgsB2Mb2d++wGV5nF0byYJ+pIOIx5yRo2VGo/9csWtunOQVeLlpAI56v3yd28Qs1RiZJigWnc9NzF+RpXhTOC01Es1JpSN6RC7lkZUovaz+aNTcmaVAQljZScyZK7+vsio1HoiA7spqRnpZW8m/usFcinZhNd+xqMkNRixRXCYCmJiMmuBDLhCZsTEEsoUt78TNqKKMmO7KtlSvOUKVknrouq5Va9xWand5PUU4QRO4Rw8uIIa3EEdmsAA4Rle4NV5ct6cd+djsVpw8ptj+APn8wcecJOi</latexit><latexit sha1_base64="7MVBSNoyoDWzaMSxoBhJg5NShso=">AAAB+HicbVDLSgNBEOyNrxhfUY9eBoPgKeyKoMegF48JmAckS5id9CZDZnaXmVkxLvkCr3r3Jl79G69+iZNkD5pY0FBUdVNNBYng2rjul1NYW9/Y3Cpul3Z29/YPyodHLR2nimGTxSJWnYBqFDzCpuFGYCdRSGUgsB2Mb2d++wGV5nF0byYJ+pIOIx5yRo2VGo/9csWtunOQVeLlpAI56v3yd28Qs1RiZJigWnc9NzF+RpXhTOC01Es1JpSN6RC7lkZUovaz+aNTcmaVAQljZScyZK7+vsio1HoiA7spqRnpZW8m/usFcinZhNd+xqMkNRixRXCYCmJiMmuBDLhCZsTEEsoUt78TNqKKMmO7KtlSvOUKVknrouq5Va9xWand5PUU4QRO4Rw8uIIa3EEdmsAA4Rle4NV5ct6cd+djsVpw8ptj+APn8wcecJOi</latexit>
y<latexit sha1_base64="IAkI/WnulkwRo6OBvH4htvdy0fo=">AAAB+HicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUL9gFtKJPpTTt0JgkzEyGGfoFb3bsTt/6NW7/EaZuFVg9cOJxzL+dygkRwbVz30ymtrW9sbpW3Kzu7e/sH1cOjjo5TxbDNYhGrXkA1Ch5h23AjsJcopDIQ2A2mt3O/+4BK8zi6N1mCvqTjiIecUWOlVjas1ty6uwD5S7yC1KBAc1j9GoxilkqMDBNU677nJsbPqTKcCZxVBqnGhLIpHWPf0ohK1H6+eHRGzqwyImGs7ESGLNSfFzmVWmcysJuSmole9ebiv14gV5JNeO3nPEpSgxFbBoepICYm8xbIiCtkRmSWUKa4/Z2wCVWUGdtVxZbirVbwl3Qu6p5b91qXtcZNUU8ZTuAUzsGDK2jAHTShDQwQnuAZXpxH59V5c96XqyWnuDmGX3A+vgEgBJOj</latexit><latexit sha1_base64="IAkI/WnulkwRo6OBvH4htvdy0fo=">AAAB+HicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUL9gFtKJPpTTt0JgkzEyGGfoFb3bsTt/6NW7/EaZuFVg9cOJxzL+dygkRwbVz30ymtrW9sbpW3Kzu7e/sH1cOjjo5TxbDNYhGrXkA1Ch5h23AjsJcopDIQ2A2mt3O/+4BK8zi6N1mCvqTjiIecUWOlVjas1ty6uwD5S7yC1KBAc1j9GoxilkqMDBNU677nJsbPqTKcCZxVBqnGhLIpHWPf0ohK1H6+eHRGzqwyImGs7ESGLNSfFzmVWmcysJuSmole9ebiv14gV5JNeO3nPEpSgxFbBoepICYm8xbIiCtkRmSWUKa4/Z2wCVWUGdtVxZbirVbwl3Qu6p5b91qXtcZNUU8ZTuAUzsGDK2jAHTShDQwQnuAZXpxH59V5c96XqyWnuDmGX3A+vgEgBJOj</latexit><latexit sha1_base64="IAkI/WnulkwRo6OBvH4htvdy0fo=">AAAB+HicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUL9gFtKJPpTTt0JgkzEyGGfoFb3bsTt/6NW7/EaZuFVg9cOJxzL+dygkRwbVz30ymtrW9sbpW3Kzu7e/sH1cOjjo5TxbDNYhGrXkA1Ch5h23AjsJcopDIQ2A2mt3O/+4BK8zi6N1mCvqTjiIecUWOlVjas1ty6uwD5S7yC1KBAc1j9GoxilkqMDBNU677nJsbPqTKcCZxVBqnGhLIpHWPf0ohK1H6+eHRGzqwyImGs7ESGLNSfFzmVWmcysJuSmole9ebiv14gV5JNeO3nPEpSgxFbBoepICYm8xbIiCtkRmSWUKa4/Z2wCVWUGdtVxZbirVbwl3Qu6p5b91qXtcZNUU8ZTuAUzsGDK2jAHTShDQwQnuAZXpxH59V5c96XqyWnuDmGX3A+vgEgBJOj</latexit><latexit sha1_base64="IAkI/WnulkwRo6OBvH4htvdy0fo=">AAAB+HicbVDLSsNAFL2pr1pfVZduBovgqiQi6LLoxmUL9gFtKJPpTTt0JgkzEyGGfoFb3bsTt/6NW7/EaZuFVg9cOJxzL+dygkRwbVz30ymtrW9sbpW3Kzu7e/sH1cOjjo5TxbDNYhGrXkA1Ch5h23AjsJcopDIQ2A2mt3O/+4BK8zi6N1mCvqTjiIecUWOlVjas1ty6uwD5S7yC1KBAc1j9GoxilkqMDBNU677nJsbPqTKcCZxVBqnGhLIpHWPf0ohK1H6+eHRGzqwyImGs7ESGLNSfFzmVWmcysJuSmole9ebiv14gV5JNeO3nPEpSgxFbBoepICYm8xbIiCtkRmSWUKa4/Z2wCVWUGdtVxZbirVbwl3Qu6p5b91qXtcZNUU8ZTuAUzsGDK2jAHTShDQwQnuAZXpxH59V5c96XqyWnuDmGX3A+vgEgBJOj</latexit>
Inputs Outputs

weight(g), height(cm)
5 6.25 7.5 8.75 10
90112.5135157.5180
●
●
●●
●
●
●
●
●●
weight(g)
height(cm)
Apple
Orange

5 6.25 7.5 8.75 10
90112.5135157.5180
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
weight(g)
height(cm)
Apple
Orange

5 6.25 7.5 8.75 10
90112.5135157.5180
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
weight(g)
height(cm)
Apple
Orange

5 6.25 7.5 8.75 10
90112.5135157.5180
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
● ●
weight(g)
height(cm)
Apple
Orange

5 6.25 7.5 8.75 10
90112.5135157.5180
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
● ●
●
weight(g)
height(cm)
Apple
Orange

5 6.25 7.5 8.75 10
90112.5135157.5180
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●●●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●● ●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
weight(g)
height(cm)
Apple
Orange

5 6.25 7.5 8.75 10
90112.5135157.5180
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●●●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●● ●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●● ●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
● ● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
weight(g)
height(cm)
Apple
Orange

5 6.25 7.5 8.75 10
90112.5135157.5180
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●●●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●● ●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●● ●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
● ● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●
● ●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
● ●
● ●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●● ●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●●
●
●
●
● ●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
● ●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●●
● ●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
● ●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
● ●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
● ●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
● ●
● ●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●● ●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●●
●
●
● ●●
● ●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●●
●●
● ●
●
●
●
● ●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●●
● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ● ●●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●● ●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●●
●
●●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
● ●●
●
●
● ●
●
●
●
● ●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●
●
●
● ●
●
●
● ●●
●
●
●
●
● ●
●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●● ●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●●
● ●
●●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
● ●●●
●●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
weight(g)
height(cm)
Apple
Orange

Fruit SorterInputs Outputs
weight(g)
height(cm)
Apple or Orange

{(x1, y1), (x2, y2), . . . , (xn, yn)}<latexit sha1_base64="7AHjj7VQQcP09rBzk8BZaX/foH0=">AAACrHichVE9T9tQFD24LQTagmkXpC4REZRIkXUdIJBOqF068hVASiLLdl6ChWNb9kvUNOIPVOrMwNRKHaqOHbrAxMIfYOAnVIwgsTBw7aRCDEmvZb9zz7vn+rx3rcB1Ikl0OaI8efpsdCw1PvH8xcvJKXX61U7kt0JblGzf9cM9y4yE63iiJB3pir0gFGbTcsWudfAh3t9tizByfG9bdgJRbZoNz6k7timZMtS3le7CJ0PPdQw9m2OUZ5TP5io1X0Zx7nHuZSuHhpohjQrLxUVKk7ZM+kqxyICosLqYT+sM4sigH+u++gcV1ODDRgtNCHiQjF2YiPgpQwchYK6KLnMhIyfZFzjEBGtbXCW4wmT2gL8Nzsp91uM87hklapv/4vIbsjKNObqgn3RN5/SL/tLdwF7dpEfspcOr1dOKwJj6MrN1+19Vk1eJ/QfVUM8SdawmXh32HiRMfAq7p29/Prreerc5152n73TF/r/RJZ3xCbz2jf1jQ2weD/FjsRe+MR7QvymkB4OdvKaTpm8sZdbe90eVwhvMYoHnsYI1fMQ6Stz/K37jBKeKpmwrZaXaK1VG+prXeBRK/R7iC53d</latexit><latexit sha1_base64="7AHjj7VQQcP09rBzk8BZaX/foH0=">AAACrHichVE9T9tQFD24LQTagmkXpC4REZRIkXUdIJBOqF068hVASiLLdl6ChWNb9kvUNOIPVOrMwNRKHaqOHbrAxMIfYOAnVIwgsTBw7aRCDEmvZb9zz7vn+rx3rcB1Ikl0OaI8efpsdCw1PvH8xcvJKXX61U7kt0JblGzf9cM9y4yE63iiJB3pir0gFGbTcsWudfAh3t9tizByfG9bdgJRbZoNz6k7timZMtS3le7CJ0PPdQw9m2OUZ5TP5io1X0Zx7nHuZSuHhpohjQrLxUVKk7ZM+kqxyICosLqYT+sM4sigH+u++gcV1ODDRgtNCHiQjF2YiPgpQwchYK6KLnMhIyfZFzjEBGtbXCW4wmT2gL8Nzsp91uM87hklapv/4vIbsjKNObqgn3RN5/SL/tLdwF7dpEfspcOr1dOKwJj6MrN1+19Vk1eJ/QfVUM8SdawmXh32HiRMfAq7p29/Prreerc5152n73TF/r/RJZ3xCbz2jf1jQ2weD/FjsRe+MR7QvymkB4OdvKaTpm8sZdbe90eVwhvMYoHnsYI1fMQ6Stz/K37jBKeKpmwrZaXaK1VG+prXeBRK/R7iC53d</latexit><latexit sha1_base64="7AHjj7VQQcP09rBzk8BZaX/foH0=">AAACrHichVE9T9tQFD24LQTagmkXpC4REZRIkXUdIJBOqF068hVASiLLdl6ChWNb9kvUNOIPVOrMwNRKHaqOHbrAxMIfYOAnVIwgsTBw7aRCDEmvZb9zz7vn+rx3rcB1Ikl0OaI8efpsdCw1PvH8xcvJKXX61U7kt0JblGzf9cM9y4yE63iiJB3pir0gFGbTcsWudfAh3t9tizByfG9bdgJRbZoNz6k7timZMtS3le7CJ0PPdQw9m2OUZ5TP5io1X0Zx7nHuZSuHhpohjQrLxUVKk7ZM+kqxyICosLqYT+sM4sigH+u++gcV1ODDRgtNCHiQjF2YiPgpQwchYK6KLnMhIyfZFzjEBGtbXCW4wmT2gL8Nzsp91uM87hklapv/4vIbsjKNObqgn3RN5/SL/tLdwF7dpEfspcOr1dOKwJj6MrN1+19Vk1eJ/QfVUM8SdawmXh32HiRMfAq7p29/Prreerc5152n73TF/r/RJZ3xCbz2jf1jQ2weD/FjsRe+MR7QvymkB4OdvKaTpm8sZdbe90eVwhvMYoHnsYI1fMQ6Stz/K37jBKeKpmwrZaXaK1VG+prXeBRK/R7iC53d</latexit><latexit sha1_base64="7AHjj7VQQcP09rBzk8BZaX/foH0=">AAACrHichVE9T9tQFD24LQTagmkXpC4REZRIkXUdIJBOqF068hVASiLLdl6ChWNb9kvUNOIPVOrMwNRKHaqOHbrAxMIfYOAnVIwgsTBw7aRCDEmvZb9zz7vn+rx3rcB1Ikl0OaI8efpsdCw1PvH8xcvJKXX61U7kt0JblGzf9cM9y4yE63iiJB3pir0gFGbTcsWudfAh3t9tizByfG9bdgJRbZoNz6k7timZMtS3le7CJ0PPdQw9m2OUZ5TP5io1X0Zx7nHuZSuHhpohjQrLxUVKk7ZM+kqxyICosLqYT+sM4sigH+u++gcV1ODDRgtNCHiQjF2YiPgpQwchYK6KLnMhIyfZFzjEBGtbXCW4wmT2gL8Nzsp91uM87hklapv/4vIbsjKNObqgn3RN5/SL/tLdwF7dpEfspcOr1dOKwJj6MrN1+19Vk1eJ/QfVUM8SdawmXh32HiRMfAq7p29/Prreerc5152n73TF/r/RJZ3xCbz2jf1jQ2weD/FjsRe+MR7QvymkB4OdvKaTpm8sZdbe90eVwhvMYoHnsYI1fMQ6Stz/K37jBKeKpmwrZaXaK1VG+prXeBRK/R7iC53d</latexit>
ML model (curve)
Inputs Outputs
Training data
Input-output
examples
(training data)
(a real number) (a real number)
Inputs
Outputs

ML model (curve)
Inputs Outputs
Training data
Input-output
examples
(training data)
Inputs
Outputs Machine learning
(curve ﬁtting)

Predicting  
by interpolating
ML model (curve)
Inputs Outputs
Training data
Input-output
examples
(training data)
Inputs
Outputs Machine learning
(curve ﬁtting)

Predicting  
by interpolating
˜x<latexit sha1_base64="x9xA4hV5CTKR/77l2BkIWmRbfo8=">AAACjHichVG7ThtBFD1seDiEhwlNJBoLC0Rl3bEQ2ERIViIhSl7mIUBodxmbEfvS7tiKsfgB2hQpgAIkCsQH8AE0+YEUfAKiJFKaFNxdGyEK4K5258y599w9M9cKHBVpotsO40NnV3dP6mPvp77+gcH00OfVyK+FtizbvuOH65YZSUd5sqyVduR6EErTtRy5Zu1/j/NrdRlGyvdWdCOQ265Z9VRF2aZmamNLK2dXNn8c7qSzlCMiIUQmBmJ6ihgUi4W8KGREnOLIoh0LfvoaW9iFDxs1uJDwoBk7MBHxswkBQsDcNprMhYxUkpc4RC9ra1wlucJkdp+/Vd5ttlmP93HPKFHb/BeH35CVGYzRH7qkB/pNV3RH/1/t1Ux6xF4avFotrQx2Bo++LP97V+XyqrH3rHrTs0YFhcSrYu9BwsSnsFv6+sGvh+WZpbHmOJ3TPfs/o1u64RN49b/2xaJcOn7Dj8Ve+MZ4QE9TyLwOVvM5QTmxOJktfWuPKoURjGKC5zGNEuaxgDL3d/ETJzg1BoxJ46sx2yo1OtqaYbwIY+4RtX2T2w==</latexit><latexit sha1_base64="x9xA4hV5CTKR/77l2BkIWmRbfo8=">AAACjHichVG7ThtBFD1seDiEhwlNJBoLC0Rl3bEQ2ERIViIhSl7mIUBodxmbEfvS7tiKsfgB2hQpgAIkCsQH8AE0+YEUfAKiJFKaFNxdGyEK4K5258y599w9M9cKHBVpotsO40NnV3dP6mPvp77+gcH00OfVyK+FtizbvuOH65YZSUd5sqyVduR6EErTtRy5Zu1/j/NrdRlGyvdWdCOQ265Z9VRF2aZmamNLK2dXNn8c7qSzlCMiIUQmBmJ6ihgUi4W8KGREnOLIoh0LfvoaW9iFDxs1uJDwoBk7MBHxswkBQsDcNprMhYxUkpc4RC9ra1wlucJkdp+/Vd5ttlmP93HPKFHb/BeH35CVGYzRH7qkB/pNV3RH/1/t1Ux6xF4avFotrQx2Bo++LP97V+XyqrH3rHrTs0YFhcSrYu9BwsSnsFv6+sGvh+WZpbHmOJ3TPfs/o1u64RN49b/2xaJcOn7Dj8Ve+MZ4QE9TyLwOVvM5QTmxOJktfWuPKoURjGKC5zGNEuaxgDL3d/ETJzg1BoxJ46sx2yo1OtqaYbwIY+4RtX2T2w==</latexit><latexit sha1_base64="x9xA4hV5CTKR/77l2BkIWmRbfo8=">AAACjHichVG7ThtBFD1seDiEhwlNJBoLC0Rl3bEQ2ERIViIhSl7mIUBodxmbEfvS7tiKsfgB2hQpgAIkCsQH8AE0+YEUfAKiJFKaFNxdGyEK4K5258y599w9M9cKHBVpotsO40NnV3dP6mPvp77+gcH00OfVyK+FtizbvuOH65YZSUd5sqyVduR6EErTtRy5Zu1/j/NrdRlGyvdWdCOQ265Z9VRF2aZmamNLK2dXNn8c7qSzlCMiIUQmBmJ6ihgUi4W8KGREnOLIoh0LfvoaW9iFDxs1uJDwoBk7MBHxswkBQsDcNprMhYxUkpc4RC9ra1wlucJkdp+/Vd5ttlmP93HPKFHb/BeH35CVGYzRH7qkB/pNV3RH/1/t1Ux6xF4avFotrQx2Bo++LP97V+XyqrH3rHrTs0YFhcSrYu9BwsSnsFv6+sGvh+WZpbHmOJ3TPfs/o1u64RN49b/2xaJcOn7Dj8Ve+MZ4QE9TyLwOVvM5QTmxOJktfWuPKoURjGKC5zGNEuaxgDL3d/ETJzg1BoxJ46sx2yo1OtqaYbwIY+4RtX2T2w==</latexit><latexit sha1_base64="x9xA4hV5CTKR/77l2BkIWmRbfo8=">AAACjHichVG7ThtBFD1seDiEhwlNJBoLC0Rl3bEQ2ERIViIhSl7mIUBodxmbEfvS7tiKsfgB2hQpgAIkCsQH8AE0+YEUfAKiJFKaFNxdGyEK4K5258y599w9M9cKHBVpotsO40NnV3dP6mPvp77+gcH00OfVyK+FtizbvuOH65YZSUd5sqyVduR6EErTtRy5Zu1/j/NrdRlGyvdWdCOQ265Z9VRF2aZmamNLK2dXNn8c7qSzlCMiIUQmBmJ6ihgUi4W8KGREnOLIoh0LfvoaW9iFDxs1uJDwoBk7MBHxswkBQsDcNprMhYxUkpc4RC9ra1wlucJkdp+/Vd5ttlmP93HPKFHb/BeH35CVGYzRH7qkB/pNV3RH/1/t1Ux6xF4avFotrQx2Bo++LP97V+XyqrH3rHrTs0YFhcSrYu9BwsSnsFv6+sGvh+WZpbHmOJ3TPfs/o1u64RN49b/2xaJcOn7Dj8Ve+MZ4QE9TyLwOVvM5QTmxOJktfWuPKoURjGKC5zGNEuaxgDL3d/ETJzg1BoxJ46sx2yo1OtqaYbwIY+4RtX2T2w==</latexit>
ˆy<latexit sha1_base64="AJk1AGRN37U7nRwFY0Rc/tyqT5Q=">AAACinichVG9TsJQFD7Uf0RBXUxciETjRM4lRkAXow6OCCImQkhbr9BQ2qa9kGDjC7g5mcikiYPxAXwAF1/AgUcwjpi4OHhaMMYBPE17v/ud851+9x7F0jVHIHYC0sjo2PjE5FRwOjQzG47MzR85ZsNWeV41ddM+VmSH65rB80ITOj+2bC7XFZ0XlNquly80ue1opnEoWhYv1eWKoZ1pqiyIKhSrsnBbF+VIDOOIyBiLeoAlN5BAOp1KsFSUeSmKGPQjY0aeoAinYIIKDagDBwMEYR1kcOg5AQYIFnElcImzCWl+nsMFBEnboCpOFTKxNfpWaHfSZw3aez0dX63SX3R6bVJGYQVf8QG7+IKP+IZfA3u5fg/PS4tWpaflVjl8uZj7/FdVp1VA9Vc11LOAM0j5XjXybvmMdwq1p2+eX3dzm9kVdxXv8J3832IHn+kERvNDvT/g2fYQPwp5oRujAf1MIToYHCXiDOPsYD22vdMf1SQswTKs0TySsA37kIG83/8KbqAthaSElJa2eqVSoK9ZgD8h7X0DoXmS8w==</latexit><latexit sha1_base64="AJk1AGRN37U7nRwFY0Rc/tyqT5Q=">AAACinichVG9TsJQFD7Uf0RBXUxciETjRM4lRkAXow6OCCImQkhbr9BQ2qa9kGDjC7g5mcikiYPxAXwAF1/AgUcwjpi4OHhaMMYBPE17v/ud851+9x7F0jVHIHYC0sjo2PjE5FRwOjQzG47MzR85ZsNWeV41ddM+VmSH65rB80ITOj+2bC7XFZ0XlNquly80ue1opnEoWhYv1eWKoZ1pqiyIKhSrsnBbF+VIDOOIyBiLeoAlN5BAOp1KsFSUeSmKGPQjY0aeoAinYIIKDagDBwMEYR1kcOg5AQYIFnElcImzCWl+nsMFBEnboCpOFTKxNfpWaHfSZw3aez0dX63SX3R6bVJGYQVf8QG7+IKP+IZfA3u5fg/PS4tWpaflVjl8uZj7/FdVp1VA9Vc11LOAM0j5XjXybvmMdwq1p2+eX3dzm9kVdxXv8J3832IHn+kERvNDvT/g2fYQPwp5oRujAf1MIToYHCXiDOPsYD22vdMf1SQswTKs0TySsA37kIG83/8KbqAthaSElJa2eqVSoK9ZgD8h7X0DoXmS8w==</latexit><latexit sha1_base64="AJk1AGRN37U7nRwFY0Rc/tyqT5Q=">AAACinichVG9TsJQFD7Uf0RBXUxciETjRM4lRkAXow6OCCImQkhbr9BQ2qa9kGDjC7g5mcikiYPxAXwAF1/AgUcwjpi4OHhaMMYBPE17v/ud851+9x7F0jVHIHYC0sjo2PjE5FRwOjQzG47MzR85ZsNWeV41ddM+VmSH65rB80ITOj+2bC7XFZ0XlNquly80ue1opnEoWhYv1eWKoZ1pqiyIKhSrsnBbF+VIDOOIyBiLeoAlN5BAOp1KsFSUeSmKGPQjY0aeoAinYIIKDagDBwMEYR1kcOg5AQYIFnElcImzCWl+nsMFBEnboCpOFTKxNfpWaHfSZw3aez0dX63SX3R6bVJGYQVf8QG7+IKP+IZfA3u5fg/PS4tWpaflVjl8uZj7/FdVp1VA9Vc11LOAM0j5XjXybvmMdwq1p2+eX3dzm9kVdxXv8J3832IHn+kERvNDvT/g2fYQPwp5oRujAf1MIToYHCXiDOPsYD22vdMf1SQswTKs0TySsA37kIG83/8KbqAthaSElJa2eqVSoK9ZgD8h7X0DoXmS8w==</latexit><latexit sha1_base64="AJk1AGRN37U7nRwFY0Rc/tyqT5Q=">AAACinichVG9TsJQFD7Uf0RBXUxciETjRM4lRkAXow6OCCImQkhbr9BQ2qa9kGDjC7g5mcikiYPxAXwAF1/AgUcwjpi4OHhaMMYBPE17v/ud851+9x7F0jVHIHYC0sjo2PjE5FRwOjQzG47MzR85ZsNWeV41ddM+VmSH65rB80ITOj+2bC7XFZ0XlNquly80ue1opnEoWhYv1eWKoZ1pqiyIKhSrsnBbF+VIDOOIyBiLeoAlN5BAOp1KsFSUeSmKGPQjY0aeoAinYIIKDagDBwMEYR1kcOg5AQYIFnElcImzCWl+nsMFBEnboCpOFTKxNfpWaHfSZw3aez0dX63SX3R6bVJGYQVf8QG7+IKP+IZfA3u5fg/PS4tWpaflVjl8uZj7/FdVp1VA9Vc11LOAM0j5XjXybvmMdwq1p2+eX3dzm9kVdxXv8J3832IHn+kERvNDvT/g2fYQPwp5oRujAf1MIToYHCXiDOPsYD22vdMf1SQswTKs0TySsA37kIG83/8KbqAthaSElJa2eqVSoK9ZgD8h7X0DoXmS8w==</latexit>
Input-output
examples
(training data)
Inputs
Outputs

Extrapolation
(Generally high-dimensional rather than 1-dimensional)
Inputs
Outputs
Extrapolation Interpolation

Extrapolation
ML Model 1
Inputs
Outputs

Extrapolation
ML Model 1
ML Model 2
Inputs
Outputs

Extrapolation
ML Model 1
ML Model 2
ML Model 3
Inputs
Outputs

1 2 3 4
5 6 7 8
9 10 11 12
1. Plugin Bayes Classiﬁer
2. 1-Nearest Neighbor Method
3. 5-Nearest Neighbor Method
4. 3-Layer Neural Networks
5. Support Vector Machine
6. Relevance Vector Machine
7. Bayes Point Machine
8. Gaussian Process Classiﬁer
9. Kernel Discriminant Analysis
10. Regression Tree (CART)
11. Random Forest
12. Gradient Boosting Machine

Data-driven
Inputs Outputs/Predictions
Training data

“Garbage in, Garbage out”

Edward
Pyro
Prob Torch
BayesFlow
LightGBM (Microsoft)
(RuleQuest) 
See5/C5.0 & Cubist
(Salford Systems) 
CART® MARS® TreeNet® 
Random Forests®
CatBoost (Yandex)
TFBoost (Google)
TenscentBoost (Tenscent)
Sherwood decision forests

http://www.sciencemag.org/news/2018/05/ai-researchers-allege-machine-
learning-alchemy

Use and Abuse of Regression (1966)
😉
 
😅

Use and Abuse of Regression (1966)
"one of the great statistical minds of the 20th century"
George E. P. Box (1919-2013)
https://en.wikipedia.org/wiki/All_models_are_wrong
"Essentially, all models are wrong, 
but some are useful"
 
😆
😫
😅

N Engl J Med 2012; 367:1562-1564
😳

http://phenomena.nationalgeographic.com/2015/09/11/nick-cage-movies-
vs-drownings-and-more-strange-but-spurious-correlations/

•
•
•
•
SAlib https://salib.readthedocs.io/en/latest/

⼊⼒出⼒x<latexit sha1_base64="BLB8K/n7QYAsE73zsDEUiBvCSV8=">AAAB/XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXRjcsK9gHtUDJppo1NMkOSEctQ/AW3uncnbv0Wt36JaTsLbT1w4XDOvZzLCRPBjfW8L1RYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0wbXis7uw4YYEkA8UjTol1UrMbyuxx0itXvKo3A14mfk4qkKPeK393+zFNJVOWCmJMx/cSG2REW04Fm5S6qWEJoSMyYB1HFZHMBNns2wk+cUofR7F2oyyeqb8vMiKNGcvQbUpih2bRm4r/eqFcSLbRZZBxlaSWKToPjlKBbYynVeA+14xaMXaEUM3d75gOiSbUusJKrhR/sYJl0jyr+l7Vvz2v1K7yeopwBMdwCj5cQA1uoA4NoHAPz/ACr+gJvaF39DFfLaD85hD+AH3+ADzJlfc=</latexit><latexit sha1_base64="BLB8K/n7QYAsE73zsDEUiBvCSV8=">AAAB/XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXRjcsK9gHtUDJppo1NMkOSEctQ/AW3uncnbv0Wt36JaTsLbT1w4XDOvZzLCRPBjfW8L1RYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0wbXis7uw4YYEkA8UjTol1UrMbyuxx0itXvKo3A14mfk4qkKPeK393+zFNJVOWCmJMx/cSG2REW04Fm5S6qWEJoSMyYB1HFZHMBNns2wk+cUofR7F2oyyeqb8vMiKNGcvQbUpih2bRm4r/eqFcSLbRZZBxlaSWKToPjlKBbYynVeA+14xaMXaEUM3d75gOiSbUusJKrhR/sYJl0jyr+l7Vvz2v1K7yeopwBMdwCj5cQA1uoA4NoHAPz/ACr+gJvaF39DFfLaD85hD+AH3+ADzJlfc=</latexit><latexit sha1_base64="BLB8K/n7QYAsE73zsDEUiBvCSV8=">AAAB/XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXRjcsK9gHtUDJppo1NMkOSEctQ/AW3uncnbv0Wt36JaTsLbT1w4XDOvZzLCRPBjfW8L1RYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0wbXis7uw4YYEkA8UjTol1UrMbyuxx0itXvKo3A14mfk4qkKPeK393+zFNJVOWCmJMx/cSG2REW04Fm5S6qWEJoSMyYB1HFZHMBNns2wk+cUofR7F2oyyeqb8vMiKNGcvQbUpih2bRm4r/eqFcSLbRZZBxlaSWKToPjlKBbYynVeA+14xaMXaEUM3d75gOiSbUusJKrhR/sYJl0jyr+l7Vvz2v1K7yeopwBMdwCj5cQA1uoA4NoHAPz/ACr+gJvaF39DFfLaD85hD+AH3+ADzJlfc=</latexit><latexit sha1_base64="BLB8K/n7QYAsE73zsDEUiBvCSV8=">AAAB/XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXRjcsK9gHtUDJppo1NMkOSEctQ/AW3uncnbv0Wt36JaTsLbT1w4XDOvZzLCRPBjfW8L1RYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0wbXis7uw4YYEkA8UjTol1UrMbyuxx0itXvKo3A14mfk4qkKPeK393+zFNJVOWCmJMx/cSG2REW04Fm5S6qWEJoSMyYB1HFZHMBNns2wk+cUofR7F2oyyeqb8vMiKNGcvQbUpih2bRm4r/eqFcSLbRZZBxlaSWKToPjlKBbYynVeA+14xaMXaEUM3d75gOiSbUusJKrhR/sYJl0jyr+l7Vvz2v1K7yeopwBMdwCj5cQA1uoA4NoHAPz/ACr+gJvaF39DFfLaD85hD+AH3+ADzJlfc=</latexit>
材料の記述⼦材料の性能
⼊⼒出⼒x<latexit sha1_base64="BLB8K/n7QYAsE73zsDEUiBvCSV8=">AAAB/XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXRjcsK9gHtUDJppo1NMkOSEctQ/AW3uncnbv0Wt36JaTsLbT1w4XDOvZzLCRPBjfW8L1RYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0wbXis7uw4YYEkA8UjTol1UrMbyuxx0itXvKo3A14mfk4qkKPeK393+zFNJVOWCmJMx/cSG2REW04Fm5S6qWEJoSMyYB1HFZHMBNns2wk+cUofR7F2oyyeqb8vMiKNGcvQbUpih2bRm4r/eqFcSLbRZZBxlaSWKToPjlKBbYynVeA+14xaMXaEUM3d75gOiSbUusJKrhR/sYJl0jyr+l7Vvz2v1K7yeopwBMdwCj5cQA1uoA4NoHAPz/ACr+gJvaF39DFfLaD85hD+AH3+ADzJlfc=</latexit><latexit sha1_base64="BLB8K/n7QYAsE73zsDEUiBvCSV8=">AAAB/XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXRjcsK9gHtUDJppo1NMkOSEctQ/AW3uncnbv0Wt36JaTsLbT1w4XDOvZzLCRPBjfW8L1RYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0wbXis7uw4YYEkA8UjTol1UrMbyuxx0itXvKo3A14mfk4qkKPeK393+zFNJVOWCmJMx/cSG2REW04Fm5S6qWEJoSMyYB1HFZHMBNns2wk+cUofR7F2oyyeqb8vMiKNGcvQbUpih2bRm4r/eqFcSLbRZZBxlaSWKToPjlKBbYynVeA+14xaMXaEUM3d75gOiSbUusJKrhR/sYJl0jyr+l7Vvz2v1K7yeopwBMdwCj5cQA1uoA4NoHAPz/ACr+gJvaF39DFfLaD85hD+AH3+ADzJlfc=</latexit><latexit sha1_base64="BLB8K/n7QYAsE73zsDEUiBvCSV8=">AAAB/XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXRjcsK9gHtUDJppo1NMkOSEctQ/AW3uncnbv0Wt36JaTsLbT1w4XDOvZzLCRPBjfW8L1RYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0wbXis7uw4YYEkA8UjTol1UrMbyuxx0itXvKo3A14mfk4qkKPeK393+zFNJVOWCmJMx/cSG2REW04Fm5S6qWEJoSMyYB1HFZHMBNns2wk+cUofR7F2oyyeqb8vMiKNGcvQbUpih2bRm4r/eqFcSLbRZZBxlaSWKToPjlKBbYynVeA+14xaMXaEUM3d75gOiSbUusJKrhR/sYJl0jyr+l7Vvz2v1K7yeopwBMdwCj5cQA1uoA4NoHAPz/ACr+gJvaF39DFfLaD85hD+AH3+ADzJlfc=</latexit><latexit sha1_base64="BLB8K/n7QYAsE73zsDEUiBvCSV8=">AAAB/XicbVDLSgMxFL2pr1pfVZdugkVwVWZE0GXRjcsK9gHtUDJppo1NMkOSEctQ/AW3uncnbv0Wt36JaTsLbT1w4XDOvZzLCRPBjfW8L1RYWV1b3yhulra2d3b3yvsHTROnmrIGjUWs2yExTHDFGpZbwdqJZkSGgrXC0fXUbz0wbXis7uw4YYEkA8UjTol1UrMbyuxx0itXvKo3A14mfk4qkKPeK393+zFNJVOWCmJMx/cSG2REW04Fm5S6qWEJoSMyYB1HFZHMBNns2wk+cUofR7F2oyyeqb8vMiKNGcvQbUpih2bRm4r/eqFcSLbRZZBxlaSWKToPjlKBbYynVeA+14xaMXaEUM3d75gOiSbUusJKrhR/sYJl0jyr+l7Vvz2v1K7yeopwBMdwCj5cQA1uoA4NoHAPz/ACr+gJvaF39DFfLaD85hD+AH3+ADzJlfc=</latexit>
材料の構造等材料の性能
機械学習
第⼀原理計算
(かかる時間の問題を除けば)機械学習を使うかどうかと無関係！

Activity cliff, Selectivity cliff, ...
 
SALI (Structure-Activity Landscape Index)

Surrogate-Based Optimization (SBO)
 
1. Initial Sampling
2. Loop:
1. Construct a Surrogate Model.
2. Search Inﬁll Criterion.
3. Add new samples.
e.g. Latin hypercube sampling (LHS)
e.g. Expected improvement (EI)
☺

Algorithm Conﬁguration & AutoML

Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design
https://arxiv.org/pdf/0912.3995.pdf
Parallelizing Exploration–Exploitation Tradeoffs with Gaussian Process Bandit Optimization
https://arxiv.org/pdf/1206.6402.pdf
Taking the Human Out of the Loop: A Review of Bayesian Optimization
https://doi.org/10.1109/JPROC.2015.2494218
Neural Processes
https://arxiv.org/abs/1807.01622
Conditional Neural Processes
https://arxiv.org/abs/1807.01613

@ SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop)

• この薬を飲めば私の病気は治るの？
• この健康⾷品⾷べていれば⻑⽣きできるの？
• この化粧品つけていれば少しでも若くいられるの？
• この⾷品たべればダイエットできるの？
• 原⼦⼒は安全なの？
科学というものには、本来限界があって、広い意味での再現可能の現象を、
⾃然界から抜き出して、それを統計学的に究明していく、そういう性質の 
学問なのである。「科学の⽅法 (中⾕宇吉郎)」

Impossible to model everything...?
 
"one of the great statistical minds of the 20th century"
George E. P. Box (1919-2013)
https://en.wikipedia.org/wiki/All_models_are_wrong
"Essentially, all models are wrong, 
but some are useful"

http://www.900910.com/mies.php

To ﬁnd out what happens to a system when you interfere with it
you have to interfere with it (not just passively observe it).

データ駆動科学と機械学習

More Related Content

What's hot (20)

More from Ichigaku Takigawa (20)

Recently uploaded (20)

データ駆動科学と機械学習