Scribd
Upload
2 views9 pages

MultiTask DL

cupidatat quis aliqua culpa ut duis cupidatat proident eiusmod aliquip pariatur magna fugiat anim reprehenderit magna sit duis adipisicing adipisicing labore ad excepteur Lorem excepteur aliquip laboris do exercitation aliquip officia voluptate quis velit do ad commodo minim exercitation mollit quis amet anim nisi veniam ipsum pariatur irure in exercitation labore aute et aliquip magna laborum nisi deserunt laborum ut velit ad nisi sunt exercitation dolor nisi enim reprehenderit non enim proiden

Uploaded by

Shishir Tamrakar
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
2 views9 pages

MultiTask DL

cupidatat quis aliqua culpa ut duis cupidatat proident eiusmod aliquip pariatur magna fugiat anim reprehenderit magna sit duis adipisicing adipisicing labore ad excepteur Lorem excepteur aliquip laboris do exercitation aliquip officia voluptate quis velit do ad commodo minim exercitation mollit quis amet anim nisi veniam ipsum pariatur irure in exercitation labore aute et aliquip magna laborum nisi deserunt laborum ut velit ad nisi sunt exercitation dolor nisi enim reprehenderit non enim proiden

Uploaded by

Shishir Tamrakar
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 9

Deep Learning Srihari

Multi-task Learning

Sargur N. Srihari
[email protected]

1
Deep Learning Srihari

Regularization Strategies
1. Parameter Norm Penalties 8. Early Stopping
2. Norm Penalties as 6. Parameter tying and
Constrained Optimization parameter sharing
3. Regularization and Under- 7. Sparse representations
constrained Problems 8. Bagging and other
4. Data Set Augmentation ensemble methods
5. Noise Robustness 9. Dropout
6. Semi-supervised learning 10. Adversarial training
7. Multi-task learning 11. Tangent methods

2
Deep Learning Srihari

Sharing parameters over tasks


• Multi-task learning is a way to improve
generalization by pooling the examples out of
several tasks
– Examples can be seen as providing soft constraints
on the parameters
• In the same way that additional training
examples put more pressure on the parameters
of the model towards values that generalize
well
3
Deep Learning Srihari

Common form of multitask learning


• Different supervised tasks, predicting y(i) given x
• Share the same input x, as well as some
intermediate representation h(shared) capturing a
common pool of factors

4
Deep Learning Srihari

Ex: Autonomous Navigation

y(1) y(2) y(3) 5


Deep Learning Srihari

Common multi-task situation


• Common input but different
target random variables
• Lower layers (whether feedforward
or includes a generative component
with downward arrows)
can be shared across such tasks.
• Task-specific parameters h(1), h(2)
can be learned on top of those yielding
a shared representation h(shared)
Common pool of factors explain variations of
Input x while each task is associated with a
Subset of these factors

6
Deep Learning Srihari

Multi-task in Unsupervised Learning

• In the unsupervised learning context


• some of the top level factors are associated with none
of the output tasks h(3)
These are factors that explain some of the input
variations but not relevant for predicting h(1), h(2)

7
Deep Learning Srihari

Model can be divided into two parts


1. Task specific parameters
– Which only benefit from the examples of their task
to achieve good generalization
• These are the upper layers of the neural network
2. Generic parameters
– Shared across all tasks
• Which benefit from the pooled data of all tasks
• These are the lower levels of the neural network

8
Deep Learning Srihari

Benefits of multi-tasking
• Improved generalization and generalization
error bounds
– achieved due to shared parameters
• For which statistical strength can be greatly improved
– In proportion to the increased no. of examples for the shared
parameters compared to the scenario of single-task models

• From the point of view of deep learning, the


underlying prior belief is the following:
– Among the factors that explain the variations
observed in the data associated with different tasks,
some are shared across two or more tasks
9

You might also like