Niklas Donges on Apr 23, 2018

Transfer Learning is the reuse of a pre-trained model on a new problem. It is currently very popular in the field of Deep Learning because it enables you to train Deep Neural Networks with comparatively little data. This is very useful since most real-world problems typically do not have millions of labeled data points to train such complex models. This blog post is intended to give you an overview of what Transfer Learning is, how it works, why you should use it and when you can use it. It will introduce you to the different approaches of Transfer Learning and provide you with some resources on already pre-trained models.

Table of Contents:

  • What is it?
  • How it works
  • Why it is used?
  • When you should use it
  • Approaches to Transfer Learning (Training a Model to Reuse it, Using a Pre-Trained Model, Feature Extraction)
  • Summary

What is it?

In Transfer Learning, the knowledge of an already trained Machine Learning model is applied to a different but related problem. For example, if you trained a simple classifier to predict whether an image contains a backpack, you could use the knowledge that the model gained during its training to recognize other objects like sunglasses.

With transfer learning, we basically try to exploit what has been learned in one task to improve generalization in another. We transfer the weights that a Network has learned at Task A to a new Task B.

The general idea is to use knowledge, that a model has learned from a task where a lot of labeled training data is available, in a new task where we don’t have a lot of data. Instead of starting the learning process from scratch, you start from patterns that have been learned from solving a related task.

Transfer Learning is mostly used in Computer Vision and Natural Language Processing Tasks like Sentiment Analysis, because of the huge amount of computational power that is needed for them.

It is not really a Machine Learning technique. Transfer Learning can be seen as a ‘design methodology’ within Machine Learning like for example, active learning. It is also not an exclusive part or study-area of Machine Learning. Nevertheless, it has become quite popular in the combination with Neural Networks, since they require huge amounts of data and computational power.

How it works

For example, in computer vision, Neural Networks usually try to detect edges in their earlier layers, shapes in their middle layer and some task-specific features in the later layers. With transfer learning, you use the early and middle layers and only re-train the latter layers. It helps us to leverage the labeled data of the task it was initially trained on.

Let’s go back to the example of a model trained for recognizing a backpack on an Image, which will be used to identify Sunglasses. In the earlier layers, the model has learned to recognize objects and because of that, we will only re-train the latter layers, so that it will learn what separates sunglasses from other objects.

In Transfer Learning, we try to transfer as much knowledge as possible from the previous task, the model was trained on, to the new task at hand. This knowledge can be in various forms depending on the problem and the data. For example, it could be how models are composed which would allow us to more easily identify novel objects.

Why it is used?

Using Transfer Learning has several benefits that we will discuss in this section. The main advantages are basically that you save training time, that your Neural Network performs better in most cases and that you don’t need a lot of data.

Usually, you need a lot of data to train a Neural Network from scratch but you don’t always have access to enough data. That is where Transfer Learning comes into play because with it you can build a solid machine Learning model with comparatively little training data because the model is already pre-trained. This is especially valuable in Natural Language Processing (NLP) because there is mostly expert knowledge required to created large labeled datasets. Therefore you also save a lot of training time, because it can sometimes take days or even weeks to train a deep Neural Network from scratch on a complex task.

According to Demis Hassabis, the CEO of DeepMind Technologies, Transfer is also one of the most promising techniques that could someday lead us to Artificial General Intelligence (AGI):

When you should use it

As it is always the case in Machine Learning, it is hard to form rules that are generally applicable. But I will provide you with some guidelines.

You would typically use Transfer Learning when (a) you don’t have enough labeled training data to train your network from scratch and/or (b) there already exists a network that is pre-trained on a similar task, which is usually trained on massive amounts of data. Another case where its use would be appropriate is when Task-1 and Task-2 have the same input.

If the original model was trained using TensorFlow, you can simply restore it and re-train some layers for your task. Note that Transfer Learning only works if the features learned from the first task are general, meaning that they can be useful for another related task as well. Also, the input of the model needs to have the same size as it was initially trained with. If you don’t have that, you need to add a preprocessing step to resize your input to the needed size.

Approaches to Transfer Learning

Now we will discuss different approaches to Transfer Learning. Note that these have different names throughout literature but the overall concept is mostly the same.

1. Training a Model to Reuse it

Imagine you want to solve Task A but don’t have enough data to train a Deep Neural Network. One way around this issue would be to find a related Task B, where you have an abundance of data. Then you could train a Deep Neural Network on Task B and use this model as starting point to solve your initial Task A. If you have to use the whole model or only a few layers of it, depends heavily on the problem you are trying to solve.

If you have the same input in both Tasks, you could maybe just reuse the model and make predictions for your new input. Alternatively, you could also just change and re-train different task-specific layers and the output layer.

2. Using a Pre-Trained Model

Approach 2 would be to use an already pre-trained model. There are a lot of these models out there, so you have to do a little bit of research. How many layers you reuse and how many you are training again, depends like I already said on your problem and it is therefore hard to form a general rule.

Keras, for example, provides nine pre-trained models that you can use for Transfer Learning, Prediction, feature extraction and fine-tuning. You can find these models and also some brief tutorial on how to use them here.

There are also many research institutions that released models they have trained. This type of Transfer Learning is most commonly used throughout Deep Learning.

3. Feature Extraction

Another approach is to use Deep Learning to discover the best representation of your problem, which means finding the most important features. This approach is also known as Representation Learning and can often result in a much better performance than can be obtained with hand-designed representation.

Most of the time in Machine Learning, features are manually hand-crafted by researchers and domain experts. Fortunately, Deep Learning can extract features automatically. Note that this does not mean that Feature Engineering and Domain knowledge isn’t important anymore because you still have to decide which features you put into your Network. But Neural Networks have the ability to learn which features, you have put into it, are really important and which ones aren’t. A representation learning algorithm can discover a good combination of features within a very short timeframe, even for complex tasks which would otherwise require a lot of human effort.

The learned representation can then be used for other problems as well. You simply use the first layers to spot the right representation of features but you don’t use the output of the network because it is too task-specific. Simply feed data into your network and use one of the intermediate layers as the output layer. This layer can then be interpreted as a representation of the raw data.

This approach is mostly used in Computer Vision because it can reduce the size of your dataset, which decreases computation time and makes it more suitable for traditional algorithms as well.

Popular Pre-Trained Models

There are a some pre-trained Machine Learning models out there that became quite popular. One of them is the Inception-v3 model, which was trained for the ImageNet “Large Visual Recognition Challenge”. In this challenge, participants had to classify images into 1000 classes, like “Zebra”, “Dalmatian”, and “Dishwasher”.

Here you can see a very good tutorial from TensorFlow on how to retrain image classifiers.

Microsoft also offers some pre-trained models which are available for both R and Python development, through the MicrosoftML R package and the microsoftml Python package.

Other quite popular models are ResNet and AlexNet. I also encourage you to visit pretrained.ml which is a sortable and searchable compilation of pre-trained deep learning models, along with demos and code.

Summary

In this post, you have learned what Transfer Learning is and why it matters. You also discovered how it is done along with some of its benefits. We talked about why it can reduce the size of your dataset, why it decreases training time and why you also need less data when you use it. We discussed when it is appropriate to do Transfer Learning and what are the different approaches to it. Lastly, I provided you with a collection of models that are already pre-trained.

Resources

https://medium.com/@14prakash/transfer-learning-using-keras-d804b2e04ef8

http://ruder.io/transfer-learning/

https://www.datacamp.com/community/tutorials/transfer-learning

Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques for Building Intelligent Systems

Deep Learning (Adaptive Computation and Machine Learning)

Collected at:  
https://towardsdatascience.com/transfer-learning-946518f95666

110 thoughts on “Transfer Learning”

  1. Autoliker, Photo Liker, ZFN Liker, Autolike, Autoliker, auto liker, Auto Like, Increase Likes, Photo Auto Liker, autolike, Status Auto Liker, Working Auto Liker, Status Liker, autoliker, auto like, Autolike International, Auto Liker

  2. Hi. I have checked your longdnguyen.com and i see you’ve got some duplicate
    content so probably it is the reason that you don’t rank
    high in google. But you can fix this issue fast. There is a tool that
    generates content like human, just search in google:
    miftolo’s tools

  3. Hmm it seems like your blog ate my first comment (it was
    extremely long) so I guess I’ll just sum it up what I submitted and
    say, I’m thoroughly enjoying your blog. I too am an aspiring blog blogger but I’m still new
    to the whole thing. Do you have any tips and hints for novice blog writers?
    I’d certainly appreciate it. fußball trikot

  4. Definitely believe that which you said. Your favorite reason appeared to be on the net the easiest thing to be aware of.

    I say to you, I definitely get irked while people think about worries that
    they plainly don’t know about. You managed to hit the nail upon the top
    and defined out the whole thing without having side effect , people can take a signal.
    Will likely be back to get more. Thanks fodboldtrøjer

  5. It’s perfect time to make some plans for the future and it’s time to be
    happy. I have read this post and if I could I desire to suggest you some interesting things or tips.
    Perhaps you can write next articles referring to this article.
    I want to read even more things about it! fotbollströjor

  6. I’ll immediately take hold of your rss feed as I can not in finding your e-mail subscription hyperlink or e-newsletter service.
    Do you’ve any? Please let me recognize in order that I could subscribe.
    Thanks. maglie calcio

  7. Do you mind if I quote a couple of your articles as long as I provide credit and sources back to your blog?
    My website is in the very same niche as yours and my visitors
    would truly benefit from some of the information you present here.
    Please let me know if this ok with you. Many thanks!

  8. Undeniably consider that that you said. Your favorite justification appeared to be on the net the easiest thing to take into account of.
    I say to you, I certainly get irked at the same time as people
    consider issues that they plainly do not realize about.
    You managed to hit the nail upon the top as well as defined
    out the whole thing with no need side effect , folks can take a
    signal. Will likely be again to get more. Thank you

  9. Hello this is kind of of off topic but I was wanting to know if blogs use WYSIWYG editors
    or if you have to manually code with HTML. I’m starting a blog soon but have no coding knowledge so I wanted to get advice from someone with experience.
    Any help would be enormously appreciated!

  10. Today, I went to the beachfront with my children. I found a sea shell and gave it to my 4 year old daughter and said “You can hear the ocean if you put this to your ear.” She put the shell to her ear and screamed.
    There was a hermit crab inside and it pinched her ear.
    She never wants to go back! LoL I know this is entirely off topic but I
    had to tell someone!

  11. Hey I know this is off topic but I was wondering if you knew of any widgets I could add
    to my blog that automatically tweet my newest twitter
    updates. I’ve been looking for a plug-in like this for quite some time and was hoping maybe
    you would have some experience with something like this.

    Please let me know if you run into anything. I truly enjoy reading your blog and
    I look forward to your new updates.

  12. When I originally commented I clicked the “Notify me when new comments are added” checkbox and now each time a comment
    is added I get four e-mails with the same comment.
    Is there any way you can remove people from that service?
    Thanks!

  13. I know this if off topic but I’m looking into starting my
    own blog and was curious what all is required to get setup?
    I’m assuming having a blog like yours would cost a pretty penny?
    I’m not very web savvy so I’m not 100% positive.
    Any tips or advice would be greatly appreciated.
    Thanks

  14. Excellent goods from you, man. I’ve understand your stuff previous to and you’re just extremely wonderful.
    I actually like what you have acquired here, really like what you are
    stating and the way in which you say it. You make it
    entertaining and you still take care of to keep it wise.
    I can’t wait to read far more from you. This is really a
    great site.

  15. I think this is one of the most vital information for me.
    And i’m glad reading your article. But want to remark on some general things, The site style is ideal, the articles is really excellent :
    D. Good job, cheers

  16. Fantastic website. Lots of helpful info here. I am sending
    it to several buddies ans additionally sharing in delicious.

    And of course, thank you in your sweat!

  17. Hi, Neat post. There is a problem together with your website in internet explorer,
    may check this? IE still is the market chief and
    a big element of folks will miss your great writing due to this problem.

  18. We’re a gaggle of volunteers and starting a new scheme in our community.
    Your website offered us with valuable information to
    work on. You’ve performed an impressive activity and our entire community will likely be thankful to
    you.

  19. Very good blog you have here but I was curious about if you knew of
    any message boards that cover the same topics talked about
    in this article? I’d really like to be a part of group where I
    can get advice from other experienced individuals that
    share the same interest. If you have any recommendations, please let me know.

    Thanks!

  20. I just couldn’t go away your website before suggesting that I really enjoyed the usual info an individual supply in your guests?
    Is gonna be again regularly in order to check up on new posts

  21. Pretty nice post. I just stumbled upon your blog and wanted to say
    that I have really enjoyed surfing around your blog posts.
    After all I’ll be subscribing to your rss feed and I hope
    you write again very soon!

  22. Thanks for some other informative web site. Where else
    could I am getting that kind of information written in such
    an ideal manner? I’ve a challenge that I am just now operating on, and I have been at
    the glance out for such info.

  23. This is the right blog for everyone who wants to find out about this topic.
    You realize so much its almost tough to argue with
    you (not that I personally would want to…HaHa).
    You definitely put a fresh spin on a topic that’s been discussed for a long time.

    Great stuff, just excellent!

  24. When I originally commented I clicked the “Notify me when new comments are added” checkbox and now each time a comment is added I get
    four emails with the same comment. Is there any way you can remove people from that
    service? Many thanks!

  25. I’m really inspired along with your writing skills and also with the format
    on your weblog. Is that this a paid theme or did
    you modify it your self? Either way stay up the excellent quality writing, it is rare to look a great blog like this one today..

  26. Pretty nice post. I just stumbled upon your weblog and wished to
    say that I’ve really enjoyed browsing your blog posts.

    After all I will be subscribing to your rss feed
    and I hope you write again soon!

  27. Thanks for one’s marvelous posting! I really enjoyed reading it,
    you happen to be a great author.I will make certain to bookmark your
    blog and will often come back later in life. I want to encourage yourself to continue your great posts, have a nice holiday weekend!

  28. you are truly a good webmaster. The web site loading velocity is amazing.
    It sort of feels that you are doing any distinctive trick.

    Also, The contents are masterwork. you’ve done a wonderful task in this topic!

  29. I’ve been browsing online more than three hours these days,
    yet I by no means discovered any fascinating article like yours.
    It’s pretty price enough for me. In my view, if all webmasters and bloggers made
    just right content material as you probably did, the internet
    will probably be a lot more helpful than ever before.

  30. Definitely imagine that which you said. Your favourite justification seemed to be on the internet the simplest thing to keep in mind of.
    I say to you, I definitely get irked whilst people consider worries
    that they just don’t know about. You controlled to hit the nail upon the highest and defined out the whole thing without having side-effects ,
    folks could take a signal. Will probably be again to get more.
    Thanks

  31. Thank you a bunch for sharing this with all of us you really recognize what you’re talking about!
    Bookmarked. Kindly also talk over with my web site =).
    We will have a link trade agreement among us

  32. After I originally commented I seem to have clicked on the -Notify me when new comments are added- checkbox and now each time a comment is added I recieve four emails with the exact same comment.
    Is there a way you are able to remove me from that service?
    Many thanks!

  33. Magnificent beat ! I wish to apprentice while you amend your website, how can i
    subscribe for a blog web site? The account aided me a acceptable deal.

    I had been a little bit acquainted of this your broadcast provided bright clear idea

  34. I have read several just right stuff here.
    Definitely price bookmarking for revisiting.
    I wonder how much effort you place to make
    this sort of wonderful informative website.

  35. Excellent beat ! I would like to apprentice at the same time as you amend your website,
    how could i subscribe for a weblog site? The account aided me
    a acceptable deal. I have been a little bit acquainted of this your broadcast provided bright clear concept

  36. Do you mind if I quote a few of your posts as long as I
    provide credit and sources back to your site? My blog is in the exact same area of interest as yours and my users
    would genuinely benefit from some of the information you present here.
    Please let me know if this ok with you. Thanks!

  37. Hi there! I just wish to give you a huge thumbs up for your excellent
    information you have here on this post. I’ll be returning to
    your site for more soon.

  38. I know this web site presents quality dependent
    posts and extra information, is there any other web
    page which gives these kinds of information in quality?

  39. Yesterday, while I was at work, my sister stole my apple ipad and tested to see if it can survive a thirty foot drop, just so she can be a youtube sensation. My iPad is now broken and she has 83 views.

    I know this is totally off topic but I had to share it with someone!

  40. This design is incredible! You certainly know how to keep a reader amused.
    Between your wit and your videos, I was almost moved to
    start my own blog (well, almost…HaHa!) Fantastic job.
    I really loved what you had to say, and more than that, how you presented it.
    Too cool!

  41. Right here is the right blog for anybody who wants to find out about this topic.
    You realize so much its almost tough to argue with you (not that I personally would want to…HaHa).
    You certainly put a fresh spin on a topic that’s been written about for
    decades. Wonderful stuff, just wonderful!

  42. I was pretty pleased to discover this great site. I need to to thank
    you for your time just for this wonderful read!! I definitely loved every part of it and I have you book marked to check out new stuff in your website.

  43. Hey, I think your blog might be having browser compatibility
    issues. When I look at your website in Ie, it looks fine but
    when opening in Internet Explorer, it has some overlapping.
    I just wanted to give you a quick heads up! Other then that,
    amazing blog!

  44. Hello my family member! I wish to say that this article is
    awesome, great written and come with almost all vital infos.

    I would like to see more posts like this .

  45. Thanks on your marvelous posting! I truly enjoyed reading it, you could be a great author.

    I will always bookmark your blog and definitely will come back from now on. I
    want to encourage you continue your great job, have a
    nice day!

  46. Hi, i think that i saw you visited my web site so i came to “return the favor”.I am attempting to find things to improve my website!I suppose its ok to use some of your ideas!!

  47. Thanks for any other informative website. The place else may I am getting that kind
    of info written in such an ideal method? I’ve a challenge that I am simply now operating on, and I’ve been on the glance out for such info.

  48. hi!,I love your writing so much! proportion we keep up a correspondence more about your article on AOL?
    I need an expert on this space to unravel my problem.
    Maybe that is you! Having a look ahead to see you.

  49. Hello would you mind letting me know which hosting company you’re using?

    I’ve loaded your blog in 3 different browsers and I must say this blog
    loads a lot quicker then most. Can you suggest a good hosting provider at
    a reasonable price? Cheers, I appreciate it!

  50. Appreciating the time and effort you put into your site and in depth
    information you present. It’s nice to come across a blog every once in a while
    that isn’t the same old rehashed material. Excellent read!
    I’ve bookmarked your site and I’m including your RSS
    feeds to my Google account.

  51. My partner and I absolutely love your blog and find a lot
    of your post’s to be exactly I’m looking for.
    Would you offer guest writers to write content to suit your needs?
    I wouldn’t mind composing a post or elaborating
    on a number of the subjects you write concerning here.
    Again, awesome web log!

  52. Heya! I just wanted to ask if you ever have any problems with hackers?
    My last blog (wordpress) was hacked and I ended up
    losing several weeks of hard work due to no data backup.
    Do you have any methods to stop hackers?

  53. Remarkable issues here. I am very satisfied to peer your article.
    Thanks so much and I am taking a look ahead to touch you.
    Will you kindly drop me a e-mail?

  54. You really make it seem so easy with your presentation but
    I find this topic to be really something that I think I would never understand.
    It seems too complicated and extremely broad for me.
    I’m looking forward for your next post, I will try to get the
    hang of it!

  55. Do you mind if I quote a couple of your articles as long as I provide credit and sources
    back to your website? My blog site is in the very same area of interest as yours and
    my visitors would definitely benefit from some of the information you provide here.
    Please let me know if this ok with you. Many thanks!

  56. Thanks for your personal marvelous posting! I quite enjoyed reading it, you will be a great author.
    I will make certain to bookmark your blog and will eventually come back
    from now on. I want to encourage yourself to continue your great work, have a nice afternoon!

  57. Hey. Do you know what essay on engineering and technology is? If not, then I recommend you read two articles: 1) How does writing an essay on engineering and technology help students choose well-paid jobs? , which really tells how grads come to choose the desired career, how to finish academy, studying in this profession, how to find job after convocation, in which academies do in order to become a guru in this field, do I need to certify in order to get a work in this field. 2) Buy an essay on engineering and technology , which tells about one of the best services that supports undergraduates not only with writing assorted essays but also with writing numerous works of varying complexity. Thanks to these two articles, you will be able to understand what engineering graph paper is and to understand whether you need to train to write. Even if you do not need to train to write to them, then learning something new in the field of education is always useful for every person.

  58. Fabricamos tipos DIN estándar según sus propios planos (ver Piezas especiales según planos). MEGA es una empresa de mecanizado de precisión ubicada en el norte de Italia (Vestone, Brescia). Como puedes observar el decoletaje sirve para una multitud de servicios y sectores que vemos día a día y hace mucho más rápida la fabricación de piezas muy necesarias.

Leave a Reply

Your email address will not be published. Required fields are marked *