What's Hidden in a Randomly Weighted Neural Network?

November 29, 2019 · Entered Twilight · 🏛 Computer Vision and Pattern Recognition

"Last commit was 5.0 years ago (≥5 year threshold)"

Evidence collected by the PWNC Scanner

Repo contents: .gitignore, LICENSE, README.md, __init__.py, args.py, configs, data, images, main.py, models, requirements.txt, simple_mnist_example.py, trainers, utils

Authors Vivek Ramanujan, Mitchell Wortsman, Aniruddha Kembhavi, Ali Farhadi, Mohammad Rastegari arXiv ID 1911.13299 Category cs.CV: Computer Vision Cross-listed cs.LG Citations 395 Venue Computer Vision and Pattern Recognition Repository https://github.com/allenai/hidden-networks ⭐ 195 Last Checked 2 months ago

Abstract

Training a neural network is synonymous with learning the values of the weights. By contrast, we demonstrate that randomly weighted neural networks contain subnetworks which achieve impressive performance without ever training the weight values. Hidden in a randomly weighted Wide ResNet-50 we show that there is a subnetwork (with random weights) that is smaller than, but matches the performance of a ResNet-34 trained on ImageNet. Not only do these "untrained subnetworks" exist, but we provide an algorithm to effectively find them. We empirically show that as randomly weighted neural networks with fixed weights grow wider and deeper, an "untrained subnetwork" approaches a network with learned weights in accuracy. Our code and pretrained models are available at https://github.com/allenai/hidden-networks.