Evolution Of Activation Functions for Neural Architecture Search

LAUR Repository

Show simple item record

dc.contributor.author Nader, Andrew
dc.date.accessioned 2022-07-21T08:40:45Z
dc.date.available 2022-07-21T08:40:45Z
dc.date.copyright 2020 en_US
dc.date.issued 2020-05-18
dc.identifier.uri http://hdl.handle.net/10725/13847
dc.description.abstract The introduction of the ReLU function in neural network architectures yielded substantial improvements over sigmoidal activation functions and allowed for the training of deep networks. Ever since, the search for new activation functions in neural networks has been an active research topic. However, to the best of our knowledge, the design of new activation functions has mostly been done by hand. In this work, we propose the use of a self-adaptive evolutionary algorithm that searches for new activation functions using a genetic programming approach, and we compare the performance of the obtained activation functions to ReLU. We also analyze the shape of the obtained activations to see if they have any common traits such as monotonicity or piece-wise linearity, and we study the effects of the self-adaptation to see which operators perform well in the context of a search for new activation functions. We perform a thorough experimental study on datasets of different sizes and types, using different types of neural network architectures. We report favorable results obtained from the mean and standard deviation of the performance metrics over multiple runs. en_US
dc.language.iso en en_US
dc.subject Computer network architectures en_US
dc.subject Neural networks (Computer science) en_US
dc.subject Machine learning en_US
dc.subject Lebanese American University -- Dissertations en_US
dc.subject Dissertations, Academic en_US
dc.title Evolution Of Activation Functions for Neural Architecture Search en_US
dc.type Thesis en_US
dc.term.submitted Spring en_US
dc.author.degree MS in Computer Science en_US
dc.author.school SAS en_US
dc.author.idnumber 201203257 en_US
dc.author.commembers Takchi, Jean
dc.author.commembers Harmanani, Haidar
dc.author.department Computer Science And Mathematics en_US
dc.description.physdesc 1 online resource (xii, 108 leaves): col. ill. en_US
dc.author.advisor Azar, Danielle
dc.keywords Machine Learning en_US
dc.keywords Genetic Algorithm en_US
dc.keywords Genetic Programming en_US
dc.keywords Neural Architecture Search en_US
dc.keywords Neural Networks en_US
dc.keywords Activation Functions en_US
dc.description.bibliographiccitations Bibliography: leaf 98-108. en_US
dc.identifier.doi https://doi.org/10.26756/th.2022.373
dc.author.email andrew.nader@lau.edu en_US
dc.identifier.tou http://libraries.lau.edu.lb/research/laur/terms-of-use/thesis.php en_US
dc.publisher.institution Lebanese American University en_US
dc.author.affiliation Lebanese American University en_US

Files in this item

This item appears in the following Collection(s)

Show simple item record

Search LAUR

Advanced Search


My Account