3 accepted papers
We design and evaluate a Bayesian optimization framework for resource efficient pre-training of Transformer-based language models (TLMs). TLM pre-training requires high computational resources and introduces many unresolved design choices, such as selecting its pre-training hyperparameters.We propos…
Deep learning has gained momentum over traditional methods in recent years due to its ability to scale up to an unforeseen rise in both volumes and dimensions of data emerging from IoT. It is suitable for modeling arbitrary complex dependencies, such as those exacerbated by asynchrony in the inputs.…
Stochastic gradient-based methods are prominent for training machine learning and deep learning models. The performance of these techniques depends on their hyperparameter tuning over time and varies for different models and problems. Manual adjustment of hyperparameters is very costly and time-cons…