Machine Learning for Mechanical Engineering

Abstract

Learn Machine Learning techniques to solve mechanical engineering problems covering: 1) Review of key concepts like cross-validation, linear models, neural networks, and probabilistic inference; 2) Advanced neural methods including transformers, GANs, VAEs, and reinforcement learning; 3) Kernel methods and probabilistic models. Includes hands-on projects and a final exam.

Objective

Part 0: Review 1. Recall key concepts from the prerequisite Stochastics and Machine Learning course, including cross-validation, linear models, basic neural networks, automatic differentiation, and probabilistic inference. 2. Apply those reviewed concepts to implement baseline models in Python for solving supplied Engineering-related problems. 3. Diagnose and correct errors in ML models or provided code using techniques such as visualization or analysis of model outputs or predictions. Part 1: Advanced Neural Methods 4. Describe the effect of advanced regularization methods for neural architectures, such as spectral normalization; apply those techniques to improve model performance in advanced neural network architectures, including transformers, self-attention mechanisms, and graph neural networks; and assess their effects on Engineering-related problems. 5. Develop generative models using GANs, VAEs, and Diffusion Models, and evaluate their effectiveness in various engineering applications. 6. Implement and evaluate reinforcement learning algorithms, including deep Q-learning, policy gradient methods, and diffusion models, for engineering problems. Part 2: Kernels and Probabilistic Models 7. Analyze how different kernels affect the function space properties of kernel methods, such as Gaussian Processes, and assess appropriate kernel choices in applications such as Bayesian Optimization. 8. Distinguish between different types of approximate probabilistic inference methods (MCMC, VI, SVI) and apply appropriate probabilistic programming methods to solve engineering problems. Part 3: Engineering-Specific ML Topics (Time Permitting) 9. Apply transfer learning methods, including fine-tuning pre-trained models, or active learning and semi-supervised learning techniques to address data-sparse regimes in engineering. 10. Evaluate and debate ethical implications of ML applications in engineering, including fairness, privacy, and intellectual property considerations. General Course Objectives 11. Collaborate effectively on projects and critique or provide feedback to others in diagnosing model errors. 12. Become aware of current research and advancements in machine learning and its applications in engineering, and develop skills for continued learning in this area beyond the present course.

Content

This course assumes as prerequisite knowledge the topics taught in the 4th semester "Stochastics and Machine Learning" course, and builds upon them to introduce more advanced techniques applicable to an Engineering context with a focus on applications in Design and Manufacturing. This course can also enable students to take more advanced or detailed courses in D-MAVT or D-INFK at the Masters level. The covered topics will be put into practice via practical programming assignments and projects in the D-MAVT domain. Part 0: Review - Weeks 1-2: Brief review of topics from prior Stochastics and ML course — Cross-Validation; Linear Models in Supervised, Unsupervised, and Reinforcement Learning; Review of simple Neural Networks (MLP, Convolutions/U-Nets, RNNs, AEs); Automatic Differentiation; Review of Probabilistic Inference (MLE, MAP, EM Algorithm). How to inspect, diagnose, and correct errors or mistakes in ML models or code. Part 1: Advanced Neural Methods - Weeks 3-4: Advanced Neural Methods — Transformers and Self-Attention, Message Passing Networks & basic Graph Neural Networks; Regularization methods (e.g., Batch or Spectral Normalization); Recurrent/Autoregressive Models - Weeks 5-6: Generative Models — Push-forward models (e.g., GANs, VAEs, Normalizing Flows) and Optimal Transport; Stochastic models (e.g., Diffusion Models), and combinations (e.g., Latent Diffusion Models). - Week 7-8: Reinforcement Learning — Deep Q-Learning, Policy Gradient methods, Diffusion Policies Part 2: Kernels and Probabilistic Models - Week 9: Kernel Methods — The Representer Theorem. RKHS Regularization and its connection to Fourier Spectra. Kernel Ridge Regression. - Weeks 10-11: Probabilistic Models — Introduction to Probabilistic Graphical Models and Probabilistic Programming Methods. Introduction to approximate inference: MCMC + Variational Inference. Basics of Stochastic Variational Inference and its use in training Bayesian Neural Networks. Gaussian Processes & Bayesian Optimization. Part 3: Engineering-Specific ML Topics (Time Permitting) - Weeks 12: Data-Sparse Regimes — Transfer Learning methods, including using and fine tuning pre-trained or foundation models; Active Learning; Semi- or Self-Supervised Learning; Data Augmentation. - Week 13: Ethics and Responsible Use of ML in Engineering + Project Work: Fairness in ML, Privacy and IP considerations. Review for final exam.

Resources