Mathematical Optimization for Engineers - Lesson 7

 

1. Machine Learning (ML)

  • Definition: Programming computers using data (experience) instead of explicit rules.

  • Types:

    • Supervised learning: Regression, classification (with labeled data).

    • Unsupervised learning: Clustering (without labels).

  • Critical aspect: The quality and format of data.

2. Data-Driven Models

  • Use past data to learn system behavior.

  • Model types vary by complexity:

    • Linear models, basis functions, neural networks, Gaussian processes, gradient-boosted trees.

  • Interpolation (within data range) is reliable.

  • Extrapolation (outside training data) is unreliable and risky.

3. Hybrid Modeling

  • Combines mechanistic models (physics-based) with data-driven components.

  • Benefits:

    • Greater accuracy.

    • Maintains physical interpretability.

  • Training topologies:

    • Serial: chain-like setup.

    • Parallel: simultaneous error correction and modeling.

4. MeLOn (Machine Learning Models for Optimization)

  • Open-source tool for training ML models (ANNs, Gaussian processes) for use in optimization.

  • Supports integration with solvers and GAMS.

5. Artificial Neural Networks (ANNs)

  • Mimic biological neurons: compute weighted sums of inputs and apply activation functions.

  • Common activations: tanh, ReLU.

  • Deep learning: Use of many layers, highly effective for large datasets (e.g., image recognition).

  • Success factors: ReLU, GPU computing, regularization (e.g., dropout).

6. Training ANNs

  • Use backpropagation: forward pass (output), backward pass (error gradient).

  • Split data into: training, validation, and test sets.

  • Regularization methods:

    • Weight decay: penalize large weights.

    • Dropout: randomly deactivate neurons during training.

  • Avoid overfitting (too complex) and underfitting (too simple).

7. Optimization with Embedded ANNs

  • Full-space formulation: Includes all ANN variables (large and complex).

  • Reduced-space formulation: Only optimization variables, ANN acts as a black-box function.

  • Reduced-space → better computational efficiency.

8. Example Applications

  • Peaks function: ANN learns and optimizes a synthetic test function.

  • Chemical process: 14 ANNs embedded in a hybrid model for process optimization.

  • Membrane synthesis: Multi-objective optimization using learned models.

9. Gaussian Processes (GPs) and Bayesian Optimization

  • GPs provide both prediction and uncertainty.

  • Useful for expensive or sparse data problems.

  • Bayesian optimization:

    • Uses GPs to guide experiments.

    • Optimizes expected improvement.

  • Embedding GPs in reduced-space optimization enhances performance.

10. MAiNGO Solver

  • In-house global optimization solver.

  • Supports deterministic optimization with ANN and GP models.

  • Outperforms traditional full-space approaches in large problems.

Subscribe to: Posts (Atom)