Introduction
Engineering is a process of iterative optimization. Large teams of engineers rely on detailed simulations, physical testing, or other forms of feedback to make refinements to their designs until (rarely) no more improvements are possible, or (more commonly) some deadline is reached and the best design available must be used. Typically, at each step of the way, the next design to try is chosen by human judgment. If an aerodynamicist spots some separated flow in a computational fluid dynamics (CFD) simulation, the surface will be manually adjusted to try and clean it up. If a structural finite element analysis (FEA) result suggests an excessive stress concentration, some more material might be added to take the load. By the nature of this process, the quality of these human judgments massively determines the performance of the end design.
How do we ensure the final design is as good as it can be? How do we know whether we’ve appropriately balanced all of the competing objectives? Maybe there’s a department responsible for overall system-level performance evaluation, but by the time they are involved, it’s often too late to make significant changes. The end result is that most devices that you can buy are more polluting, costly, and more expensive than they need to be.
This culture is a natural consequence of the lack of any optimization module beyond the very basics in most engineering university courses. If such a module exists, it is often relegated to an optional section in the last year, often hosted by another department (see MIT’s Mech Eng or Imperial’s Aeronautics and EEE courses). The problem is compounded by the lack of any serious statistical modeling approaches for epistemic uncertainty, the type of uncertainty that is usually dominant in any engineering context with expensive physics to be resolved. In effect, this means that engineers are unprepared for one of the most significant tasks that they will be asked to perform: producing efficient designs that optimize performance objectives and meet constraints. This is an almost ludicrous state of affairs. Engineers, ever resilient and resourceful, cope by relying on canned software packages and iterating with much manual effort until a solution is found. These are inadequate alternatives when powerful methods are available to address the problems.
Although some engineers eventually learn optimization on the job, or by reading texts on the subject, many books on optimization aimed at engineers focus on introducing as many different methods as possible, which might only confuse the reader. Often, highly relevant methods such as Bayesian optimization are glossed over, and sometimes even referred to by a different, obscure name, such as “Efficient Global Optimization (EGO)” in Martins and Ning (2021).
Meanwhile, industry players focus on ever more complex models, performance and accuracy improvements for predicting phenomena, or growing hardware requirements to run finer simulations. While all these are beneficial, they absorb a great deal of resources compared to the often-overlooked optimization task. Although some software packages offer integrated design optimization tools, such as FEA topology optimization or adjoint CFD solvers, these are underused in the industry, in part due to high computational cost and the difficulty in appropriately defining the optimization problem.
This is not unique to engineering. In data science, most oxygen tends to be consumed by supervised and unsupervised machine learning, while simple and well-understood optimization topics like linear programming or search are relegated to the periphery. In biology and chemistry, experiments are tuned one parameter at a time despite an abundance of techniques for multivariate optimization in the lab. In all cases, the “what” (predictive modeling) is given far more attention than the “what next” (optimization).
We identify the lack of proper treatment for optimization within the industrial engineering practice and issue a call for reformation. We mention what we think are the socio-cultural elements of the problem, and outline some general ways to overcome common technical hurdles in applying existing optimization methods in the context of geometry design tasks. The article is organized into two main parts: (1) a review of the steps necessary to properly define an optimization task, accompanied by requirements for effective optimization and usual hurdles; and (2) a case study that demonstrates the hazards of “doing optimization wrong.”
Optimization Recipes: Ingredients and Fire Hazards
Many engineering problems can naturally be framed as optimization problems, where the task is to find the best design or configuration, subject to constraints. Despite this, it is often the case that the objectives, and especially the constraints, are hard to articulate. Stakeholders may say “I want the most performant wing design”, when what they really mean is “I want the most performant wing design, which is thick enough to sustain stress loads; big enough that it can carry a payload; manufacturable; small enough to fit on a runway; has tapered edges so it can attach to an aircraft body, etc.” That is, there is a lot of tacit knowledge and assumptions that engineers carry around with them when manually iterating to create better designs, which need to be codified into the optimization problem.
On the other hand, even if the constraints can be well articulated, there are significant technical challenges in optimizing over geometry. Fortunately, the past few decades have seen major advancements in both methodology and hardware that can be harnessed effectively for this purpose.
We shall begin by introducing the most prominent socio-cultural barriers to employing optimization methods in engineering, based on our experience in the industry so far. Some problems are termed artificial, as they are mostly self-imposed and entirely human-made: they emerge from behavioral patterns that can be avoided, or misconceptions that can be lifted. Others are natural, as they are fundamentally difficult — but not impossible — to solve.
Artificial Hazards
The first challenge is defining a space of geometries that allows for efficient exploration of candidate solutions. In such a space, proximity should be a good indicator of similarity in performance: nearby geometries should behave similarly. Crafting a solution space that is easy to traverse (continuous), relatively low-dimensional, covers most viable designs, and is useful in predicting the objective as well as any constraint violation is half the battle in solving such problems. In engineering domains, we usually have that in the CAD/CAE configuration. While geometric assumptions and constraints are tricky to deal with, computer-aided engineering (CAE) has been part and parcel of engineering practice over the past few decades and can be a powerful ally in this task. Consider that most design problems are already translated into some form of parametrized computer-aided design (CAD), which is imbued with domain knowledge of the engineer. More on the limitations of CAD parametrization as a solution space, later.
The second challenge is crafting the objective function to be optimized. This is tricky because often the objective is not a single well-defined quantity but instead a constellation of metrics, some of which may be qualitative. For any optimization method to be effective, all relevant objectives must be quantified. While there are methods that can deal with multiple objectives and present a Pareto front, they are typically more cumbersome to work with, and the result will eventually have to be whittled down to a single-candidate solution. As such, crafting a suitable objective is paramount — ideally, a single scalar metric that balances all the relevant concerns.
The third challenge is to account for constraints. Optimizers optimize objectives ruthlessly. Therefore, all relevant constraints must be clearly articulated and the cost of violation must be made explicit. Hard constraints, which are to be respected at all costs, must be framed as such: a candidate that violates these can never be preferred over a candidate that doesn’t, regardless of objectives. Soft constraints must be balanced against objectives and so, in some sense, belong more in the family of objectives than constraints (although we find that stakeholders often tend to think by default that any important quantity is an objective to be minimized or maximized when, in reality, it only matters whether it’s above or below some critical value). Articulating all relevant constraints is another difficulty for experienced practitioners, who develop a sense of intuition and common sense about what is acceptable and good over time without explicitly assessing it. Most of us do this naturally in our everyday lives — for example, you could save eight seconds on your walk home by cutting across the grass instead of staying on the path, but you’d end up with muddy shoes.
If these requirements aren’t expressed in a way that an optimizer will understand, the resulting solutions might be problematic — sometimes in subtle ways that go unnoticed until later in the design process.
Natural Hazards
We now move to the technical set of challenges engineers face when striving for optimization. Once the objective landscape is constructed and the constraints formalized, a suitable algorithm must be employed — one that can perform optimization of highly non-convex functions while dealing with multiple constraints. There is a range of algorithms designed for this setting, with the biggest taxonomical distinction being gradient-based and gradient-free. Since objectives and constraints typically do not come with gradients with respect to the solution dimensions (CAD parameters), we rely on gradient-free methods for global optimization. The gradient-free taxon admits further distinction between model-based and model-free approaches. The latter has gained popularity in engineering, especially evolutionary algorithms (CMA-ES, GA) that optimize without constructing any explicit model for the objectives and constraints. Model-based approaches have also seen use in some engineering domains, but perhaps to a limited extent. Bayesian optimization supported by Gaussian processes to model the objective and constraints has seen some success in computation-heavy domains (Frazier, 2018; Hvarfner et al., 2024; Siivola et al., 2021).
.png){kind=spacer}
Evaluating candidate solutions in the engineering context often involves expensive simulations, both in time and compute resources; for example, a CFD simulation can take hours or even days. In such cases, effective optimization requires building an underlying model that captures the relationships between objectives and constraints, along with defining a suitable design space. This model is optimized as a surrogate for the expensive simulation. While machine learning literature offers a wide range of options that can theoretically recover (almost) any such relationships that might exist, these are often so general that many evaluations are required before any kind of reasonable model fit emerges. This is not a great problem in theory, as the method is eventually guaranteed to find a good solution, but in practice, “eventually” is not competitive.
Finally, hurdles with adequate modeling extend to a lack of unified common representation when it comes to representing geometry. In the course of a project, or multiple projects, the CAD representation of related geometries evolves, either by changing slightly (e.g., adding new parameters) or completely (scrapping the original parametrization and starting fresh). This makes it hard to benefit from prior work. Still, even within the confines of a single CAD parametrization with no prior compatible data, proper optimization methods have a lot to offer.
Optimization Recipes: Best Practices
We reviewed some basic ingredients for optimization in engineering and highlighted a few major hazards. The “artificial hazards” presented above are a consequence of engineering disciplines prioritizing modeling at the expense of optimization but also relate to knowledge organization, knowledge codification, and cultural practices within a company. We leave these aside for now, while we focus on the “natural hazards” that are amenable to technical treatments: modeling and geometry representation. Thankfully, we have an arsenal of tactics at our disposal to overcome these issues. We will focus on the challenging situation where the optimization problem involves an expensive simulation, and so we must construct a surrogate model for this simulation in order to successfully optimize. The setting in which the underlying simulation is easy to evaluate is more amenable to direct optimization.
To effectively model objectives and constraints, we can employ models that are well-suited for the domain at hand. These might be purely data-driven machine learning models or models that are informed by physics. In general, it is beneficial if the model can quantify its uncertainty in order to direct the search for new and better candidates, although even surrogates with no uncertainty quantification can be highly effective.
To reduce the time taken to reach useful solutions, we can also take advantage of high-performance computing (HPC) capabilities, which are now ubiquitous in engineering — instead of sequentially evaluating individual candidates with a CAE simulation on a workstation, a batch of candidates can be simulated concurrently on a cluster. The dataset for training an initial model of the objective and constraints can also be collected in this way. The key to success here is to define consistent processes and automate as much as possible, including CAD geometry creation from parameters, meshing, simulation configuration, resource scheduling, simulation convergence criteria, and post-processing.
The last challenge of placing available data in a common solution space, across diverse CAD parametrizations, is a harder one to address. In recent months, we have turned our attention to variational autoencoders (VAEs) (Kingma and Welling, 2022) that construct low-dimensional latent spaces for geometries in an unsupervised manner (Zhang et al., 2023; Park et al., 2019). Regardless of how the geometries were constructed, a VAE learns to place them in a common, continuous, latent space, which can then be used as the solution space for the optimizer. This enables us to combine data from different parametrizations, or even extend the available solution space beyond the confines of a particular CAD configuration. This is the technology powering PhysicsX’s recent Ai.rplane tool, which allows users to create new aircraft designs drawn from the VAE latent space.
Optimization Recipes: Designing an Aircraft
We will use a simplified conceptual aircraft design task to illustrate the pitfalls of incorrectly specifying an optimization problem. For this demonstration, we use AeroSandbox (Sharpe, 2021) as we can create a 3D aircraft geometry and run a rough aerodynamic analysis in under a second.
First, we define a parameter space containing a wide variety of aircraft. There are 20 geometric parameters in total, allowing shape variation of the wing, fuselage, and tailplane. Random samples from this space are visualized in Figure 2. We will also include three operational variables in the optimization space — altitude, airspeed, and angle of attack — leading to a 23-dimensional optimization problem.
Basic Optimization
Most engineering graduates can at least be expected to understand the concept of maximizing a function — for example, they may have used Newton-Raphson or similar basic methods. Given such a design space, one can imagine posing a simple unconstrained single-objective function. Having fixed the solution space as the geometric parameters, there are then two decisions to be made:
- What optimization algorithm should I use?
- What is the objective?
The answer to the first question is situation-dependent, and some familiarity is required to make a good choice. Two important factors are whether the simulation is differentiable and how long it takes to evaluate it. Although the analysis methods in AeroSandbox are typically fast and differentiable (in fact, it has its own design optimization framework), for now, we will use it as a placeholder for an expensive, non-differentiable CFD simulation. In such situations, Bayesian optimization with a Gaussian process surrogate is often a good choice and has already found success in aerospace design problems, remaining an active area of research (Lam et al., 2018). For this demonstration, we used BoTorch (Balandat et al., 2020), although there are a range of other suitable BO frameworks (Picheny et al., 2023; Song et al., 2022).
The second question requires a good understanding of what you are actually trying to achieve, and making the right choice is critical. To illustrate this, let us start by simply trying to maximize lift: an aircraft has to generate lift in order to fly, so clearly more is better. Unsurprisingly, this is probably not the right objective, as the optimizer just makes the wing as large as possible with a very high incidence of the oncoming flow. The operational variables have also been tuned so that the aircraft flies as low and fast as possible. This will certainly make for a high lift force, but clearly, we are likely to run into trouble if we attempt to fly this particular design.
Multi-objective Optimization
If lift is something that should be high, then drag is something that should be low; why not ask our optimizer to do both? For multi-objective problems, we form a Pareto front from our objectives and search for candidates that push this out as much as possible. Our objective becomes to maximize the area (or, with more than two objectives, the hypervolume) under the Pareto front. We will also make some other improvements: the airspeed and altitude do not need to be optimization variables, because we know they affect lift and drag only through dynamic pressure, neglecting compressibility effects. We can remove them and focus on the non-dimensional lift and drag coefficients.
As far as this optimization is concerned, all of the points on the Pareto front are optimal — an aircraft with a lift coefficient of 2 and a drag coefficient of 0.1 is as valuable as one with lift and drag coefficients of 0.5 and 0.01 respectively. The decision of which design to pick is left to us. This is helpful if we are just exploring the set of high-performing aircraft, but we will likely need to apply a rule to decide which of these designs to move forward with. In practice, if we already know this rule, then we would be better off using it in our objective function rather than wasting computations on exploring the entire Pareto front.
Constrained Optimization
An obvious way to unify lift and drag coefficients is by computing the aerodynamic efficiency, $\frac{L}{D} \equiv \frac{C_L}{C_D}$. Maximizing this gives us a good chance of developing an aircraft with both long range and endurance. Now we return to having a single objective, but maybe we should think about more than just that. For instance, we could also evaluate the longitudinal static stability of the aircraft — if the aircraft is uncontrollable, then reasoning about how far it can fly is worthless. Of course, modern fighter jets are famously unstable, which allows them to have higher performance, but let’s assume for now that we don’t want to use advanced control systems. This gives us two constraints that we want all of our designs to satisfy:
- The pitching moment about the center of mass, $C_M$, has to be zero, otherwise, the aircraft will instantly pitch up or down away from the design point.
- The pitching moment derivative with respect to the angle of attack, $\frac{\partial{C_M}}{\partial\alpha}$, must be negative, otherwise a small increase in angle will typically result in the aircraft flipping nose-over-tail.
The first of these is known as an equality constraint, and the second is an inequality constraint. Inequality constraints are easier to deal with as there is an area of feasibility, but equality constraints typically have to be relaxed so that there is some allowable tolerance above and below the target value. The solution to this optimization problem presents a very different picture than before. The main wing is now long and skinny (high aspect ratio), and the horizontal stabilizer now acts to balance out the moment generated by the main wing about the center of gravity (rather than being used as a lift-generating device). The angle of attack is also no longer maximized, as it ensures that the now slender fuselage is more aligned with the oncoming flow. The estimated L/D of this aircraft is about 50, which is about the same as a high-performance glider. However, it is worth noting that some real-world gliders have L/D ratios as high as 80; a major contribution to the performance gap here is the lack of wingtip devices in our design space, which act to increase the effective aspect ratio of the wing. This demonstrates how important it is to thoughtfully set up the design space with enough degrees of freedom based on expert input; no matter how well we optimize over this parametrization, the final candidates would likely always be inferior to those chosen from a space where wingtips were included. Of course, we have to draw the line somewhere when constructing an explicit design space.
.png){kind=spacer}
Next Steps
We’ve set up an optimization problem that results in aerodynamically efficient gliders, but what if we want to carry cargo, or a pilot? What if we want to achieve powered flight with motors and battery packs? What if there are regulatory constraints on aircraft dimensions that must be respected? The engineer defining the optimization problem must be adept at translating the intended use case into an appropriate set of objectives and constraints. Furthermore, the real design task is not just an aerodynamic problem with packaging constraints but also includes the structural design of the vehicle. This can also be included in the optimization problem, but this often results in an explosion in the size of the design space, as internal geometry is less amenable to explicit low-dimensional parametrization than external geometry. The VAE approach mentioned earlier can help with this.
One rule to bear in mind is that the closer your optimization objective is to what you really want (for most companies, this is profit), the better your result will be. Aerodynamic efficiency is a proxy for the energy required to transport a mass over a distance, which is an approximation of energy expenditure to achieve a desired mission profile, which is just another way of expressing operational cost. In an ideal world, we would like to have a model for the entire lifetime value of a design and optimize that. Since this is typically an intractable and infeasible task, we settle for optimizing smaller components in a modular fashion and with the best proxy to the overall value that we can find. The modularity also offers redundancy and separation of concerns, which in the real world can translate to robustness, compatibility standards, economies of scale, and distributed supply chains.
The process we have followed here is as applicable to component-level design as it is to system-level design, although system-level design tasks tend to be more interesting as demonstrations. The point we wish to underscore is simply that framing tasks in an optimization paradigm, with objectives and constraints, can yield tangible improvements for an organization’s goals. Of course, we are not suggesting that an entire aircraft can be designed fully autonomously in an optimization loop — at least, not yet. We’re working on it.
Engineering today is driven by iterative refinement, but too often, optimization remains an afterthought. Our technology brings it to the forefront, systematically improving designs with deep learning surrogates while balancing competing objectives. If you’re ready to move beyond trial and error and embed effective optimization into your engineering process, contact us at info@physicsx.ai. Let’s redefine how engineering problems are solved.
References
- M. Balandat, B. Karrer, D. R. Jiang, S. Daulton, B. Letham, A. G. Wilson, and E. Bakshy. BoTorch: A Framework for Efficient Monte-Carlo Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020. URL http://arxiv.org/abs/1910.06403.
- P. I. Frazier. A tutorial on Bayesian optimization, 2018. URL https://arxiv.org/ abs/1807.02811.
- C. Hvarfner, E. O. Hellsten, and L. Nardi. Vanilla Bayesian optimization performs great in high dimensions. In R. Salakhutdinov, Z. Kolter, K. Heller, A. Weller, N.
- Oliver, J. Scarlett, and F. Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 20793–20817. PMLR, 21–27 Jul 2024. URL https: //proceedings.mlr.press/v235/hvarfner24a.html.
- D. P. Kingma and M. Welling. Auto-encoding variational Bayes, 2022. URL https: //arxiv.org/abs/1312.6114.
- R. Lam, M. Poloczek, P. Frazier, and K. E. Willcox. Advances in Bayesian optimization with applications in aerospace engineering. In 2018 AIAA Non-Deterministic Approaches Conference, 2018. doi: 10.2514/6.2018-1656. URL https://arc.aiaa. org/doi/abs/10.2514/6.2018-1656.
- J. R. R. A. Martins and A. Ning. Engineering Design Optimization. Cambridge University Press, 2021.
- J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
- V. Picheny, J. Berkeley, H. B. Moss, H. Stojic, U. Granta, S. W. Ober, A. Artemev, K. Ghani, A. Goodall, A. Paleyes, S. Vakili, S. Pascual-Diaz, S. Markou, J. Qing, N.
- R. B. S. Loka, and I. Couckuyt. Trieste: Efficiently exploring the depths of black-box functions with tensorflow, 2023. URL https://arxiv.org/abs/2302.08436.
- P. D. Sharpe. AeroSandbox: A differentiable framework for aircraft design optimization. Master’s thesis, Massachusetts Institute of Technology, 2021. Available at https: //dspace.mit.edu/handle/1721.1/140023.
- E. Siivola, A. Paleyes, J. Gonz ́alez, and A. Vehtari. Good practices for Bayesian optimization of high dimensional structured spaces. Applied AI Letters, 2(2):e24, 2021. doi: https://doi.org/10.1002/ail2.24. URL https://onlinelibrary.wiley. com/doi/abs/10.1002/ail2.24.
- X. Song, S. Perel, C. Lee, G. Kochanski, and D. Golovin. Open source vizier: Distributed infrastructure and API for reliable and flexible black-box optimization. In Automated Machine Learning Conference, Systems Track (AutoML-Conf Systems), 2022.
- B. Zhang, J. Tang, M. Nießner, and P. Wonka. 3DShape2VecSet: A 3D shape representation for neural fields and generative diffusion models. ACM Trans. Graph., 42(4), jul 2023. ISSN 0730-0301. doi: 10.1145/3592442. URL https://doi.org/10.1145/ 3592442.