WorksComputer Science
Bayesian Optimization For Multi-Objective Mixed-Variable Problems
Abstract
Optimizing multiple, non-preferential objectives for mixed-variable, expensive black-box problems is important in many areas of engineering and science. The expensive, noisy, black-box nature of these problems makes them ideal candidates for Bayesian optimization (BO). Mixed-variable and multi-objective problems, however, are a challenge due to BO's underlying smooth Gaussian process surrogate model. Current multi-objective BO algorithms cannot deal with mixed-variable problems. We present MixMOBO, the first mixed-variable, multi-objective Bayesian optimization framework for such problems. Using MixMOBO, optimal Pareto-fronts for multi-objective, mixed-variable design spaces can be found efficiently while ensuring diverse solutions. The method is sufficiently flexible to incorporate different kernels and acquisition functions, including those that were developed for mixed-variable or multi-objective problems by other authors. We also present HedgeMO, a modified Hedge strategy that uses a portfolio of acquisition functions for multi-objective problems. We present a new acquisition function, SMC. Our results show that MixMOBO performs well against other mixed-variable algorithms on synthetic problems. We apply MixMOBO to the real-world design of an architected material and show that our optimal design, which was experimentally fabricated and validated, has a normalized strain energy density times greater than existing structures.
Introduction
Optimization is an inherent part of design for complex physical systems. Often optimization problems are posed as noisy black-box problems subject to constraints, where each function call requires an extremely expensive computation or a physical experiment. Many of these problems require optimizing a mixed-variable design space (combinatorial, discrete, and continuous) for multiple non-preferential objectives. Architected material design [1, 2, 3, 4, 5, 6], hyper-parameter tuning for machine learning algorithms [7, 8, 9], drug design [10, 11], fluid machinery [12? –14] and, controller sensor placement [15] pose such problems. Due to their cost of evaluation, Bayesian optimization is a natural candidate for their optimization.
Much research has gone into Bayesian optimization for continuous design spaces using Gaussian processes (GP) as a surrogate model and efficiently optimizing this design space with a minimum number of expensive function calls [16, 17, 18]. Despite the success of continuous variable Bayesian optimization strategies, multi-objective and mixed-variable problems remain an area of open research. The inherent continuous nature of GP makes dealing with mixed-variable problems challenging. Finding a Pareto-front for multi-objective problems, and parallelizing function calls for batch updates, Q-batch, also remain challenges in the sequential setting of the BO algorithm. Hedge strategies, which use a portfolio of acquisition functions to reduce the effect of choosing a particular acquisition function, have not been formulated for multi-objective problems.
Mixed-Variable BO Algorithms:
We provide a brief description of the current approaches in recent studies for dealing with mixed variables.
One Hot Encoding Approach: Most BO schemes use Gaussian processes as surrogate models. When dealing with categorical variables, a common method is ‘one-hot encoding’ [19]. Popular BO packages, such as GPyOpt and Spearmint [7], use this strategy. However, this can result in inefficiency when searching the parameter space because the surrogate model is continuous. For categorical variables, this approach also leads to a quick explosion in dimensional space [20].
Multi-Armed Bandit (MAB) Approach: Some studies use the MAB approach when dealing with categorical variables where a surrogate surface for continuous variables is defined for each bandit arm. These strategies can be expensive in terms of the number of samples required [21, 22], and they do not share information across categories. An interesting approach, where coupling is introduced between continuous and categorical variables, is presented in the CoCaBO algorithm [20], and it is one of the baselines that we test MixMOBO against.
Latent Space Approach: A latent variable approach has also been proposed to model categorical variables [23, 24, 25, 26]. This approach embeds each categorical variable in a latent variable space. However, the embedding is dependent on the kernel chosen, and for small-data settings can be inefficient.
Modified Kernel Approach: There is a rich collection of studies in which the underlying kernel is modified to work with ordinal or categorical variables.
For example, Ru et al. [20] considers the sum + product kernel; Deshwal et al. [27] proposes hybrid diffusion kernels, HyBO; and Oh et al. [28] proposes frequency modulated kernels. The BOCS algorithm [29] for categorical variables uses a scalable modified acquisition function. Nguyen et al. [22], Pelamatti et al. [30], Oh et al. [31], Garrido-Merchán and Hernández-Lobato [32] all use modified kernels to adapt the underlying surrogate surface. Our approach is unique in that any modified kernel can be incorporated into our framework. Currently we use the modified radial basis function (RBF) kernel for modelling the surrogate surface, with our future research focused on using different kernels in our framework.
Other Surrogate Models: Other surrogate models can be used in place of the GP to model mixed-variable problems such as random forests, an approach used by SMAC3 [33] or tree based estimators, used by the Tree-Parzen Estimator (TPE) [34]. Daxberger et al. [35] considers a linear model with cross-product features. BORE [36] leverages the connection to density ratio estimation.
Multi-Objective BO Algorithms:
Multi-objective Bayesian optimization (MOBO) has been the subject of some recent studies. BoTorch [37], the popular BO framework, uses the EHVI and ParEGO based on the works of Fonseca et al. [38] and Daulton et al. [39, 40]. Hyper-volume improvement is the main mechanism used to ensure diversity in generations. ‘-batch’ parallel settings of the above two acquisition functions use hyper-volume improvement and the previously selected point in the same batch to choose the next set of points. For most single-objective BO algorithms with parallel batch selection, the next batch of test points is selected by adding the ‘fantasy’ cost-function evaluation, usually the predicted mean, to the previously selected test point within that batch. However, this commonly used method often leads to overly confident test point selection, and the surrogate surface then needs to be optimized, and sometimes refitted times. Using a genetic algorithm (GA), we can select a ‘-batch’ of points with a single optimization of the surrogate surface from the GA generation.
Suzuki et al. [41] provide an interesting Pareto-frontier entropy method as an acquisition function, and Shu et al. [42] use Pareto-frontier heuristics to formulate new acquisition functions. Their approaches were not extended to mixed-variable problems.
Hedge Strategies
Hedge algorithms have proven to be efficient in dealing with a diverse set of problems by using a portfolio of acquisition functions. ‘GP-Hedge’, introduced by Brochu et al. [43] is a well-known and efficient algorithm. However, current Hedge algorithms have not been extended for multi-objective problems and, to the authors’ knowledge, there is no existing Hedge strategy implementation that solves such problems.
MixMOBO
In this paper, we present a Mixed-variable, Multi-Objective Bayesian Optimization (MixMOBO) algorithm, the first generalized framework that can deal with mixed-variable, multi-objective problems in small data setting and can optimize a noisy black-box function with a small number of function calls.
Genetic algorithms, such as NSGA-II [44], are well known for dealing with mixed-variable spaces and finding an optimal Pareto-frontier. However, these algorithms require a large number of black-box function calls and are not well suited to expensive small-data problems. Our approach is to use a GA to optimize the surrogate model itself and find a Pareto-frontier. Diversification is ensured by the distance metrics used while optimizing the surrogate model. This method allows cheap -batch samples from within the GA generation, and also allows the use of commonly used acquisition functions such as Expected Improvement (EI), Probability of Improvement (PI) and Upper Confidence Bound (UCB) [43], which work well for single objective problems. We note here that other metrics can easily be incorporated instead of a distance metric within the GA setting and is one of the areas of our future work. Using a GA on a mixed variable surrogate model in a multi-objective setting allows us to work with modified kernels that were developed for mixed-variable problems in literature. We also present a new acquisition function, ‘Stochastic Monte-Carlo’ (SMC), which performs well for multi-objective categorical variable problems [6].
Hedge strategies for Bayesian optimization are efficient for single objective algorithms. We present here our Hedge Multi-Objective (HedgeMO) algorithm, which uses a portfolio of acquisition functions for multi-objective problems and can work with -batch updates. It is an extension of GP-Hedge [43], which has regret bounds, and the same bounds hold for HedgeMO.
We note here that MixMOBO is designed for mixed-variable, multi-objective problems. Although there are algorithms in the literature that can solve problems with a subset of these attributes (e.g. mixed-variable single-objective or multi-objective continuous variable problems), no algorithm, to our knowledge, can deal with all of these attributes. In addition, MixMOBO outputs a batch of query points and uses HedgeMO, the first multi-objective hedging strategy. To the authors’ knowledge, no existing approaches can achieve all this within a single framework.
In summary, the main contributions of our work are as follows:
We present Mixed-variable, Multi-Objective Bayesian Optimization (MixMOBO), the first algorithm that can deal with mixed-variable, multi-objective problems. The framework uses GA to optimize the acquisition function on a surrogate surface, so it can use modified kernels or surrogate surfaces developed to deal with mixed-variable problems in previous studies. This extends the capabilities of previous approaches in literature to handle mixed-variable and multi-objective problems as well if adopted within our framework, since our framework is agnostic to the underlying GP kernel over mixed-variables.
GA is used to optimize surrogate models, which allows the optimization of multi-objective problems. ‘-batch’ samples can be extracted in parallel from within the GA generation without sacrificing diversification.
We present a Hedge Multi-Objective (HedgeMO) strategy for multiple objectives for which regret bounds hold. We also present an acquisition function, Stochastic Monte-Carlo (SMC), which performs well for combinatorial multi-objective problems, and use it as part of our HedgeMO portfolio.
We benchmark our algorithm against other mixed-variable algorithms and prove that MixMOBO performs well on test functions. We applied MixMOBO to a practical engineering problem: the design of a new architected meta-material that was optimized to have the maximum possible strain-energy density within the constraints of a design space. The fabrication and testing of this new material showed that is has a normalized strain energy density that is times greater than existing unblemished microlattice structures in literature.
The rest of the paper is organized in the following manner: § 3 defines the optimization problem to be solved with MixMOBO. The detailed workings of MixMOBO and HedgeMO are presented in § 4. § 5 details the validation tests performed on our framework to test its efficiency and comparison to existing algorithms. Our application of MixMOBO for design of architected materials and its results are presented in § 6.
MixMOBO Problem Statement
We pose the multi-objective and mixed-variable problem as:
for maximizing the objective. Here are the non-preferential objectives to be maximized, and is a mixed-variable vector, defined as . is an -dimensional vector defined over a bounded set representing continuous variables. Ordinal and categorical variables are defined as and , respectively. Each variable takes one of ordinal ‘levels’ (discrete numbers on the real-number line) and each categorical variable takes a value from unordered categories (that cannot, by definition, be ordered on the real-number line). and are the ordinal and combinatorial spaces respectively.
Generally, is a set of Pareto-optimal solution vectors i.e., vectors that are not Pareto-dominated by any other vector. A vector is Pareto-dominated by , iff . This is the optimal set found by MixMOBO, details of which are presented in the following section.
Methodology
Preliminaries
Single-objective Bayesian optimization is a sequential optimization technique, aimed at finding the global optimum of a single objective noisy black-box function with minimum number of evaluations of . For every iteration, a surrogate model, , is fit over the existing data set . An acquisition function then determines the next point for evaluation with , balancing exploration and exploitation. Data is appended for the next iteration, , and the process is repeated until the evaluation budget for or the global optimum is reached.
Gaussian processes are often used as surrogate models for BO [17, 45]. A GP is defined as a stochastic process such that a linear combination of a finite set of the random variables is a multivariate Gaussian distribution. A GP is uniquely specified by its mean and covariance function . The GP is a distribution over functions, and is a function sampled from this GP:
Here, is the covariance between input variables and . Once a GP has been defined, at any the GP returns the mean and variance . The acquisition function , balances exploration and exploitation, and is optimized to find the next optimal point . The success of BO comes from the fact that evaluating is much cheaper than evaluating .
MixMOBO Approach
Our Mixed-variable Multi-Objective Bayesian Optimization (MixMOBO) algorithm extends the single-objective, continuous variable BO approach presented in the preceding section, to more generalized optimization problems and is detailed in Algorithm 1.
A single noisy GP surrogate surface is fit for multiple objectives, . Note that this is different from Eq. (2), since the GP would predict mean for multiple objectives. For details on fitting a single GP to multi-objective data, we refer the reader to [17 Eq. 2.25-2.26]. For multiple objectives, the response vector, with -data points, is of size . The predicted variance remains the same, but the predicted mean is a vector. This is equivalent to fitting GP surfaces with the same kernel for all of the surfaces, where is the total number of objectives. All objectives are assumed to have equal noise levels. Only one set of hyper-parameters needs to be fit over this single surface, rather than fitting sets of hyper-parameters for different surfaces; thus, when is large, the overall computational cost for the algorithm is reduced. Note that we could fit different GP surfaces, particularly if different noise levels for different objectives is to be considered, with different hyper-parameters to the data to add further flexibility to the fitted surfaces. This idea will be investigated in our future work. We use LOOCV [45] for estimating hyper-parameters since we are dealing with small-data problems.
Gaussian processes are defined for continuous variables. For mixed variables, we need to adapt the kernel so that a GP can be fit over these variables. Cited works in § 1 dealt with modified kernels that were designed to model mixed variables. Those kernels can be used in the MixMOBO algorithm. For the current study, we use a simple modified squared exponential kernel:
where is a vector containing all the hyper-parameters, is the covariance hyper-parameter matrix and is the vector of covariance lengths. The distance metric, , is an concatenated vector, with the distance between categorical variables defined to be the Hamming distance, and the distance between continuous variables and the distance between ordinal variables defined to be their Euclidean distances. Noise is added to the diagonal of the covariance matrix. We emphasize that any modified kernel discussed in the citations of § 1 can be used within our framework and is a focus of our future work.
Algorithm 1 Mixed-variable Multi-Objective Bayesian Optimization (MixMOBO) Algorithm
Input: Black-box function , initial data set size , batch points per epoch , total epochs , mutation rate
Initialize: Sample black-box function for
for to do
Fit a noisy Gaussian process surrogate function
For total acquisition functions, from each acquisition function, propose -batch test-points, with the constrained search space using multi-objective GA
Mutate point within the search space with probability rate if -norm of its difference with any other member in set is below tolerance
Select batch of points using HedgeMO,
Evaluate the selected points from the black-box function,
Update
end for
return Pareto-optimal solution set
Once the GP is fit over multi-objective data, acquisition functions, , explore the surrogate model to maximize reward by balancing exploration and exploitation. Using a standard optimization scheme is problematic when dealing with mixed-variable and multi-objective problems due to non-smooth surrogate surface and conflicting objectives. We propose using a constrained, multi-objective GA to optimize the acquisition functions, which, although expensive to use on an actual black-box function, is an ideal candidate for optimizing the acquisition function working on the surrogate surface. For multi-objective problems, multi-objective GA algorithms, such as [44], are ideal candidates for obtaining a Pareto-front of optimal solutions.
Within a GA generation, for multi-objectives, diversification is ensured by a ‘distance crowding function’ which ranks the members of a non-dominated Pareto-front. The ‘distance crowding function’ can be computed in decision-variable space, in function space or a hybrid of the two, and ensures that the generations are distinct and diverse. This inherent feature of GA is exploited to ensure diversity in the ‘Q-batch’ of points. The ranking takes place when choosing the test points from an acquisition function for a multi-objective problem because the choice must take into account the diversity of the solution and propagate the Pareto-front. Because the members of the population are ranked by the GA, we can easily extract a ‘Q-batch’ of points from each of the acquisition functions without needing to add any ‘fantasy’ cost function evaluations or optimizing the acquisition functions again. This is a great advantage of using GA as our optimizer since we can output a ‘Q-batch’ of diverse query points using the inherent GA features.
For dealing with mixed-variable problems, GA are again ideal candidates. Genetic algorithms (GA) can be easily be constrained to work in mixed variable spaces. These variables can be dealt with by using probabilistic mutation rates. The genes are allowed to mutate within their prescribed categories, thereby constraining the proposed test points to the space.
Common acquisition functions, such as EI, PI, and UCB, can be used within this framework and can be used to nominate a ‘Q-batch’ of points. If a candidate in a Q-batch is within the tolerance limit of another candidate in the same batch or a previous data point (for convex functions), we mutate the proposed point within to avoid sampling the same data point again.
Test points are selected from to evaluate their using HedgeMO algorithm which is detailed in the next section. HedgeMO selects a ‘Q-batch’ of test-points from the candidates proposed by each of the acquisition functions. These points are then, along with their function evaluations , appended to the data set.
HedgeMO Algorithm
Hedge strategies use a portfolio of acquisition functions, rather than a single acquisition function. It
| Algorithm 2 HedgeMO Algorithm |
| 1: Input: Surrogate function , proposed test points by AFs , batch points per epoch , current epoch , total objective , parameter |
| 2: for to do |
| 3: For acquisition function, find rewards for -batch points nominated by that AF from epochs , by sampling from , , where for each objective |
| 4: end for |
| 5: Normalize rewards for each AF and objective, , where is defined as |
| 6: Calculate probability for selecting nominees from acquisition function, |
| 7: for to do |
| 8: Select nominee as test-point from AF with probability |
| 9: end for |
| 10: return Batch of test points |
Table 2.
is an extension to multi-objective problems of GP-Hedge as proposed by [43]. HedgeMO is part of our MixMOBO algorithm that not only extends the Hedge strategy to multi-objective problems, but also allows ‘Q-batches’. Our algorithm is shown in Algorithm 2.
Extending the methodology presented by [43], HedgeMO chooses the next ‘Q-batch’ of test points from the history of the candidates nominated by all of the acquisition functions. Rewards are calculated for each acquisition function from the surrogate surface for the entire history of the nominated points by the acquisition functions. The rewards are then normalized to scale them to the same range for each objective. This step is fundamentally important because it prevents biasing the probability of any objective. This type of bias, of course, cannot occur in single objective problems. The rewards for different objectives are then summed and the probability, , of choosing a nominee from a specific acquisition function is calculated using step 6 in Algorithm 2. For a ‘Q-batch’ of tests points, the test points are chosen times.
Regret Bounds: The regret bounds derived by Brochu et al. [43] hold for HedgeMO if and only if the Upper Confidence Bound (UCB) acquisition function is a part of the portfolio of acquisition functions. The regret bounds follow from the work of Srinivas et al. [46] who derived cumulative regret bounds for UCB. In essence, the cumulative regret in our case is bounded by two sublinear terms as for UCB and an additional term which depends on proximity of the chosen point with the test point proposed by UCB. The interested reader is directed to Srinivas et al. [46] and Brochu et al. [43] for a description of the exact regret bounds and their derivation.
SMC Acquisition Function
We introduce a new acquisition function, Stochastic Monte-Carlo (SMC), which for the maximization of an objective, is defined as:
where is sampled from , and and are the mean and standard deviation returned by the GP at , respectively. This is equivalent to taking Monte-Carlo samples from a truncated distribution. For categorical and ordinal variable problems, this acquisition function performs well across a range of benchmark tests [6]. We use this acquisition function as part of our portfolio of HedgeMO in the MixMOBO algorithm.
Validation Tests
MixMOBO is designed to deal with mixed-variable, multi-objective problems. However, no other small-data algorithm, to the authors’ knowledge, can similarly deal with all the attributes of such problems to provide an honest comparison. In the absence of such competition, we use the specific case of mixed-variable, single-objective problems to provide a comparison to state-of-the-art algorithms present for such problems and prove that even for this subset case, MixMOBO is able to perform better than existing algorithms in the literature. We then perform further experiments in both single
and multi-objective settings to show the efficacy of the HedgeMO algorithm compared to stand-alone acquisition functions and the performance of SMC in the multi-objective setting.
We benchmarked MixMOBO against a range of existing state-of-the-art optimization strategies that are commonly used for optimizing expensive black-box functions with mixed-variable design spaces. We chose the following single objective optimization algorithms for comparison: CoCaBO [20], which combines the multi-armed bandit (MAB) and Bayesian optimization approaches by using a mixing kernel. CoCaBO has been shown to be more efficient than GPyOpt (one-hot encoding approach [47]) and EXP3BO (multi-armed bandit (MAB approach [21]). We used CoCaBO with a mixing parameter of 0.5. We also tested MixMOBO against GBRT, a sequential optimization technique using gradient boosted regression trees [48]. TPE-Hyperopt (Tree-structured Parzen Estimator) is a sequential method for optimizing expensive black-box functions, introduced by Bergstra et al. [49]. SMAC3 is a popular Bayesian optimization algorithm in combination with an aggressive racing mechanism [50]. Both of these algorithms, in addition to Random Sampling, were used as baselines. Publicly available libraries for these algorithms were used.
Six different test functions for mixed variables were chosen as benchmarks. A brief description of these test functions and their properties is given below with further details in Appendix § 10:
Contamination Problem: This problem, introduced by Hu et al. [51], considers a food supply chain with various stages in the chain where food may be contaminated with pathogens. The objective is to maximize the reward of prevention efforts while making sure the chain does not get contaminated. It is widely used as a benchmark for binary categorical variables. We use the problem as a benchmark with 21 binary categorical variables.
Encrypted Amalgamated: An anisotropic, mixed-variable function created using a combination of other commonly used test functions [52]. We modify the combined function so that it can be used with mixed variables. In particular, it is adapted for categorical variables by encrypting the input space with a random vector, which produces a random landscape mimicking categorical variables [6]. Our Encrypted Amalgamated function has 13 inputs: 8 categorical, 3 ordinal variables (with 5 categories or states each) and 2 continuous.
NK Landscapes: This is a popular benchmark for simulating categorical variable problems using randomly rugged, interconnected landscapes [53, 54]. The fitness landscape can be produced with random connectivity and number of optima. The problem is widely used in evolutionary biology and control optimization and is NP-complete. The probability of connectivity between NK is controlled by a ‘ruggedness’ parameter, which we set at 20%. We test the Li et al. [54] variant with 8 categorical variables with 4 categories each.
Rastringin: This is an isotropic test function, commonly used for continuous design spaces [52]. We use a 9-D Rastringin function for testing a design space of 3 continuous and 6 ordinal variables with 5 discrete states.
Encrypted Syblinski-Tang: This function is isotropic [52], and we have modified it as we did with the Encrypted Amalgamated test function so that it can work with categorical variables and was used as a representative benchmark for -categorical variable problems. The 10-D variant tested here consists only of categorical variables with 5 categories each.
Encrypted ZDT6: This is a multi-objective test function introduced by Zitzler et al. [55] that we modified with encryption so that it can deal with mixed variables. The test function is non-convex and non-uniform in the parameter space. We test ZDT6 with 10 categorical variables with 5 states each. ZDT6 was only used for testing HedgeMO.
To the extent of our knowledge, no other optimization algorithm is capable of handling mixed-variable, multi-objective problems in small-data settings. Thus, we have no direct comparisons between MixMOBO and other published algorithms. Therefore, we tested MixMOBO against a variant of NSGA-II [44] with the ZDT4 and ZDT6 test functions with mixed variables. However, we found that using a GA required more than more function calls to find the Pareto front to a similar tolerance. For visualization purposes, we do not plot the GA results.
All of the optimization algorithms were run as maximizers, with a 0.005 noise variance built into all the benchmarks. The budget for each benchmark test was fixed at 250 function calls including the evaluations of 50 initial randomly sampled data points for all algorithms, except for SMAC3 which determines its own initial sample size. The algorithms were run in single output setting (GBRT, CoCaBO and MixMOBO’s batch mode was not used for fair comparison). Each algorithm was run 10 times for every benchmark. Our metric for optimization is the ‘Normalized Reward’, defined as (current optimum - random sampling optimum)/(global optimum - random sampling optimum). Figure 1 shows the Normalized Rewards versus the number of black-box function evaluations for MixMOBO and five other algorithms. The mean and standard deviation of the Normalized

Figure 1. Performance comparison of MixMOBO against other mixed-variable algorithms
Rewards of the 10 runs for each algorithm, along with their standard deviations (S.D.), are plotted. The width of each of the translucent colored bands is equal to 1/5 of their S.D.
MixMOBO outperforms all of the other baselines and is significantly better in dealing with mixed-variable problems. GBRT is the next best algorithm and performs better than MixMOBO on the Rastrigin function; however, note that the Rastrigin function does not include any categorical variables. For problems involving categorical variables, MixMOBO clearly outperforms the others. TPE and CoCaBO have similar performances, and SMAC3 has the poorest performance. All three are outdistanced by MixMOBO.
We then perform experiments to test the performance of our HedgeMO algorithm by comparing it to four different acquisition functions which make up the entirety of its portfolio. These acquisition functions, namely, EI, PI, UCB, and SMC, along with HedgeMO are tested on three different test functions: the Encrypted Amalgamated, Encrypted Syblinski-Tang, and Encrypted ZDT6. The latter is used as the multi-objective test function. The Normalized Reward for the multi-objective Encrypted ZDT6 is defined as (current -optimum - random sampling -optimum)/(global -optimum - random sampling -optimum). Here,
where is the number of global Pareto-optimal points.
The results of our acquisition function comparisons are shown in Figure 2, which shows that HedgeMO performs well across all three test functions. For single-objective test functions, PI outperforms HedgeMO for Encrypted Amalgamated test function. However, for the multi-objective Encrypted ZDT6 test function, PI performs significantly worse and is outperformed by both SMC and UCB. SMC performs well on multi-objective problems combinatorial problems and hence should be a part of portfolio for a hedging algorithm.

Figure 2. Performance comparison of HedgeMO against other acquisition functions
These results prove that for a range of different problems, acquisition function choice can play a huge role in the performance of the algorithm. For a black-box function, this information can not be known a priori, making hedging necessary. HedgeMO consistently performs well in all scenarios and ensures efficiency across a range of different problems. Thus, for unknown black-box functions, HedgeMO should be the hedge strategy of choice for multi-objective problems.
Application to Architected Materials
We applied our MixMOBO framework to the optimization of the design of architected, microlattice structures. Advances in modeling, fabrication, and testing of architected materials have promulgated their utility in engineering applications, such as ultralight [56, 57, 58], reconfigurable [59], and high-energy-absorption materials [60], and in bio-implants [61]. The optimization of architected materials [57, 58, 59, 62] often requires searching huge combinatorial design spaces, where the evaluation of each design is expensive. [63, 64, 65] The design space for the architected material we optimize here has billion possible combinations of its 17 categorical inputs (one with 2 possible states, and the other 16 with 4 possible states). Our goal is to maximize the strain energy density of a microlattice structure. We maximize the strain energy density (which is extremely expensive to compute, even for one design) by minimizing the buckling load , while maintaining the lattice’s structural integrity and stiffness before fracturing. Minimizing (a proxy for maximizing the strain energy density by instigating buckling which leads to the densification of the deformed lattice members) is a more computationally tractable cost function to evaluate (but, it is still expensive and involves solving a numerical finite element code for each evaluation of the cost function.) The manufacturing and testing details of our methodology are included in Vangelatos et al. [6], which focuses on the material aspects of the problem.
The design space consists of choosing one of four possible unit cells (shown in the upper left of Figure 3, each with one or more defects (shown in color) in them, at each of the 16 independent lattice sites) creating 16 of the categorical inputs with 4 possible values; and the choice of whether the cells are connected along their faces or along their edges on 45°-diagonals (shown in the upper right panel of Figure 3) creating the 17th categorical input with 2 possible values.

Figure 3. Top Left: The 4 unit cells, labelled A – D. Top Right: The 2 orientations in which they can be joined. Bottom Left: Optimization results using MixMOBO. Bottom Right: SEM images of Unblemished and Optimum structures.
The minimization of using MixMOBO was initialized with 50 random structures and the evaluation budget, including initial samples, was set at 250. The algorithm achieved a 42% improvement in the of the lattice structure over the best structure obtained with the first 50 random samples (Figure 3). The optimal microlattice obtained using as a proxy with MixMOBO has an experimentally measured normalized strain energy density that is 12,030 times greater than that of the unblemished microlattice structure with no defects that is cited in the literature to have the best strain energy density [66], a 4 orders of magnitude improvement. Table 1 shows the properties of the fabricated and experimentally measured design created with MixMOBO. The choices of the unit cells in the optimally designed lattice that were determined by MixMOBO are not intuitive and have no obvious pattern or structure. Images of our optimized structures using Helium Ion Microscopy (HIM) [4], shows the comparison of the Unblemished structure from literature with our MixMOBO Optimal structure, before and after loading. It is evident that the MixMOBO Optimal structure due to its densification mechanism, can handle much higher loads without breaking [6].
| Structure | |||||
| Unblemished | 3814.5 | 1.08 | 0.071 | 388.21 | 1 |
| Random Sampling Optimal | 996.2 | 0.08 | 2.85 | 347.19 | 526 |
| MixMOBO Optimal | 545.1 | 0.02 | 14.71 | 460.35 | 12030 |
Table 1. Experimental values of the critical buckling (minimization objective for MixMOBO), strain energy density at buckling and fracture, and respectively, elastic stiffness , and ratio of normalized strain energy density compared to the Unblemished structure.
Conclusions
The existing optimization literature does not offer an algorithm for optimizing multi-objective, mixed-variable problems with expensive black-box functions. We have introduced Mixed-variable Multi-Objective Bayesian Optimization (MixMOBO), the first BO based algorithm for optimizing such problems. MixMOBO is agnostic to the underlying kernel. It is compatible with modified kernels and other surrogate methods developed in previous studies for mixed-variable problems. MixMOBO allows for parallel batch updates without repeated evaluations of the surrogate surface, while maintaining diversification within the solution set. We presented the Hedge Multi-Objective (HedgeMO) algorithm, a novel Hedge strategy for which regret bounds hold for multi-objective problems. A new acquisition function, Stochastic Monte-Carlo (SMC) was also proposed as part of the HedgeMO portfolio. MixMOBO and HedgeMO were benchmarked and shown to be significantly better on a variety of test problems compared to existing mixed-variable optimization algorithms.

Figure 4. HIM images of the loaded and unloaded unblemished and optimum structures. (a) Unloaded Unblemished structure (b) Unblemished structure after loading, showing severe fracture and collapse of many beam members. (c) Focused image revealing several fractured beams and the internal collapse of the upper layer that subsequently instigated the accumulation of damage in the underlying layers. (d) Unloaded MixMOBO Optimum structure. (e) MixMOBO Optimum structure after the structure was subjected to the same maximum compressive load as the structure shown in (b). Unloading of the optimum structure showed only excessive plastic deformation without catastrophic collapse and the manifestation of the buckling mode. (f) Focused revealing the effect of buckling that led to deformation but no fracture due to the occurrence of densification. (g) Side view of the unloaded optimum structure. (h) Side view of the unloaded optimum structure revealing that fracture was inhibited throughout the structure due to the densification precipitated by the low critical buckling load. Scale: Each scale bar is equal to 10 m.
MixMOBO was then applied to the real-world optimization of an architected micro-lattice, and we increased the structure’s strain-energy density by compared to existing Unblemished structures in the literature reported to have highest strain energy density. Our future work entails further testing multi-objective and ‘Q-batch’ settings. We have also applied MixMOBO for optimization of draft-tubes for hydrokinetic turbines [13] and Cauchy symmetric meta-material structures and are currently applying it for optimization of vertical-axis wind turbines.
Acknowledgments
The authors would like to thank Chiyu ‘Max’ Jiang, research scientist at Waymo Research, and Professor Uros Seljak, Department of Physics, University of California at Berkeley (UCB) for insightful discussions regarding Bayesian optimization. We would also like to thank Zacharias Vangelatos and Professor Costas P. Grigoropoulos, Department of Mechanical Engineering, University of California at Berkeley (UCB) for the collaboration to design and manufacture architected materials, conduct nanoindentation, SEM, and HIM experiments. This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation grant number ACI-1548562 through allocation TG-CTS190047.
Declarations
Funding and Competing Interests
The authors declare funding sources, conflicts or competing interests that need to be disclosed.
Availability of Data and Materials
Complete data sets to reproduce any and all experiments which were generated during and analysed during the current study and MixMOBO code are available from the corresponding author on reasonable request.
Authors’ Contributions
H.M.S conceptualized the algorithm, designed the methodology and performed experiments under the supervision of P.S.M. H.M.S and P.S.M then wrote and edited the manuscript.
Replication of Results
All the results in this manuscript can be replicated. The complete data sets, MixMOBO algorithm or any other supplementary material and information required for replication are available from the corresponding author on reasonable request.
Benchmark Test Functions
In this section, we define the benchmark test functions, all of which are set to be maximized during our optimizations.
Contamination Problem
The contamination problem was introduced by Hu et al. [51] and is used to test categorical variables with binary categories. The problem aims to maximize the reward function for applying a preventative measure to stop contamination in a food supply chain with stages. At each stage, where , decontamination efforts can be applied. However, this effort comes at a cost and will decrease the contamination by a random rate . If no prevention effort is taken, the contamination spreads with a rate of . At each stage , the fraction of contaminated food is given by the recursive relation:
here and is the decision variable to determine if preventative measures are taken at stage or not. The goal is to decide which stages action should be taken to make sure does not exceed an upper limit . and are determined by a uniform distribution. We consider the problem setup with Lagrangian relaxation [29]:
Here violation of is penalized by and summing the contaminated stages if the limit is violated and our total stages or dimensions are . The cost is set to be and . As in the setup for [29], we use stages, , and .
Encrypted Amalgamated
Analytic test functions generally cannot mimic mixed variables. To map the continuous output of a function into discrete ordinal or categorical variables, the continuous range of the test function’s output is first discretized into discrete subranges by selecting break points, often equally spaced, within the bounds of the range. Then, the continuous output variable is assigned the integer round-off value of the subrange defined by its surrounding pair of break points. If necessary, the domain of the test function’s output is first mapped into a larger domain so that each subrange has a unique integer value. To mimic ordinal variables, we are done, but for categorical variables, a random vector for each categorical variable is then generated which scrambles or ‘encrypts’ the indices of these values, thus creating random landscapes as is the case for categorical variables with a latent space. The optimization algorithm only sees the encrypted space and the random vector is only used when evaluating the black-box function.
We also define a new test function that we call the Amalgamated function, a piece-wise function formed from commonly used analytical test functions with different features (for more details on these functions we refer to Tušar et al. [52]). The Amalgamated function is non-convex and anisotropic, unlike conventional test functions where isotropy can be exploited.
For , :
where
To create the Encrypted Amalgamated function, for categorical and ordinal variables, equally spaced points are taken within the bounds defined above. For our current work, we use a with 8 categorical and 3 ordinal variables with 5 states each, and 2 continuous variables.
NK Landscapes
NK Landscapes were introduced by Kauffman and Levin [53] as a way of creating optimization problems with categorical variables. describes the number of genes or number of dimensions and is the number of epistatic links of each gene to other genes, which describes the ‘ruggedness’ of the landscape. A large number of random landscapes can be created for given and values. The global optimum of a generated landscape for experimentation can only be computed through complete enumeration. The landscape cost for any vector is calculated as an average of each component cost. Each component cost is based on the random values generated for the categories, not only by its own alleles, but also by the alleles in the other genes connected through the random epistasis matrix, with probability or ruggedness. A ruggedness translates to a fully connected genome.
The Landscapes from Kauffman and Levin [53] were formulated only for binary variables. They were extended by Li et al. [54] for multi-categorical problems, which is the formulation we use. Details of the Landscape test-functions we use can be found in Li et al. [54]. For the current study, we use with 4 categories each and ruggedness .
Rastringin
Rastringin function is a commonly used non-convex optimization function [52] with a large number of local optima. It is defined as:
We use for testing with 6 ordinal with 5 discrete states and 3 continuous variables. The ordinal variables are equally spaced within the bounds.
Encrypted Syblinski-Tang
We use the Syblinski-Tang function [52], an isotropic non-convex function. The function is considered difficult to optimize because many search algorithms get ‘stuck’ at a local optimum. For use with categorical variables, we encrypt it as described previously. The Syblinski-Tang function, in terms of input vector , is defined as:
For the current study, this function was tested with categorical variables and 5 categories for each variable.
Encrypted ZDT6
benchmarks are a suite of multi-objective problems, suggested by Zitzler et al. [55], and most commonly used for testing such problems. We use , which is non-convex and non-uniform in its parameter space. We again modify the function by encrypting it to work with categorical problems. is defined as:
Here and for . The function was tested for with 5 categories each. We note that to evaluate the performance of MixMOBO, we compared it against the NSGA-II variant [44] that can deal with mixed variables (by running in a mixed variable setting and with categorical variables). No encryption is necessary for GAs. GAs required, on average, more function calls compared to MixMOBO.
References
- [1]Peter I. Frazier and Jialei Wang. Bayesian optimization for materials design. Springer Series in Materials Science, page 45–75, Dec 2015. ISSN 2196-2812. doi: 10.1007/978-3-319-23871-5_3.DOI
- [2]Desai Chen, Mélina Skouras, Bo Zhu, and Wojciech Matusik. Computational discovery of extremal microstructure families. Science Advances, 4(1):eaao7005, 2018.DOI
- [3]Wen Chen, Seth Watts, Julie A Jackson, William L Smith, Daniel A Tortorelli, and Christopher M Spadaccini. Stiff isotropic lattices beyond the maxwell criterion. Science Advances, 5(9):eaaw1937, 2019.DOI
- [4]Lucas A Shaw, Frederick Sun, Carlos M Portela, Rodolfo I Barranco, Julia R Greer, and Jonathan B Hopkins. Computationally efficient design of directionally compliant metamaterials. Nature Communications, 10 (1):1–13, 2019.DOI
- [5]Jian Song, Yuejiao Wang, Wenzhao Zhou, Rong Fan, Bin Yu, Yang Lu, and Lixiao Li. Topology optimization-guided lattice composites and their mechanical characterizations. Composites Part B: Engineering, 160:402–411, 2019.DOI
- [6]Zacharias Vangelatos, Haris Moazam Sheikh, Philip S. Marcus, Costas P. Grigoropoulos, Victor Z. Lopez, George Flamourakis, and Maria Farsari. Strength through defects: A novel bayesian approach for the optimization of architected materials. Science Advances, 7 (41), 2021. doi: 10.1126/sciadv.abk2218.DOI
- [7]Jasper Snoek, Hugo Larochelle, and Ryan P. Adams. Practical bayesian optimization of machine learning algorithms, 2012.arxiv.org/abs/1206.2944
- [8]Yutian Chen, Aja Huang, Ziyu Wang, Ioannis Antonoglou, Julian Schrittwieser, David Silver, and Nando de Freitas. Bayesian optimization in alphago. CoRR, abs/1812.06855, 2018.arxiv.org/abs/1812.06855
- [9]ChangYong Oh, Efstratios Gavves, and Max Welling. BOCK : Bayesian optimization with cylindrical kernels. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3868–3877. PMLR, 10–15 Jul 2018.arxiv.org/abs/1806.01619
- [10]Edward Pyzer-Knapp. Bayesian optimization for accelerated drug discovery. IBM Journal of Research and Development, PP:1–1, 11 2018. doi: 10.1147/JRD.2018.2881731.
- [11]Ksenia Korovina, Sailun Xu, Kirthevasan Kandasamy, Willie Neiswanger, Barnabas Poczos, Jeff Schneider, and Eric Xing. Chembo: Bayesian optimization of small organic molecules with synthesizable recommendations. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 3393–3403. PMLR, 26–28 Aug 2020.arxiv.org/abs/1908.01425
- [13]Haris Moazam Sheikh, Tess A. Callan, Kealan J. Hennessy, and Philip S. Marcus. Optimization of the shape of a hydrokinetic turbine’s draft tube and hub assembly using design-by-morphing with bayesian optimization, 2022. URL https://arxiv.org/abs/2207.11451.
- [15]Andreas Krause, Ajit Singh, and Carlos Guestrin. Near-optimal sensor placements in gaussian processes: Theory, efficient algorithms and empirical studies. Journal of Machine Learning Research, 9(8):235–284, 2008.
- [16]J. Mockus. Application of bayesian approach to numerical methods of global and stochastic optimization. Journal of Global Optimization, 4:347–365, 1994.DOI
- [17]Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. Adaptive computation and machine learning. MIT Press, 2006. ISBN 026218253X.
- [18]Eric Brochu, Vlad M. Cora, and Nando de Freitas. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning, 2010.arxiv.org/abs/1012.2599
- [19]Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. Google vizier: A service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’17, page 1487–1495, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 978145034874. doi: 10.1145/3097983.3098043.DOI
- [20]Binxin Ru, Ahsan S. Alvi, Vu Nguyen, Michael A. Osborne, and Stephen J Roberts. Bayesian optimisation over multiple continuous and categorical inputs, 2020.arxiv.org/abs/1906.08878
- [21]Shivapratap Gopakumar, Sunil Gupta, Santu Rana, Vu Nguyen, and Svetha Venkatesh. Algorithmic assurance: An active approach to algorithmic testing using bayesian optimisation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
- [22]Dang Nguyen, Sunil Gupta, Santu Rana, Alistair Shilton, and Svetha Venkatesh. Bayesian optimization for categorical and category-specific continuous inputs, 2019. 004.
- [23]Peter Z G Qian, Huaiqing Wu, and CF Jeff Wu. Gaussian process models for computer experiments with qualitative and quantitative factors. Technometrics, 50(3):383–396, 2008.
- [24]Qiang Zhou, Peter ZG Qian, and Shiyu Zhou. A simple approach to emulation for computer models with qualitative and quantitative factors. Technometrics, 53(3):266–273, 2011.DOI
- [25]Yichi Zhang, Daniel W Apley, and Wei Chen. Bayesian optimization for materials design with mixed quantitative and qualitative variables. Scientific Reports, 10(1):1–13, 2020.arxiv.org/abs/1910.01688
- [26]Aryan Deshwal and Janardhan Rao Doppa. Combining latent space and structured kernels for bayesian optimization over combinatorial spaces. CoRR, abs/2111.01186, 2021.arxiv.org/abs/2111.01186
- [27]Aryan Deshwal, Syrine Belakaria, and Janardhan Rao Doppa. Bayesian optimization over hybrid spaces, 2021.arxiv.org/abs/2106.04682
- [28]Changyong Oh, Efstratios Gavves, and Max Welling. Mixed variable bayesian optimization with frequency modulated kernels, 2021.arxiv.org/abs/2102.12792
- [29]Ricardo Baptista and Matthias Poloczek. Bayesian optimization of combinatorial structures, 2018.arxiv.org/abs/1806.08838
- [30]Julien Pelamatti, Loic Brevault, Mathieu Balesdent, El-Ghazali Talbi, and Yannick Guerin. Efficient global optimization of constrained mixed variable problems, 2018.arxiv.org/abs/1806.03975
- [31]Changyong Oh, Jakub M. Tomczak, Efstratios Gavves, and Max Welling. Combinatorial bayesian optimization using the graph cartesian product, 2019.arxiv.org/abs/1902.00448
- [32]Eduardo C. Garrido-Merchán and Daniel Hernández-Lobato. Dealing with categorical and integer-valued variables in bayesian optimization with gaussian processes. Neurocomputing, 380:20–35, Mar 2020. ISSN 0925-2312. doi: 10.1016/j.neucom.2019.11.004.
- [33]Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, René Sass, and Frank Hutter. Smac3: A versatile bayesian optimization package for hyperparameter optimization, 2021.arxiv.org/abs/2109.09831
- [34]James Bergstra, Daniel Yamins, and David Cox. Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures. In Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pages 115–123, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR.
- [35]Erik Daxberger, Anastasia Makarova, Matteo Turchetta, and Andreas Krause. Mixed-variable bayesian optimization. Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, Jul 2020. doi: 10.24963/ijcai.2020/365.DOI
- [36]Louis C. Tiao, Aaron Klein, Matthias Seeger, Edwin V. Bonilla, Cedric Archambeau, and Fabio Ramos. Bore: Bayesian optimization by density-ratio estimation, 2021.arxiv.org/abs/2102.09009
- [37]Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. Botorch: A framework for efficient monte-carlo bayesian optimization, 2020.arxiv.org/abs/1910.06403
- [38]C.M. Fonseca, L. Paquete, and M. Lopez-Ibanez. An improved dimension-sweep algorithm for the hypervolume indicator. In 2006 IEEE International Conference on Evolutionary Computation, pages 1157–1163, 2006. doi: 10.1109/CEC.2006.1688440.DOI
- [39]Samuel Daulton, Maximilian Balandat, and Eytan Bakshy. Differentiable expected hypervolume improvement for parallel multi-objective bayesian optimization, 2020.arxiv.org/abs/2006.05078
- [40]Samuel Daulton, David Eriksson, Maximilian Balandat, and Eytan Bakshy. Multi-objective bayesian optimization over high-dimensional search spaces, 2021.arxiv.org/abs/2109.10964
- [41]Shinya Suzuki, Shion Takeno, Tomoyuki Tamura, Kazuki Shitara, and Masayuki Karasuyama. Multi-objective bayesian optimization using pareto-frontier entropy, 2020.arxiv.org/abs/1906.00127
- [42]Leshi Shu, Ping Jiang, Xinyu Shao, and Yan Wang. A New Multi-Objective Bayesian Optimization Formulation With the Acquisition Function for Convergence and Diversity. Journal of Mechanical Design, 142(9), 03 2020. ISSN 1050-0472. doi: 10.1115/1.4046508. 091703.DOI
- [43]Eric Brochu, Matthew W. Hoffman, and Nando de Freitas. Portfolio allocation for bayesian optimization, 2011.arxiv.org/abs/1009.5419
- [44]K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. A fast and elitist multiobjective genetic algorithm: Nsga-ii. IEEE Transactions on Evolutionary Computations, 6(2):182–197, 2002. doi: 10.1109/4235.996017.DOI
- [45]K. P. Murphy. Machine learning: a probabilistic perspective. MIT Press, 2012.
- [46]Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Information-theoretic regret bounds for gaussian process optimization in the bandit setting. IEEE Transactions on Information Theory, 58(5):3250–3265, May 2012. ISSN 1557-9654. doi: 10.1109/tit.2011.2182033.DOI
- [47]GPyOptAuthors. GPyOpt: A bayesian optimization framework in python. http://github.com/SheffieldML/GPyOpt, 2016.
- [48]Scikit-learn. scikit-optimize. https://scikit-optimize.github.io/stable/, 2021.
- [49]James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. Algorithms for hyper-parameter optimization. In Proceedings of the 24th International Conference on Neural Information Processing Systems, NIPS’11, page 2546–2554, Red Hook, NY, USA, 2011. Curran Associates Inc. ISBN 9781618395993.
- [50]Frank Hutter, Holger H. Hoos, and Kevin Leyton-Brown. Sequential model-based optimization for general algorithm configuration. In Carlos A. Coello Coello, editor, Learning and Intelligent Optimization, pages 507–523, Berlin, Heidelberg, 2011. Springer Berlin Heidelberg. ISBN 978-3-642-25566-3.DOI
- [51]Yingjie Hu, Jianqiang Hu, Yifan Xu, Fengchun Wang, and Rong Cao. Contamination control in food supply chain. pages 2678 – 2681, 01 2011. doi: 10.1109/WSC.2010.5678963.DOI
- [52]Tea Tušar, Dimo Brockhoff, and Nikolaus Hansen. Mixed-integer benchmark problems for single- and bi-objective optimization. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’19, page 718–726, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450361118. doi: 10.1145/3321707.3321868.DOI
- [53]Stuart Kauffman and Simon Levin. Towards a general theory of adaptive walks on rugged landscapes. Journal of Theoretical Biology, 128(1):11–45, 1987. ISSN 0022-5193. doi: https://doi.org/10.1016/S0022-5193(87)80029-2.
- [54]Rui Li, Michael Emmerich, Jeroen Eggermont, Ernst V. Bovekamp, Thomas Bäck, Jouke Dijkstra, and Johan Reiber. Mixed-integer nk landscapes. volume 4193, pages 42–51, 01 2006. ISBN 978-3-540-38990-3. doi: 10.1007/11844297_5.DOI
- [55]Eckart Zitzler, Kalyanmoy Deb, and Lothar Thiele. Comparison of multiobjective evolutionary algorithms: Empirical results. Evol. Comput., 8(2):173–195, jun 2000. ISSN 1063-6560. doi: 10.1162/106365600568202.DOI
- [56]Xiaoyu Zheng, Howon Lee, Todd H Weisgraber, Maxim Shusteff, Joshua DeOtte, Eric B Duoss, Joshua D Kuntz, Monika M Biener, Qi Ge, Julie A Jackson, et al. Ultra-light, ultrastiff mechanical metamaterials. Science, 344(6190):1373–1377, 2014.
- [57]Minh-Son Pham, Chen Liu, Iain Todd, and Jedsada Lertthanasarn. Damage-tolerant architected materials inspired by crystal microstructure. Nature, 565(7739):305–311, 2019.DOI
- [58]Xuan Zhang, Andrey Vyatskikh, Huajian Gao, Julia R Greer, and Xiaoyan Li. Lightweight, flaw-tolerant, and ultrastrong nanoarchitected carbon. Proceedings of the National Academy of Sciences, 116(14):6665–6672, 2019.DOI
- [59]Xiaoxing Xia, Arman Afshar, Heng Yang, Carlos M Portela, Dennis M Kochmann, Claudio V Di Leo, and Julia R Greer. Electrochemically reconfigurable architected materials. Nature, 573(7773):205–213, 2019.DOI
- [60]Jian Song, Wenzhao Zhou, Yuejiao Wang, Rong Fan, Yinchu Wang, Junying Chen, Yang Lu, and Lixiao Li. Octet-truss cellular materials for improved mechanical properties and specific energy absorption. Materials & Design, 173:107773, 2019.DOI
- [61]Jiaxi Song, Christos Michas, Christopher S Chen, Alice E White, and Mark W Grinstaff. From simple to architecturally complex hydrogel scaffolds for cell and tissue engineering applications: Opportunities presented by two-photon polymerization. Advanced Healthcare Materials, 9(1):1901217, 2020.DOI
- [62]Jens Bauer, Almut Schroer, Ruth Schwaiger, and Oliver Kraft. Approaching theoretical strength in glassy carbon nanolattices. Nature Materials, 15(4):438–443, 2016.DOI
- [63]Lucas R Meza, Satyajit Das, and Julia R Greer. Strong, lightweight, and recoverable three-dimensional ceramic nanolattices. Science, 345(6202):1322–1326, 2014.DOI
- [64]JB Berger, HNG Wadley, and RM McMeeking. Mechanical metamaterials at the theoretical limit of isotropic elastic stiffness. Nature, 543(7646):533–537, 2017.DOI
- [65]Thomas Tancogne-Dejean, Marianna Diamantopoulou, Maysam B Gorji, Colin Bonatti, and Dirk Mohr. 3d plate-lattices: An emerging class of low-density metamaterial exhibiting optimal isotropic stiffness. Advanced Materials, 30(45):1803334, 2018.DOI
- [66]Z Evangelatos, K Komvopoulos, and C Grigoropoulos. Regulating the mechanical behavior of metamaterial microlattices by tactical structure modification. Journal of the Mechanics and Physics of Solids, page 104112, 2020.DOI