WorksComputer Science
DreamLander: A 6-DOF World Model based Reinforcement Learning Planetary Lander
Abstract
Soft powered descent landings are becoming evermore important in the field of space exploration and scientific advancement, because of their ability to deliver mission critical payloads, both highly accurately and safely. The high efficiency restriction on planetary landings due to the severely limited fuel supply results in the need for optimal guidance trajectories and control. Model based reinforcement learning is proving to be an important tool in simplifying the highly complex process of generating optimal guidance trajectories and control. In this work, we present DreamLander, a state of the art model based reinforcement learning algorithm based on DreamerV3, to generate optimal guidance and control for a realistic 6 degree of freedom entry descent landing problem. The model based approach creates a sample efficient and highly generalizable model that can be applied to a variety of scenarios whilst maintaining a high degree of accuracy, landing within 2 meters of the designated landing zone, and with velocity norm less than 0.3 meters per second. DreamLander makes use of state sensor observations, making progress toward End-to-End solutions of the Entry Descent Landing problem.
Introduction
Landing on extraterrestrial bodies, be it the Moon, other planets or even asteroids is a crucial part of any exploratory mission, and the task must be achieved safely and efficiently. Being able to safely deliver, resources, astronauts or robots to the surface of a celestial body is becoming even more paramount in missions, as seen with NASA’s Discovery and Artemis Programs, JAXA’s SLIM lander, SpaceX’s recapture of launch vehicles, as well as other upcoming missions, and with that the need for highly accurate, automated, reproducible guidance and control systems are becoming evermore sought after. Because of the delicate nature of the equipment and payloads onboard the lander, hard landing methods such as the use of parachutes and inflatable balloons are often discarded for more favorable soft, powered entry descent landings (EDL’s). A major benefit of such powered landings is that they can be autonomously guided and controlled [1, 2] such that a high degree of accuracy (within 5 meters of the target) can be achieved as well as small approach velocities of around 2 meters per second.
Automated guidance methods for the powered descent (PD) phase of the landings have been investigated since the Apollo program [3, 4, 5], however due to the severely limited computational capability of that era, complex constraints on the optimization problems resulted in the use of polynomial guidance methods rather than optimal control theory. As computational prowess increased and algorithms in nonlinear programming [6, 7, 8] improved the capability of these guidance and control algorithms by allowing the deviation from explicit guidance laws, simplified models, or process constraints [9, 10, 11]. Further improvements introduced variational gravitational acceleration and aerodynamic forces to the EDL problem, resulting in methods such as successive convex programming [12, 13], the Jet Propulsion Laboratory (JPL) Guidance for Fuel-Optimal Large Diverts (G-FOLD) [14, 15, 16], Legendre pseudospectral method [17] which all aim to improve the performance of the optimal guidance through convexification of the non convex problem. Control laws such as model predictive control (MPC), receding horizon control (RHC) and zero-effort-miss/zero-effort-velocity may then be used in conjunction with the optimal guidance algorithms to efficiently and accurately maneuver the lander vehicle [1, 18, 19].
With the recent popularity of neural network based approaches in a wide range of scientific fields, due to their powerful predictive and modeling capabilities, [20, 21, 22] it is no surprise that deep reinforcement learning (RL) algorithms have also been used in the discovery of guidance and control algorithms for the EDL problem [23, 24]. Deep RL is a class of machine learning control optimization algorithms that can be broken down into either model-based or model-free, each with its own advantages and disadvantages. Unlike Imitation Learning (IL) where an expert trajectory is used to guide the learning of the IL agent, RL algorithms make use of unsupervised learning in order to arrive at an optimal policy that maximizes the given reward function for the problem [25, 26, 27]. This results in a policy that is more robust and has the potential to learn and generalize to new environments far more quickly that an IL agent. Subclasses of RL include model-free algorithms such as REINFORCE [28], PPO [29], DDPG [30], TRPO [31] etc. As the name suggests, these classes of models do not learn a representative model of the dynamics of the world but simply learn to output actions based on the observations presented. The benefits of such a model is that is often easier to implement and can perform better in more complex environments [32, 33], however it is usually less sample efficient than model based RL algorithms. Model based RL however, is usually more sample efficient as well as allows the agent to plan ahead using certain outcomes and can generalize to different contexts and tasks within environments [34, 35], as well as include methods for safety and stability assurances[36, 37, 38].
A number of methods in developing high fidelity planetary lander agents using model free RL have been developed previously, using algorithms such as DDPG [24], TD3 and SAC [23] and PPO [39] and show success both in generalization and accuracy, however the use of a model-based approach is largely understudied, with only an implementation of the DreamerV3 model being applied within the Space Robotics Bench benchmark [40]. [24] employs transfer learning to generalize from the trained Martian environment to an astroid environment, but the highly generalizable properties of DreamerV3 suggest that with little hyperparameter tuning or few shot learning, our agent can successfully land on multiple environments. Our contributions in this work are as follows:
We propose a DreamerV3 based MBRL model to solve the late stage EDL problem using realistic physical parameters from the Apollo Moon landings.
Our model, DreamLander, is able to successfully solve the problem with higher performance and stricter accuracy than the original missions, ensuring greater safety.
We integrate realistic sensor based observations in the solution, making significant progress toward the full End-to-End approach to realistic, space robotics based problems.
Methodology
In this paper we develop DreamLander, which utilizes the popular and successful DreamerV3 model based reinforcement learning (MBRL) algorithm in order to generate optimal guidance trajectories and a control policy for EDL problems. DreamLander can be used in an End-to-End fashion to autonomously guide and control the lander to a successful, highly precise landing or, the generated guidance trajectories can be easily tracked by simple, reliable controllers such as a PID, adaptive, or MPC. The benefit of using the DreamerV3 algorithm is that, unlike model-free algorithms, because DreamerV3 learns a representation of the world through its latent vector, it is possible to generalize to new and varying environments with little hyperparameter tuning (few or zero shot adaptation [41, 42]). The robustness of this particular world model is that it can quickly be transferred to varying terrains, bodies and external forces so that a generalized framework that is suited for any range of EDL problems can be learned. We also note that due to the existence of the world model and latent space representation of the dynamics, that more rigorous and provable safety and stability guarantees with probabilistic guarantees can be formulated. The ability to ensure safety of the entire system can be performed in numerous methods including, Control Barrier Functions (CBFs) [43, 44], Hamilton-Jacobi Reachability Analysis [37, 45] or Lagrangian method constrained Markov Decision Processes [46, 47].
In the original work [33], DreamerV3 was shown to generalize and work well for a range of different environments, ranging from state based locomotion to pixel based 2D games, each with a number of tasks. However, many of the environments that DreamerV3 was applied to, were either toy problems or simple demonstrative physical problems [33]. The application of this framework to challenging, complex engineering problems has only recently begun to be explored, seen in the case of autonomous driving scenarios [48, 49], but as of yet very limited applications into safety critical systems such as spacecraft and planetary landers have been conducted [40]. We aim to show that world model RL frameworks such as DreamerV3 are usable as guidance and/or control systems in these space-based applications.
In a conventional guidance, navigation and control (GNC) pipeline, a number of systems must be developed in unison to complete the full stack on-board computer, such as state estimation, sensor fusion, measurement filtering and the controller development, which is both highly complex and time consuming. With the advent of RL models, a new approach to the GNC pipeline, known as End-to-End, in which the RL agent receives raw observations such as camera imagery, LiDAR data, IMU and other sensors, and outputs directly the next required control, can be designed. It has been shown that DreamerV3 can be utilized in this End-to-End fashion, and given we can solve the optimal guidance problem with the states of the system, hence the next step would be to integrate DreamerV3 in an End-to-End fashion.
This objective can be supported through NVIDIA’s IsaacLab, which is built on Isaac Sim, a simulation framework with high fidelity simulation, rendering, asset management and compatibility with RL algorithms.
In this work, we partially integrate raw sensor data into the observations, complimented with state data, to achieve the desired performance. We utilize the built-in Ray-Cast sensor which operates similar to a LiDAR to achieve the height mapping of the lander over the terrain. We incorporate an IMU to deliver velocity components, as well as a contact sensor on the legs of the lander to detect whether contact with the surface has been made and for how long contact has been made. Each of the sensors contain realistic noise levels with a mean noise of 0.08m for the LiDAR and and for velocity and angular velocity respectively on the IMU.
Simulation Framework
DreamLander is built on the Dreamer world model framework, first popularized as a method of RL in [32]. It consists of a recurrent neural network (RNN) that is able to learn and predict a representation of the environment in a specified latent space. It then uses this latent representation of the world to perform two tasks, the first being passing this latent information of the observations to the actor-critic policy that is responsible for determining the next best action to take to achieve the goal. And second, because of the inherent memory of the RNN, the world model can be used to create new latent states of the future, similar to the process of dreaming, and then use these imagined latent states to further train the actor-critic policy. A representation of the process for both tasks is given in Figure 1.

Figure 1. Process diagram showing the latent imagination of the model from an initial state to further states and the input of only the latent states to the agent to determine the action.
The environment upon which DreamLander trains on is based on the 6-degrees-of-freedom (6-DOF) equations of motion between the lander and the desired landing point, which are elaborated on in the next section. Because we use IsaacLab for the construction of the environment in which we train DreamLander, we can make use of the NVIDIA PhysX engine that is built-in to IsaacLab to handle the dynamics propagation. PhysX integrates collision capabilities and full 6-DOF constraint support, which makes it an ideal candidate for our simulation with multiple sensors. We also focus our simulation on the near-landing phase of the EDL problem, hence we assume that the gravity vector points directly down.
The terrain generation within the simulation environment in IsaacLab is achieved using FBX files that are specifically designed and textured to have similar color and geometric features such as craters, mountains and rocks, to that of the desired planet or asteroid being simulated. Having high fidelity renders of the terrain are especially important for photorealistic observations from cameras, and help to reduce the Sim2Real gap [50]. In this work we do not include pixel based observations, so the quality of the rendering of the textures need not but ultra-high, but high fidelity geometric features are required, to enable accurate geometric representation of the terrain.
The 3D model of the lander is another important aspect of the simulation framework. IsaacLab internally computes the inertia tensor of the vehicle based on user-defined mass or density parameters, so the choice of a lander that has geometrically similar features to that of a real vehicle, improves the overall fidelity of the simulation. We do not include a texture on the lander vehicle model as this does not impact the simulation. The Moon terrain and lander model can be visualized in Figure 2a and Figure 2b.


Figure 2. (a) 3D model of the lander used in simulation. The flame is purely for visual effect. (b) 3D model of the terrain used in simulation, emphasizing the shadows, craters, mountains and valleys that improve realism of the simulation. 3D models of the lander and terrain used to simulate the environment of an EDL problem in IsaacLab.
Entry Descent Landing Dynamics
Because of the fact that for many powered entry descent landing missions, a heat shield is utilized to protect the vehicle from the harsh environment of the initial reentry phase, and this shield is often jettisoned close to the ground with a short amount of time until landing, the dynamics of this problem are often only considered shortly after the jettison of the heat shield. As such, this results in the need for a control system that can orient itself quickly and provide robust control authority and guidance to make a safe and highly precise landing.
As the vehicle is near to the surface of the body it is landing on, we can make use of three reference frames to provide a complete description of the problem. The first is the fixed inertial frame with respect to the celestial body, similar to the Earth Centered Earth Fixed (ECEF) frame on Earth. The origin of the frame, is at the center of the body, is aligned with the axis of rotation of the celestial body, points East along the equator of the body and completes the right handed coordinate system. We can the define a second reference frame, the East North Up (ENU) frame, which has its origin at the location of the landing zone (LZ), and is oriented by rotating the Inertial frame by the desired Longitude and Latitude . For the purposes of the EDL simulation, the ENU frame is considered to be inertially fixed and denoted by with unit vectors , , . Finally the body frame is defined with its origin at the center of mass of the vehicle, the z-axis pointing upward out of the vehicle, the x-axis pointing toward the front of the vehicle and the y-axis completing the right hand coordinate system. The three coordinate systems can be visualized in Figure 3.

Figure 3. Coordinate frames for the 6-DOF environment used to represent the dynamics of the EDL problem.
The lander itself has a number of key physical properties that are assumed based on historical information from the Apollo missions [5] and prior works [51]. The mass of the vehicle is a constant 15000 kg whilst the vehicle encompasses a 4 m 4 m 4 m volume and the lander’s inertia tensor calculated by IsaacLab is given as:
Even though it is assumed that the lander expels propellant to generate thrust, the mass and inertia tensors are kept constant in this simulation. The thruster arrangement on our model is assumed to be similar to that of the Apollo lander, with one main thruster beneath the lander and 4 small RCS thruster clusters at each corner of the lander as shown in Figure 4. DreamLander is designed to output actions that correspond to the total forces and moments in the body frame as a result of all thrusters, since the alignment of the thrusters with respect to the body frame is constant at all times. Given that each RCS thruster is capable of a thrust range of 0N to 400N, the Main Engine is throttle-able between 0N to 43000N [52] and the mounting configuration, it can be calculated that the range of forces and moments in the body frames are as shown in Table 1.
| Input | Min | Max |
| (N) | -800 | 800 |
| (N) | -800 | 800 |
| (N) | 0 | 43000 |
| (Nm) | -3200 | 3200 |
| (Nm) | -3200 | 3200 |
| (Nm) | -3200 | 3200 |
Table 1. Minimum and Maximum values of force and moment that the lander vehicle is capable of in the body frame.

Figure 4. Diagram of the Reaction Control Subsystem thruster configuration of the Lunar Module Descent Stage [52].
The relationship between the force that is produced by each engine and the total force and moment that the vehicle experiences is given by,
Where, is the thrust commanded by each of the 16 RCS thrusters and main engine, and denote whether the vector is in the body or inertial frame and and are the unit thrust direction vectors and position vectors, in the body frame, of each thruster respectively. The definitions for both and are given in Table 2 and Table 3.
| Thruster | Unit Direction Vector |
| Main Engine | |
| RCS FR1 | |
| RCS FR2 | |
| RCS FR3 | |
| RCS FR4 | |
| RCS FL1 | |
| RCS FL2 | |
| RCS FL3 | |
| RCS FL4 | |
| RCS BL1 | |
| RCS BL2 | |
| RCS BL3 | |
| RCS BL4 | |
| RCS BR1 | |
| RCS BR2 | |
| RCS BR3 | |
| RCS BR4 |
Table 2. Unit Direction vectors for each thruster. FR, FL, BL, BR designate front right, front left, back left, back right. RCS cluster is counted from top going clockwise looking out from inside the lander.
| Thruster | Position Vector |
| Main Engine | |
| RCS FR1 | |
| RCS FR2 | |
| RCS FR3 | |
| RCS FR4 | |
| RCS FL1 | |
| RCS FL2 | |
| RCS FL3 | |
| RCS FL4 | |
| RCS BL1 | |
| RCS BL2 | |
| RCS BL3 | |
| RCS BL4 | |
| RCS BR1 | |
| RCS BR2 | |
| RCS BR3 | |
| RCS BR4 |
Table 3. Position vectors for each thrusters from the center of mass of the vehicle.
As DreamLander only outputs total force and moment control inputs, it is assumed that the total force or moment required is evenly split between each of the thrusters responsible for producing that force or moment. For example, if a desired yaw moment of 1600Nm around the body z-axis was commanded, then RCS thrusters FR2, FL2, BL2, BR2 would each contribute 200N of thrust to achieve the desired yawing moment.
The equations of motion for the lander, upon approach to the LZ can now be described by the traditional 6-DOF equations of motion using quaternions. Although it is unlikely that the vehicle will reach a state where gimble-lock becomes an issue, during training of DreamLander, a large space of the state space is often searched due to initially high entropy values, hence it is imperative that the dynamics can account for this. It is important to note that the states referenced by the dynamics are not the same states that are referenced by the RL algorithm, as in, the states used by DreamLander are the observations from the environment. As such we denote the state space state vector as:
Where is the quaternion that represents the rotation between the body frame and the inertial ENU coordinate frame , is the position vector of the lander in the inertial frame, is the angular velocity of the lander in the body frame, and is the velocity of the lander in the inertial frame. The control input can then be defined as
Hence, the controlled, nonlinear, continuous-time dynamical system is propagated according to the following mathematical expression
To evolve the dynamics of the lander, the PhysX engine uses a semi-implicit Euler Integration method, with a Temporal Gauss-Seidel (TGS) solver in order to account for the contact dynamics that are not included in Equation (6) [53]. Hence, the time derivatives of each of the states are required and can be given as follows:
Where, represents the quaternion product. Since quaternions are of dimension 4 and vectors are of dimension 3, the vectors must have a 0 appended in front of them to allow for valid quaternion multiplication. is the inertia tensor defined in Equation (1), is the gravity vector and is the mass of the vehicle.
The expression denotes a skew-symmetric matrix that is equivalent to performing the cross product and is defined as:
Finally, the value for can be found via the following relation between the body frame force, and the quaternion representing the rotation between the body and inertial frame:
The simulation and control policy that results is run at 10Hz, whereas the solver and simulation run at 60Hz to ensure stable and smooth dynamics and solutions to the differential equations.
DreamerLander Architecture
DreamLander is built from DreamerV3 which is a world-model based RL algorithm that incorporates a autoencoder model to map observations to and from the latent space, a Recurrent State Space Model (RSSM) to learn and propagate the dynamics of the system in the latent space as well as a simple Actor-Critic agent to learn and execute the controls required to succeed in the given task. The high-level breakdown of the World Model (WM) includes the RSSM, which is comprised contains the encoder, dynamics predictor and sequence model, as well as a series of heads which are the reward and continuation predictors and decoder shown in Figure 5

Figure 5. Architecture of the DreamerV3 WM.
The autoencoder needs to map raw observation inputs from the observation space to the usually lower dimensional latent space, hence for simulations in which the observations arrive as images, the pixels are encoded and decoded through a convolutional neural network (CNN) autoencoder. If the observation space is a vector then the encoding and decoding is performed via a fully connected multilayer perceptron (MLP) network.
The structure of the DreamerV3 network is closely based on the architecture of the DreamerV2 [54] world model that was shown to have good predictive accuracy on discrete world models. The RSSM [55], consists of multiple smaller networks that all play a specific role. As shown in Figure 1 there is a deterministic recurrent state that is an output of the RNN, which stores memory from past observations and states. At each step, the recurrent state is used to help predict a posterior latent state that contains information from the encoded image , as well as a prior latent state that does not include information from the current observation but is supposed to closely predict the posterior state . In training this is enforced by including the KL divergence between and in loss function. After training, since the prior state can accurately predict what would be the latent state with the next image, this is what is used for dreaming trajectories, without actual observations.
The model then uses a concatenated state of both the deterministic RNN state and the stochastic posterior state to feed 3 separate networks that generate a prediction of the next observation , the reward and the discount factor .
The behavior policy of the RL agent is a simple actor critic framework that utilizes the REINFORCE [28] algorithm to iteratively improve the actor toward the optimal control policy. The actor and critic are trained solely on latent state representations of the world, generated by the world model, hence this allows for significantly faster training and the ability to learn in the imagined trajectories created by the learned world model. During training of the actor and critic, the world model is fixed.
The actor and critic are trained cooperatively, meaning that the actor generates actions that lead to states that maximize the value output of the critic, whilst the critic uses the actions and states generated by the actor to accurately estimate the sum of rewards. Hence they both have separate loss functions that they try to optimize. The critic is parameterized on whilst the actor is parameterized on :
Where,
Here is the distribution of the return estimates, is the imagined model states, is the critics predicted values, is the actor output, and are discount factors, is the exponential moving average, is the percentile of some value and is the entropy of the model. Further details regarding the DreamerV3 architecture can be found in the original work [33].
Reinforcement Learning Problem
RL generates optimal solutions because of its nature of maximizing reward functions that have been specifically designed to improve the models behavior in reaching the goal. For any planetary lander, the objective function is to find the thrust program that minimizes the fuel consumption of the based on the thrust , specific impulse and reference gravity :
Subject to the constraints from the dynamics of the system and boundary conditions:
Where and are the position and velocity vectors of the lander in the inertial frame, respectively, denotes the initial state and denotes the final, desired state. In order to achieve this minimization of the objective function, the RL agent needs to have a reward function defined, whereby maximizing the reward function leads to a minimization of the objective function. For the EDL problem, constructing a reward function that is solely based on the optimization of the fuel leads to reward spaces that are both sparse and do not properly guide the agent to the right policy that optimizes the objective function. As such, the reward function that is used penalizes errors in the tracking position and velocity, penalizes excessive yaw, pitch and roll angles of the lander, penalizes excessive control effort, penalizes as the attitude approaches the attitude limits, rewards a bonus for successfully landing and penalizes a crash landing. Mathematically this looks like:
Where are weighting parameters for each of the rewards that are either positive or negative based on whether that portion of the reward function penalizes or rewards the agent and are detailed in Table 4. , , reward the agent for meeting the criteria of a successful landing, penalize for crashing and reward heavily for an aligned landing, respectively, is the potential function used in the potential-based reward for both position and velocity, is the error quaternion between the current and desired orientation, measuring the alignment of the vehicle with respect to its orientation and the conditions if any and if any apply the respective penalizes if those conditions are satisfied. penalizes large deviations from alignment, whilst penalizes the use of the thrusters when the vehicle is hovering above the LZ, to approximate shutting down the engines when close enough to land completely.
| Parameter | Value |
| -1.3 | |
| -1.3 | |
| 10 | |
| -0.05 | |
| 20 | |
| -10 | |
| 50 | |
| 2m | |
Table 4. Environment parameters for the Lunar Lander EDL problem
The use of potential-based reward shaping enables faster progress toward a goal without the notion of getting ‘distracted’ [56]. Thus, the potential function is defined as
Where, to induce a better trajectory that discourages hovering at low altitude whilst adjusting the x-y position, the weighted norm [57], is used to more strongly punish errors in the and directions. The intuition follows that because in this EDL problem, the distance from the LZ is significantly larger in the direction than the and directions, using an unweighted two-norm can potentially lead to an optimal policy in which the vehicle reduces its altitude faster than it can correct its error in the x-y directions. As such we define the weighted norm and its corresponding weighing matrix as
The second term in (24) penalizes misalignment between the body axis and the reference axis at the LZ. For simplicity, and because we utilize the quaternion in the expression of the orientation, we define the reference quaternion as . Hence the error quaternion can be found by
Where is the quaternion product and denotes the quaternion conjugate [58]. We then convert the error quaternion to axis-angle representation to find the angular error, in alignment between the two frames, as this can be used in the inverse exponential reward function to guide DreamLander to align the lander. The angle can be found via the following,
Where and are the vector and scalar part of the error quaternion, respectively. If then the alignment is within 3 degrees, and considered aligned.
Based on the reinforcement learning problem presented above, it is necessary to define an observation that can be computed based off the dynamical state, and fed into DreamLander. The observation is defined by the state vector given in (4), with the substitution of the position and velocity vectors with their corresponding error state, concatenated with the contact sensor observation , which indicates the current amount of time in contact between the feet of the lander and the terrain. Hence, the observation vector is then given as:
We again emphasize that both velocity (angular and linear) components of the state vector are obtained using the built-in IMU sensor within IsaacLab’s simulation framework. The altitude, corresponding to the z-axis of the position vector , is obtained via a downward facing LiDAR through IsaacLab’s Ray-Cast sensor, and the contact time is directly measured via the contact sensor that measures the intersection of boundaries between two objects in the simulation. The use of the error state in DreamLander’s observation vector allows for dynamic selection of the LZ, which may prove useful with more complex guidance or pixel based guidance that DreamerV3 is capable of.
Simulation Results & Discussion
DreamLander is specifically trained on solving the EDL problem for end-stage landing. The training process is conducted through IsaacLab which enables the use of multiple environments to enhance and speed up data collection and training as well as the incorporation of multiple sensors that cannot be analytically expressed through dynamic equations. However, due to difference in how IsaacLab and the DreamerV3 architecture treat multiple environments (IsaacLab only allows the creation of one instance of the world, with multiple environments inside that world, whereas DreamerV3, which uses Gym environments, expects to be able to create separate instances of the environments), a wrapper to synchronize the two architectures is needed to ensure proper training that utilized the efficiency of all environments. We trained DreamLander on a single NVIDIA GeForce RTX 3090 GPU, with 8 environments and the training hyperparameters outlined in Table 5. Training was completed on average in 17 hours. The related code can be found at: https://github.com/george-nehma/IsaacLab-DreamerV3.
| Parameter | Value |
| steps | 5e5 |
| action_repeat | 2 |
| prefill | 8000 |
| dyn_hidden | 256 |
| dyn_deter | 256 |
| units | 1024 |
| batch_size | 18 |
| batch_length | 128 |
| train_ratio | 1024 |
| envs | 8 |
Table 5. DreamLander main hyperparameters corresponding to the DreamerV3 PyTorch version [59].
Lunar Environment
In all simulations, both training and evaluation, the initial conditions were randomly distributed from the following bounds: and position; , position; , Euler angles ; , and velocity; , velocity; , angular velocities; . Specifically for the Moon environment, the gravitational acceleration was set at a constant . In order to determine whether the vehicle successfully landed, limits on the terminal state of the simulation were imposed, with the LZ set as and the origin of the ENU frame, , the condition for landing is as follows
Where is the limit on the distance from the desired landing point, and is the limit on the landing velocities. It should be noted that these landing conditions are stricter than those of the Apollo missions [5] and are deliberately chosen to emphasize the precise and high performance capability of MBRL algorithms. Each environment had a maximum time sec however, once DreamLander was trained and converged, all simulations were completed within 40 sec.
Our training session was run with a budget of 500,000 iteration steps as we found this to be long enough for DreamLander to learn the model of the world and converge to the optimal value for each of the individual losses. Within the DreamerV3 architecture, the train ratio hyperparameter controls the number of iteration steps taken per step of the actual environment. In our training, this value was set to 1024, with a batch size of 18 and batch length of 128. The RSSM dynamic predictor model has 256 GRU recurrent units, 1024 dense hidden units while the actor and critic heads each had 4 layers in their respective MLPs, with the continuation head only having 2 layers and the reward having 4. In total, this resulted in World Model network with 18,418,702 parameters, the actor with 4,476,940 and the critic with 4,726,015.
In order to evaluate our trained model, we twice ran 100 separate simulations, with randomly chosen initial conditions consistent with the training initial conditions, and landing criteria consistent with the conditions in Equation (30). The first set of 100 trajectories had the LZ set to the origin of the ENU frame, whilst the second set had it’s LZ set to the X-Y coordinates . Out of the 100 evaluation trajectories in the first set, 92 simulations successfully landed, whilst the other 8 simulations crashed. This resulted in DreamLander achieving a success rate of 98%. It should be noted however, that of the 8 crashed trajectories, 2 did not meet the landing criteria solely because of the norm of landing velocity being between 0.14 and 0.98 larger than the target landing velocity of 0.3 , 1 was unable to land within the precise landing radius despite having good velocities, and 5 missed both the landing velocity and accuracy criteria. For the two trajectories that missed the landing velocity, the velocity in the direction was satisfactory.
For the second set of trajectories, 81 successfully landed, 12 had an increased landing norm velocity between 0.01 and 1.24 larger than the threshold, whilst 7 crashed completely. Of the 12 that missed the landing velocity, the largest velocity was only 0.91 greater than the threshold, hence the primary cause of failure was due to an excessive direction velocity. Figure 6a and Figure 6b depict each of these scenarios in the evaluation simulations to emphasize the precise nature of DreamLander. Successful landing is marked by a green circle, missing due to position is marked by a yellow circle, crashing due to velocity is marked by an orange circle and a complete miss and crash is marked by a red cross.
The increased number of failure cases in the second set of trajectories can be attributed to uneven terrain, causing premature collision because of the inherent policy that is learned, since DreamLander appears to hover close to the ground and then translate toward the LZ. Although undesirable, through tuning of the reward function in future work, this can be alleviated.

(a) (a) Landing points for the 100 simulations with LZ at the origin.
!Landing points for the 100 simulations with LZ at [10, −5] bbox=0.58,0.12,0.79,0.28
(b) (b) Landing points for the 100 simulations with LZ at [10 −5].
Figure 6. Top-down view of the LZ with the designated landing points of each of the evaluations simulations, showing whether it was a successful, partially successful, or failed landing.
Figure 7a and Figure 7b show the landing trajectories of the 200 evaluation simulations and that every trajectory was able to regulate toward the LZ as would be expected. We note the set of evaluation trajectories where the coordinates of the LZ were set to the location [10 −5], to highlight the ability of the model to regulate not only to the origin but also to any desired location, which will be important when the LZ is dynamically allocated based on pixel observations in future work.

(a) (a) 3D view of each of the evaluation trajectories with the origin as the designated LZ.
!3D view of each of the evaluation trajectories with [10, −5] as the designated LZ bbox=0.59,0.50,0.77,0.84
(b) (b) 3D view of each of the evaluation trajectories with [10 −5] as the designated LZ.
Figure 7. 3D view of the 200 total evaluation trajectories.
A particularly interesting control strategy emerges as a result of DreamLander’s training. Whilst the initial position in the and directions is not substantially large, it can be large enough that simply using the force from the RCS thrusters may not allow the vehicle to converge precisely to the LZ. As such, whilst the vehicle is decreasing in altitude, full main engine use is not explicitly needed, DreamLander reorients the lander to tilt toward the direction of the LZ, and fire the main engine to allow for faster directional alignment, whilst also maintaining an appropriate descent rate. This effect can be seen in the both rows of images in Figure 8. This control strategy is both intuitive and akin to thrust vectoring controlled landing that is utilized by SpaceX in the recapture of their boosters [60] and other lander vehicles [61]. While we impose a penalty on the alignment of the vehicle with the ENU axis, balancing the rewards to achieve a plausible yet optimal control strategy is key.
Figure 8. Snapshots of a landing trajectory of DreamLander, highlighting its use of attitude control to utlize the main engine thrust in positional alignment.
In order to demonstrate the capability of DreamLander to generate trajectories that are smooth and trackable by when used as a guidance algorithm, we display the following position and velocity trajectories. Figure 9 displays the position trajectories for all three axes for each of the evaluation simulations, while Figure 10 displays the velocities. Although we penalize DreamLander for not aligning with the ENU frame, because of the natural control strategy it learns, throughout the trajectory, the vehicle may need to deviate from exact alignment, hence why the trajectory of the Euler angles does not exponentially converge like the position and velocities. Figure 11 shows this and the fact that despite this fact, they each stay close to zero.

Figure 9. Position trajectories in the x,y and z directions of each of the evaluation simulations.

Figure 10. Velocity trajectories in the x,y and z directions of each of the evaluation simulations.

Figure 11. Euler angle trajectories in the roll, pitch, yaw directions of each of the evaluation simulations.
In order to demonstrate the convergence of the training losses of both the models and prediction heads of DreamLander, we include the following plots, left to right, top to bottom in Figure 12: dynamic prediction loss, state prediction loss, reward prediction loss, continuation prediction loss, actor loss and value loss.

Figure 12. Training Loss plots for each of the key predictors, actor and value functions for DreamLander, which is based on the DreamerV3 architecture.
Conclusion
In this work we present DreamLander, a DreamerV3 based MBRL framework capable of a highly precise and efficient planetary landing control policy. It has shown to capably learn the dynamics of the EDL problem and then formulate a control policy capable of landing the vehicle with pinpoint accuracy. The model trains efficiently on data provided in simulation and with the latent imagination it produces. We demonstrated its capability to successfully solve the EDL over a large range of initial conditions, and learn a policy that is intuitive. The model based nature of this framework also enables the further inclusion of safety based algorithms to be included so that undesired states can be avoided, further demonstrating the capability of MBRL applications in space related domains. Future work can also investigate the ability of this architecture to learn in real-time or with few-shot simulations to generalize to other environments such as the Moon or Mars, or the inclusion of pixel based observations to improve the autonomy of the guidance algorithm.
References
- [1]D’Souza, C., and D’Souza, C., “An optimal guidance law for planetary landing,” Guidance, Navigation, and Control Conference, American Institute of Aeronautics and Astronautics, Reston, Virginia, 1997.
- [2]Bai, C., Guo, J., and Zheng, H., “Optimal guidance for planetary landing in hazardous terrains,” IEEE Trans. Aerosp. Electron. Syst., Vol. 56, No. 4, 2020, pp. 2896–2909.
- [3]Meditch, J., “On the problem of optimal thrust programming for a lunar soft landing,” IEEE Trans. Automat. Contr., Vol. 9, No. 4, 1964, pp. 477–484.
- [4]Klumpp, A. R., “Apollo lunar descent guidance,” Automatica (Oxf.), Vol. 10, No. 2, 1974, pp. 133–146.DOI
- [5]NASA, “Apollo 14 Mission Report,” Apr 1971. URL https://apollojournals.org/afj/ap14fj/pdf/a14_mission-report.pdf.
- [6]Wright, S. J., Primal-dual interior-point methods, Society for Industrial and Applied Mathematics, USA, 1997.
- [7]Boyd, S., and Vandenberghe, L., Convex Optimization, Cambridge University Press, 2004.
- [8]Bomze, I. M., Demyanov, V. F., Fletcher, R., Terlaky, T., and Polik, I., Nonlinear Optimization: Lectures given at the C.I.M.E. Summer School held in Cetraro, Italy, July 1-7, 2007, 2010th ed., Lecture notes in mathematics, Springer, Berlin, Germany, 2010.DOI
- [9]Biegler, L. T., and Zavala, V. M., “Large-scale nonlinear programming using IPOPT: An integrating framework for enterprise-wide dynamic optimization,” Comput. Chem. Eng., Vol. 33, No. 3, 2009, pp. 575–582.DOI
- [10]Domahidi, A., Chu, E., and Boyd, S., “ECOS: An SOCP solver for embedded systems,” 2013 European Control Conference (ECC), IEEE, 2013.DOI
- [11]Reynolds, T., and Mesbahi, M., “Small body precision landing via convex model predictive control,” AIAA SPACE and Astronautics Forum and Exposition, American Institute of Aeronautics and Astronautics, Reston, Virginia, 2017.DOI
- [12]Lu, P., and Liu, X., “Autonomous trajectory planning for rendezvous and proximity operations by conic optimization,” J. Guid. Control Dyn., Vol. 36, No. 2, 2013, pp. 375–389.
- [13]Liu, X., and Lu, P., “Solving nonconvex optimal control problems by convex optimization,” J. Guid. Control Dyn., Vol. 37, No. 3, 2014, pp. 750–765.DOI
- [14]Acıkmese, B., Aung, M., Casoliva, J., Mohan, S., Johnson, A. E., Scharf, D., Masten, D., Scotkin, J., Wolf, A., and Regehr, M., “Flight testing of trajectories computed by G-FOLD: Fuel optimal large divert guidance algorithm for planetary landing,” 2013.
- [15]Scharf, D. P., Regehr, M. W., Vaughan, G. M., Benito, J., Ansari, H., Aung, M., Johnson, A., Casoliva, J., Mohan, S., Dueri, D., Acıkmese, B., Masten, D., and Nietfeld, S., “ADAPT demonstrations of onboard large-divert Guidance with a VTVL rocket,” 2014 IEEE Aerospace Conference, IEEE, 2014.DOI
- [16]Scharf, D. P., Acıkmese, B., Dueri, D., Benito, J., and Casoliva, J., “Implementation and experimental demonstration of onboard powered-descent guidance,” J. Guid. Control Dyn., Vol. 40, No. 2, 2017, pp. 213–229.DOI
- [17]Sostaric, R., and Rea, J., “Powered descent guidance methods for the moon and mars,” AIAA Guidance, Navigation, and Control Conference and Exhibit, American Institute of Aeronautics and Astronautics, Reston, Virginia, 2005.DOI
- [18]Sibley, G., Matthies, L., and Sukhatme, G., “Sliding window filter with application to planetary landing: Sibley et al.: Sliding Window Filter,” J. Field Robot., Vol. 27, No. 5, 2010, pp. 587–608.DOI
- [19]Song, Z.-Y., Wang, C., Theil, S., Seelbinder, D., Sagliano, M., Liu, X.-F., and Shao, Z.-J., “Survey of autonomous guidance methods for powered planetary landing,” Front. Inf. Technol. Electron. Eng., Vol. 21, No. 5, 2020, pp. 652–674.DOI
- [20]Tiwari, M., Nehma, G., and Lusch, B., “Computationally efficient data-driven discovery and linear representation of nonlinear systems for control,” arXiv [eess.SY], 2023.arxiv.org/abs/2309.04074
- [21]Nehma, G., Tiwari, M., and Lingam, M., “Deep learning based dynamics identification and linearization of orbital problems using Koopman theory,” Aerosp. Sci. Technol., Vol. 163, No. 110295, 2025, p. 110295.
- [22]Quinn, P., Nehma, G., and Tiwari, M., “End-to-end imitation learning for optimal asteroid proximity operations,” arXiv [cs.RO], 2025.arxiv.org/abs/2502.01034
- [23]Xu, X., Chen, Y., and Bai, C., “Deep Reinforcement Learning-based accurate control of planetary soft landing,” Sensors (Basel), Vol. 21, No. 23, 2021, p. 8161.DOI
- [24]Ciabatti, G., Daftry, S., and Capobianco, R., “Autonomous planetary landing via deep reinforcement learning and transfer learning,” 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), IEEE, 2021, pp. 2031–2038.DOI
- [25]Fujimoto, S., van Hoof, H., and Meger, D., “Addressing function approximation error in actor-critic methods,” arXiv [cs.AI], 2018.
- [26]Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M., “Playing Atari with deep reinforcement learning,” arXiv [cs.LG], 2013.arxiv.org/abs/1312.5602
- [27]Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S., “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” arXiv [cs.LG], 2018.
- [28]Williams, R. J., “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Mach. Learn., Vol. 8, No. 3–4, 1992, pp. 229–256.
- [29]Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O., “Proximal Policy Optimization Algorithms,” arXiv [cs.LG], 2017.arxiv.org/abs/1707.06347
- [30]Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D., and Riedmiller, M. A., “Deterministic policy gradient algorithms,” ICML, 2014, pp. 387–395.
- [31]Schulman, J., Levine, S., Moritz, P., Jordan, M. I., and Abbeel, P., “Trust Region Policy Optimization,” arXiv [cs.LG], 2015.arxiv.org/abs/1502.05477
- [32]Ha, D., and Schmidhuber, J., “World Models,” arXiv [cs.LG], 2018.DOI
- [33]Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T., “Mastering diverse domains through world models,” arXiv [cs.AI], 2023.arxiv.org/abs/2301.04104
- [34]Rimon, Z., Jurgenson, T., Krupnik, O., Adler, G., and Tamar, A., “MAMBA: An effective world model approach for meta-reinforcement learning,” arXiv [cs.LG], 2024.arxiv.org/abs/2403.09859
- [35]Chen, B., Zhu, C., Agrawal, P., Zhang, K., and Gupta, A., “Self-Supervised Reinforcement Learning that Transfers using Random Features,” , 2023.arxiv.org/abs/2305.17250
- [36]Cheng, R., Orosz, G., Murray, R. M., and Burdick, J. W., “End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks,” Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI Press, 2019. https://doi.org/10.1609/aaai.v33i01.33013387, URL https://doi.org/10.1609/aaai.v33i01.33013387.
- [37]Fisac, J. F., Lugovoy, N. F., Rubies-Royo, V., Ghosh, S., and Tomlin, C. J., “Bridging Hamilton-Jacobi Safety Analysis and Reinforcement Learning,” 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 8550–8556. https://doi.org/10.1109/ICRA.2019.8794107.
- [38]Berkenkamp, F., Turchetta, M., Schoellig, A., and Krause, A., “Safe Model-based Reinforcement Learning with Stability Guarantees,” Advances in Neural Information Processing Systems, Vol. 30, edited by I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper_files/paper/2017/file/766ebcd59621e305170616ba3d3dac32-Paper.pdf.arxiv.org/abs/1705.08551
- [39]Gaudet, B., Linares, R., and Furfaro, R., “Deep reinforcement learning for six degree-of-freedom planetary landing,” Adv. Space Res., Vol. 65, No. 7, 2020, pp. 1723–1741.DOI
- [40]Orsula, A., Geist, M., Olivares-Mendez, M., and Martinez, C., “Space Robotics Bench: Robot Learning Beyond Earth,”, 2025. URL https://arxiv.org/abs/2509.23328.
- [41]Xiong, X., Mu, N., Xie, R., Yang, S., Wang, Y., Wang, L., Luan, Y., Li, S., Xu, S., Yang, Y., and Xu, B., “MrCoM: A Meta-Regularized World-Model Generalizing Across Multi-Scenarios,” 2025. URL https://arxiv.org/abs/2511.06252.
- [42]Rimon, Z., Jurgenson, T., Krupnik, O., Adler, G., and Tamar, A., “MAMBA: an Effective World Model Approach for Meta-Reinforcement Learning,”, 2024. URL https://arxiv.org/abs/2403.09859.
- [43]Cohen, M. H., and Belta, C., “Safe exploration in model-based reinforcement learning using control barrier functions,” Automatica, Vol. 147, 2023, p. 110684. https://doi.org/10.1016/j.automatica.2022.110684, URL http://dx.doi.org/10.1016/j.automatica.2022.110684.
- [44]Mahmud, S. N., “Safety-aware model-based reinforcement learning using barrier transformation,” Master’s thesis, Oklahoma State University, 2021.
- [45]Nakamura, K., Peters, L., and Bajcsy, A., “Generalizing Safety Beyond Collision-Avoidance via Latent-Space Reachability Analysis,” 2025.DOI
- [46]Lindner, D., Chen, X., Tschiatschek, S., Hofmann, K., and Krause, A., “Learning Safety Constraints from Demonstrations with Unknown Rewards,”, 2024. URL https://arxiv.org/abs/2305.16147.
- [47]Hsu*, K.-C., Rubies-Royo*, V., Tomlin, C., and Fisac, J., “Safety and Liveness Guarantees through Reach-Avoid Reinforcement Learning,” Robotics: Science and Systems XVII, Robotics: Science and Systems Foundation, 2021. https://doi.org/10.15607/rss.2021.xvii.077, URL http://dx.doi.org/10.15607/RSS.2021.XVII.077.
- [48]Gao, D., Cai, S., Zhou, H., Wang, H., Soltani, I., and Zhang, J., “CarDreamer: Open-source learning platform for world model based autonomous driving,” arXiv [cs.RO], 2024.arxiv.org/abs/2405.09111
- [49]Zhao, G., Wang, X., Zhu, Z., Chen, X., Huang, G., Bao, X., and Wang, X., “DriveDreamer-2: LLM-enhanced world models for diverse driving video generation,” arXiv [cs.CV], 2024.arxiv.org/abs/2403.06845
- [50]Verraest, A., Bahnam, S., Ferede, R., de Croon, G., and Wagter, C. D., “SkyDreamer: Interpretable End-to-End Vision-Based Drone Racing with Model-Based Reinforcement Learning,”, 2025. URL https://arxiv.org/abs/2510.14783.
- [51]Cox, K. J., “Design Development of The Apollo Lunar Module,”, 1978.
- [52]Grumman Aerospace Corporation, “Lunar Module LM10 through LM14 Vehicle Familiarization Manual,”, Aug 1969. URL https://www.nasa.gov/wp-content/uploads/static/history/alsj/a14/a14-43939523-LM10-LM14-Fam-Manual.pdf.
- [53]Corp, N., Nov 2025. URL https://docs.isaacsim.omniverse.nvidia.com/5.1.0/physics/simulation_fundamentals.html#simulation-components.
- [54]Hafner, D., Lillicrap, T., Norouzi, M., and Ba, J., “Mastering Atari with discrete world models,” arXiv [cs.LG], 2020.arxiv.org/abs/2010.02193
- [55]Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M., “Dream to control: Learning behaviors by latent imagination,” arXiv [cs.LG], 2019.arxiv.org/abs/1912.01603
- [56]Ng, A., Harada, D., and Russell, S. J., “Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping,” International Conference on Machine Learning, 1999. URL https://api.semanticscholar.org/CorpusID:5730166.
- [57]Olver, P. J., and Shakiban, C., Inner Products and Norms, Springer International Publishing, Cham, 2018, pp. 129–182. https://doi.org/10.1007/978-3-319-91041-3_3, URL https://doi.org/10.1007/978-3-319-91041-3_3.
- [58]Fresk, E., and Nikolakopoulos, G., “Full quaternion based attitude control for a quadrotor,” 2013 European Control Conference (ECC), 2013, pp. 3864–3869. https://doi.org/10.23919/ECC.2013.6669617.
- [59]Morihira, N., “dreamerv3-torch,” urlhttps://github.com/NM512/dreamerv3-torch, 2023. URL https://github.com/NM512/dreamerv3-torch.
- [60]Carrodori, J., “6-DOF Atmospheric Rocket Landing Guidance using Meta-Reinforcement Learning,” Master’s thesis, TU Delft, 2024.
- [61]Li, W., Zhang, X., Dong, Y., Lin, Y., and Li, H., “Powered Landing Control of Reusable Rockets Based on Softmax Double DDPG,” Aerospace, Vol. 10, No. 7, 2023. https://doi.org/10.3390/aerospace10070590, URL https://www.mdpi.com/2226-4310/10/7/590.
