betaFeedback

WorksComputer Science

Adversarial Learning for Neural Dialogue Generation

Abstract

In this paper, drawing intuition from the Turing test, we propose using adversarial training for open-domain dialogue generation: the system is trained to produce sequences that are indistinguishable from human-generated dialogue utterances. We cast the task as a reinforcement learning (RL) problem where we jointly train two systems, a generative model to produce response sequences, and a discriminator—analagous to the human evaluator in the Turing test— to distinguish between the human-generated dialogues and the machine-generated ones. The outputs from the discriminator are then used as rewards for the generative model, pushing the system to generate dialogues that mostly resemble human dialogues. In addition to adversarial training we describe a model for adversarial evaluation that uses success in fooling an adversary as a dialogue evaluation metric, while avoiding a number of potential pitfalls. Experimental results on several metrics, including adversarial evaluation, demonstrate that the adversarially-trained system generates higher-quality responses than previous baselines.

Introduction

Open domain dialogue generation ([27, 38, 46, 43, 16, 31, 33 }) aims at generating meaningful and coherent dialogue responses given the dialogue history. Prior systems, e.g., phrase-based machine translation systems ({citep}]Ritter et al., 2011; Sordoni et al., 2015) or end-to-end neural systems ({citep}Shang et al., 2015; Vinyals and Le, 2015; Li et al., 2016a; Yao et al., 2015; Luan et al., 2016) approximate such a goal by predicting the next dialogue utterance given the dialogue history using the maximum likelihood estimation (MLE) objective. Despite its success, this oversimplified training objective leads to problems: responses are dull, generic ({citep}Sordoni et al., 2015; Serban et al., 2016a; Li et al., 2016a), repetitive, and short-sighted ({citep}Li et al., 2016d`).

Solutions to these problems require answering a few fundamental questions: what are the crucial aspects that characterize an ideal conversation, how can we quantitatively measure them, and how can we incorporate them into a machine learning system? For example, Li et al. [19] manually define three ideal dialogue properties (ease of answering, informativeness and coherence) and use a reinforcement-learning framework to train the model to generate highly rewarded responses. Yu et al. [49] use keyword retrieval confidence as a reward. However, it is widely acknowledged that manually defined reward functions can’t possibly cover all crucial aspects and can lead to sub-optimal generated utterances.

A good dialogue model should generate utterances indistinguishable from human dialogues. Such a goal suggests a training objective resembling the idea of the Turing test (Turing, [41]). We borrow the idea of adversarial training ([10, 8]) in computer vision, in which we jointly train two models, a generator (a neural Seq2Seq model) that defines the probability of generating a dialogue sequence, and a discriminator that labels dialogues as human-generated or machine-generated. This discriminator is analogous to the evaluator in the Turing test. We cast the task as a reinforcement learning problem, in which the quality of machine-generated utterances is measured by its ability to fool the discriminator into believing that it is a human-generated one. The output from the discriminator is used as a reward to the generator, pushing it to generate utterances indistinguishable from human-generated dialogues.

The idea of a Turing test—employing an evaluator to distinguish machine-generated texts from human-generated ones—can be applied not only to training but also testing, where it goes by the name of adversarial evaluation. Adversarial evaluation was first employed in Bowman et al. [5] to evaluate sentence generation quality, and preliminarily studied for dialogue generation by Kannan and Vinyals [13]. In this paper, we discuss potential pitfalls of adversarial evaluations and necessary steps to avoid them and make evaluation reliable.

Experimental results demonstrate that our approach produces more interactive, interesting, and non-repetitive responses than standard SEQ2SEQ models trained using the MLE objective function.

Related Work

Dialogue generation Response generation for dialogue can be viewed as a source-to-target transduction problem. Ritter et al. [27] frame the generation problem as a machine translation problem. Sordoni et al. [38] improved Ritter et al.’s system by rescoring the outputs of a phrasal MT-based conversation system with a neural model incorporating prior context. Recent progress in SEQ2SEQ models have inspired several efforts (Vinyals and Le [42]; Serban et al. [29]; Luan et al. [23]) to build end-to-end conversational systems that first apply an encoder to map a message to a distributed vector representing its meaning and then generate a response from the vector.

Our work adapts the encoder-decoder model to RL training, and can thus be viewed as an extension of Li et al. [19], but with more general RL rewards. Li et al. [19] simulate dialogues between two virtual agents, using policy gradient methods to reward sequences that display three useful conversational properties: informativity, coherence, and ease of answering. Our work is also related to recent efforts to integrate the SEQ2SEQ and reinforcement learning paradigms, drawing on the advantages of both (Wen et al. [43]). For example, Su et al. [39] combine reinforcement learning with neural generation on tasks with real users. Asghar et al. [2] train an end-to-end RL dialogue model using human users.

Dialogue quality is traditionally evaluated (Sordoni et al. [38], e.g.) using word-overlap metrics such as BLEU and METEOR scores used for machine translation. Some recent work (Liu et al. [20]) has started to look at more flexible and reliable evaluation metrics such as human-rating prediction (Lowe et al. [21]) and next utterance classification (Lowe et al. [22]).

Adversarial networks The idea of generative adversarial networks has enjoyed great success in computer vision ([25, 6, 28]). Training is formalized as a game in which the generative model is trained to generate outputs to fool the discriminator; the technique has been successfully applied to image generation.

However, to the best of our knowledge, this idea has not achieved comparable success in NLP. This is due to the fact that unlike in vision, text generation is discrete, which makes the error outputted from the discriminator hard to backpropagate to the generator. Some recent work has begun to address this issue: Lamb et al. [14] propose providing the discriminator with the intermediate hidden vectors of the generator rather than its sequence outputs. Such a strategy makes the system differentiable and achieves promising results in tasks like character-level language modeling and handwriting generation. Yu et al. [48] use policy gradient reinforcement learning to backpropagate the error from the discriminator, showing improvement in multiple generation tasks such as poem generation, speech language generation and music generation. Outside of sequence generation, Chen et al. [7] apply the idea of adversarial training to sentiment analysis and Zhang et al. [50] apply the idea to domain adaptation tasks.

Our work is distantly related to recent work that formalizes sequence generation as an action-taking problem in reinforcement learning. Ranzato et al. [26] train RNN decoders in a SEQ2SEQ model using policy gradient to obtain competitive machine translation results. Bahdanau et al. [3] take this a step further by training an actor-critic RL model for machine translation. Also related is recent work ([36, 45]) to address the issues of exposure bias and loss-evaluation mismatch in neural translation.

Adversarial Training for Dialogue Generation

In this section, we describe in detail the components of the proposed adversarial reinforcement learning model. The problem can be framed as follows: given a dialogue history xx consisting of a sequence of dialogue utterances,1 the model needs to generate a response y={y1,y2,…,yT}y = \{y_1, y_2, \ldots, y_T\}. We view the process of sentence generation as a sequence of actions that are taken according to a policy defined by an encoder-decoder recurrent neural network.

Adversarial REINFORCE

The adversarial REINFORCE algorithm consists of two components: a generative model GG and a discriminative model DD.

Generative model The generative model GG defines the policy that generates a response yy given dialogue history xx. It takes a form similar to SEQ2SEQ models, which first map the source input to a vector representation using a recurrent net and then compute the probability of generating each token in the target using a softmax function.

Discriminative model The discriminative model DD is a binary classifier that takes as input a sequence of dialogue utterances {x,y}\{x, y\} and outputs a label indicating whether the input is generated by humans or machines. The input dialogue is encoded into a vector representation using a hierarchical encoder (Li et al., [17]; Serban et al., [30]),2 which is then fed to a 2-class softmax function, returning the probability of the input dialogue episode being a machine-generated dialogue (denoted Q−({x,y})Q_-(\{x, y\})) or a human-generated dialogue (denoted Q+({x,y})Q_+(\{x, y\})).

Policy Gradient Training The key idea of the system is to encourage the generator to generate utterances that are indistinguishable from human generated dialogues. We use policy gradient methods to achieve such a goal, in which the score of current utterances being human-generated ones assigned by the discriminator (i.e., Q+({x,y})Q_+(\{x, y\})) is used as a reward for the generator, which is trained to maximize the expected reward of generated utterance(s) using the REINFORCE algorithm (Williams, [44]):

J(θ)=Ey∼p(y∣x)(Q+({x,y})∣θ)(1)J(\theta) = \mathbb{E}_{y \sim p(y|x)}\left(Q_+(\{x,y\}) \mid\theta\right) \tag*{(1)}

Given the input dialogue history xx, the bot generates a dialogue utterance yy by sampling from the policy. The concatenation of the generated utterance yy and the input xx is fed to the discriminator. The gradient of (1) is approximated using the likelihood ratio trick (Williams, [44]; Glynn, [9]; Aleksandrov et al., 1968):

∇J(θ)≈[Q+({x,y})−b({x,y})]∇log⁡π(y∣x)=[Q+({x,y})−b({x,y})]∇∑tlog⁡p(yt∣x,y1:t−1)(2)\begin{aligned} \nabla J(\theta) &\approx[Q_+(\{x,y\}) - b(\{x,y\})] \\ &\qquad\nabla\log\pi(y|x) \\ &= [Q_+(\{x,y\}) - b(\{x,y\})] \\ &\qquad\nabla\sum_t \log p(y_t|x,y_{1:t-1}) \tag*{(2)} \end{aligned}

where π\pi denotes the probability of the generated responses. b({x,y})b(\{x,y\}) denotes the baseline value to reduce the variance of the estimate while keeping it unbiased.3 The discriminator is simultaneously updated with the human generated dialogue that contains dialogue history xx as a positive example and the machine-generated dialogue as a negative example.

Reward for Every Generation Step (REGS)

The REINFORCE algorithm described has the disadvantage that the expectation of the reward is approximated by only one sample, and the reward associated with this sample (i.e., [Q+({x,y})−b({x,y})][Q_+(\{x,y\}) - b(\{x,y\})] in Eq. (2)) is used for all actions (the generation of each token) in the generated sequence. Suppose, for example, the input history is what’s your name, the human-generated response is I am John, and the machine-generated response is I don’t know. The vanilla REINFORCE model assigns the same negative reward to all tokens within the human-generated response (i.e., I, don’t, know), whereas proper credit assignment in training would give separate rewards, most likely a neutral reward for the token I, and negative rewards to don’t and know. We call this reward for every generation step, abbreviated REGS.

Rewards for intermediate steps or partially decoded sequences are thus necessary. Unfortunately, the discriminator is trained to assign scores to fully generated sequences, but not partially decoded ones. We propose two strategies for computing intermediate step rewards by (1) using Monte Carlo (MC) search and (2) training a discriminator that is able to assign rewards to partially decoded sequences.

In (1) Monte Carlo search, given a partially decoded sPs_P, the model keeps sampling tokens from the distribution until the decoding finishes. Such a process is repeated NN (set to 5) times and the NN generated sequences will share a common prefix sPs_P. These NN sequences are fed to the discriminator, the average score of which is used as a reward for the sPs_P. A similar strategy is adopted in Yu et al. [48]. The downside of MC is that it requires repeating the sampling process for each prefix of each sequence and is thus significantly time-consuming.4

In (2), we directly train a discriminator that is able to assign rewards to both fully and partially decoded sequences. We break the generated sequences into partial sequences, namely {y1:t+}t=1NY+\{y^+_{1:t}\}_{t=1}^{N_{Y^+}} and {y1:t−}t=1NY−\{y^-_{1:t}\}_{t=1}^{N_{Y^-}} and use all instances in {y1:t+}t=1NY+\{y^+_{1:t}\}_{t=1}^{N_{Y^+}} as positive examples and instances {y1:t−}t=1NY−\{y^-_{1:t}\}_{t=1}^{N_{Y^-}} as negative examples. The problem with such a strategy is that earlier actions in a sequence are shared among multiple training examples for the discriminator (for example, token y1+y^+_1 is contained in all partially generated sequences, which results in overfitting. To mitigate this problem, we adopt a strategy similar to when training value networks in AlphaGo (Silver et al. [37]), in which for each collection of subsequences of YY, we randomly sample only one example from {y1:t+}t=1NY+\{y^+_{1:t}\}_{t=1}^{N_{Y^+}} and one example from {y1:t−}t=1NY−\{y^-_{1:t}\}_{t=1}^{N_{Y^-}}, which are treated as positive and negative examples to update the discriminator. Compared with the Monte Carlo search model, this strategy is significantly more time-effective, but comes with the weakness that the discriminator becomes less accurate after partially decoded sequences are added in as training examples. We find that the MC model performs better when training time is less of an issue.

For each partially-generated sequence Yt=y1:tY_t = y_{1:t}, the discriminator gives a classification score Q+(x,Yt)Q_+(x,Y_t). We compute the baseline b(x,Yt)b(x,Y_t) using a similar model to the vanilla REINFORCE model. This yields the following gradient to update the generator:

∇J(θ)≈∑t(Q+(x,Yt)−b(x,Yt))∇log⁡p(yt∣x,Y1:t−1)(3)\nabla J(\theta) \approx\sum_t \left(Q_+(x,Y_t) - b(x,Y_t)\right) \nabla\log p(y_t \mid x,Y_{1:t-1}) \tag*{(3)}

Comparing (3) with (2), we can see that the values for rewards and baselines are different among generated tokens in the same response.

Teacher Forcing Practically, we find that updating the generative model only using Eq. (1) leads to unstable training for both vanilla Reinforce and REGS, with the perplexity value skyrocketing after training the model for a few hours (even when the generator is initialized using a pre-trained Seq2Seq model). The reason this happens is that the generative model can only be indirectly exposed to the gold-standard target sequences through the reward passed back from the discriminator, and this reward is used to promote or discourage its (the generator’s) own generated sequences. Such a training strategy is fragile: once the generator (accidentally) deteriorates in some training batches and the discriminator consequently does an extremely good job in recognizing sequences from the generator, the generator immediately gets lost. It knows that its generated sequences are bad based on the rewards outputted from the discriminator, but it does not know what sequences are good and how to push itself to generate these good sequences (the odds of generating a good response from random sampling are minute, due to the vast size of the space of possible sequences). Loss of the reward signal leads to a breakdown in the training process.

To alleviate this issue and give the generator more direct access to the gold-standard targets, we propose also feeding human generated responses to the generator for model updates. The most straightforward strategy is for the discriminator to automatically assign a reward of 1 (or other positive values) to the human generated responses and for the generator to use this reward to update itself on human generated examples. This can be seen as having a teacher intervene with the generator some fraction of the time and force it to generate the true responses, an approach that is similar to the professor-forcing algorithm of Lamb et al. [14].

A closer look reveals that this modification is the same as the standard training of SEQ2SEQ models, making the final training alternately update the SEQ2SEQ model using the adversarial objective and the MLE objective. One can think of the professor-forcing model as a regularizer to regulate the generator once it starts deviating from the training dataset.

We also propose another workaround, in which the discriminator first assigns a reward to a human generated example using its own model, and the generator then updates itself using this reward on the human generated example only if the reward is larger than the baseline value. Such a strategy has the advantage that different weights for model updates are assigned to different human generated examples (in the form of different reward values produced by the generator) and that human generated examples are always associated with non-negative weights.

A sketch of the proposed model is shown in Figure 1.

Adversarial reinforcement algorithm

Figure 1. A brief review of the proposed adversarial reinforcement algorithm for training the generator GG and discriminator DD. The reward rr from the discriminator DD can be computed using different strategies according to whether using REINFORCE or REGS. The update of the generator GG on (X,Y^)(X,\hat{Y}) can be done by either using Eq. (2) or Eq. (3). D-steps is set to 5 and G-steps is set to 1.

Training Details

We first pre-train the generative model by predicting target sequences given the dialogue history. We trained a SEQ2SEQ model ([40]) with an attention mechanism ([4, 24]) on the OpenSubtitles dataset. We followed protocols recommended by Sutskever et al. [40], such as gradient clipping, mini-batch and learning rate decay. We also pre-train the discriminator. To generate negative examples, we decode part of the training data. Half of the negative examples are generated using beam-search with mutual information reranking as described in Li et al. [15], and the other half is generated from sampling.

For data processing, model training and decoding (both the proposed adversarial training model and the standard SEQ2SEQ models), we employ a few strategies that improve response quality, including: (2) Remove training examples with length of responses shorter than a threshold (set to 5). We find that this significantly improves the general response quality.5 (2) Instead of using the same learning rate for all examples, using a weighted learning rate that considers the average tf-idf score for tokens within the response. Such a strategy decreases the influence from dull and generic utterances.6 (3) Penalizing intra-sibling ranking when doing beam search decoding to promote N-best list diversity as described in Li et al. [18]. (4) Penalizing word types (stop words excluded) that have already been generated. Such a strategy dramatically decreases the rate of repetitive responses such as no, no, no, no, or contradictory responses such as I don’t like oranges but i like oranges.

Adversarial Evaluation

In this section, we discuss strategies for successful adversarial evaluation. Note that the proposed adversarial training and adversarial evaluation are separate procedures. They are independent of each other and share no common parameters. The idea of adversarial evaluation, first proposed by Bowman et al. [5], is to train a discriminant function to separate generated and true sentences, in an attempt to evaluate the model’s sentence generation capability. The idea has been preliminarily studied by Kannan and Vinyals [13] in the context of dialogue generation. Adversarial evaluation also resembles the idea of the Turing test, which requires a human evaluator to distinguish machine-generated texts from human-generated ones. Since it is time-consuming and costly to ask a human to talk to a model and give judgements, we train a machine evaluator in place of the human evaluator to distinguish the human dialogues and machine dialogues, and we use it to measure the general quality of the generated responses.

Adversarial evaluation involves both training and testing. At training time, the evaluator is trained to label dialogues as machine-generated (negative) or human-generated (positive). At test time, the trained evaluator is evaluated on a held-out dataset. If the human-generated dialogues and machine-generated ones are indistinguishable, the model will achieve 50 percent accuracy at test time.

Adversarial Success

We define Adversarial Success (AdverSuc for short) to be the fraction of instances in which a model is capable of fooling the evaluator. AdverSuc is the difference between 1 and the accuracy achieved by the evaluator. Higher values of AdverSuc for a dialogue generation model are better.

Testing the Evaluator’s Ability

One caveat with the adversarial evaluation methods is that they are model-dependent. We approximate the human evaluator in the Turing test with an automatic evaluator and assume that the evaluator is perfect: low accuracy of the discriminator should indicate high quality of the responses, since we interpret this to mean the generated responses are indistinguishable from the human ones. Unfortunately, there is another factor that can lead to low discriminative accuracy: a poor discriminative model. Consider a discriminator that always gives random labels or always gives the same label. Such an evaluator always yields a high AdverSuc value of 0.5. Bowman et al. [5] propose two different discriminator models separately using unigram features and neural features. It is hard to tell which feature set is more reliable. The standard strategy of testing the model on a held-out development set is not suited to this case, since a model that overfits the development set is necessarily superior.

To deal with this issue, we propose setting up a few manually-invented situations to test the ability of the automatic evaluator. This is akin to setting up examinations to test the ability of the human evaluator in the Turing test. We report not only the AdverSuc values, but also the scores that the evaluator achieves in these manually-designed test cases, indicating how much we can trust the reported AdverSuc. We develop scenarios in which we know in advance how a perfect evaluator should behave, and then compare AdverSuc from a discriminative model with the gold-standard AdverSuc. Scenarios we design include:

  • We use human-generated dialogues as both positive examples and negative examples. A perfect evaluator should give an AdverSuc of 0.5 (accuracy 50%), which is the gold-standard result.

  • We use machine-generated dialogues as both positive examples and negative examples. A perfect evaluator should give an AdverSuc of 0.5 (accuracy 50%).

  • We use original human-generated dialogues as positive examples and dialogues consisting of random utterances as negative examples. A perfect evaluator should give an AdverSuc of 0 (accuracy 100%).

  • We use original human-generated dialogues as positive examples and dialogues with responses replaced by the utterances immediately following the true responses. A perfect evaluation should give an AdverSuc of 0 (accuracy 100%).

The evaluator reliability error (ERE) is the average deviation of an evaluator’s adversarial error from the gold-standard error in the above tasks, with equal weight for each task. The smaller the error, the more reliable the evaluator is.

Machine-vs-Random Accuracy

Evaluator reliability error uses scenarios constructed from human-generated dialogues to assess feature or hyper-parameter choice for the evaluator. Unfortunately, no machine-generated responses are involved in the ERE metric. The following example illustrates the serious weakness resulting from this strategy: as will be shown in the experiment section, when inputs are decoded using greedy or beam search models, most generation systems to date yield an adversarial success less than 10 percent (evaluator accuracy 90 percent). But when using sampling for decoding, the adversarial success skyrockets to around 40 percent,7 only 10 percent less than what’s needed to pass the Turing test. A close look at the decoded sequences using sampling tells a different story: the responses from sampling are sometimes incoherent, irrelevant or even ungrammatical.

Inputtell me ... how long have you had this falling sickness ?
Vanilla-MLEi ’m not a doctor .
Vanilla-Samplewell everything you did was totally untrue .
REINFORCEi don ’t know how long it ’s been .
REGS Monte CarloA few months, I guess .
Inputso i had the doctors test sammy ’s response to conditioning .
Vanilla-MLEsammy wrote the test sammy wrote the test .
Vanilla-Sampleobjects pick objects up objects objects objects objects objects
REINFORCEi ’m not sure that ’s relevant .
REGS Monte Carloso he took the pills .
Inputwhat are you going to do ?
Vanilla-MLEi ’m going to the movies .
Vanilla-Sampleget him outta here first !
REINFORCEi ’m going to get you .
REGS Monte Carloi ’m going to get you a new car .
Inputthey fear your power your intellect .
Vanilla-MLEyou ’re the only one who knows what ’s going on .
Vanilla-Samplewhen they are conquered and you surrender they will control all of us .
REINFORCEi ’m afraid i ’m not ready yet .
REGS Monte Carloi ’m not afraid of your power .

Table 1. Sampled responses from different models. More in Appendix Tables 5 and 6.

We thus propose an additional sanity check, in which we report the accuracy of distinguishing between machine-generated responses and randomly sampled responses (machine-vs-random for short). This resembles the N-choose-1 metric described in Shao et al. [35]. Higher accuracy indicates that the generated responses are distinguishable from randomly sampled human responses, indicating that the generative model is not fooling the generator simply by introducing randomness. As we will show in § 5, using sampling results in high AdverSuc values but low machine-vs-random accuracy.

Experimental Results

In this section, we detail experimental results on adversarial success and human evaluation.

Adversarial Evaluation

ERE We first test adversarial evaluation models with different feature sets and model architectures for reliability, as measured by evaluator reliability error (ERE). We explore the following models: (1) SVM+Unigram: SVM using unigram features.8 A multi-utterance dialogue (i.e., input messages and responses) is transformed to a unigram representation; (2) Concat Neural: a neural classification model with a softmax function that takes as input the concatenation of representations of constituent dialogues sentences; (3) Hierarchical Neural: a hierarchical encoder with a structure similar to the discriminator used in the reinforcement; and (4) SVM+Neural+multi-lex-features: a SVM model that uses the following features: unigrams, neural representations of dialogues obtained by the neural model trained using strategy (3),9 the forward likelihood log p(t∣s)p(t|s) and backward likelihood p(s∣t)p(s|t).

ERE scores obtained by different models are reported in Table 2. As can be seen, the hierarchical neural evaluator (model 3) is more reliable than simply concatenating the sentence-level representations (model 2). Using the combination of neural features and lexicalized features yields the most reliable evaluator. For the rest of this section, we report results obtained by the Hierarchical Neural setting due to its end-to-end nature, despite its inferiority to SVM+Neural+multil-features.

SettingERE
SVM+Unigram0.232
Concat Neural0.209
Hierarchical Neural0.193
SVM+Neural+multi-features0.152

Table 2. ERE scores obtained by different models.

Table 3 presents AdverSuc values for different models, along with machine-vs-random accuracy described in § 4.3. Higher values of AdverSuc and machine-vs-random are better.

ModelAdverSucmachine-vs-random
MLE-BS0.0370.942
MLE-Greedy0.0490.945
MMI+p(t∣s)p(t \mid s)0.0730.953
MMI-p(t)p(t)0.0900.880
Sampling0.3720.679
Adver-Reinforce0.0800.945
Adver-REGS0.0980.952

Table 3. AdverSuc and machine-vs-random scores achieved by different training/decoding strategies.

Baselines we consider include standard SEQ2SEQ models using greedy decoding (MLE-greedy), beam-search (MLE+BS) and sampling, as well as the mutual information reranking model of Li et al. [15] with two algorithmic variations: (1) MMI+p(t∣s)p(t \mid s), in which a large N-best list is first generated using a pre-trained SEQ2SEQ model and then reranked by the backward probability p(s∣t)p(s \mid t) and (2) MMI−p(t)p(t), in which language model probability is penalized during decoding.

Results are shown in Table 3. What first stands out is decoding using sampling (as discussed in § 4.3), achieving a significantly higher AdverSuc number than all the rest models. However, this does not indicate the superiority of the sampling decoding model, since the machine-vs-random accuracy is at the same time significantly lower. This means that sampled responses based on SEQ2SEQ models are not only hard for an evaluator to distinguish from real human responses, but also from randomly sampled responses. A similar, though much less extreme, effect is observed for MMI−p(t)p(t), which has an AdverSuc value slightly higher than Adver-Reinforce, but a significantly lower machine-vs-random score.

By comparing different baselines, we find that MMI+p(t∣s)p(t \mid s) is better than MLE-greedy, which is in turn better than MLE+BS. This result is in line with human-evaluation results from Li et al. [15]. The two proposed adversarial algorithms achieve better performance than the baselines. We expect this to be the case, since the adversarial algorithms are trained on an objective function more similar to the evaluation metric (i.e., adversarial success). REGS performs slightly better than the vanilla REINFORCE algorithm.

Human Evaluation

For human evaluation, we follow protocols defined in Li et al. [19], employing crowdsourced judges to evaluate a random sample of 200 items. We present both an input message and

the generated outputs to 3 judges and ask them to decide which of the two outputs is better (single-turn general quality). Ties are permitted. Identical strings are assigned the same score. We also present the judges with multi-turn conversations simulated between the two agents. Each conversation consists of 3 turns. Results are presented in Table 4. We observe a significant quality improvement on both single-turn quality and multi-turn quality from the proposed adversarial model. It is worth noting that the reinforcement learning system described in Li et al. [19], which simulates conversations between two bots and is trained based on manually designed reward functions, only improves multi-turn dialogue quality, while the model described in this paper improves both single-turn and multi-turn dialogue generation quality. This confirms that the reward adopted in adversarial training is more general, natural and effective in training dialogue systems.

Settingadver-winadver-losetie
single-turn0.620.180.20
multi-turn0.720.100.18

Table 4. The gain from the proposed adversarial model over the mutual information system based on pairwise human judgments.

Conclusion and Future Work

In this paper, drawing intuitions from the Turing test, we propose using an adversarial training approach for response generation. We cast the model in the framework of reinforcement learning and train a generator based on the signal from a discriminator to generate response sequences indistinguishable from human-generated dialogues. We observe clear performance improvements on multiple metrics from the adversarial training strategy.

The adversarial training model should theoretically benefit a variety of generation tasks in NLP. Unfortunately, in preliminary experimentsышәтә applying the same training paradigm to machine translation, we did not observe a clear performance boost. We conjecture that this is because the adversarial training strategy is more beneficial to tasks in which there is a big discrepancy between the distributions of the generated sequences and the reference target sequences. In other words, the adversarial approach is more beneficial on tasks in which entropy of the targets is high. Exploring this relationship further is a focus of our future work.

Acknowledgements

The authors thank Michel Galley, Bill Dolan, Chris Brockett, Jianfeng Gao and other members of the NLP group at Microsoft Research, as well as Sumit Chopra and Marc’Aurelio Ranzato from Facebook AI Research for helpful discussions and comments. Jiwei Li is supported by a Facebook Fellowship, which we gratefully acknowledge. This work is also partially supported by the NSF under award IIS-1514268, and the DARPA Communicating with Computers (CwC) program under ARO prime contract no. W911NF-15-1-0462, IIS-1464128. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of DARPA, the NSF, or Facebook.

Inputmaybe his eyes were too big for his stomach .
Vanilla-MLEhe ’s got a big mouth .
Vanilla-Samplei ’m sure he ’s unhappy with the fact .
Reinforcehe ’s a little bit of a pain .
MCi don ’t think so .
Inputthey didn ’t have much success with this problem commander .
Vanilla-MLEthey ’re not the only ones who have been in the system .
Vanilla-Sampleit ’s the new priority only .
Reinforcethey ’re not the only ones who can help us .
MCcan we find someone else ?
Inputand he carried them all the time wherever he went .
Vanilla-MLEi don ’t know who he is .
Vanilla-Samplethe church ... suffered the winds of 900 meters with horse aid .
Reinforcehe was a good man .
MChe was a very good man .
Inputsomebody probably flushed their pet snake down the toilet .
Vanilla-MLEi ’m sure the dog will be here soon .
Vanilla-Samplei could at least think .
Reinforcei don ’t think so .
MCi don ’t think so .
Inputexcuse me but how did we pay for these suits ?
Vanilla-MLEi ’m not a beggar .
Vanilla-Sampleis this the money eugene ?
Reinforcei don ’t know what to do .
MChow much money you owe me ?
Input... while the beautiful and happy world was madly destroyed ;
Vanilla-MLEi was born in the world of the world
Vanilla-Sampleas in childhood i took you to one tenth of a postcard
Reinforceand the world was full of joy .
MCand the world was full of joy .
Inputi was kinda hoping that you were my dad .
Vanilla-MLEi ’m just a little bit of a little bit of a little bit of a little bit of a
Vanilla-Sampleshiny gold gold and silver in the mountains .
Reinforcei ’m not gonna be a baby .
MCi ’m not going to be a father .
Inputthe man who does that is not afraid to die .
Vanilla-MLEhe ’s not afraid of the dark .
Vanilla-Sampleeven the program is the only truth
Reinforcehe ’s also not afraid of me .
MChe ’s such a brave man !
Inputdo you have any idea what caused the explosion ?
Vanilla-MLEi think it ’s the same thing .
Vanilla-Samplei ’m still very nervous .
Reinforcemaybe it ’s just a coincidence .
MCi don ’t know if it ’s a coincidence .
Inputdon ’t bother the citizens in hong kong island anymore
Vanilla-MLEi ’ll be back in a few days
Vanilla-Sampleyou wiii be transferred to the target court
Reinforcei ’m not sure if you can handle it without bothering them .
MCi ’m not going to be a nuisance
Inputyou know you ought to find yourself a girl .
Vanilla-MLEyou know what i mean ?
Vanilla-Sampleit ’s full of crime ...
Reinforcei don ’t know what you ’re talking about .
MCi don ’t want to be a girl .
Inputi got you the big screen tv deluxe UNknown machine
Vanilla-MLEi ’m gonna be a little late for the movie .
Vanilla-Sampleyou mean you ’re living ?
Reinforcei ’m gonna get you a new one .
MCi ’m gonna get you a new tv set .
Inputwe won ’t give her the chance to surprise us .
Vanilla-MLEshe ’s not going to be a party member .
Vanilla-Samplethe timing ’s good and bad .
Reinforcewe ’ll have to get her back .
MCi don ’t think we have to worry about her .
Inputdoes it live in this matter we call dna ?
Vanilla-MLEi don ’t think so .
Vanilla-Sampleand it requires time to be wrong and no one believes it .
Reinforceit ’s a genetic disorder .
MCi don ’t think so .

Table 5. Appendix: Responses sampled from different models.

Inputi ’ll have to drive home .
Vanilla-MLEi ’ll be back in a minute .
Vanilla-Samplelet ’s go for some fun .
Reinforcei ’ll drive you home .
MCi don ’t have a car .
Inputi had to work late and then i went for a walk .
Vanilla-MLEi was in the middle of the road .
Vanilla-Samplemy body was just pounding .
Reinforcei was just trying to help you .
MCi was just trying to get a little sleep .
Inputlots of cream lots of sugar .
Vanilla-MLEi ’m gonna be in the kitchen .
Vanilla-Samplewas this gay or red ?
Reinforcei don ’t want to be a vegetable .
MCi ’m not a very good cook .
Inputso they just let him go ?
Vanilla-MLEhe ’s not gonna let them go through the motions .
Vanilla-Samplethey asked him whether we could go see ernesto ?
Reinforcei don ’t know what to do .
MChe ’s not going to be here any longer .
Inputyou must not bomb this town .
Vanilla-MLEyou have to go back to the city .
Vanilla-Samplethe one is a multiple choice .
ReinforceI think I have to .
MCI think I have to .
Inputcommunism recently failed in eastern europe .
Vanilla-MLEthe war is not over .
Vanilla-Samplethere are people who hope that in tokyo they won ’t risk losing their departing .
Reinforceand the people who are not the same .
MCthe war is still going on .
Inputyou should stop in more often .
Vanilla-MLEI don ’t think so .
Vanilla-Sampleis it possible as old as you ?
Reinforcei ’m not going to be a burden on you .
MCi don ’t want to be a burden .
Inputhe couldn ’t have heard us .
Vanilla-MLEhe was in the middle of the night .
Vanilla-Samplehe should have been somewhere out here .
Reinforcehe ’s not a good guy .
MChe ’s got a lot of nerve.
Inputthat ’s our wake up call .
Vanilla-MLEwe ’re gonna be late for the meeting .
Vanilla-Samplec ’mon c ’mon let ’s get outta here .
Reinforcei ’ll be right back .
MCwe ’re gonna have to get to the station .
Inputi had ’em when i started .
Vanilla-MLEi ’m gonna get ’em .
Vanilla-Samplethose weak rotten guys had me in a rush on a broken swoop .
Reinforcei don ’t know what to do .
MCi was just a little bit nervous .
Inputoatmeal delicious start into a new day
Vanilla-MLEi ’ll be right back
Vanilla-Samplesure if you don ’t put it into the water
Reinforcei ’m gonna be a little busy with the dishes .
MCi ’m gonna make you a little dinner .

Table 6. Appendix: More responses sampled from different models.

References

  1. [2]Nabiha Asghar, Pasca Poupart, Jiang Xin, and Hang Li. 2016. Online sequence-to-sequence reinforcement learning for open-domain conversational agents. arXiv preprint arXiv:1612.03929.
  2. [3]Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. An actor-critic algorithm for sequence prediction. ICLR.arxiv.org/abs/1607.07086
  3. [4]Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of ICLR.arxiv.org/abs/1409.0473
  4. [5]Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. CoNLL.arxiv.org/abs/1511.06349
  5. [6]Xi Chen, Yan Duan, Rein Houthooft, John Schulman, Ilya Sutskever, and Pieter Abbeel. 2016a. Info-gan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances In Neural Information Processing Systems. pages 2172–2180.
  6. [7]Xilun Chen, Ben Athiwaratkun, Yu Sun, Kilian Weinberger, and Claire Cardie. 2016b. Adversarial deep averaging networks for cross-lingual sentiment classification. arXiv preprint arXiv:1606.01614.arxiv.org/abs/1606.01614
  7. [8]Emily L Denton, Soumith Chintala, Rob Fergus, et al. 2015. Deep generative image models using a? laplacian pyramid of adversarial networks. In Advances in neural information processing systems. pages 1486–1494.arxiv.org/abs/1506.05751
  8. [9]Peter W Glynn. 1990. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM 33(10):75–84.DOI
  9. [10]Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems. pages 2672–2680.
  10. [11]Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735–1780.
  11. [12]Thorsten Joachims. 2002. Learning to classify text using support vector machines: Methods, theory and algorithms. Kluwer Academic Publishers.
  12. [13]Anjuli Kannan and Oriol Vinyals. 2016. Adversarial evaluation of dialogue models. In NIPS 2016 Workshop on Adversarial Training.arxiv.org/abs/1701.08198
  13. [14]Alex Lamb, Anirudh Goyal, Ying Zhang, Saizheng Zhang, Aaron Courville, and Yoshua Bengio. 2016. Professor forcing: A new algorithm for training recurrent networks. In Advances In Neural Information Processing Systems. pages 4601–4609.arxiv.org/abs/1610.09038
  14. [15]Jiwei Li, Michel Galley, Chris Brockett, Jiafeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proc. of NAACL-HLT.arxiv.org/abs/1510.03055
  15. [16]Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jiafeng Gao, and Bill Dolan. 2016b. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Berlin, Germany, pages 994–1003. http://www.aclweb.org/anthology/P16-1094.arxiv.org/abs/1603.06155
  16. [17]Jiwei Li, Minh-Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. ACL.arxiv.org/abs/1506.01057
  17. [18]Jiwei Li, Will Monroe, and Dan Jurafsky. 2016c. A simple, fast diverse decoding algorithm for neural generation. arXiv preprint arXiv:1611.08562.arxiv.org/abs/1611.08562
  18. [19]Jiwei Li, Will Monroe, Alan Ritter, and Dan Jurafsky. 2016d. Deep reinforcement learning for dialogue generation. EMNLP.arxiv.org/abs/1606.01541
  19. [20]Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. EMNLP.arxiv.org/abs/1603.08023
  20. [21]Ryan Lowe, Michael Noseworthy, Iulian Serban, Nicolas Angeliard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an automatic turing test: Learning to evaluate dialogue responses. ACL.arxiv.org/abs/1708.07149
  21. [22]Ryan Lowe, Iulian V Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. On the evaluation of dialogue systems with next utterance classification. SIGDIAL.arxiv.org/abs/1605.05414
  22. [23]Yi Luan, Yangfeng Ji, and Mari Ostendorf. 2016. LSTM based conversation models. arXiv preprint arXiv:1603.09457.arxiv.org/abs/1603.09457
  23. [24]Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015. Effective approaches to attention-based neural machine translation. ACL.arxiv.org/abs/1508.04025
  24. [25]Alec Radford, Luke Metz, and Soumith Chintala. 2015. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434.arxiv.org/abs/1511.06434
  25. [26]Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. ICLR.
  26. [27]Alan Ritter, Colin Cherry, and William B Dolan. 2011. Data-driven response generation in social media. In Proceedings of EMNLP 2011. pages 583–593.
  27. [28]Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. 2016. Improved techniques for training gans. In Advances in Neural Information Processing Systems. pages 2226–2234.arxiv.org/abs/1606.03498
  28. [29]Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016a. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of AAAI.arxiv.org/abs/1507.04808
  29. [30]Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016b. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the 30th AAAI Conference on Artificial Intelligence (AAAI-16).arxiv.org/abs/1507.04808
  30. [31]Iulian Vlad Serban, Tim Klinger, Gerald Tesauro, Kartik Talamadupula, Bowen Zhou, Yoshua Bengio, and Aaron Courville. 2016c. Multiresolution recurrent neural networks: An application to dialogue response generation. arXiv preprint arXiv:1606.00776.arxiv.org/abs/1606.00776
  31. [33]Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017. A hierarchical latent variable encoder-decoder model for generating dialogues. AAAI.arxiv.org/abs/1605.06069
  32. [35]Louis Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2017. Generating long and diverse responses with neural conversational models. ICLR.
  33. [36]Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. ACL.arxiv.org/abs/1512.02433
  34. [37]David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lancot, et al. 2016. Mastering the game of Go with deep neural networks and tree search. Nature 529(7587):484–489.DOI
  35. [38]Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Meg Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of NAACL-HLT.arxiv.org/abs/1506.06714
  36. [39]Pei-Hao Su, Milica Gasic, Nikola Mrksic, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Continuously learning neural dialogue management. arXiv.arxiv.org/abs/1606.02689
  37. [40]Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems. pages 3104–3112.arxiv.org/abs/1409.3215
  38. [41]Alan M Turing. 1950. Computing machinery and intelligence. Mind 59(236):433–460.
  39. [42]Oriol Vinyals and Quoc Le. 2015. A neural conversational model. In Proceedings of ICML Deep Learning Workshop.arxiv.org/abs/1506.05869
  40. [43]Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Lina M Rojas-Barahona, Pei-Hao Su, Stefan Ultes, David Vandyke, and Steve Young. 2016. A network-based end-to-end trainable task-oriented dialogue system. arXiv preprint arXiv:1604.04562.arxiv.org/abs/1604.04562
  41. [44]Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning 8(3-4):229–256.
  42. [45]Sam Wiseman and Alexander M Rush. 2016. Sequence-to-sequence learning as beam-search optimization. ACL.arxiv.org/abs/1606.02960
  43. [46]Zhen Xu, Bingquan Liu, Baoxun Wang, Chengjie Sun, and Xiaolong Wang. 2016. Incorporating loose-structured knowledge into LSTM with recall gate for conversation modeling. arXiv preprint arXiv:1605.05110.arxiv.org/abs/1605.05110
  44. [48]Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. 2016a. Seggan: sequence generative adversarial nets with policy gradient. arXiv preprint arXiv:1609.05473.arxiv.org/abs/1609.05473
  45. [49]Zhou Yu, Ziyu Xu, Alan W Black, and Alex I Rudnicky. 2016b. Strategy and policy learning for non-task-oriented conversational systems. In 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. page 404.DOI
  46. [50]Yuan Zhang, Regina Barzilay, and Tommi Jaakkola. 2017. Aspect-augmented adversarial networks for domain adaptation. arXiv preprint arXiv:1701.00188.arxiv.org/abs/1701.00188

Paper details

vlnc:W_7pmheuz7
Version
v1 · latest
Published
Fresh — added to the Hub within the last 30 days
Added
License
CC BY 4.0
Usage

views · 1

downloads · 0

A view counts once per browser per day, a download each time the PDF is requested.

  • Computer Science

Not recorded for this paper

View statement
Looking for related papers…
Contents