Distributed Denial of Service (DDoS) attack is one of the major attacks that incur large financial loss in the cloud system. This motivated the research community to develop various detection techniques for controlling the effects of the DDoS attack. However, the existing techniques are not mature to satisfy the requirements of a cloud-based attack detection system, as they manage the devious strategies that exploit the elastic and multi-tenant properties of the cloud and ignore the resource constraints of the cloud system. This paper proposes a new solution that allows the hypervisor to establish trust-based relationships towards the guest Virtual Machines (VMs). The Bayesian inference is applied to aggregate the objective and subjective trust sources. A trust-based maximin game between DDoS attackers is designed. A hypervisor tries to maximize the attack minimization under a limited amount of resources. The game solution guides the hypervisor to determine the distribution of optimal detection load among VMs to improve the real-time detection rate of DDoS attack. The Least Squares Support Vector Machine (LS-SVM) classification is applied to classify the normal VMs and malicious VMs. The file is allocated to the VM based on the storage capacity of the VM. The experimental result shows that the proposed approach achieves high DDoS attack detection rate with minimum false positive and negative rate, when compared to the existing attack detection models. © 2019 John Wiley & Sons, Ltd.