Abstract

This paper investigates the diffusive predator-prey system with nonmonotonic functional response and fear effect. Firstly, we discussed the stability of the equilibrium solution for a corresponding ODE system. Secondly, we established a priori positive upper and lower bounds for the positive solutions of the PDE system. Thirdly, sufficient conditions for the local asymptotical stability of two positive equilibrium solutions of the system are given by using the method of eigenvalue spectrum analysis of linearization operator. Finally, the existence and nonexistence of nonconstant positive steady states of this reaction-diffusion system are established by the Leray–Schauder degree theory and Poincaré inequality.

1. Introduction

In order to describe the evolution of biological populations in the ecosystem, some mathematical theories and methods have been used to establish the corresponding biological mathematical model, which has become a research hotspot. In recent years, the research on biological models such as the predator-prey model has aroused the attention of many scientists and biologists. The predator-prey model of PDE forms is an important branch of reaction-diffusion equations. The dynamic relationship between predator and their prey is one of the dominant themes in ecology and mathematical ecology. During these thirty years, the investigation on the prey-predator models has been developed, and more realistic models are derived in view of laboratory experiments. Moreover, the research on the prey-predator models has been studied from various views and obtained many good results (see [1–22] and the references therein).

However, many studies have shown that only the presence of predators in front of the prey can affect the size of the prey population, and the effect is even greater than the effect of direct predation. Although some biologists have realized that the relationship between the prey and the predator cannot be simply described as direct killing, we should take the fear of the prey population into account. At present, there are few research studies on establishing corresponding mathematical models to explain this phenomenon.

For every specific prey-predator system, we know that the functional response of the predator to the prey density is very important, which represents the specific transformation rule of the two organisms. In [8], Pang and Wang considered a predator-prey model incorporating a nonmonotonic functional response which is called the Monod–Haldane or Holling type IV function:where is the outward directional derivative normal to . Model (1) describes a prey population which serves as food for a predator with population . The parameters are assumed to be only positive values: the positive constant is the carrying capacity of the prey and the positive constant is the death rate of the predator; is the growth rate of prey ; and the positive constants are the diffusion coefficients.

In this paper, based on the above model, in order to describe the evolution law of the population in the ecosystem more specifically, we will consider the natural mortality and fear effect of the prey population and establish the corresponding PDE model within a fixed bounded domain with smooth boundary at any given time and the natural tendency of each species to diffuse to areas of smaller population concentration [7–10]. Hence, we will investigate the following reaction-diffusion system under the homogeneous Neumann boundary conditions as follows:where are continuous functions of . and stand for the densities of prey and predators, respectively. The parameters are assumed to be only positive constants. and denote the intrinsic death rate of prey and predator , respectively. stands for the fear factor of prey to predator. The remaining parameters refer to (1). Here, stands for Monod–Haldane functional response.

The main aim of this paper is to study the nonconstant positive steady states of (2), that is, the existence and nonexistence of nonconstant positive classical solutions of the following elliptic system:

The rest of this paper is arranged as follows. In Section 2, we discuss the stability of the equilibrium of the ODE system which corresponds to system (2). In Section 3, we establish a priori positive upper and lower bounds for the positive solutions of the PDE system. In Section 4, sufficient conditions for the local asymptotical stability of two positive equilibrium solutions of the system are established by using the method of eigenvalue spectrum analysis of linearization operator. In Section 5, the existence and nonexistence of nonconstant positive steady states of this reaction-diffusion system are established by using the Leray–Schauder degree theory, which demonstrates the effect of large diffusivity.

2. Stability of the ODE Model

The goal of this section is to discuss the stability of the ODE model; we give the ordinary differential equation of system (3) as follows:

By the similar method to [7], for (4), we can get the following result.

Lemma 1. Under initial conditions , the solution of system (4) is nonnegative and ultimately bounded which implies

Next, we will calculate the equilibrium point of system (4), and the result is given as follows.

Theorem 1. System (4) always has an extinction equilibrium point . If , then system (4) has only the equilibrium point . If , , and , then system (4) has two positive constant equilibrium points , i = 1 and 2.

Proof. It is easy to see that all equilibrium points of system (4) satisfy the following equations:It follows that system (4) obviously has equilibrium points and with . Next, we consider the existence of positive constant equilibrium point . By calculating the second equation of (6), we directly getwhere ensures that . Substituting into (6) and combining the two equations of system (6), we can obtain the following equation:Through the same solution deformation calculation, we can getwhere . According to the Vieta theorem, we getObviously, if , then has no positive constant solution; if , then has only one positive constant solution. Thanks to the same sign and , implies , which ensures that has only one positive constant solution denoted by . Thus, system (4) has two positive constant equilibrium points , i = 1 and 2. The proof is complete.

Theorem 2. If , then is globally asymptotically stable. If , then is unstable.

Proof. The proof of Theorem 2 is similar to that of Theorem 2 of [9]; hence, we omit it.

Theorem 3. Assume . If , then is locally asymptotically stable. If , then is unstable.

Proof. Through mathematical calculation, we obtain the Jacobian matrix of system (5) at the equilibrium point as follows:Obviously, when and both eigenvalues of have negative real parts, then is locally asymptotically stable; when and has a positive eigenvalue, then is unstable. The proof is complete.

Theorem 4. Assume . If (i = 1 and 2), then is locally asymptotically stable. If (i = 1 and 2), then is unstable.

Proof. For , the corresponding Jacobian matrix is given byBy simplifying, we can getwhereIt is easy to get that and under these conditions . Then, two eigenvalues of the matrix have negative real parts. Therefore, the equilibrium is locally asymptotically stable. If and the matrix has one positive eigenvalue, then is unstable.

3. A Priori Estimates on Equation (3)

The main purpose of this section is to give a priori upper and lower bounds for the positive solutions. To this aim, we first recall the following maximum principle due to [23, 24].

Lemma 2. Suppose .
If satisfies

and , then .

Theorem 5. If and , is a positive solution of (3). Then, the solution of (3) yieldswhere and .

Proof. By Lemma 2, if reaches its maximum at , it follows from the first equation of (3) thatHence, . Setting and combining two equations of system (3), we obtainthat is,If reaches its maximum at , thenwhich results inThus,Thanks to , we know thatLet , then,that is,If reaches its maximum at , thenwhere , which means thatLetting , it is easy to get the maximum of , that is, . Thus, :By (23) and (28), we have proved Theorem 5.
According to Theorems 5 and 1, we can easily get the following conclusion.

Theorem 6. If , , and , then system (3) has two positive constant solutions , i = 1 and 2.

Theorem 7. Suppose that is a nonnegative classical solution of (3). If , then is always zero solution.

Proof. Integrating the equation for in (3) over by parts, we getThus,Hence, . Substituting into the second equation of (3), we getand then, The proof is complete.

4. Stability of the Equilibrium of Equation (3)

The goal of this section is to investigate the local and global stability of the positive constant steady state . We first discuss the local stability of . To this end, we need to introduce some notations for developing our result.

Let

Therefore, system (3) becomes the following forms:

It follows that two positive solutions satisfywhere satisfies

In order to get the linearization operator of (3) at the positive constant solution , for (33), we calculate the partial derivatives with respect to and , respectively, at the equilibrium point , as follows: