This chapter is dedicated to studying the impact of the curriculum learning process and the Swish activation function for finding the solution of Differential Equations (DEs) with initial conditions. Then we have compared the result of the proposed training algorithm and the usual training algorithm. Also, we have compared the neural result using Swish, Tanh, and Sigmoid activation functions. The artificial neural network (ANN) trial solution of the differential equation is written as a sum of two terms. Here the first term satisfies boundary conditions and the second term involves containing adjustable parameters so as the trial solution can solve the differential equations. In this investigation, first we have trained our neural network in a small domain and gradually expanded the domain. Feedforward neural network (FFNN) and error backpropagation algorithm have been used to minimize the error function and modification of weights and biases. Finally, several problems have been solved to illustrate the proposed training method, and analytical results have been compared with neural results.