The aim of this work is to establish a basic microscopic picture, which stands behind complex magnetic properties of hexagonal manganites. For these purposes, we consider two characteristic compounds: YMnO and LuMnO, which form different magnetic structures in the ground state (P6̲3cm̲ and P6̲3c̲m, respectively). First, we construct an electronic low-energy model for the Mn bands of YMnO and LuMnO, and derive parameters of this model from the first-principles calculations. From the solution of this model, we conclude that, despite strong frustration effects in the hexagonal lattice, the relativistic spin-orbit interaction lifts the degeneracy of the magnetic ground state. Furthermore, the experimentally observed magnetic structures are successfully reproduced by the low-energy model. Then, we analyze this result in terms of interatomic magnetic interactions, which were computed using different types of approximations (starting from the model Hamiltonian as well as directly from the first-principles electronic structure calculations in the local-spin-density approximation). We argue that the main reason why YMnO and LuMnO tend to form different magnetic structures is related to the behavior of the single-ion anisotropy, which reflects the directional dependence of the lattice distortion: namely, the expansion and contraction of the Mn-trimers, which take place in YMnO and LuMnO, respectively. On the other hand, the magnetic coupling between the planes is controlled by the next-nearest-neighbor interactions, which are less sensitive to the direction of the trimerization. In the P6̲3cm̲ structure of YMnO, the Dzyaloshinskii-Moriya interactions lead to the spin canting out of the hexagonal plane, which is additive to the effect of the single-ion anisotropy. Finally, using the Berry-phase formalism, we evaluate the magnetic-state dependence of the ferroelectric polarization, and discuss potential applications of the latter in magnetoelectric switching phenomena.