Hostile species are those known to violently attack others without provocation, whether or not they see their own behavior as perfectly reasonable and/or justifiable in their own alien sense of morality. They often seek the annihilation, enslavement or subjugation of entire races.
They're different from aggressive species: in which individuals regularly act with hostility towards their own kind rather than, or in addition to, others. As such, all aggressive species are hostile, but not all hostile species are aggressive. Predatory species are different too, in that they attack not out of sheer violence, but to obtain something from their victims, be it resources, power or something else.