Haldane’s rule is a particularly well-known example of hybrid inviability and sterility. J.B.S. Haldane noticed that, when inviability or sterility is displayed in only one sex of F1 hybrids, it is generally the heterogametic (ZW or XY) sex rather than the homogametic sex (XX or ZZ). Thus, in Drosophila and mammals, it is usually the male hybrids which suffer the most pronounced inviability or sterility. In birds and butterflies, it is generally the females. This was wonderful! We here actually have a rule (if not a law, because it is sometimes disobeyed). As you have probably realized, apart from the trivial Hardy-Weinberg law, generally obeyed laws are very unusual in evolution!
However, opinions have differed, indeed still do differ as to how this might be caused. Now, through the work of Turelli and Orr, building on earlier work by Coyne and Orr, and especially by the famous geneticist H.J. Müller in the 1940s (who discovered, among other things that mutation could be enhanced by X rays and certain mutagenic chemicals), it seems now well established that it is chiefly due to sex-linked recessive genes acting epistatically in hybrids.
First, we must realize that these genes that cause hybrid sterility and inviability must be epistatic. Unless genes individually cause heterozygous disadvantage (and the chromosomal differences between species are an example), the genes of species B must be causing deleterious effects in A/B hybrids because of genetic interactions between the alleles in species A and and alleles at different genes in species B.
This is especially true when alleles are on the X chromosome in the heterogametic sex -- they can’t be causing difficulties due to heterozygous disadvantage since the alleles are hemizygous (i.e. there is only one copy). These alleles must be interacting with genes either on the Y chromosome or on the rest of the genome (probably the latter, which is almost always more genetically active). Turelli and Orr realized that an extremely simple explanation of Haldane’s rule was also probably the best.
The explanation requires only one extremely simple assumption: that most alleles causing hybrid sterility and inviability are recessive. It is not required that they preferentially appear on any chromosome, but because the sex chromosomes are heterogametic in one sex, sterility or inviability is predicted for the heterogametic sex of F1 hybrids, because the sex chromosomes are the only ones which have only one copy in each cell, and therefore express the recessive alleles.
It is not entirely clear why such genes should normally have recessive effects on hybrid inviability. But they do (see below). Possibly it is because deleterious alleles, wherever they occur, must create a loss of function. We are used to this in normal protein genes. If a stop codon mutation happens in the middle of an important functional protein, there is usually a recessive deleterious effect, because heterozygotes retain one normal copy of the gene which can make the protein. Think of the sickle cell allele of beta-haemoglobin. The same might be true for epistatic genes that cause hybrid inviability: whereas the allele in species A evolves to cause problems in AxB hybrids, the allele in species B has not necessarily diverged from the ancestral function, and can substitute for the epistatic deleterious effect of A’s allele. And vice-versa for genes that evolve in B.
Why should these alleles evolve to cause problems in hybrids at all? The answer is probably that the alleles evolve adaptively in response to selection pressures within their own species. An allele that evolves in species A must coevolve with epistatic loci in its own species, but this will not be the case for epistatic loci in other species. This kind of evolution will not always interfere with hybrids, but if it does, the selection pressure against it happening will be weak. Provided that the gene is sufficiently advantageous, it will overcome any effects of gene flow (which will be low) from the other species.
Now for Turelli & Orr’s explanation:
As two species diverge evolutionarily, recessive epistatic alleles that nobble hybrids evolve all over the genome. Because they act recessively in hybrids, they only cause damage in F1 hybrids when they are on the X chromosome, and then only when they are in the heterogametic sex.
As evolution proceeds, rarer forms of hybrid inviability and sterility (genes with dominant effects in hybrids, or genes that individually cause heterozygote disadvantage) also evolve. These genes will nobble hybrids wherever they are in the genome.
Therefore, we expect that the first F1 inviability and sterility to appear will be of the Haldane’s rule type. Later, inviability will appear in both sexes. If the genetics of hybrid inviability and sterility is studied, there would not, under this hypothesis be expected to be much of an X-chromosome bias for evolution (though there would be an X-chromosome bias of gene action). Also, a preponderance of recessive genes would be expected, wherever one looks in the genome.
Studies by Hollocher et al. and True et al. on Drosophila showing hybrid inviability have now largely confirmed these results. In Drosophila, for some reason, there are many more male hybrid sterility genes than equivalent genes for female sterility, whereas genes for inviability are largely the same -- possibly this is due to strong sexual selection for seminal chemistry which affect male/male sperm competition in the female reproductive tract. But hybrid sterility and viability genes are both largely recessive, and evolve on the autosomes roughly as commonly as on sex chromosomes.