HOM is a frequently-used abbreviation for the Hall of Mirrors Effect. This effect is caused by an error in the map which results in a funky blending or bleeding of textures. When you look around the area where the error is, it seems to have the same qualities of the old Fun House Hall of Mirrors, hence the name.
HOMs are not quite the same as a BSP Hole, though a BSP hole is often the cause of a HOM.
This is the scourge of 3D level design. Implemented by the UnrealEd Goblin himself, this is a hole in geometry that you may view what is the black expanse in UnrealEd.
Sometimes this is fixed with a rebuild, sometimes not. One effective technique is to add another brush in such a way that it is flush with the surface.
The best way is to study zone/portal view to work out which surface might be responsible. Reordering brushes or changing the way an area is built may help.
If for some reason, you WANT to introduce a hall of mirrors effect, you can do so by setting a surfaces 'invisible' flag to true. This, of course, only causes a HOM effect on surfaces that "border" the world, like those that would normally be designated fake backdrop.
GRAF1K: I'll bet there's a few little-know HOM fixes floating around. Here's a good place to post them.
MasterOfTheDark: I made a relatively simple (bad for very large levels, but hey, it worked) fix for the HOM effect: I subracted a hollow box from around a room that caused the HOM effect and made a tiny hole between the two. You can find my sample here (22 KB): http://motd.4d2.org/UT/fixHOM.ut2 A skybox might also fix the problem, though I haven't tried.