Lamp holder with rectangular cover