New Zealand's latest batting failure saw them dismissed for 174 on the stroke of tea and that left them 180 runs behind England's first innings 354.
Off-spinner Swann took four for 42 in nine overs after fast bowler Steven Finn had rocked the top order on his way to three for 36.
But although England captain Alastair Cook was in a position to enforce the follow-on -- New Zealand needed to reach 205 and be within 150, rather than 200, of the hosts' first innings following Friday's first day washout to be certain of avoiding it --- he decided to bat again.
But they made almost as many for the first wicket Sunday until Peter Fulton, trying to turn Finn legside, got a leading edge and was caught and bowled by the fast bowler for 28 to leave New Zealand 55 for one.
Then a superb over to Hamish Rutherford, the other opening batsman, saw Finn beat the left-hander three times in a row outside off-stump.