They were arrested while getting off from the boat off Dhanushkodi coast.
The arrested have been identified as Nirmalan, his wife Nishandhini, their one-year-old son Dickson and boat operator Kandhasamy.
The couple told the naval personnel that they wanted to come to India as their parents in Sri Lanka opposed their marriage. They also confessed that the boat fitted with two engines was a stolen one.
Meanwhile, the Sri Lankan navy allegedly destroyed fishing nets of more than 60 boats from Rameswaram.