Foreign Minister Julie Bishop said senior ministers decided today not to proceed with the treaty after the opposition Labor Party declared it would block it in the Senate.
Bishop says she will renew negotiations with China and Labor to find a compromise that the Senate will accept.
Former conservative Prime Minister John Howard's government signed the treaty in 2007 only weeks before it was defeated in general elections, so it was never ratified.
Chinese Premier Li Keqiang urged Prime Minister Malcolm Turnbull to ratify the deal during a visit last week which focused on closer economic ties.