OTTAWA—Canada’s financial-intelligence agency said Tuesday it has fined Royal Bank of Canada, the country’s largest lender by market capitalization, for failing to comply with the country’s rules aimed at deterring money laundering.

The Financial Transactions and Reports Analysis Centre of Canada, or Fintrac, said the fine, of 7.5 million Canadian dollars, or the equivalent of $5.5 million, is the largest financial penalty it has yet levied.