WASHINGTON—A conservative appeals court dealt a setback to the Securities and Exchange Commission’s plan to give investors more information about stock buybacks after business groups challenged the rule.

Responding to a May lawsuit by the U.S. Chamber of Commerce and two Texas-based business groups, a three-judge panel on the Fifth Circuit Court of Appeals ruled that the SEC’s new requirements for share repurchases were “arbitrary and capricious.” The judges agreed with the groups’ assertion that the SEC didn’t adequately explain the rule’s costs and benefits.