To cancel an order, you need to create a Credit Memo. This will reverse out the payment record for the order and cancel the order.
To cancel an order:
Click on an order to view the details.
Then click the Credit Memo button in the toolbar at the top.
Finally, click the blue Create Credit Memo and Cancel Order button at the bottom of this page.
This will mark the order as Cancelled.