You can delete podcasts from either the podcast context menu, the Sidebar -> Remove menu, or with the Delete or Backspace key.
To remove a single podcast:
Right-click (Mac users: control-click) on the name of the show in the sidebar
To remove several podcasts:
Use Shift or CTRL (CMD) click to select them
From the Sidebar Menu choose Remove
Click the delete button that appears in the main view section
Accept the Delete confirmation