Please check the version of your RStudio. The tutorials have been recorded on RStudio version 1.1.456 and 1.1.463. If you are also using the same version, the broom icon should be visible in the upper right corner of the Console window. Please note that this icon is a bit faded. So, you might need to look carefully.
Broom symbol might not be colored. It is always present below or towards the left of the minimize and maximize screen options of the console window in RStudio.