Firstly, you need to check whether your server has been integrated the GD library or not. This is essential for the Captcha.
If you cannot login the back-end of the software without the Captcha, please modify the database to cease the Captcha function with the following tutorial:
Please run SQL the below with MySQL manager tool (e.g. phpmyadmin, mysql-front etc) and disable this feature first.
UPDATE `is_setting` SET `use_captcha`=0 WHERE `id`=1;