Selective feature extraction from single-channel electroencephalogram signal provides appropriate classification of mental tasks, which is crucial for designing mobile brain-computer interface and neuro-bio-feedback systems. However, existing features deteriorate in the presence of artifacts. Therefore, we propose a mental task classification method using variational mode decomposition (VMD)-based novel feature extraction from single-channel EEG, with three stages: Signal decomposition using VMD; computation of proposed variational mode energy ratio; classification using adaptive boosting algorithm. The proposed method is evaluated using artifact-free and contaminated EEG signals from EEG during mental arithmetic task (EEGMAT) database and self-acquired (SA) database recorded using single-channel device. Average subject-specific accuracies of 93\% and 96\% for classification of baseline and serial-subtraction task have been achieved in EEGMAT and SA databases respectively. Extensive comparative analysis exhibits the superiority of proposed feature as compared to existing features in terms of accurate classification of baseline-mental task, and robustness under artifactual EEG signals. © 2020 IEEE.