Телеканал CNN RGB в оттенки серого против ввода проблемного обучения ?

Hi. I've been learning about CNNs and reinforcement learning and I've find this project on github to play with: https://github.com/adityajn105/flappy-bird-deep-q-learning which uses double dueling deep Q learning. Input is in grayscale and there's no background so I've found background and been trying to train with RGB input, but didn't have good results at all.
Should CNN architecture be more complex or what's the main problem here when training with RGB input?

Only things I changed, since I'm trying RGB input, are:
I'm not converting to grayscale obviously and I've changed tensor shapes to fit RGB input channels. 
I also tried to add more CNN layers, change number of neurons etc.