People are sensitive to bitter taste because bitterness serves as a warning sign of poisons. Previous research reported that taste receptors in the oral cavity are more sensitive to bitter taste than to sweet taste. Other sensory systems, i.e., auditory, olfactory, tactile and visual, have also been shown to be involved in the sense of taste, a phenomenon known as cross-modal correspondence. It has been shown that the color of food or its packaging influences taste expectation and perception. However, it remains unknown whether cross-modal correspondence between visual information and taste depends on the degree of bitterness. Here, we investigated the effect of background color on taste using chocolate and green tea with varying bitterness levels. Undergraduate students were asked to evaluate how bitter and how sweet milk chocolate (Experiment 1) or less bitter and bitter chocolate (Experiment 2) tasted when served from black or pink wrapping paper. In Experiment 3, undergraduate students were ask to evaluate how bitter and how sweet less bitter and bitter green tea tasted when served from a clear blue or a clear cup. The experiments showed that when chocolate and green tea were bitter, visual information influenced taste evaluation. In contrast, no visual effect on taste was seen for less bitter chocolate, milk chocolate or less bitter green tea. The results suggest the possibility that cross-modal correspondence of visual information to taste changes depending on the degree of bitterness.