How Buddhism rose abruptly in India?

" Today it is the fashion to talk of Buddhism and Buddhistic agnosticism, especially in the South. Little do they dream that this degradation which is with us today has been left by Buddhism. This is the legacy which Buddhism has left to us. You read in books written by men who had never studied the rise and fall of Buddhism that the spread of Buddhism was owing to the wonderful ethics and the wonderful personality of Gautama Buddha. I have every respect and veneration for Lord Buddha, but mark my words, the spread of Buddhism was less owing to the doctrines and the personality of the great preacher, than to the temples that were built, the idols that were erected, and the gorgeous ceremonials that were put before the nation. Thus Buddhism progressed. The little fire-places in the houses in which the people poured their libations were not strong enough to hold their own against these gorgeous temples and ceremonies; but later on the whole thing degenerated. It became a mass of corruption of which I cannot speak before this audience; but those who want to know about it may see a little of it in those big temples, full of sculptures, in Southern India; and this is all the inheritance we have from the Buddhists. " - Swami Vivekananda

(Complete Works of Swami Vivekananda . Swamiji's speech at Madras)

How Buddhism rose abruptly in India? How far is it responsible for some of the evils in the Indian society? These critical views of swamiji seems to be in line with the view point of some scholars that  the idol worship and temple culture in Hinduism are the result of Buddhism's influence which has brought in both positive and negative impacts.