Here is the actual quote.
I am afraid I am not in the flight for “aerial navigation”. I was greatly interested in your work with kites; but I have not the smallest molecule of faith in aerial navigation other than ballooning or of expectation of good results from any of the trials we hear of. So you will understand that I would not care to be a member of the aëronautical Society.
The statement is frequently misquoted in the way you have. But even if we take your version of the quote as valid, I don't recall Kelvin being an authority on fluid dynamics, mechanics, materials, or anything even remotely related to flight. His work was in thermodynamics and electricity. So his lack of enthusiasm is understandable.
Scientists are people and they often speak of things they don't understand, just like everyone else. The difference is that they also can speak on things that they have great deal of understanding about. And things that scientists working in the field can agree on are the sort of things where you can say, "Science thinks that..."
In contrast to Kelvin's statement, take a look at the works by such people as Kutta and Joukowski. These guys are effectively responsible for modern aerodynamics. Joukowski's papers on theory of lift predate Kelvin's statement. So people who actually worked in the field knew the heavier than air flight was possible and were working hard on making it a reality.