Scientific Machine Learning (SciML) is an interdisciplinary field combining data-driven machine learning with physics-based modeling, for accurate predictions across scientific and engineering applications. Despite the great successes of SciML, the widespread deployment in real-world applications remains challenged by model uncertainty, limited data, and potential model misspecification, which can undermine reliability and trustworthiness. Bayesian learning offers a principled solution by providing probabilistic predictions, quantifying uncertainty, and supporting continual adaptation as new data becomes available. However, the integration of Bayesian methods into SciML, particularly Bayesian Deep Learning, remains relatively underexplored. In this review, we provide a comprehensive overview of Bayesian perspectives in SciML, focusing on methodological advances, current challenges, and emerging opportunities. We highlight how Bayesian approaches can enhance uncertainty quantification, continual learning, interpretability, and robustness in scientific applications, bridging the gap between statistical rigor and the practical demands of large-scale applications and high-dimensional models. By systematically examining these contributions, we outline promising research directions and opportunities at the intersection of Bayesian methods and Scientific Machine Learning.