As a burgeoning medical imaging method based on hybrid fusion of light and ultrasound, photoacoustic imaging (PAI) has demonstrated high potential in various biomedical applications, especially in revealing the functional and molecular information to improve diagnostic accuracy. However, stemming from weak amplitude and unavoidable random noise, caused by limited laser power and severe attenuation in deep tissue imaging, PA signals are usually of low signal-to-noise ratio (SNR), and reconstructed PA images are of low quality. Despite that conventional Kalman Filter (KF) can remove Gaussian noise in time domain, it lacks adaptability in real-time estimation due to its fixed model. Moreover, KF-based de-noising algorithm has not been applied in PAI before. In this paper, we propose an adaptive Modified Kalman Filter (MKF) targeted at PAI de-noising by tuning system noise matrix Q and measurement noise matrix R in the conventional KF model. Additionally, in order to compensate the signal skewing caused by MKF, we cascade the backward part of Rauch-Tung-Striebel smoother (BRTS), which utilizes the newly determined Q. Finally, as a supplement, we add a commonly used differential filter (DF) to remove in-band reflection artifacts. Experimental results using phantom and ex vivo colorectal tissue are provided to prove validity of the algorithm. This article is protected by copyright. All rights reserved.This article is protected by copyright. All rights reserved.