In this paper an approach for estimating the three dimensional motion field of the observed world from stereo image sequences is proposed. This approach combines dense optical flow estimation, including spatial regularization, and dense stereo information using Kalman filters for temporal smoothness and robustness. The result is a dense, robust and accurate reconstruction of the three-dimensional motion field of the observed scene. Parallel implementation on a GPU and a FPGA yields a real-time vision-system which is directly applicable in real-world scenarios, like automotive driver assistance systems, robotics or in the field of surveillance.