Dynamic Programming and Reinforcement Learning

To solve those three question use any software to code the dynamic programming equations and solve.However, you cannot use an optimization solver to solve the problems. Please, provide printouts for your solution.