MDP Assignment / Operations Research

Hi,I need help in operations research assignment / MDP assignment. Please check attachment. Please provide step by step solution and provide all required descriptions.