Advanced mathematics question

please explain the answer