Everyone is unique, everyone is different but potentially the fitter you are the more recovery time you may need. You are able to complete more intense exercise in less time than people who are not so fit. You will have more lean muscle mass than not so fit people so when the muscles tear and break down after exercise the more lean muscle tissue there is to repair by the body.