Yeah, there is a lot of confusion and contradiction over it.
The way i understand it is this:
After the Executor crashed, command transferred to the next in the chain of command, being Chimaera (as secondary command ship). As the fighting continued, her captain was killed, leaving Pellaeon in command (both of Chimaera, and accidentally the fleet).
When Palpatine died, his Force Meditation was cut, leading to (as Thrawn later surmised) the loss of coordination and morale among the remaining fleet. This followed almost immediately by the destruction of the Death Star, caused massive panic and confusion among the fleet (natural, considering the huge loss the Death Star represented, plus the probable death of the Emperor).
Pellaeon then ordered the retreat towards the edge of the system (in realspace, partly due to Endor's gravity, and i assume also to allow for collection of survivors from ships, and Death Star) assuming the Rebels wouldnt follow. The fleet was slow in responding to this, and following Teshik's reappearance (having survived the Death Star's destruction) was cancelled.
The fight continued for some hours until Teshik's ship was neutralised, at which point Chimaera/Pellaeon regained command, and issued the to order to retreat to Annaj (which some commanders ignored, including Harrsk).