The feedback phase in a chaotic system consisting of a semiconductor laser subject to delayed optical feedback is considered for the first time as a secret key for secure chaotic communications not exclusively based on hardware uniqueness. Extensive numerical simulations illustrate that the feedback phase is of extreme importance as far as synchronization is concerned. The ability of an eavesdropper to attack the intensity-modulated message when a pseudorandom variation of the feedback phase is imposed at the transmitter's side is numerically quantified by bit-error-rate calculations. The analysis demonstrates that the eavesdropper is not able to synchronize and hence to extract the message when he is not aware of the phase variations even if he is equipped with an identical chaotic device.