The Supreme Court's definition of "custody" has evolved. Originally, the Court spoke of the need to guard against compulsion "whenever the suspect is taken into custody or is otherwise deprived of his freedom of action in any significant way." (Miranda v. Arizona.) This guideline proved unworkable. Some lower courts thought that Miranda applied whenever the interrogated person was not free to leave (the traditional test for a Fourth Amendment detention); still other opinions examined whether there was probable cause to arrest, or whether the officer had subjectively decided to make an arrest, or whether questioning had occurred in a police station.
The Supreme Court later narrowed the definition of "custody" to mean either an actual arrest or the level of physical restraint typically imposed on an arrestee. "The ultimate inquiry is simply whether there is a formal arrest, or restraint on freedom of movement of the degree associated with a formal arrest." (California v. Beheler.)
This makes it reasonably easy to tell if someone is in custody, for Miranda purposes. If he has been told he is under arrest, he's in custody. Or, even if no announcement has been made and even though no Fourth Amendment arrest would necessarily be found, if he is nevertheless treated like an arrested person, he's in custody for purposes of receiving Miranda warnings.
For example, during a high-risk stop, a person who is ordered down at gunpoint is in Miranda custody. A person who is detained in handcuffs, or is locked in the rear-seat cage of a police car, or is detained in a police facility, or is surrounded by an inordinate number of officers is in Miranda custody. (Orozco v. Texas.) And a person who is in jail is normally considered to be in custody, even though he may have been arrested for some other crime. (Mathis v. United States.)
One by one, the alternative definitions of "custody" have been rejected by the Supreme Court. See, for example, Stansbury v. California (officer's focus of suspicion does not create custody); Berkemer v. McCarty (neither probable cause to arrest nor officer's intent to arrest constitutes custody); Pennsylvania v. Bruder (ordinary street detention without arrest-like restraints is not custody); and California v. Beheler (voluntary stationhouse appearance for questioning is not custody).