Let’s start with Cris Crawford’s definition of interaction. According to him interaction is a cyclic process in which two (purposeful) actors alternatively listen(input), think(process) and speak(output).
So two people talking (communicating verbally) is a form of interaction. A person watching a movie is not interaction.
This part is simple, but the idea of actors being purposeful is where it gets tricky. For example if I am putting a nail on a wall using a hammer, am I interacting? What am I interacting with? The nail, the hammer or the wall? The answer might be none and all. For example, technically I am speaking to the hammer with the force of my arm and it is listening and doing my work for me. But it is easy to dismiss a dumb hammer as not being a purposeful actor. What if it was a smart hammer? A hammer that can assess the job and judge how much force is required and does the job on press of a button? Will that be interaction with a purposeful actor? Maybe.
So yes, interaction is subjective. There are degrees to it. Why do we want to define interaction again? Oh yeah because we want to design interactions. Ok so for us adding that purposeful actor clause in the definition makes sense. Because sure one can design for interaction with a dumb hammer by making it very ergonomic, giving a safe and convenient way to store it etc, but we are generally not taking about that kind of interaction. We are taking about interaction at a level of conversation. Object A conversing with Object B, one of the two object would be a human because, again, we are not designing for machines. Although there is nothing wrong with it.
So if interaction is all about conversations, what is physical interaction? Well it is interaction with a physical object. When you are using the Spotify app on your phone, your physical interaction is with the phone. Someone (Apple) designed the physical interaction paradigms of the phone. Spotify has to design interaction of its app under the restrictions of physical interaction a user can have with the phone. Touch yes smell no. If Spotify wants to design a music player that works on smell, it will have to take its app out of the phone environment and put it in a device that takes smell as input. They will have to think about how will the user physically interact with that device, everything from its physical design, user touch-points etc.
Bret Victor wrote in 2011 about future of interaction design. His frustration was that even our vision of the future of interaction design is one where people are interacting with glass screens. In the five years since he wrote the article we have seen Tesla cars that are basically computer on wheels, Google glass and rise of virtual reality, Nest and other home assistants. I think he would be happy with the progress we have made in the field of physical interaction.
So yeah, coming back to the question in the title of this post, in a nutshell physical interaction is interaction with a physical object (one that is purposeful/smart for our purpose).