What does it mean to be haunted by the memory of something? First we need to figure out what memories are and how they are made. Let's say that a memory is an imprint of sensory experience from the first person perspective. It can be any combination of sense experiences like touching, hearing, or seeing, or even more complex emotions like being sad or angry. And it has to be from the first person perspective because we assume someone is/was having these experiences for it to leave an imprint on. Memories are fascinating phenomena and play a huge role in regular human function. With that said, sometimes it feels like there are some memories we would like to block out, or even forget completely. Since memories are essentially past sense experiences and humans have limited cognitive resources at any given moment, being able to focus one's attention on certain memories instead of others becomes an important skill. So, to answer our initial question, to be haunted by memories just means that some collection of past sense experiences keep making their way into the person's attention at the expense of the person's cognitive resources.
This is essentially what is happening to Rokka, the female protagonist in Natsuyuki Rendezvous. Around 30 years old, her husband died 6 years ago due to cancer. Rokka continues to work at the flower shop that she and her late husband founded. The story of Natsuyuki Rendezvous is about Hazuki, a young man who falls in love with Rokka and tries to make her happy, but Rokka isn't exactly ready to move on. No matter what Rokka does with Hazuki, she still has the memories of her deceased husband at the back of her head.
There are two things that make Natsuyuki Rendezvous interesting. First is the presence of Rokka's dead husband as a ghost. Rokka's dead husband, named Atsushi, wanders around the flower shop as a ghost. The problem is no one can see or hear him except Hazuki. Atsushi tries to distract and deter Hazuki from getting closer to Rokka. As a result, Hazuki essentially becomes love rivals with a deceased man both because of the fact that Rokka can't get the memories of Atsushi out of her head and because Atsushi is still around as a ghost. Rokka's memories of Atsushi probably persist longer because she's in the same environment (the flower shop) that she was in when Atsushi was still alive and working with her at the shop. It also doesn't help that their flower shop uses their last name (The Shimao Flower Shop). All around Rokka she has these reminders of Atsushi which help bring up memories of him. In a way, you could compare ghosts to memories. Ghosts are supposedly the imprint of a living person, or spirit, left behind because they have something that they want to finish or do.
Whether or not you believe in ghosts or spirits is not the point, the question is how ghosts or memories can affect the day-to-day life of a person by 'sticking around'. In a way, both Hazuki and Rokka are being haunted or bothered by the memory of Atsushi. Likewise, all three of the characters are clinging onto (or at least have trouble not thinking about) something that might just be better off letting go. Rokka keeps thinking about Atsushi, even though he is dead. Atsushi wants to protect Rokka from having another partner, even though she might be happier with someone new. And Hazuki wants to make Rokka happy, even though it's clear she has some.. unfinished business with her dead husband.
The second thing that makes Natsuyuki Rendezvous interesting is the use of flowers and the comparisons that can be drawn from plant life to human life. Flowers don't have minds or memories (that we know of), but they do go through a cycle of growth and death each year. In spring, flowers blossom, then die in winter, leaving behind seeds for another new batch of flowers to bloom next spring. It's less straight forward when dealing with humans, especially when relationships are involved. This is because of the phenomenon of memories. It wouldn't be a problem for Rokka to go out with Hazuki, if memories had some sort of fixed lifespan, like after a few years after a particular sense experience, the memory just disappears forever. But we all know human memory is a lot more complicated than that. Now you might be thinking, wait there's amnesia. All Hazuki has to do is give Rokka a serious concussion that will erase all her memories of Atsushi. =D But.. would that really be the right way to go about forming a relationship? Is there a right way? Some people spend lots of time and money on dating and their partners, and many a drama can result from lingering feelings for exes. What would you do? It's hard to say because we're not the ones experiencing Rokka or Hazuki or Atsushi's emotions, thoughts or memories, they are! Likewise, they can never really know what each other is thinking.
At the end of episode 4, Hazuki is about ready to give up, having gotten drunk, and stumbling into the flower shop, he gives in to Atsushi's request to lend him his body. With Atsushi having control over Hazuki's body, he is no longer just an imprint or a ghost. What will he do with the chance that he's given? It's not clear how long Atsushi is planning to borrow Hazuki's body (because this happened at the end of the episode), but now Atsushi has a chance to talk and interact with Rokka to make new memories with her to help her move on... or not. ;)