Jay tolerates Javier Delgado (Benjamin Bratt) because he's Manny's dad. That makes him a member of their "Modern Family," but it doesn't mean Jay has to like him. Javier only shows up at the Pritchett house about twice a year, and Jay tries to keep things cordial for Manny's sake, although he can't help tossing out a sarcastic remark or two. He didn't even complain when Javier knocked on their door at 10 pm one night. He'd missed Manny's birthday and wanted to celebrate his becoming a man with a day at the racetrack.
Before his dad showed up, Manny was excited over a golf outing Jay had planned for the next day. Golfing meant he could wear knickers and a vest like Matt Damon wore in "Bagger Vance." Still he jumped at the chance to spend a day at the racetrack with his father. Jay knew he had no control over Manny where his father was concerned, but that wasn't the only issue.
Jay was trustworthy and Javier wasn't
Jay didn't really want to spend the day at the racetrack with Manny and Javier, but Gloria demanded he tag along. She trusted Jay with her son, but didn't trust her ex. Javier once took Manny to a petting zoo; and she found out later that Petting Zoo was the name of a strip club.
Jay was logical and Javier was a dreamer
Before Jay bet on a race horse, he read the handicap sheets and analyzed the stats. Javier's methods were a bit more esoteric. He looked into a horse's eyes to "...see their souls" and decided to bet or not to bet. Javier's methods were annoying and Jay made a few snide remarks about his soul-seeing ability, but he and Manny won some cash, while Jay lost.
Javier is handsome, Jay, not so much
Javier's swarthy good looks made him the perfect match for ex wife, Gloria. He's so proud of that fact, he introduced himself to Claire as "...Gloria's ex husband and lover." Jay was young once upon a time. He might have been good looking as well, but those days are well behind him.
Jay is old and Javier is young
Last season, age had Jay feeling so insecure he got Botox treatments. Javier is the far younger man of the two, and he used that fact to push Jay's buttons. When Jay mentioned the golf outing, Javier explained that golf was for "... old men, and for women who love women." That was an insult no matter how you looked at it.
Javier is Manny's father and Jay isn't
Javier only showed up a few times a year to take Manny on father/son outings. Jay used to enjoy the break from fatherhood, but now he and Manny have their own relationship. The dynamic shifts when Javier shows up. Jay knows he's not his dad, but he admitted, "Maybe I don't like the reminder."
Source: "Modern Family": Season 3, Episode 11: "Lifetime Supply"