What do I do about the guy at work?

I work with a guy and 3 days ago I drove him home cause his car wasn't working.He ask me to come in for a drink.Inside I saw a nude picture of him on the table and his dick was huge.He seen me looking at his picture.He ask me if i want to see it for real and He got naked.He was ten inches and thick.He ask me to go in the bedroom and i did.The sex was great. Do i keep on having sex with him?He is married to a coworker who was out of town.I told him I won't tell anyone.I am also married.Also I am having a affair with his wife that no one know that we are bisexual.