Mr Pence and Kim Yong-nam were being hosted by South Korean President Moon Jae-in before the opening ceremony in Pyeongchang.
But the US vice-president left the reception venue after five minutes, South Korea's Yonhap said.
While Mr Moon and Japanese Prime Minister Shinzo Abe both shook hands with Mr Kim, Mr Pence did not, South Korean officials said.
At the event Mr Moon said he hoped the Winter Olympics would be remembered as the "day peace began". He is due to meet the North Korean delegation for talks on Saturday according to Yonhap.
At the opening ceremony, Mr Pence, North Korean leader Kim Jong-un's sister Kim Yo-jong and Kim Yong-nam were seated in close proximity to each other.
Who is Kim Yo-jong?
The highest profile member of the North Korean delegation to the Games, she is the first immediate member of the North's ruling family to visit the South since the 1950-1953 Korean war.
Ms Kim, who is said to be very close to her brother, was promoted to the North's powerful politburo last year.
She is on a US sanctions list over alleged links to human rights abuses in North Korea.
Ms Kim is thought to be about 30 years old, around four years younger than her brother.
Image copyrightREUTERSImage captionNorth Korea's Ms Kim shook hands with South Korea's Mr Moon at the opening ceremony
Her visit is being seen as a sign that Kim Jong-un is serious about improving ties with the South, the BBC's South Korea correspondent Laura Bicker reports.
She adds that some are also speculating that Ms Kim might be bringing a message from her brother.
Athletes from both North and South Korea marched under one flag at the opening ceremony, bringing spectators to their feet.
They are also fielding a joint women's ice hockey team.
Alongside 22 athletes, Pyongyang has sent more than 400 delegates to the Games, including a team of cheerleaders and an orchestra.
Image copyrightAFP/GETTYImage captionNorth and South Korean athletes marched under one flag at the opening ceremony
However the opening ceremony was not shown on North Korean state TV, which was broadcasting patriotic songs and slogans celebrating industry and the armed forces.
Have relations warmed?
The sports diplomacy comes at a time of improved ties between the two Koreas, although experts have cautioned that it does not put an end to underlying regional tensions.
The Korean peninsula has been divided since the 1950-53 war and the two sides have never signed a peace treaty.