A starter the first half of his career, Kline was converted to relief and collected 108 saves. As a Pirate starter, his 18 losses in 1956 and 16 in 1958 were NL highs. In 1965 he led the AL with 29 saves and was second with 74 appearances, giving him the single-season Senator club records for saves, games pitched, and games finished. Returning to the Pirates in 1968, Kline had 12 relief wins (10 in succession) to lead the NL.
Kline had one of the most peculiar pitching rituals in baseball, touching his cap, belt, and shirt before each pitch. One of the poorer-hitting pitchers in the game, Kline managed only a lifetime .092 average.