Before his NFL career, Mean Joe Greene had an outstanding college football career at North Texas State University (now the University of North Texas) (1966-68), leading the team to a 23-5-1 record during his 3 seasons. In his 29 games at defensive tackle, North Texas State held the opposition to 2,507 yards gained on 1,276 rushes: a per carry average of less than two yards per attempt. His collegiate coach, Rod Rust, said of the 1968 consensus All-America, "There are two factors behind Joe's success. First, he has the ability to make the big defensive play and turn the tempo of a game around. Second, he has the speed to be an excellent pursuit player." A pro scout said, "He's tough and mean and comes to hit people. He has good killer instincts. He's mobile and hostile."

The North Texas mascot, the Mean Green, is named after him. In 1984, he was enshrined in the College Football Hall of Fame. In 2006 was voted to the East-West Shrine Game Hall of Fame.

In 1969, he was selected by the Pittsburgh Steelers with the 4th pick of the NFL draft and spent his entire career with them until his retirement in 1981. When Joe Greene was drafted, a newspaper headline asked Who's Joe Greene? The question was quickly answered as Greene became so good, that teams double-teamed, and even triple-teamed, him throughout his entire career.

After he was drafted Greene quickly established himself as a dominant defensive player. He was strong, quick and intense. He was the NFL's rookie of the year in 1969, even though he played on a poor Steelers team that went 1-13 and had a long standing reputation of being a doormat for other NFL teams. Greene later admitted that he was upset with being drafted by the Steelers due to their long history of losing. He often showed his displeasure on the field.

In his early years with the Steelers, Greene was at times uncontrollable and often let his temper get the best of him. At one time during a 1975 game against the rival Cleveland Browns at Cleveland Municipal Stadium in which the Steelers won decisively 42-6, Greene repeatedly kicked Browns lineman Bob McKay in the groin while McKay was lying on the ground. Another incident had Greene snap the ball away from the center while the opposing team was lining up for a play. He had no tolerance for losing, and the team veterans quickly took notice. His intense desire to win rallied the veterans around him, and with great drafts as well as superb coaching the Steelers franchise soon began to undergo a dramatic makeover. Joe Greene was credited as the cornerstone of the great Steelers dynasty and the most important player in team history. He was maniacal, relentless and utterly unstoppable.

Mean Joe Greene was the leader and the anchor of the Steel Curtain defense that won four Super Bowls in the 1970's. He was twice recognized as the NFL defensive player of the year in 1972 and 1974. He, along with other members of the Steelers' front four (L.C. Greenwood, Dwight White and Ernie Holmes) even appeared on the cover of Time magazine. That defense held NFL MVP Fran Tarkenton and the Vikings offense scoreless in Super Bowl IX (the Vikings only scored on a blocked punt, for which they missed the extra point), the only time that has occurred in Super Bowl history. It was also Greene's best championship performance, when he became the first player ever to record an interception, a forced fumble, and a fumble recovery in a single Super Bowl. He went to the Pro Bowl 10 times during his career.

Greene is also well-known for the "stunt 4-3" defense in which he would line up at an angle, between the center and guard, and would explode into the line taking up 2-3 blockers. He started doing this sometime in the 1974 season, and while it cut down on the number of sacks he racked up it freed up his other defensive teammates like middle linebacker Jack Lambert to make tackles with ease.

After leading the Steelers to another Super Bowl win after the 1975 season over the Dallas Cowboys in Super Bowl X, Greene missed the first several games of the 1976 season with a back injury. The Steelers started off the season 1-4 and looked like they would not make the playoffs. Quarterback Terry Bradshaw also got injured and was replaced by rookie Mike Kruczek. The season looked lost. But Greene and the Steelers defense carried the Steelers to nine straight wins and the playoffs. In what was probably the greatest NFL defense in the modern, the 1976 Steelers held opponents to an average of less than 10 points per game (138 points over 14 games). During their nine-game winning streak, the Steelers defense recorded five shutouts, another modern record, and gave up a total of just 28 points. The defense allowed only two touchdowns over nine games.

Middle linebacker Jack Lambert had, along with Greene, become the emotional leader of the defense and over the next several years became the dominant player at his position while Greene continued to perform at an all-pro level, becoming a 5-time All-Pro (1972-74,77,79) and in 1969 receiving the first of his 10 Pro Bowl invitations. He retired after the 1981 season after 13 years in the league.

After retiring from the NFL, Mean Joe Greene spent one year (1982) as a color analyst for CBS' NFL coverage before becoming an assistant coach under Steelers' head coach Chuck Noll in 1987. He spent the next 16 years as an assistant coach with the Pittsburgh Steelers, Miami Dolphins, and Arizona Cardinals. In 2004, he retired from coaching and was named the special assistant for player personnel for the Steelers. In this position he earned his 5th Super Bowl ring after the Steelers won Super Bowl XL, and a sixth from Super Bowl XLIII. Greene is one of four people outside the Rooney family to have Super Bowl rings from the first six championship teams.

Although the Steelers do not officially retire jersey numbers, Greene's number 75 has not been issued since his retirement and is understood to be "unofficially retired". Greene also briefly wore number 72 during his rookie season before switching to his more familiar 75 midseason.

Greene now resides in Flower Mound, Texas.