Brimfield was first settled in 1706 and was officially incorporated in 1731. Until 1812, Brimfield was originally part of Hampshire County. The original Hampshire County included territory that is now in modern-day Hampden County, Franklin County, and Berkshire County as well as small parts of modern-day Worcester County.

According to the United States Census Bureau, the town has a total area of 35.2 square miles (91 km²), of which 34.7 square miles (90 km²) is land and 0.5 square miles (1.3 km²) (1.45%) is water. Brimfield is located in Western Massachusetts. Brimfield is bordered by Sturbridge on the east, Holland and Wales on the south, Monson and Palmer on the west, and Warren on the north. U.S. Route 20 and Route 19 both run through Brimfield. Interstate 90 cuts through the northern portion of the town in two sections.

Brimfield is located 22 miles (35 km) away from Springfield, 25 miles (40 km) away from Worcester, 41 miles (66 km) away from Hartford, 65 miles (105 km) away from Boston, 105 miles (169 km) from Albany, 71 miles (114 km) away from Pittsfield and 153 miles (246 km) from New York City.

As of the census of 2000, there were 3,339 people, 1,250 households, and 886 families residing in the town. The population density was 96.2 people per square mile (37.1/km²). There were 1,396 housing units at an average density of 40.2 per square mile (15.5/km²). The racial makeup of the town was 97.69% White, 0.51% African American, 0.36% Native American, 0.06% Asian, 0.06% Pacific Islander, 0.72% from other races, and 0.60% from two or more races. Hispanic or Latino of any race were 1.29% of the population.

There were 1,250 households, out of which 35.4% had children under the age of 18 living with them, 61.1% were married couples living together, 6.9% had a female householder with no husband present, and 29.1% were non-families. 23.1% of all households were made up of individuals, and 11.2% had someone living alone who was 65 years of age or older. The average household size was 2.67 and the average family size was 3.21.

In the town, the population was spread out, with 27.3% under the age of 18, 5.7% from 18 to 24, 28.9% from 25 to 44, 27.0% from 45 to 64, and 11.0% who were 65 years of age or older. The median age was 39 years. For every 100 females, there were 98.2 males. For every 100 females age 18 and over, there were 97.2 males.

The median income for a household in the town was $50,181, and the median income for a family was $59,943. Males had a median income of $44,000 versus $26,695 for females. The per capita income for the town was $23,711. About 2.2% of families and 4.4% of the population were below the poverty line, including 2.8% of those under age 18 and 3.8% of those age 65 or over.