AMSTeX is a TeX macro package, originally written by Michael Spivak for the American Mathematical Society (AMS) during 1983-1985 and is described in the book "The Joy of TeX". It is based on Plain TeX, and provides many features for producing more professional-looking maths formulas with less burden on authors. It pays attention to the finer details of sizing and positioning that mathematical publishers care about. The aspects covered include multi-line displayed equations, equation numbering, ellipsis dots, matrices, double accents, multi-line subscripts, syntax checking (faster processing on initial error-checking TeX runs), and other things.
As LaTeX increased in popularity, authors asked to submit papers to the AMS in LaTeX, and so the AMS developed AMSLaTeX, which is a collection of LaTeX packages and classes that offer authors most of the functionality of AMSTeX.