Astrophysical black holes are expected to be described by the Kerr metric. This is the only stationary, vacuum, axisymmetric metric, without electromagnetic charge, that satisfies Einstein’s equations and does not have pathologies outside of the event horizon. We present new constraints on potential deviations from the Kerr prediction based on $2017$ EHT observations of Sagittarius A$^\ast$ (Sgr A$^\ast$). We calibrate the relationship between the geometrically defined black hole shadow and the observed size of the ring-like images using a library that includes both Kerr and non-Kerr simulations. We use the exquisite prior constraints on the mass-to-distance ratio for Sgr A$^\ast$ to show that the observed image size is within $\sim 10$% of the Kerr predictions. We use these bounds to constrain metrics that are parametrically different from Kerr, as well as the charges of several known spacetimes. To consider alternatives to the presence of an event horizon, we explore the possibility that Sgr A$^\ast$ is a compact object with a surface that either absorbs and thermally reemits incident radiation or partially reflects it. Using the observed image size and the broadband spectrum of Sgr A$^\ast$, we conclude that a thermal surface can be ruled out and a fully reflective one is unlikely. We compare our results to the broader landscape of gravitational tests. Together with the bounds found for stellar-mass black holes and the M87 black hole, our observations provide further support that the external spacetimes of all black holes are described by the Kerr metric, independent of their mass.