As climate change causes the environment to shift away from the local optimum that populations have adapted to, fitness declines are predicted to occur. Recently, methods known as (GOs) have become a popular tool to predict population responses to climate change from landscape genomic data. Populations with a high GO have been interpreted to have a high "genomic vulnerability" to climate change. GOs are often implicitly interpreted as a fitness offset, or a change in fitness of an individual or population in a new environment compared to a reference. However, there are several different types of fitness offset that can be calculated, and the appropriate choice depends on the management goals. This study uses hypothetical and empirical data to explore situations in which different types of fitness offsets may or may not be correlated with each other or with a GO. The examples reveal that even when GOs predict fitness offsets in a common garden experiment, this does not necessarily validate their ability to predict fitness offsets to environmental change. Conceptual examples are also used to show how a large GO can arise under a positive fitness offset, and thus cannot be interpreted as a population vulnerability. These issues can be resolved with robust validation experiments that can evaluate which fitness offsets are correlated with GOs.