It appears that volatility in equity markets is asymmetric: returns and conditional volatility are negatively correlated. We provide a unified framework to simultaneously investigate asymmetric volatility at the firm and the market level and to examine two potential explanations of the asymmetry: leverage effects and time-varying risk premiums. Our empirical application uses the market portfolio and portfolios with different leverage constructed from Nikkei 225 stocks, extending the empirical evidence on asymmetry to Japanese stocks. Although volatility asymmetry is present and significant at the market and the portfolio levels, its source differs across portfolios. We find it is important to include leverage ratios in the volatility dynamics but that their economic effects are mostly dwarfed by the volatility feedback mechanism. Volatility feedback is enhanced by a phenomenon that we term covariance asymmetry: conditional covariances with the market increase only significantly following negative market news. We do not find significant asymmetrics in conditional betas.