Many current approaches to shrinkage within the time-varying parameter framework assume that each state is equipped with only one innovation variance for all time points. Sparsity is then induced by shrinking this variance towards zero. We argue that this is not sufficient if the states display large jumps or structural changes, something which is often the case in time series analysis. To remedy this, we propose the dynamic triple gamma prior, a stochastic process that has a well-known triple gamma marginal form, while still allowing for autocorrelation. Crucially, the triple gamma has many interesting limiting and special cases (including the horseshoe shrinkage prior) which can also be chosen as the marginal distribution. Not only is the marginal form well understood, we further derive many interesting properties of the dynamic triple gamma, which showcase its dynamic shrinkage characteristics. We develop an efficient Markov chain Monte Carlo algorithm to sample from the posterior and demonstrate the performance through sparse covariance modeling and forecasting of the returns of the components of the EURO STOXX 50 index.