Driven by the multi-billion dollar video gaming industry, the exponential growth in floating point power of graphics processing units (GPUs) has led to a new computing platform to be exploited. In this talk I shall describe the challenges in algorithmic design and huge potential gains obtained from using GPUs to solve systems of linear equations. Examples will be taken from solving the lattice QCD Dirac operator.