In this factorization, formula used:
(a - b)^2 = a^2 - 2.a.b + b^2
a^2 - b^2 = (a + b) (a - b)

Explanation

x^2 – y^2 – 6ax + 2ay + 8a^2
\rightarrow\ x^2\ -\ y^2\ -\ 6ax\ +\ 2ay\ +\ 9a^2\ -\ a^2
\rightarrow\ \left(x^2\ -\ 6ax\ +\ 9a^2\right)\ -\ \left(a^2\ -\ 2ay\ +\ y^2\right) _[rearrange]
\rightarrow\ \{\left(x\right)^2\ -\ 2.x.3a\ +\ \left(3a\right)^2 \}\ -\ \{ \left(a\right)^2\ -\ 2.a.y\ +\ \left(y\right)^2 \}
\rightarrow\ \{\left(x\ -\ 3a\right)^2\ -\ \left(a\ -\ y\right)^2\}
\rightarrow\ \{\left(x\ -\ 3a\right)\ +\ \left(a\ -\ y\right)\}\ \{\left(x\ -\ 3a\right)\ -\ \left(a\ -\ y\right)\}
\rightarrow\ \left(x\ -\ 3a\ +\ a\ -\ y\right)\ \left(x\ -\ 3a\ -\ a\ +\ y\right)
\rightarrow\ \left(x\ -\ 2a\ -\ y\right)\ \left(x\ -\ 4a\ +\ y\right)
Ans: (x – 2a – y) (x – 4a + y).