formula use:
1st formula
\mathbf{4ab\ =\ \left(a+b\right)^2\ -\ \left(a-b\right)^2}
prove: \left(a+b\right)^2\ -\ \left(a-b\right)^2
\rightarrow\left(a^2+2ab+b^2\right)-\left(a^2-2ab+b^2\right)
\rightarrow\ a^2\ +\ 2ab\ +\ b^2\ -\ a^2\ +\ 2ab\ -\ b^2
\rightarrow\ 4ab
\therefore\ 4ab\ =\ \left(a+b\right)^2\ -\ \left(a-b\right)^2 [proved]

2nd formula
\mathbf{2\left(a^2+b^2\right)=\left(a\ +\ b\right)^2+\left(a-b\right)^2}
prove: \left(a+b\right)^2+\left(a-b\right)^2
\rightarrow\left(a^2+2ab+b^2\right)+\left(a^2-2ab+b^2\right)
\rightarrow\ a^2\ +\ 2ab\ +b^2\ +\ a^2\ -\ 2ab\ +\ b^2
\rightarrow\ 2a^2\ +\ 2b^2
\rightarrow2\left(a^2+b^2\right)
\therefore2\left(a^2+b^2\right)=\left(a\ +\ b\right)^2+\left(a-b\right)^2 [proved]

Explanation

Here,
a\ +\ b\ =\ 5
a\ -\ b\ =\ 3
Now,
8ab\left(a^2+b^2\right)
\rightarrow4ab\times2\left(a^2+b^2\right) _{[put the value of ‘ 4ab’ & ‘2(a² + b²)’]}
\rightarrow\ \{\left(a+b\right)^2\ -\ \left(a-b\right)^2\}\times \{\left(a\ +\ b\right)^2+\left(a-b\right)^2\}
\rightarrow\ \{\left(5\right)^2\ -\ \left(3\right)^2\}\times \{\left(5\right)^2+\left(3\right)^2\}
\rightarrow\ \left(25\ -\ 9\right)\times\left(25\ +\ 9\right)
\rightarrow\ 16\ \times\ 34
\rightarrow\ 544
Ans: The value of 8ab(a^2 + b^2) = 544.