Get your DH's test result first before worrying. If there are problems, there are loads of ways around it e.g refer to this link
here.
Also, as Kobec says, it only takes ONE sperm to work. My DH has a similar issue to yours, and the specialist is quite hopeful, although we'll probably end up going down the IVF path.