A common question being asked by many people who are moving from LWJGL2 to LWJGL3 is what they should use as a math library for their project as the old lwjgl_util.jar is no longer part of LWJGL3.
A rather good solution is the Java OpenGL Math Library (aka JOML). Its small, fast, supports NIO buffers (compatible with how LWJGL handles buffer arguments), multi-threading and is allocation free (i.e. will not allocate any memory unless specifically requested to do so).
It can be found at https://github.com/JOML-CI/JOML
The latest release can be downloaded from here