Normal maps themselves are lightweight—any "heaviness" in a 3D model using them comes from pairing them with high-resolution normals, complex base geometry, or large companion textures (like 4K albedo).
Normal maps add surface detail via 2D textures (no extra geometry), so their impact is indirect. The "heaviness" usually stems from combining high-res normals with detailed models or other big textures—these together increase file size or performance load.
To lighten the model, try reducing the normal map resolution (e.g., 4K → 2K) or simplifying non-essential base geometry first. This balances detail and performance without losing much visual quality.