When I try tu run config utility and click "scan", I get the "invalid floating point operation". Any ideas how to solve it?

My specs (I have the newest drivers for all software):
windows xp prof 32bit
radeon hd 5770 1gb
core2duo 3ghz
3gb ram ddr2
abit ip35-e motherboard

have you checked your computer for viruses/malware? i guess this could be happening because of faulty ram/cpu but i'd bet it's a virus or something.

on the other hand i dunno if there's any use in scanning the system (don't remember if you have to do it the first time when running config) since NWN is so old game you can simply max out everything with your computer.

I'm pretty sure my system is clean. Well, without proper scan the game sets wrong parameters in .ini files like no opengl driver installed or wrong number of cpu cores. Also whenever I try to enable vsync the game crashes (I had to force it in ccc panel). On my previous gpu (hd 4850) the scan was succesful and I didn't have any vsync problems.

It's not a matter of life and death but why should I tolerate game errors if there is a solution to them ?:)

you shouldn't :p

anyways was simply trying to offer a quickfix until someone more knowledgeable comes along but since the scan is needed it was in vain and unfortunately i have no idea why it crashes, sorry.

I've seen this once before, but with Intel integrated chipsets. Does Event Viewer reports any errors?

[WinKey + R, type eventvwr]

Well, after the "invalid floating point operation" comes this (see image, it reads: Instruction at "0x5f1baed8" use memory address "0x00000564". Can't be "read".).

The crash is always at the same exact point, right after reaching 11% of the scan bar.

But if you open Event Viewer you should see a faulty module as well, which would tell us where the problem is located, i.e:

AppName: nwnmain.exe AppVer: 1.6.4.0 ModName: atioglxx.dll
etc.

Edit:
If by any chance the problem is indeed atioglxx.dll, grab this file from NWVault and put .dll to your NWN directory. (where nwconfig.exe and nwmain.exe are)

Thanks for your quick response. The only error I see in the event manager is the same I described in my previous post.

After copying atioglxx.dll the scan is actually successful but it also sets wrong parameners like 255 cpu cores, 2gb ram instead of 3gb, 2gb vram instead of 1gb or still no opengl driver. Also after running the game, there is only music and black screen after the intro movies and I have to manually kill the nwn process in the task manager.

EDIT: Maybe I need newer atioglxx.dll file because the one from NWVault is for older generations Ati cards.
EDIT2: I copied the newer atioglxx.dll from my system32 folder but I get the same error as before.

Newer file is what causing trouble actually. If you search the internet, you'll find lots of issues with lack of OGL compatibility and newer ATI drivers. Usually you'd need .dll file from older drivers package, but the problem is to find the right one that will work for you.

I'd prefer if someone with more experience with ATI chimed in, as I'm mostly an Nvidia user.

Ignore the errors and change the parameters manually. The config utility is scrap. AFAIK Its only good upto earlier patches - used to be a whole topic devoted to this issue on the original Bioware forums.

You should see what you get when you run it on a Vista or Win 7 system.

The black screen after intro is a driver issue as Thiev noted. However, this issue with ATI was listed as resolved sometime back. Have you tried running the 1.69 Critical Rebuild?

Also make sure that the game is NOT installed in the programs folder. Your path to the game files should be C:/NeverwinterNights/NWN.

Yes, I'm using 1.69 rebuild patch and the game is not installed in programs folder (I have a separate partition for game installations). Oh, well, the game runs fine without this scan. The only issue is that I can't get NWshader to work and I thought that bad scan was the issue.

The only way I've ever gotten shader to work is by having the game in the c:/ drive. I used to have my games setup like you do and shader wiuld never work. I moved the game to the c:/ drive and, voila, it worked. Contact Shader's developer directly - he might be able to solve your issue. He has a thread over at Bioware Social.