I try to test how my application behave under console. Even I switch to VT1 (console), it opens window on VT3 (KWIN_Wayland). I remove also WAYLAND_DISPLAY and DISPLAY from terminal emulator on VT3 and same effect.
Is there any other way of logout from GUI?
Is this the same user on VT1 (CLI) and VT3 (GUI)?
I would remove the process environment variable from the CLI session on VT1 (if they are there, and how where they defined there?).
Doing it on the GUI session on VT3 will have no effect at all on the child processes running in VT1 (and when you do this from a terminal emulator inside the GUI session, that will not even influence any other process that is running in the GUI session BTW).
What you probably try to do is blocking on VT3 access from outside the GUI session on the GUI session. With X11 this is done with xhosts. I have no idea how this basic security protection works in Wayland.
I log under VT1, so there should not be WAYLAND_DISPLAY variable. On the terminal emulator under KWIN_WAYLAND, I unset WAYLAND_DISPLAY variable.
That is useless. You only unset it for the bash process that is running on the terminal and it will also not be set for child processes of that bash process. But it will still be set for all other processes that run in the Wayland session. You can not change anything on process variables of parent processes.
So, what you tried will not help. As said you must find out (maybe with help from others here) how this protection functions in Wayland.
Unless I misunderstand, the obvious method is to
I start my program from terminal, after unset WAYLAND_DISPLAY. I suggest to test prior, before answering. I do not blame you, of course, but there is something wrong on my PC or you did never try to do something similar.
Maybe this is related to autoreconnect? I think both QT/GTK support this on Wayland. What do you think?
I am afraid I am not completely understanding you here. What I think I understand:
user (a normal user, that I will name user because you did not tell us) is logged in in a Wayland based GUI session on the logical console VT3;user is also logged in in a CLI (bash) session on VT1;user on VT1 starts a program and that program apparently is a GUI program that tries to open a window, you detect that that window is opened on VT3, something that should not happen without at least active allowance action on VT3 (like the xhost +` on X11);Is my understanding correct so far?
Now I understand that in trying to block outside sessions to open windows (or worse, spying on what user is doing there) you opened a terminal emulator (like konsole) in the session on VT3 and unset the process environment variable WAYLAND_DISPLAY.
I tried to explain already two times that that will NOT bring you one step further in a solution for your problem. Because process environment variables are only valid in the process they are created/set/changed/unset and their child processes. Thus all other processes in the session on VT3 still have it (if it was there from the start of the session). And I also do not believe that WAYLAND_DISPLAY set in any process on VT# will influence the ability to open windows from outside the session.
For the third time I try to explain that you must find out about the protection mechanism on Wayland for this. Either try to find Wayland docs or hope for someone here to help.
I am not a Wayland users, but I see that you are on the wrong path to a solution (because of basic Unix/Linux concepts) and try to steer you to a better path.
Are you able to make this happen only after having first run the app directly from the graphical session?
If you switch to another VT (while the graphical desktop session is active), login as the same user, and do
env -u DBUS_SESSION_BUS_ADDRESS <application>
I assume that it fails?
What is “it” in the above sentence? I miss description of what you did.
Please show the result of
echo $XDG_SESSION_TYPE
form your VT1 shell.
Even if WAYLAND_DISPLAY is unset, the Wayland client can still leverage $XDG_RUNTIME_DIR/wayland-0by default, and connect to that socket.
Examine
ls -l $XDG_RUNTIME_DIR
for yourself in a VT with an active Wayland session running as the same user.
As long as the session is running under the same user and the socket is present and accessible, Wayland clients can still auto-connect to the compositor as expected.
@Lachu Why do you consider this a problem? You already have access to the user account - whether it is for accessing the graphical Wayland session or a VT.
When this (being a session of the same user) is indeed required for intruding in the GUI session, then I have less concerns. But for me coming from more then thirty years of X Windows it still triggers a security alert. 
App still ran. I also unset WAYLAND_DISPLAY and DISPLAY and app still run in GUI.
> echo $XDG_SESSION_TYPE
tty
I consider this is a problem, because someone may ran app on tty/no GUI. My app automatically detects if GUI is accessible. Someone may ran app on VT1 (tty) and it ran on VT3 (KWin_Wayland), so user may think that app is misbehave/it is broken, because it does not display anything.
In this case, user must force to load console module of my library. My library is in some manner similar to WXWidgets, but it loads backend/module dynamically.
Ok. Close this topic, if there is not solution to detect it properly. I will test if XDG_SESSION_TYPE is a tty and in this case try to load console backend first. I think this is the best solution.
When I understand the above from @deano_ferrari correct, then it can NOT be “someone”. It can only be the same user that is at the same time running a GUI session…
Yes that is what you should do as the application developer. Test the environment you are launching from. This is the question you should have been asking.
Sorry for be too offending. I thought there is no newer standards about session type. I remember that unset DISPLAY on X11 works and even probably unset WAYLAND_DISPLAY works too. That means were change in GTK/Qt libs. So there was problem in my library. Sorry again I do not test normal GTK/Qt apps and suggest you are doing thinks wrong in answering.